Background: Understanding the magnitude and variability of the radiation dose absorbed by the breast fibroglandular tissue during mammography and digital breast tomosynthesis (DBT) is of paramount importance to assess risks versus benefits. Although homogeneous breast models have been proposed and used for decades for this purpose, they do not accurately reflect the actual heterogeneous distribution of the fibroglandular tissue in the breast, leading to biases in the estimation of dose from these modalities. Purpose: To develop and validate a method to generate patient-derived, heterogeneous digital breast phantoms for breast dosimetry in mammography and DBT. Methods: The proposed phantoms were developed starting from patient-based models of compressed breasts, generated for multiple thicknesses and representing the two standard views acquired in mammography and DBT, that is, cranio-caudal (CC) and medio-lateral-oblique (MLO). Internally, the breast phantoms were defined as consisting of an adipose/fibroglandular tissue mixture, with a nonspatially uniform relative concentration. The parenchyma distributions were obtained from a previously described model based on patient breast computed tomography data that underwent simulated compression. Following these distributions, phantoms with any glandular fraction (1%–100%) and breast thickness (12–125 mm) can be generated, for both views. The phantoms were validated, in terms of their accuracy for average normalized glandular dose (DgN) estimation across samples of patient breasts, using 88 patient-specific phantoms involving actual patient distribution of the fibroglandular tissue in the breast, and compared to that obtained using a homogeneous model similar to those currently used for breast dosimetry. Results: The average DgN estimated for the proposed phantoms was concordant with that absorbed by the patient-specific phantoms to within 5% (CC) and 4% (MLO). These DgN estimates were over 30% lower than those estimated with the homogeneous models, which overestimated the average DgN by 43% (CC), and 32% (MLO) compared to the patient-specific phantoms. Conclusions: The developed phantoms can be used for dosimetry simulations to improve the accuracy of dose estimates in mammography and DBT.