Asteroids smaller than 10 km are thought to be rubble piles formed from the reaccumulation of fragments produced in the catastrophic disruption of parent bodies. Ground-based observations reveal that some of these asteroids are today binary systems, in which a smaller secondary orbits a larger primary asteroid. However, how these asteroids became binary systems remains unclear. Here, we report the analysis of boulders on the surface of the stony asteroid (65803) Didymos and its moonlet, Dimorphos, from data collected by the NASA DART mission. The size-frequency distribution of boulders larger than 5 m on Dimorphos and larger than 22.8 m on Didymos confirms that both asteroids are piles of fragments produced in the catastrophic disruption of their progenitors. Dimorphos boulders smaller than 5 m have size best-fit by a Weibull distribution, which we attribute to a multi-phase fragmentation process either occurring during coalescence or during surface evolution. The density per km2 of Dimorphos boulders ≥1 m is 2.3x with respect to the one obtained for (101955) Bennu, while it is 3.0x with respect to (162173) Ryugu. Such values increase once Dimorphos boulders ≥5 m are compared with Bennu (3.5x), Ryugu (3.9x) and (25143) Itokawa (5.1x). This is of interest in the context of asteroid studies because it means that contrarily to the single bodies visited so far, binary systems might be affected by subsequential fragmentation processes that largely increase their block density per km2 . Direct comparison between the surface distribution and shapes of the boulders on Didymos and Dimorphos suggest that the latter inherited its material from the former. This finding supports the hypothesis that some asteroid binary systems form through the spin up and mass shedding of a fraction of the primary asteroid.