Chondrules are millimeter-sized spherules that dominate primitive meteorites (chondrites) originating from the asteroid belt. The incorporation of chondrules into asteroidal bodies must be an important step in planet formation, but the mechanism is not understood. We show that the main growth of asteroids can result from gas drag–assisted accretion of chondrules. The largest planetesimals of a population with a characteristic radius of 100 km undergo runaway accretion of chondrules within ~3 My, forming planetary embryos up to Mars’s size along with smaller asteroids whose size distribution matches that of main belt asteroids. The aerodynamical accretion leads to size sorting of chondrules consistent with chondrites. Accretion of millimeter-sized chondrules and ice particles drives the growth of planetesimals beyond the ice line as well, but the growth time increases above the disc lifetime outside of 25 AU. The contribution of direct planetesimal accretion to the growth of both asteroids and Kuiper belt objects is minor. In contrast, planetesimal accretion and chondrule accretion play more equal roles in the formation of Moon-sized embryos in the terrestrial planet formation region. These embryos are isolated from each other and accrete planetesimals only at a low rate. However, the continued accretion of chondrules destabilizes the oligarchic configuration and leads to the formation of Mars-sized embryos and terrestrial planets by a combination of direct chondrule accretion and giant impacts.