ABSTRACT The recent discovery of high-redshift dusty galaxies implies a rapid dust enrichment of their interstellar medium (ISM). To interpret these observations, we run a cosmological simulation in a 30 h−1 cMpc/size volume down to z ≈ 4. We use the hydrodynamical code dustygadget, which accounts for the production of dust by stellar populations and its evolution in the ISM. We find that the cosmic dust density parameter (Ωd) is mainly driven by stellar dust at z ≳ 10, so that mass- and metallicity-dependent yields are required to assess the dust content in the first galaxies. At z ≲ 9, the growth of grains in the ISM of evolved systems [log(M⋆/M⊙) > 8.5] significantly increases their dust mass, in agreement with observations in the redshift range 4 ≲ z < 8. Our simulation shows that the variety of high-redshift galaxies observed with the Atacama Large Millimeter Array can naturally be accounted for by modelling the grain growth time-scale as a function of the physical conditions in the gas cold phase. In addition, the trends of dust-to-metal and dust-to-gas (${\cal D}$) ratios are compatible with the available data. A qualitative investigation of the inhomogeneous dust distribution in a representative massive halo at z ≈ 4 shows that dust is found from the central galaxy up to the closest satellites along polluted filaments with $\rm log({\cal D}) \le -2.4$, but sharply declines at distances d ≳ 30 kpc along many lines of sight, where $\rm log({\cal D}) \lesssim -4.0$.