Context. The Earth is strongly depleted in carbon compared to the dust in the ISM, implying efficient removal of refractory carbon before parent body formation. It has been argued that grains get rid of their carbon through oxidation and photolysis in the exposed upper disk layers. Aims. We assess the efficacy of these C-removal mechanisms, while accounting for the vertical and radial transport of grains. Methods. We obtained the carbon and carbon free mass budget of solids by solving two 1D advection-diffusion equations, accounting for the dust grain size distribution and radial transport. The carbon removal acts on the fraction of the grains that are in the exposed layer and requires efficient vertical transport. Results. In models without radial transport, oxidation and photolysis can destroy most of the refractory carbon in terrestrial planet formation region. But it only reaches the observed depletion levels for extreme parameter combinations and requires that parent body formation was delayed by 1 Myr. Adding radial transport of solids prevents the depletion entirely, leaving refractory carbon equally distributed throughout the disk. Conclusions. It is unlikely that the observed carbon depletion can ultimately be attributed to mechanisms operating on small grains in the disk surface layers. Other mechanisms of removing carbon quickly and deeply must be studied, for example flash heating events or FU Ori outbursts. However, a sustained drift barrier or strongly reduced radial grain mobility is necessary to prevent replenishment of carbon from the outer disk.