Worldwide erosion rates seem to have increased strongly since the beginning of the Quaternary, but there is still discussion about the role of glaciation as a potential driver and even whether the increase is real at all or an artefact due to losses in the long-term sedimentary record. In this study we derive estimates of average erosion rates on the time scale of some tens of million years from the terrestrial impact crater inventory. This approach is completely independent from all other methods to infer erosion rates such as river loads, preserved sediments, cosmogenic nuclides and thermochronometry. Our approach yields average erosion rates as a function of present-day topography and climate. The results confirm that topography accounts for the main part of the huge variation of erosion on Earth, but also identifies a significant systematic dependence on climate in contrast to several previous studies. We found a fivefold increase in erosional efficacy from the cold regimes to the tropical zone and that temperate and arid climates are very similar in this context. Combining our results to a worldwide mean erosion rate we found that erosion rates on the time scale of some tens of million years are at least as high as present-day rates and suggest that glaciation has a rather regional effect with a limited impact at the continental scale.