ABSTRACT Redshifts have been so easy to measure for so long that we tend to neglect the fact that they too have uncertainties and are susceptible to systematic error. As we strive to measure cosmological parameters to better than 1 per cent it is worth reviewing the accuracy of our redshift measurements. Surprisingly small systematic redshift errors, as low as 10−4, can have a significant impact on the cosmological parameters we infer, such as H0. Here, we investigate an extensive (but not exhaustive) list of ways in which redshift estimation can go systematically astray. We review common theoretical errors, such as adding redshifts instead of multiplying by (1 + z); using v = cz; and using only cosmological redshift in the estimates of luminosity and angular diameter distances. We consider potential observational errors, such as rest wavelength precision, air to vacuum conversion, and spectrograph wavelength calibration. Finally, we explore physical effects, such as peculiar velocity corrections, galaxy internal velocities, gravitational redshifts, and overcorrecting within a bulk flow. We conclude that it would be quite easy for small systematic redshift errors to have infiltrated our data and be impacting our cosmological results. While it is unlikely that these errors are large enough to resolve the current H0 tension, it remains possible, and redshift accuracy may become a limiting factor in near future experiments. With the enormous efforts going into calibrating the vertical axis of our plots (standard candles, rulers, clocks, and sirens) we argue that it is now worth paying a little more attention to the horizontal axis (redshifts).