We present a model for the relative velocity of inertial particles in turbulent flows that provides new physical insight into this problem. Our general formulation shows that the relative velocity has contributions from two terms, referred to as the ‘generalized acceleration’ and ‘generalized shear’, because they reduce to the well-known acceleration and shear terms in the Saffman–Turner limit. The generalized shear term represents particles' memory of the flow velocity difference along their trajectories and depends on the inertial particle pair dispersion backward in time. The importance of this backward dispersion in determining the particle relative velocity is emphasized. We find that our model with a two-phase separation behaviour, an early ballistic phase and a later tracer-like phase, as found by recent simulations for the forward (in time) dispersion of inertial particle pairs, gives good fits to the measured relative speeds from simulations at low Reynolds numbers. In the monodisperse case with identical particles, the generalized acceleration term vanishes and the relative velocity is determined by the generalized shear term. At large Reynolds numbers, our model gives a St^1/2-dependence of the relative velocity on the Stokes number St in the inertial range for both the ballistic behaviour and the Richardson separation law. This leads to the same inertial-range scaling for the two-phase separation that well fits the simulation results. Our calculations for the bidisperse case show that, with the friction timescale of one particle fixed, the relative speed as a function of the other particle's friction time has a dip when the two timescales are similar. This indicates that similar-size particles tend to have stronger velocity correlation than different ones. We find that the primary contribution at the dip, i.e. for similar particles, is from the generalized shear term, while the generalized acceleration term is dominant for particles of very different sizes. Future numerical studies are motivated to check the accuracy of the assumptions made in our model and to investigate the backward-in-time dispersion of inertial particle pairs in turbulent flows.