ABSTRACT Type Iax supernovae (SNe Iax) are the most common class of peculiar SNe. While they are thought to be thermonuclear white-dwarf (WD) SNe, SNe Iax are observationally similar to, but distinct from SNe Ia. Unlike SNe Ia, where roughly 30 per cent occur in early-type galaxies, only one SN Iax has been discovered in an early-type galaxy, suggesting a relatively short delay time and a distinct progenitor system. Furthermore, one SN Iax progenitor system has been detected in pre-explosion images with its properties consistent with either of two models: a short-lived (<100 Myr) progenitor system consisting of a WD primary and a He-star companion, or a singular Wolf–Rayet progenitor star. Using deep Hubble Space Telescope images of nine nearby SN Iax host galaxies, we measure the properties of stars within 200 pc of the SN position. The ages of local stars, some of which formed with the SN progenitor system, can constrain the time between star formation and SN, known as the delay time. We compare the local stellar properties to synthetic photometry of single-stellar populations, fitting to a range of possible delay times for each SN. With this sample, we uniquely constrain the delay-time distribution for SNe Iax, with a median and 1σ confidence interval delay time of $63_{- 15}^{+ 58} \times 10^{6}$ yr. The measured delay-time distribution provides an excellent constraint on the progenitor system for the class, indicating a preference for a WD progenitor system over a Wolf–Rayet progenitor star.