Context. An important aspect of quenching star formation is the removal of the cold interstellar medium (ISM; non-ionised gas and dust) from a galaxy. In addition, dust grains can be destroyed in a hot or turbulent medium. The adopted timescale of dust removal usually relies on uncertain theoretical estimates. It is tricky to track dust removal because the dust is constantly being replenished by consecutive generations of stars. Aims. Our objective is to carry out an observational measurement of the timescale of dust removal. Methods. We explored an approach to select galaxies that demonstrate detectable amounts of dust and cold ISM coupled with a low current dust production rate. Any decrease of the dust and gas content as a function of the age of such galaxies must, therefore, be attributed to processes governing ISM removal. We used a sample of the galaxies detected by Herschel in the far-infrared with visually assigned early-type morphology or spirals with red colours. We also obtained JCMT/SCUBA-2 observations for five of these galaxies. Results. We discovered an exponential decline of the dust-to-stellar mass ratio with age, which we interpret as an evolutionary trend for the dust removal of these galaxies. For the first time, we have directly measured the dust removal timescale for such galaxies, with a result of τ = (2.5 ± 0.4) Gyr (the corresponding half-life time is (1.75 ± 0.25) Gyr). This quantity may be applied to models in which it must be assumed a priori and cannot be derived. Conclusions. Any process which removes dust in these galaxies, such as dust grain destruction, cannot happen on shorter timescales. The timescale is comparable to the quenching timescales found in simulations for galaxies with similar stellar masses. The dust is likely of internal, not external origin. It was either formed in the past directly by supernovae (SNe) or from seeds produced by SNe, and with grain growth in the ISM contributing substantially to the dust mass accumulation.