ABSTRACT Our study aims to recognize M-type stars which are classified as ‘UNKNOWN’ due to poor quality in the Large sky Area Multi-Object fiber Spectroscopic Telescope (LAMOST) DR5 V1. A binary nonlinear hashing algorithm based on Multi-Layer Pseudo-Inverse Learning (ML-PIL) is proposed to effectively learn spectral features for M-type-star detection, which can overcome the bad fitting problem of template matching, particularly for low S/N spectra. The key steps and the performance of the search scheme are presented. A positive data set is obtained by clustering the existing M-type spectra to train the ML-PIL networks. By employing this new method, we find 11 410 M-type spectra out of 642 178 ‘UNKNOWN’ spectra, and provide a supplemental catalogue. Both the supplemental objects and released M-type stars in DR5 V1 are composed of a whole M-type sample, which will be released in the official DR5 to the public in June 2019. All the M-type stars in the data set are classified as giants and dwarfs by two suggested separators: (1) a colour diagram of H versus J − K from 2MASS, (2) line indices CaOH versus CaH1, and the separation is validated with the Hertzsprung–Russell diagram (HRD) derived from Gaia DR2. The magnetic activities and kinematics of M dwarfs are also provided with the equivalent width (EW) of the Hα emission line and the astrometric data from Gaia DR2 respectively.