Context. Analyzing the properties of dust and its evolution in the early phases of star formation is crucial to put constraints on the collapse and accretion processes as well as on the pristine properties of planet-forming seeds. Aims. In this paper, we aim to investigate the variations of the dust grain size in the envelopes of the youngest protostars. Methods. We analyzed Plateau de Bure interferometric observations at 1.3 and 3.2 mm for 12 Class 0 protostars obtained as part of the CALYPSO survey. We performed our analysis in the visibility domain and derived dust emissivity index (β_1−3mm) profiles as a function of the envelope radius at 200–2000 au scales. Results. Most of the protostellar envelopes show low dust emissivity indices decreasing toward the central regions. The decreasing trend remains after correction of the (potentially optically thick) central region emission, with surprisingly low β_1−3mm < 1 values across most of the envelope radii of NGC 1333-IRAS 4A, NGC 1333-IRAS 4B, SVS13B, and Serpens-SMM4. Conclusions. We discuss the various processes that could explain such low and varying dust emissivity indices at envelope radii 200–2000 au. Our observations of extremely low dust emissivity indices could trace the presence of large (millimeter-size) grains in Class 0 envelopes, in which case our results would point to a radial increase of the dust grain size toward the inner envelope regions. While it is expected that large grains in young protostellar envelopes could be built via grain growth and coagulation, we stress that the typical timescales required to build millimeter grains in current coagulation models are at odds with the youth of our Class 0 protostars. Additional variations in the dust composition could also partly contribute to the low β_1−3mm we observe. We find that the steepness of the β_1−3mm radial gradient depends strongly on the envelope mass, which might favor a scenario in which large grains are built in high-density protostellar disks and transported to the intermediate envelope radii, for example with the help of outflows and winds.