ABSTRACT The study of the cosmic near-infrared background (CIB) light after subtraction of resolved sources can push the limits of current observations and yield information on galaxies and quasars in the early universe. Spatial fluctuations of the CIB exhibit a clustering excess at angular scales ∼1° whose origin has not been conclusively identified, but disentangling the relative contribution from low- and high-redshift sources is not trivial. We explore the likelihood that this signal is dominated by emission from galaxies and accreting black holes (BHs) in the early Universe. We find that, the measured fluctuation signal is too large to be produced by galaxies at redshifts z > 8, which only contribute ∼0.01–0.05 nW m−2 sr−1 to the CIB. Additionally, if the first small mass galaxies have a normal initial mass function, the light of their ageing stars (fossils) integrated over cosmic time contributes a comparable amount to the CIB as their pre-reionization progenitors. In order to produce the observed level of CIB fluctuation without violating constraints from galaxy counts and the electron optical depth of the IGM, minihaloes at z > 12 must form preferably top-heavy stars with efficiency f* ≳ 0.1 and at the same time maintain a very low escape fraction of ionizing radiation, fesc < 0.1 per cent. If instead the CIB fluctuations are produced by high-z BHs, one requires vigorous accretion in the early universe reaching ρacc ≳ 105 M⊙ Mpc−3 by z ≃ 10. This growth must stop by z ∼ 6 and be significantly obscured not to overproduce the soft cosmic X-ray background and its observed coherence with the CIB. We therefore find the range of suitable high-z explanations to be narrow, but could possibly be widened by including additional physics and evolution at those epochs.