ABSTRACT There exist hitherto unexplained fluctuations in the cosmic infrared background on arcminute scales and larger. These have been shown to cross-correlate with the cosmic X-ray background, leading several authors to attribute the excess to a high-redshift growing black hole population. In order to investigate potential sources that could explain this excess, in this paper, we develop a new framework to compute the power spectrum of undetected sources that do not have constant flux as a function of halo mass. In this formulation, we combine a semi-analytic model for black hole growth and their simulated spectra from hydrodynamical simulations. Revisiting the possible contribution of a high-redshift black hole population, we find that too much black hole growth is required at early epochs for z > 6 accretion to explain these fluctuations. Examining a population of accreting black holes at more moderate redshifts, z ∼ 2–3, we find that such models produce a poor fit to the observed fluctuations while simultaneously overproducing the local black hole mass density. Additionally, we rule out the hypothesis of a missing Galactic foreground of warm dust that produces coherent fluctuations in the X-ray via reflection of Galactic X-ray binary emission. Although we firmly rule out accreting massive black holes as the source of these missing fluctuations, additional studies will be required to determine their origin.