AbstractThe mass and dynamics of protoplanetary disks are dominated by molecular hydrogen (H₂). However, observationally very little is known about the H₂. In this paper, we discuss two projects aimed to constrain the properties of H₂ in the disk's planet forming region (R<50AU). First, we present a sensitive survey for pure-rotational H₂ emission at 12.278 and 17.035 μm in a sample of nearby Herbig Ae/Be and T Tauri stars using VISIR, ESO's VLT high-resolution mid-infrared spectrograph. Second, we report on a search for H₂ ro-vibrational emission at 2.1228, 2.2233 and 2.2477 μm in the classical T Tauri star LkHα 264 and the debris disk 49 Cet employing CRIRES, ESO's VLT high-resolution near-infrared spectrograph.VISIR project: none of the sources show H₂ mid-IR emission. The observed disks contain less than a few tenths of M_Jupiter of optically thin H₂ at 150 K, and less than a few M_Earth at T>300 K. % and higher T. Our non-detections are consistent with the low flux levels expected from the small amount of H₂ gas in the surface layer of a Chiang and Goldreich (1997) Herbig Ae two-layer disk model. In our sources the H₂ and dust in the surface layer have not significantly departed from thermal coupling (T_gas/T_dust<2) and the gas-to-dust ratio in the surface layer is very likely <1000.CRIRES project: The H₂ lines at 2.1218 μm and 2.2233 μm are detected in LkHα 264. An upper limit on the 2.2477 μm H₂ line flux in LkHα 264 is derived. 49 Cet does not exhibit H₂ emission in any of observed lines. There are a few M_Moon of optically thin hot H₂ in the inner disk (∼0.1 AU) of LkHα 264, and less than a tenth of a M_Moon of hot H₂ in the inner disk of 49 Cet. The shape of the 1–0 S(0) line indicates that LkHα disk is close to face-on (i<35^o). The measured 1–0 S(0)/1–0 S(1) and 2–1 S(1)/1–0 S(1) line ratios in LkHα 264 indicate that the H₂ is thermally excited at T<1500 K. The lack of H₂ emission in the NIR spectra of 49 Cet and the absence of Hα emission suggest that the gas in the inner disk of 49 Cet has dissipated.