Different land uses have varying degrees of impact on nitrogen transport in the catchments. In recent decades, rapid urbanization has dramatically changed the Earth’s land surface, which may cause excessive nitrogen losses and a negative influence on the environment. In the long-term scale, it is important to explore how the nitrogen transport responds to land use change and its effects on aquatic habitats. In this study, the water and sediment samples were collected from northern Taihu Basin, and nine periods of land use data were obtained using the techniques of supervised classification. Results revealed that the proportion of farmland area decreased from 28.33% to 7.09%, while that of constructed land area increased from 23.85% to 61.72% during 1990–2017. Most of the constructed land originated from farmland, which makes it the dominant land use type due to rapid urbanization. In spatial distribution, high total nitrogen (TN) losses regions remain distributed over constructed land and farmland, which may aggravate the trend of local water quality deterioration. Of these regions, constructed land was the dominant contributor (46.29%–63.62%) of TN losses from surface runoff. In temporal variation, the TN losses of runoff decreased by 47% from 175 t N·a−1 in 1990. However, they increased by 2.91% from 75.28 t N·a−1 after 2013 with rapid population growth and high fertilizer application (>570 kg·ha−1). The nitrogen load in sediments also has a significant response (t = 2.43, p = 0.02) to the effects of land use change on the overlying water, indicating that the role of nitrogen in the sediment as a source and/or sink to the waterbody may change frequently. Given the increasing accumulation of nitrogen loads in highly urbanized regions, water quality would cause more aggravation in the long-term without reasonable land management measures.