This set of data comprises temporal temperature gradient electrophoresis (TTGE) and soil process measurements, used to analyse the effects of perturbations (sludge and/or lime application) on the structure, community development and activity of bacteria that catalyse fundamental processes in upland soils. These were collected to address the following questions: Do soil improvement treatments select for particular components of bacterial populations and hence drive community development? If so, at what functional and phylogenetic level is this selection expressed? Can any changes in community structure be related to changes in the function of the community or is biogeochemical function independent of community structure and controlled by other mechanisms? The work was part of the NERC Soil Biodiversity Thematic Programme, which was established in 1999 and was centred upon the intensive study of a large field experiment located at the Macaulay Land Use Research Institute (now the James Hutton Institute) farm at Sourhope in the Scottish Borders. During the experiment, the site was monitored to assess changes in above-ground biomass production (productivity), species composition and relative abundance (diversity). Full details about this dataset can be found at https://doi.org/10.5285/1cebca07-dd82-4ba2-823b-274868abda42