Graph from Pullinger, M., Kilgour, J., Goddard, N. et al. The IDEAL household energy dataset, electricity, gas, contextual sensor data and survey data for 255 UK homes. Sci Data 8, 146 (2021). https://doi.org/10.1038/s41597-021-00921-y
-
Topic:Energy Industry
In short:
As more home owners install batteries and smart appliances, they’re creating an increasingly valuable resource that can feed power to the grid when needed.
But there are concerns the companies offering to manage these services are looking to capitalise on this market and cut consumers out of the profits.
What’s next:
Consumer advocates say the smart meter market is in urgent need of regulation to prevent industry players from using customer data for their own benefit instead of the consumer’s.
Kerry Bradbury’s three adult children are still occasional tenants in her Chifley home in Sydney’s south-eastern suburbs.
And like countless Australians, she’s been feeling the sting of soaring electricity prices.
“It got to the stage where you’re actually afraid to open your electricity bill,” Ms Bradbury says.
“Especially with the transient kids, definitely.
“I could get one bill for electricity, which would be around the $600, $700 mark.
“If I had one, two, three [kids] here over summer, the bills could be as high as $2,000.”
To deal with the shock, a couple of years ago Kerry Bradbury took a radical step — she installed batteries, new solar panels and a heap of smart tech in her home.
It wasn’t cheap, costing her $28,000.
But thanks to that tech and the small retailer that coordinates it for her, she won’t have to pay a power bill for seven years.
“All of it’s automated,” Ms Bradbury explains.
“Me being an average person, I don’t have time to try and work out what’s the best price, when’s the best price to be using it.
“I am quite happy for somebody else to be doing that for me, and I am still getting some benefit.”
Sitting on a gold mine
While Kerry Bradbury is grateful for someone taking over to generate savings for her, she’s actually paying to hand over control of an asset that has huge money-making potential.
Australia’s energy authorities envision a world where smart, clean tech — from solar panels and batteries to electric vehicles to pool pumps, air conditioners and heat pumps — can help prop up the grid while slashing consumers’ bills.
The smart technology behind those devices enables them to be turned on and off, up and down through software platforms known as home energy management systems.
And the ability to control households’ energy systems is becoming an increasingly valuable asset to the grid and power companies.
“A rooftop solar system coupled with a small-scale battery installation can make a meaningful difference to a single household’s energy bill,” the Australian Energy Regulator wrote in its most recent state of the energy market report.
“But aggregated across thousands of households, these technologies can enhance system reliability and security.”
That potential could be used to soak up excess electricity when there is too much solar power in the system and not enough demand.
Or it could be used to provide electricity to the grid when there isn’t enough solar and there’s too much demand.
Thousands of households like Kerry’s being able to work together to reduce demand or feed power into the grid is a powerful tool that the energy market would be happy to pay for.
The data holds the key
But what underpins the ability to allow households to work together for the grid is data collected by the smart meter attached to your power box.
Kerry Bradbury’s smart meter shows exactly how she’s using power and when she’s using it — a valuable insight for authorities running the grid and anyone trying to save or make money from the electricity market.
But it’s not clear whether she owns this data. Smart meter rollouts are controlled by energy retailers, and a few companies are responsible for most of the installations.
If the householder doesn’t pay for their own smart meter, the data showing how and when they use power, among other valuable insights, is effectively owned by the company that provided the smart meter.
If Kerry wants to sell the ability to feed power back to the grid to a company, they are going to have to go through whoever owns the smart meter.
Douglas McCloskey, from the Justice and Equity Centre, is sceptical about the supposed benefits for consumers under the current system.
“Consumers do have a right to access that data, but it’s quite complicated and clunky,” he said.
He says the metering companies’ effective control over consumers’ data can make it extremely difficult for anyone else to use that information.
This, Mr McCloskey notes, would make it all but impossible for third parties such as aggregators or small electricity retailers to provide services to punters unless the customer is able to afford an entirely new, and extra, smart meter.
“In a lot of cases, there’s potential for the meter and what it can do and the data that comes through it to be controlled by metering providers,” he says.
“Where there aren’t any explicit provisions, it’s possible for metering providers to do things that aren’t technically illegal but wouldn’t necessarily be in the consumer’s best interest.”
Shutting out competition
Jonathan Jutsen, the founder of consultancy Energetics, says it is imperative that regulators prevent “walled gardens”, a term that refers to the practice by technology companies of closing access to a platform so they can profit from the data.
A well-known example for this is Apple, which is unwilling to share its systems — from messaging to video-calling — with other tech providers.
“That is surely what regulators are there for — to protect consumers’ interests.”
According to Mr McCloskey, the stakes are high.
He says the great prize of the energy transition is smarter and cleaner power that can cut bills and hand greater control to consumers.
But he warns this prize is likely to remain out of reach for most households unless yawning gaps in the regulations covering the smart meter market can be plugged.
As it currently stands, Mr McCloskey says the market is only likely to serve the interests of the metering providers and their retailer customers.
“If we get this wrong, the energy transition will be more expensive for consumers and there will be a lot more complications than there should be,” he says.
“And there’s the potential we won’t have the full range of services that some consumers may want and the protections for data and information that consumers need.”
‘Conflict of interest’
Giving consumers greater control of their own data and real-time access to it would be a good start, he says.
He says it should be entirely up to consumers to decide who — or which company — can use their data to provide them energy services.
Sydney householder Kerry Bradbury agrees.
She questions how metering companies can best serve consumers’ interests when their own customer is the power retailer.
how metering companies can best serve consumers’ interests when their own customer is the power retailer.
In her opinion, the “conflict of interests” seems obvious.
“They [the metering companies] can’t be dealing with the little person like me … and the bigger players in the system,” Ms Bradbury says.
“I know who would win.
“It would be the big boys who would win over a little person like me.
“This is where there has to be a regulator of some kind to step in and stop anyone dominating the market.”
The IDEAL household energy dataset, electricity, gas, contextual sensor data and survey data for 255 UK homes
- Martin Pullinger,
- Jonathan Kilgour,
- Nigel Goddard,
- Niklas Berliner,
- Lynda Webb,
- Myroslava Dzikovska,
- Heather Lovell,
- Janek Mann,
- Charles Sutton,
- Janette Webb &
- Mingjun Zhong
Scientific Data volume 8, Article number: 146 (2021) Cite this article
Abstract
The IDEAL household energy dataset described here comprises electricity, gas and contextual data from 255 UK homes over a 23-month period ending in June 2018, with a mean participation duration of 286 days. Sensors gathered 1-second electricity data, pulse-level gas data, 12-second temperature, humidity and light data for each room, and 12-second temperature data from boiler pipes for central heating and hot water. 39 homes also included plug-level monitoring of selected electrical appliances, real-power measurement of mains electricity and key sub-circuits, and more detailed temperature monitoring of gas- and heat-using equipment, including radiators and taps. Survey data included occupant demographics, values, attitudes and self-reported energy awareness, household income, energy tariffs, and building, room and appliance characteristics. Linked secondary data comprises weather and level of urbanisation. The data is provided in comma-separated format with a custom-built API to facilitate usage, and has been cleaned and documented. The data has a wide range of applications, including investigating energy demand patterns and drivers, modelling building performance, and undertaking Non-Intrusive Load Monitoring research.
Measurement(s) | electrical current • Gas • room temperature ambient air • humidity • light • temperature • individual appliance power usage • sociodemographics values • physical characteristics of building • appliance characteristics • Knowledge, Attitudes, Behaviors • Perception |
Technology Type(s) | current clamp • pulse block device • Temperature Sensor Device • Sensor Device • Temperature Probe Device • Monitor Device • Survey |
Sample Characteristic – Organism | Homo sapiens |
Sample Characteristic – Environment | anthropogenic environment • building |
Sample Characteristic – Location | United Kingdom |
Machine-accessible metadata file describing the reported data: https://doi.org/10.6084/m9.figshare.14096993
Background & Summary
Countries around the world have committed to the Paris Agreement goal of keeping the global average temperature rise to well below 2 °C above pre-industrial levels, whilst aiming to keep the rise to no more than 1.5 °C1. An essential component of achieving this goal is a rapid and deep decarbonisation of energy systems, with most scenarios involving substantial electrification of demand and increased end-use energy efficiency, among other things2. Empirical data on energy use, as well as on the factors which shape and predict it and the outcomes for end users, can play an important role in understanding the nature of this challenge and how to address it effectively and efficiently.
Within this context, we describe here the newly released IDEAL household energy dataset. The dataset consists of high-resolution sensor data on electricity and gas usage, linked to a wide range of predictor and outcome variables collected via additional sensors and surveys. Data was collected from a sample of 255 homes from Edinburgh and the nearby regions of the Lothians and south Fife, in Scotland, UK, in the 23 months up to 30 June 2018. Sensor data comprises electricity apparent power (1 Hz) and gas usage (per fixed-volume pulse) recorded at the meter, 12-second temperature, humidity and light data for each room, and 12-second boiler pipe temperatures for central heating and hot water pipes. Linked survey data includes occupant demographics, values, attitudes, and self-reported energy awareness, household income and energy tariffs, as well as a range of building, room and appliance attributes. Also included are variables describing local weather and each locality’s level of urbanisation, from secondary data sources.
In addition, 39 homes formed an ‘enhanced group’, having additional sensors to measure real power electricity use for the whole home, selected sub-circuits, and a selection of high power and user-controllable electrical appliances, and temperature sensors to indirectly indicate usage of gas-using or boiler-using appliances like hot water outlets, individual radiators and cooker hobs.
Data was collected from participating homes for between 55 and 673 days, with a mean of 286 days, median 267 days. Participants were going about their normal daily lives, although their involvement in the research project is likely to have prompted some behaviour change. In all homes, heating and hot water were provided primarily by a gas-fired combi-boiler system. The homes included a variety of building types and sizes and family structures, including single-occupancy, multiple adults and families with children.
Data was collected as part of two EPSRC-funded projects, IDEAL and BIGSMALL. The projects had several aims, including:
- To develop a long-life, battery-powered, wireless sensor system providing high frequency measurements;
- To investigate residential energy demand patterns, drivers and outcomes;
- To advance Machine Learning methods to infer: (a) appliance use in homes based on whole-home electricity data (Non-Intrusive Load Monitoring, NILM); (b) usage of gas-using appliances, boilers, and individual radiators and hot water outlets based on whole-home gas data and ambient room temperature and humidity data;
- To co-design with participants the ‘IDEAL app’: a selection of digital energy feedback and advice features available to participants via an app on a project-provided tablet, as well as via web browsers;
- To evaluate the energy and other effects of making different sets of features available through the IDEAL app, in a Randomised Controlled Trial experimental study.
Figure 1 provides an overview of the data collected and IDEAL app features available to different groups of participants in the study.
The data is likely to be of value for a wide range of research purposes, including:
- Evaluating Machine Learning methods for inferring patterns of appliance usage using whole-home energy data;
- Studying daily and seasonal patterns of energy use; the factors which ‘drive’ those patterns, including occupant, building, appliance and weather factors; and the outcomes, including levels of expenditure, indoor temperature and humidity, and occupant satisfaction.
- Evaluating assumptions about occupant behaviours that are incorporated into building performance models, such as the UK’s Standard Assessment Procedure (SAP).
Related datasets, suitable for research into one or more of these subjects, are listed in Table 1. The table focuses on UK household energy datasets for highest comparability, although it also includes non-UK-based datasets that are commonly used in NILM research. In many cases, the IDEAL household energy dataset includes a wider range of sensor and survey data, and/or a larger sample size and duration of participation, which may provide greater opportunities for researching these topics. The research team are in the process of disseminating analyses and results from the projects using this dataset. A list of these works is available on the lead team’s website (https://wp.inf.ed.ac.uk/sustainlab/), which will be updated as new work is released.
References
-
United Nations. Paris Agreement. (United Nations, 2015).
-
IPCC. Global Warming of 1.5 °C: An IPCC Special Report on the Impacts of Global Warming of 1.5 °C Above Pre-industrial Levels and Related Global Greenhouse Gas Emission Pathways, in the Context of Strengthening the Global Response to the Threat of Climate Change, Sustainable Development, and Efforts to Eradicate Poverty. (Intergovernmental Panel on Climate Change, World Meteorological Organization, 2018).
-
DECC. Smart Metering Implementation Programme: Smart Metering Equipment Technical Specifications Version 1.57. (Department for Energy and Climate Change, 2014).
-
Hagenaars, A. J. M., de Vos, K. & Zaidi, M. A. Poverty Statistics in the Late 1980s: Research Based on Micro-Data (Office for Official Publications of the European Communities, 1994).
-
HMRC. National Statistics: Percentile points from 1 to 99 for total income before and after tax. (HM Revenue & Customs, 2020).
-
Scottish Government Geographic Information Science & Analysis Team, Rural and Environment Science and Analytical Services Division Scottish Government Urban Rural Classification 2016. (Scottish Government, 2018).
-
Shove, E., Pantzar, M. & Watson, M. The Dynamics of Social Practice: Everyday Life and How It Changes (Sage, 2012).
-
Akrich, M. In Shaping Technology/Building Society: Studies in Sociotechnical Change(eds. Bijker, W. E. & Law, J.) 205–224 (MIT Press, 1992).
-
Steg, L. Promoting household energy conservation. Energy Policy 36, 4449–4453 (2008).
-
Lindenberg, S. & Steg, L. Normative, gain and hedonic goal frames guiding environmental behavior. Journal of Social Issues 63, 117–137 (2007).
-
Becker, G. A Treatise on the Family (Harvard University Press, 1981).
-
Ferrer-i-Carbonell, A. & van den Bergh, J. A micro-econometric analysis of determinants of unsustainable consumption in the Netherlands. Environmental & Resource Economics 27, 367–389 (2004).
-
Hazas, M., Friday, A. & Scott, J. Look back before leaping forward: four decades of domestic energy inquiry. IEEE Pervasive Computing 10, 13–19 (2011).
-
Froehlich, J., Findlater, L. & Landay, J. The design of eco-feedback technology. CHI ’10: Proceedings of the 28th International Conference on Human Factors in Computing Systems 1999–2008 (2010).
-
Pullinger, M., Lovell, H. & Webb, J. Influencing household energy practices: a critical review of UK smart metering standards and commercial feedback devices. Technology Analysis & Strategic Management 26, 1144–1162 (2014).
-
Zimmermann, J. P. et al. Household Electricity Survey, a study of domestic electrical product usage. Final report issue 4. Report No. Intertek Report R66141 (Intertek Testing & Certification Ltd, 2012).
-
Weather data Weather Underground https://www.wunderground.com (2018).
-
Goddard, N. et al. The IDEAL Household Energy Dataset. Edinburgh DataSharehttps://doi.org/10.7488/ds/2836 (2021).
-
Ofgem. Typical Domestic Consumption Values Ofgemhttps://www.ofgem.gov.uk/gas/retail-market/monitoring-data-and-statistics/typical-domestic-consumption-values (2020).
-
Browne, A. L., Anderson, B. & Medd, W. Lancaster SPRG/ARCC Water Surveyhttps://eprints.soton.ac.uk/439637 (2011).
-
Axsen, J., TyreeHageman, J. & Lentz, A. Lifestyle practices and pro-environmental technology. Ecological Economics 82, 64–74 (2012).
-
Ministry of Housing, Communities & Local Government. English Housing Survey 2013 to 2014: Questionnaire and physical survey formhttps://www.gov.uk/government/publications/english-housing-survey-2013-to-2014-questionnaire-and-physical-survey-form (HM Government, 2015).
-
ONS Social Surveys. Living Costs and Food Survey Volume B: The Household Questionnaire User Guide, January – December 2011, Great Britain and Northern Ireland. Study No. 7047 http://doc.ukdataservice.ac.uk/doc/7047/mrdoc/pdf/2011/7047volume_b_household_questionnaire_2011.pdf (UK Data Archive, 2012).
-
Schulenberg, S. E. & Melton, A. M. A. The Computer Aversion, Attitudes, and Familiarity Index (CAAFI): a validity study. Computers in Human Behavior 24, 2620–2638 (2008).
-
Eurofound. Third European Quality of Life Survey: Questionnairehttps://www.eurofound.europa.eu/publications/other/2012/quality-of-life/third-european-quality-of-life-survey-questionnaire (Foundation for the Improvement of Living and Working Conditions, 2012).
-
University of Essex, Institute for Social and Economic Research. Understanding Society: Waves 1-10, 2009-2019 and Harmonised BHPS: Waves 1-18, 1991-2009. 13th Edition. Study No. 6614 https://doi.org/10.5255/UKDA-SN-6614-14 (UK Data Archive, 2019).
-
Inglehart, R. et al. World Values Survey Wave 6 (2010-2014), Official Questionnaire v.4, June 2012 http://www.worldvaluessurvey.org/WVSDocumentationWV6.jsp (JD Systems Institute, 2014).
-
Webb, J., Hawkey, D., McCrone, D. & Tingey, M. Bringing Warmth to Wyndford: Household Survey of New District Heating. Heat and the City Project https://heatandthecity.org.uk/project/heat-and-the-city/ (2014).
-
Webb, J., Hawkey, D., Mccrone, D. & Tingey, M. House, home and transforming energy in a cold climate. Families, Relationships and Societies 5, 411–429 (2016).
-
Haines, V. et al. The DEFACTO field trial: Methodology and data sets. Loughborough University https://doi.org/10.17028/rd.lboro.7837940.v1 (2019).
-
Buswell, R. et al. LEEDR Project Home Energy Dataset. Loughborough Universityhttps://doi.org/10.17028/rd.lboro.6176450.v1 (2018).
-
Richardson, I. & Thomson, M. One-Minute Resolution Domestic Electricity Use Data, 2008–2009. UK Data Service https://doi.org/10.5255/UKDA-SN-6583-1 (2010).
-
Kolter, J. Z. & Johnson, M. J. REDD: A public data set for energy disaggregation research. Proceedings of the SustKDD Workshop on Data Mining Applications in Sustainability (2011).
-
Murray, D., Stankovic, L. & Stankovic, V. An electrical load measurements dataset of United Kingdom households from a two-year longitudinal study. Sci. Data 4, 1–12 (2017).
-
Webborn, E., Elam, S., McKenna, E. & Oreszczyn, T. Utilising smart meter data for research and innovation in the UK. ECEEE Summer Study Proceedings, 1387–1396 (2019).
-
Kelly, J. & Knottenbelt, W. The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes. Scientific Data 2 (2015).
“We MUST respect this earth - it is all we have
Claudio Dametto - South Australia
“I will always Vote to Preserve Our World.
Liam McGregor - Western Australia
“A simple message that even a politician can understand
Felicity Crombach - Victoria
“Please show you care about our future generations!!
Phil Harmer - New South Wales
“Save our world , Life & health before profits.
Kerry Lillian - New South Wales
“Close down all coal mines and Do not mine gas . Make these Companies
Daniel Johnson - New South Wales
“We want carbon free energy!
Edan Clarke - New South Wales
“Feels good to be taking a voter action step
Beaver Hudson - New South Wales
“Great Initiative. Let’s Hold elected officials Accountable to their bosses, us!
John Paul Posada - New South Wales
“We need actions not words we need honest democratic govt We need a pm
Bob Pearce - South Australia
“Thank you for this great resource. I was feeling helpless. Even this small step
Silvia Anderson - Victoria
“If political parties continue receiving political donations, we will rarely have politicians working for
Dan Chicos - New South Wales
“I only vote for people who will take urgent action to restore a safe
Susie Burke - Victoria
“Current government is not representing the opinion of the majority of Australian to meet
Neil Price - Tasmania
“We are fighting to rescue our kids' future from those who seek to steal
Vanessa Norimi - Queensland
“No time to waste Now or Never My vote is for NOW
Rosalie White - Victoria
“I am only 9 but I already care
Ava Bell - New South Wales
“From New Lambton Uniting Church - Caring for our world is a moral imperative.
Niall McKay - New South Wales
“Our federal govt is an International climate Embarrassment - its about time they stepped
Oriana Tolo - Victoria
“Vote earth this time!
Sue Cooke - Queensland
“We are in one on the wealthiest countries in the world. we have the
rowan huxtable - New South Wales
“The climate Emergency is the public health opportunity and urgent priority of the 21st
Mike Forrester - Victoria
“If they want my vote they better act now
Barbara McNiff - New South Wales
“We need to act locally now for the earth. Our only home. Vote Earth
Anne Miller - New South Wales
“I often look at the places I've known all my life and see how
Jim Baird - New South Wales
“Strike one For people power!!! Democracy might prevail outside the current cronyism that faces
Lorraine Bridger - New South Wales
“Our federal politicians Are Afraid to make action on climate change a major election
Jennifer Martin - New South Wales
“climate election, let's go!
Fahimah Badrulhisham - New South Wales
“Great to see this website that is a focus on action for climate change
Lynette Sinclair - New South Wales
“Let’s show politicians and the Murdoch media that climate change is by far the
Jane Aitken - Australian Capital Territory
“If you want to stay in power You need to take action to stop
Jane Bulter - New South Wales
“We are all that stands between terminal climate change and the vulnerable. We are
Carol Khan - Queensland
“We need a Government that Believes this is real and not taking money from
Ken Gray - New South Wales
“I'm voting for my childrens future
Anneliese Alexander - New South Wales