Researchers across the UK have been hacking their way through Terabytes of climate data ahead of the 26th UN Climate Change Conference of the Parties (COP26) in Glasgow on 1st-12th November 2021. Three virtual hackathon events were run, during May and June 2021, by a range of UK Universities and the Met Office, covering various topics:
Aim: to generate usable and impactful climate narratives for decision makers in East and West Africa
Aim: to produce cutting-edge research using Climate Model Intercomparison Project 6 (CMIP6) data, via a variety of projects
Aim: to produce novel solutions for understanding and quantifying climate risks in future energy systems
All three hackathons conducted their data storage and analysis on JASMIN - the UK’s data analysis facility for environmental science. Access to climate data (such as CMIP6) held in the CEDA Archive, alongside the processing capabilities of JASMIN, were essential for these hackathons. The hackathon organisers hope to present outputs at COP26 in November and through peer-reviewed papers.
The hackathons had over 150 attendees in various teams, investigating topics ranging from climate change to oceanography, biogeochemistry, and more.
Julia Crook, organiser of the Africa hackathon, said “During the hackathon, we had two groups looking at the effects of climate change, one focused on how it will affect hydroelectric power in Kenya, the other on agriculture in Ghana. Having a shared workspace, alongside analysis tools such as the Notebook Service, with direct access to CMIP6 data was essential for the success of our hackathon.”
Attendees made use of a variety of JASMIN services, depending on their experience and research needs. James Thomas, organiser of the CMIP6 Data hackathon, said “The ability to give participants access to the JASMIN infrastructure in such a variety of ways meant we were able to cater for a wider range of users at the hackathon, and allow researchers to concentrate on the science as much as possible. Whether this be Jupyter Notebooks for newer users, command line access, or even batch processing systems for our more advanced users, there was something for everyone”.
A five Terabyte shared workspace allowed teams to collaboratively share and analyse data across the many different academic organisations.
The JASMIN Notebook Service provided a web interface where the attendees could interactively analyse data using Python. The Notebook Service is perfect for non-experts - allowing code, documentation and visualisations to be edited and shared. At one point, over 70 notebooks were running simultaneously, the largest number to date!
Training accounts were provided to some attendees, allowing short-term access to data and services for those who may not usually be able to access JASMIN. James Thomas explains how these accounts helped “The ability to provide short-term access to researchers without accounts definitely removed a barrier to getting involved. In fact I’m happy to say that based on their experiences during the hackathon, some have now gone on to request full JASMIN accounts and intend to utilise the platform in the future.”
Energy hackathon organiser, Hannah Bloomfield, said “I don't think there is anything the JASMIN team could have done to make this easier for us. The online help is really good - once users got into the swing of things they were able to answer their own questions by searching the help pages. The JASMIN team also helped us to get some specialist software, called Gurobi, set up at very short notice - this software turned out to be key to a couple of projects' success. We couldn’t have done it without the staff expertise at JASMIN.”
If you are interested in using JASMIN for a training event or hackathon, please get in touch with the JASMIN helpdesk.
The CMIP6 hackathon has created a template virtual hackathon website for future events. This is freely available with instructions on how to copy and adapt for your specific event.
Image: One hackathon team investigated how future weather would impact agriculture in Ghana. In particular they looked at how drought intensity and duration changed under future climate predictions. Photo by Anaya Katlego on Unsplash
Purpose: to generate usable and impactful climate narratives for decision makers in East and West Africa
Organised by: University of Leeds, Met Office
For further details about the event take a look at the webpage
Contact: Julia Crook
Some details about the two projects are below, further details will be published in a future paper:
The KenGen team looked at the impact of climate change on hydroelectric power in Kenya and built a dam level model to estimate shutdown and overflow events from precipitation and evaporation data; the model was parameterised against observed weather and dam response. Future CMIP6 scenarios were used to feed the model to estimate future responses.
The Agriculture team looked at how future weather would impact agriculture in Ghana. In particular they looked at how drought intensity and duration changed under future climate predictions. They found droughts of a similar magnitude to the very rare 1983 drought occur a bit more frequently under SSP119 but much more frequently under SSP585.
CMIP6 Data hackathon
Purpose: to produce cutting-edge research using Climate Model Intercomparison Project 6 (CMIP6) data, via a variety of projects.
Organised by: University of Bristol’s Cabot Institute and Jean Golding Institute, in association with the Met Office and universities of Exeter, Leeds and UCL.
Contact: James Thomas
The projects covered in this hackathon are described here.
Purpose: to produce novel solutions for understanding and quantifying climate risks in future energy systems
Organised by: University of Oxford and University of Reading
During the hackathon, groups looked into the following topics related to energy systems:
Building a forecasting model to improve predictions of solar power generation over a network of sites in Reading using smart meter data.
Exploring the adverse weather scenarios for renewable energy systems using a dataset from the Met Office with the aim of finding “How extreme is too extreme?”. Team members focussed on visualisation, comparisons, resilience and how the data can be used as an input into a power system model.
Supporting power system modelers in choosing appropriate meteorological data through climate input sensitivity experiments using a European scale power system model (PyPSAEUR)
Building a workflow to quantify climate risk in power system design using distributed computing. The developed workflow converts hourly climate data to dependent energy variables which can then be used as inputs for a power system model (Calliope). Performing the computation in the distributed framework removes the need to store gridded hourly climate output.
Downscaling daily reanalysis/climate model output to daily resolution as an input for power system models using a variety of machine learning techniques.
CEDA and JASMIN
The Centre for Environmental Data Analysis (CEDA) provides services on behalf of the Natural Environment Research Council (NERC) via the National Centre for Atmospheric Science (NCAS) and the National Centre for Earth Observation (NCEO). CEDA is based within RAL Space, a department of the Science and Technology Facilities Council (STFC).
This work used the JASMIN infrastructure. JASMIN is hosted and managed jointly between STFC's Scientific Computing Department and CEDA.
For more information about using JASMIN or data held in the CEDA Archive, please contact our helpdesk.