HOW do you crash a supercomputer? Try crunching the numbers from 150 terabytes of data water simulation models, reading inputs and writing outputs simultaneously.
The recently-completed Bioregional Assessments Programme is a prime example of the value of big data. It explored the risks to water resources, and the assets that rely on them, posed by an estimated $2.6 billion of proposed coal mines and coal seam gas developments.
Six-years and 200 experts in the making, it covered regions across 860,000 km² in central and eastern Australia.
It included running complex simulation models – frequently too much input/output for even the world’s most powerful supercomputers.
One of the challenges of the program was how to make supercomputers more flexible to cope with the motherlode of data being processed.
Creating information from data
The Australian Government Department of the Environment and Energy commissioned CSIRO, the Bureau of Meteorology and Geoscience Australia, to undertake the BA program in 2012 to provide the best science available to inform decisions around major coal and coal seam gas (CSG) developments, and provide a common, objective information base for governments, landowners, communities, businesses and investors.
Six years on, the invention and implementation of new scientific methods through the $62 million BA technical program has led to the world’s most comprehensive biophysical assessment of the cumulative impacts of resource extraction, and the communication of findings in forms that regulators, communities and industry can easily understand and use.
“The science moved the goal posts,” says Mr Peter Baker, the BA program’s Technical Assurance Reference Group chair and Principal Science Advisor of the Australian Department of the Environment and Energy. “It is new; it is innovative. We were all working together to get the best outcomes that we could.”
It has changed the way we do collaborative science.
From day one it involved consultation across communities and governments to establish an understanding of the water-dependent assets most valued in each region. Assets identified included surface and groundwater management areas including bores and water rights providing economic benefits; environmental assets, such as wetlands and springs, and the landscapes reliant upon them; and sociocultural assets such as important cultural and social sites and habitats of species with cultural significance.
“What the community thought was important was a key part of the establishment of the BA program,” says Mr Baker.
Upping the ante on environmental protection approvals
The issue of water resources in relation to CSG and large coal mining developments was elevated to a matter of national environmental significance in 2013 through the introduction of ‘water trigger’ provisions under the Commonwealth’s Environment Protection and Biodiversity Conservation Act 1999 (EPBC Act).
This means that CSG and large coal mining developments that are likely to have a significant impact on water resources now require Federal assessment and approval. Historically, state regulators assessed individual proposals independent of other existing or proposed mines, creating an environmental risk if cumulative impacts are not properly accounted for.
The BA program developed an impact assessment framework and comprehensive regional information base which can be used to inform and evaluate site-specific development proposals.
CSIRO’s BA project leader Dr David Post says the complex assessment process included regional context setting, building development pathways, undertaking hazard analyses, and modelling the movement of water through the landscape. To understand the potential impacts on the ecology, it meant creating landscape classes to categorise different parts of the environment, determining key water-dependent assets, and understanding causal pathways of connectivity within those environments.
Motherlode of data
This involved developing geological models to understand the barriers and conduits for water movement under the earth’s surface, and incorporating groundwater and surface water models to investigate changes in surface and groundwater, and then investigating how the ecology might respond in the face of changes to water regimes.
This meant lots and lots of data. “We repeatedly broke some of the world’s biggest computers,” says groundwater modeller Dr Russell Crosbie, about the challenge of making super computers more flexible for the BA’s big data needs.
“The biggest problem we had was that we could only use about 200 nodes on machines that have 10,000s nodes due to our requirements for reading and writing of input and output files.
“Supercomputers are designed for making huge numbers of computations very quickly. Our use case required fast computations as well as the ability to read input files of many GBs and then write output files that were even bigger. The throughput of the link between the compute nodes and the storage only allowed us to run 200 models at once when the compute resources available could have run thousands of models.”
The program has left a significant legacy of tools and information, including:
- outputs for 13 regions covering 860,000 km2 , consisting of 104 technical reports, more than 2000 datasets, and generating 150 terabytes of data
- identifying about 26,000 km2 with potential for hydrological changes due to coal and gas developments, with about 1,800 ecological, 250 economic and 90 sociocultural assets that could potentially be impacted by hydrological changes. The assessment also ruled out impacts for another 14,000 ecological, 1600 economic and 860 sociocultural assets
- geological information across 10 of Australia’s major sedimentary basins, providing a better understanding of the movement of groundwater and interactions between different aquifers
- an understanding of the cumulative impacts of multiple types of developments, such as CSG development adjacent to open cut or underground mines, the impact of underground mines on surface water, and at multiple locations across landscapes, particularly in areas where there were assets of interest. This approach to cumulative assessments can be used across other resource sectors
- development of an uncertainty framework which allows us to understand the likelihood of something happening. This innovation has allowed researchers to rule out, potentially-impacted streams, and to look at the significance of potential changes to groundwater and streamflow. Since the release of the BA reports, uncertainty analysis is now required as part of all future environmental impact statements submitted to the IESC
- advances in analysing risks through the development of landscape quantitative risk analyses allowing us to understand the way things could change, but using a deliberate and rigorous approach to identifying the potential hazards and the likelihood of them happening
- a new ‘ecosystem services thinking’ approach to investigating water-dependent ecosystems through development of a conceptual understanding of water dependency using landscape classes to understand how a system fits together, how much of a system is likely to be impacted, and (for instance) how much of a river system or the habitat of a particular species sits within that area
- information accessible to broader audiences through an online explorer tool. The BA Explorer is a spatial mapping and data visualisation web-based tool that allows the user to access information from regions of interest to better understand potential impacts on parts of the environment of particular interest to the end user.
- the registration and ongoing ability to identify and trace data sets from the program, when combined with their open publication, which not only allows complete transparency on the data driving the analyses within the program, but also allows their reuse, and future impacts to be tracked over time, helping to quantify the legacy of the research.
Working with the results
The results of the BA program are being used by the Independent Expert Scientific Committee on Coal Seam Gas and Large Coal Mining Development, the statutory committee of leading scientists that independently advises government regulators on the impacts that CSG and coal mining may have on Australia’s water resources.
Results are also being used by a range of Commonwealth and State regulators, industry and communities. As a science-based regulator, the Australian Government Department of the Environment and Energy relies on science such as this for decision making.
A 2017 Jacobs Group (Australia) independent evaluation described the BA program as “the most inter-locked and connected program (the collaborators) been involved in”.
The program was awarded the CSIRO’s Chairman’s Medal for Science Excellence, recognising teams that have made significant scientific or technical advances that create value through innovation that delivers positive impact for Australia.
CSIRO Land and Water Director Jane Coram says the program’s inception was a courageous step on part of government.
“It was a precedent-setting step that challenged us all scientifically, and incorporated high-level communications, project management and stakeholder engagement,” she says.
“It required an enormous amount of interaction between scientific disciplines, and between science organisations. It required relationship building across and within institutions; it required, at times, difficult conversations because it challenged us. But we will reap the benefits from it for a long time to come.”
Former BA Technical Programme Director Bronwyn Ray, from the Bureau of Meteorology, says the program was “a fantastic example of what happens when you build trust between disciplines. The gaps in between is where the innovation happens, and how we solve problems”.
A key achievement was the program’s delivery of open access to data, models, workflows and reports, through a curated information platform and a map-based interface to query the scientific results, she said.
As well as creating new benchmarks in transparency, the lasting legacy of science, data and systems developed through the program helped change the way we think about doing science in the Commonwealth, according to Geoscience Australia’s Dr Stuart Minchin.
“That’s enormous credit to the collaboration that happened across the team,” Dr Minchin says.
The Australian Government has commissioned a follow-up four year, $30m Geological and Bioregional Assessment (GBA) program. This builds on the BA approach to examine the potential impacts of shale and tight gas extraction, informing future development decisions.