CSIRO's GPU cluster being installed at the Data Centre.

CSIRO's GPU cluster being installed at the Data Centre.

Speeding up science: CSIRO's CPU-GPU supercomputer

CSIRO's Bragg supercomputer cluster was among the world's first to combine traditional CPUs with more powerful graphics processing units or GPUs, providing a world class computational and simulation science facility to advance priority CSIRO science.

  • 19 November 2010 | Updated 28 November 2013

Scientific research is constantly generating vast quantities of highly complicated data. Handling and processing this information requires advances in data management, analytics, computing and collaboration tools. 

Graphics processing units (GPUs) have typically been used to render complex graphics in computer games. However they can also be used to accelerate scientific computing by multi-tasking on hundreds of computing cores.

Using GPUs to analyse large and complex research data sets is a relatively new global trend in computing.

CSIRO aims to be at the forefront of GPU super computing for scientific and industrial research. Our investment will enable the technology to be applied across CSIRO from desktop GPU workstations to our GPU supercomputer cluster, helping to solve big data challenges in many research fields from bioscience, image analysis, fluid dynamics modelling and environmental science.

CSIRO's GPU cluster

CSIRO’s GPU cluster was named Bragg, after Australia’s first Nobel prize-winners – father-and-son physicists Lawrence and Henry Bragg. It was the first of its kind in Australia combining traditional central processing units (CPUs) with graphics processing units (GPUs) when it launched in 2009.

Running dual Linux and Windows applications it caters for the needs of scientists who use different operating systems for running software codes and analysing data from experiments.

It is currently one of the fastest supercomputers running a Windows HPC operating system in the world.

Ranking in the TOP500 list of the world's fastest and most powerful supercomputers, Bragg is also ranked highly on the Green500 list - a ranking determined by energy efficiency, making it one of the most energy efficient supercomputers in Australia.

The GPU cluster was supplied by Xenon Systems of Melbourne and is located at a data centre in Canberra, Australia.

GPU technology

Watch this video about the CSIRO GPU cluster.

The player will show in this paragraph

The CSIRO GPU cluster

Transcript

[Dr John Taylor, CSIRO Computational and Simulation Sciences appears on screen]

Dr Taylor: The CSIRO GPU cluster is now open for business.

[Image changes to a man walking down the corridor]

Narrator: In 2009 Dr John Taylor switched on CSIRO’s GPU Cluster.

[Image changes to show a data centre]

Dr Taylor: GPU computing really is a revolution in computing.

[Image moves back to Dr Taylor]

We’ve been able to take this new technology that was originally developed to support computer games and apply it across the board in science.

[Camera zooms back in on the data centre]

Narrator: A GPU or Graphics Processing Unit, processes data much faster than the central processing unit or CPU found in your home computer. Joining many hundreds of GPUs with CPUs in one super computer means they can churn through masses of data in very short time. Speeding up image rendering, scientific calculations and computational models, hundreds of times faster than CSIRO’s scientists were used to.

[Image changes back to Dr Taylor]

Dr Taylor: Probably the most exciting moments are when I get a call or an email or somebody tells me that something’s running a thousand times faster and they’re really thrilled about it, that’s probably the most exciting thing. So, it’s a real sign that the technology is very beneficial to peoples research and that people are getting great results.

[Image changes to a 3D rendered picture of a sharp tooth]

Dr Mayo: It’s a 3D rendered view of 3D data set of a micro fossil, which is a tiny tooth from a prehistoric animal. The whole tooth is less than a millimetre across.

[Image changes to an image of a large silicone sphere with lots of bubbles inside]

[Image changes Dr Sherry Mayo, CSIRO Materials Science and Engineering]

It’s a ceramic microsphere, it’s made of silicone and it has little bubble s in it and the whole thing’s about 100 microns across, so that’s only about half again as thick as a human hair, so it’s quite small.

[Image changes back to image of a large silicone sphere with lots of bubbles inside]

Narrator: CSIRO Materials Scientists are using GPUs to create highly detailed images from 3D data sets collected at the Australian Synchrotron.

[Image changes to show the Australian Synchrotron building]

[Image changes to inside the Australian Synchrotron building]

When electrons are fired around this Olympic stadium sized tracked, powerful magnets force them take a circular path.

[Image changes to show a Scientist setting up equipment]

As they do they omit light a million times brighter than the sun, this powerful beam can be focused at any wavelength down beam lines. X-rays, for example, filled up a highly detailed picture of any target in their path.

[Camera zooms in on computer on screens]

But it’s a data intensive task. Where Scientists may have waited hours for their results in the past, the GPU cluster churns through the raw data quickly, displaying amazing images in real time.

[Image changes to a 3D rendered picture]

[Image changes back to Dr Mayo]

Dr Mayo: It’s actually been really fun just from a sheer visual point of view and then we can get science from it too, of course.

[Image changes to Dr Andrew Stephenson, CSIRO Materials Science and Engineering]

Dr Stephenson: I think the fact that we’ve got everything in one place, so we’ve got our own synchrotron now and it’s quite close and we’ve got ready access to a world class beam line.

[Image changes to a 3D rendered picture]

And if we have world class computing facilities to use with that, I think we can capitalise on that to a great extent and get a lot more out of our data and get our data much more quickly than we could in the past.

[Image changes to Dr Lawrence Murray, CSIRO Mathematics, Informatics and Statistics]

Dr Murray: What we’re looking to do is model large scale environmental systems. Using the GPU cluster we can do that more accurately, more precisely and we quantify uncertainty, which is very important from a management perspective.

[Image changes to a computer generated graph]

When you do have a powerful machine like this, your productivity increases in a sense, what might have taken a few days, might take a few hours now. What might have taken a week, or was completely infeasible because it was going to take a month, well that’s out of the question in terms of productivity and the turnaround in your work. If we can get results in a few hours, or overnight and be looking at them the next morning, that really makes a difference as to how we can then say, look at those results and make any adjustments we need to the model, or our inputs or anything like this and in terms of our work flow then, you know, it really opens up some opportunities for us.

[Image changes to Dr Emma Huang, CSIRO Mathematics, Informatics and Statistics]

Dr Huang: The problem that I’ve been working on the GPU most for is to construct genetic maps in wheat.

[Image changes to a camera panning over a wheat field and zooms in on a field]

And the issue there is that you actually have to calculate the distance between every position that you have on the genome in order to construct a map and so that actually means you’re looking at the square of the number of positions that you have, which is where the issues with computation come in.

[Image changes to show a computer generated image]

[Camera zooms back in on the wheat field]

But it does break up quite nicely for a GPU because you can look at each position individually and do it for each wheat variety.

[Image changes to a bread roll]

And the idea there is really to improve the quality of wheat, worldwide, and to do it in a faster way than has been done before.

[Image moves back to Dr Murray]

Dr Murray: Really, using the GPU cluster is quite interesting because it’s a shift in the way we think about computation. I mean, we’ve gone from this sort of sequential computation, which is almost like following a recipe, a sort of a cookbook recipe, to large scale parallelism, so doing lots of tasks at a time, as opposed to one long task very quickly.

[Image changes to people working on laptops]

So people really need to rethink the algorithms that they use. Rethink the approach that they use to their particular computation challenges in science.

[Image moves back to Dr Murray]

You need to be thinking about how can I divide up the work into lots of little small things that I can do concurrently as opposed to one sort of long recipe to follow.

[Image changes to someone working on the Top 500 list website]

Dr Taylor: The Top 500 list it’s very important, it provides us with some indication of how successful we’ve been in building and exploiting computational facilities.

[Image changes to a computer screen with the The Green 500 site opened]

The fact that also that we’re very close to the top of The Green 500 indicates that, not only, are we achieving a very high performance, but we’re delivering that at very low energy costs.

[Image moves back to the data centre]

We’re reducing our carbon footprint, so we’re able to deliver this fantastic capability at very low cost and very low energy cost.

[Image changes to Andy Keane, General Manger, Tesla Business, NVIDIA]

Andy Keane: GPUs are an entirely new way of computing information. Rather than separating that information among hundreds of processors with very few cores, a GPU has hundreds of computing cores inside. So you can take that application and divide it among all those hundreds of energy efficient cores for very efficient processing of HPC applications.

This is why the GPU is among the systems that are leading on The Green 500.

[Image moves back to Dr Taylor]

Dr Taylor: The CSIRO GPU cluster is currently delivering 52.55 TFlops on the standard Linpac test in double precision, its running at very high efficiency and its consuming about 94.65 kilowatts of electricity when it’s doing that. That means that we’re able to achieve about 555 mega flops per watt of energy consumed. So that puts us very close to the top of The Green 500.

[Image changes to picture of servers in a rack]

The cluster was operated from the original Tesla S1070 to the latest generation S2050s.

[Camera zooms in on a computer screen]

We’ve seen about a five to six X improvement in performance.

[Image moves back to Dr Taylor]

So yeah, I think it’s been a fantastic year or so and I think a lot of people have really enjoyed the benefits of our new GPU based cluster.

[Image moves back to Dr Huang]

Dr Huang: It allows you to think of problems that you wouldn’t have been able to address before.

[Image moves back to Dr Murray]

Dr Murray: When we first started on this, the few of us who are now talking at these workshops, were the only ones in the GPU cluster and we could submit jobs and they would all run immediately, and we see now, we submit jobs and they all go into quite a long queue because people are actually starting to use it and starting to realise the benefits of it.

[Image moves back to Dr Taylor]

Dr Taylor: A year ago we didn’t really know what the uptake was going to be within CSIRO, now we know that it’s been quite strong and we’ve been very successful, so I think, yeah, a year on we can see that we’re on the right track and we can also see a track that’s leading forward with this technology for the next few years.

[Credits role: CSIRO, Dr John Taylor, Dr Sherry Mayo, Dr Andrew Stephenson, Dr Lawrence Murray, Dr Emma Huang, NVIDIA Andy Keane. Thanks to NVIDIA, The Australian Synchrotron, Canberra Data Centre, XENON, APC, Screen Craft, Luke Domanski, Tad Matuszkiewicz, Dragan Dimitrovici. Production Assistance, Sarah Wood, Carrie Bengston. Music, Harley Oliver]

[Produced and Directed by Harley Oliver]

[NVIDIA© CUDA™ RESEARCH CENTER]

[CSIRO © 2010]

The unique feature of the CSIRO GPU cluster is the use of NVIDIA GPU technology to deliver outstanding computational performance at low cost and low energy demand.

Compared to the latest quad-core CPUs, the Tesla C2050s inside the CSIRO GPU cluster deliver the equivalent supercomputing performance at 1/10th the cost and 1/20th the power consumption.

The GPU technology can be accessed using the CUDA parallel computing architecture [external link] or using new compiler technology released by the Portland Group [external link].

CSIRO science applications have already seen 6-200x speedups on NVIDIA GPUs.

Driving computational and simulation science

The CSIRO GPU supercomputer supports computational and simulation science research across CSIRO.

The technology is used to support CSIRO research in the areas of:

We are using the GPU cluster to speed up modelling of CSIRO’s air cargo scanner. The speed up is 5000 to 10 000 times that of a normal desktop computer when we use most of the cluster. With this improvement, simulations that normally take hours can be run interactively in real-time.

Dr James Tickner, Minerals Down Under Flagship, CSIRO

  • computational biology
  • climate and weather
  • multi-scale modelling
  • computational fluid dynamics
  • computational chemistry
  • astronomy and astrophysics
  • computational imaging and visualisation
  • advanced materials modelling
  • computational geosciences.

CSIRO projects being undertaken using the cluster include:

  • high-content analysis of nerve cell images for medical research and drug discovery
  • deconvolving (un-blurring) three-dimensional (3D) images from astronomy, microscopy and medical imaging
  • reconstructing large 3D computer tomography (CT) images from medicine and materials science and developing specialised software to run on the Australian Synchrotron
  • quantifying uncertainty in complex environmental models
  • simulating in high resolution biomechanical processes like the movements of a person swimming.

CSIRO CUDA Research Centre

In 2010 CSIRO was selected as part of NVIDIA’s prestigious CUDA Research Centre Program [external link]. We join a community of research institutions that are embracing GPU computing across multiple research fields including; Johns Hopkins University in the Unites States, Nanyan University in Singapore, Technical University of Ostrava in the Czech Republic, and SINTEF (Norway).

As a CUDA Research Centre, CSIRO gains access to the latest developments in GPU technology and is part of a wider community of organisations with GPU facilities, sharing information and ideas.

E-research agenda

E-research underpins the future delivery of great science.

Modern scientific research is increasingly generating vast quantities of highly complicated data. Making the most of the rich information it contains is the key to success.

Handling and interrogating this information requires advances in data management, computing and collaboration tools – of which the CSIRO GPU Supercomputer is one.

Our e-research agenda is supporting CSIRO’s Transformational Capability Platforms, building our capability to sustain and accelerate the delivery of solutions for national challenges.

The Platforms are:

  • computational and simulation science
  • transformational biology
  • sensors and sensor networks
  • advanced materials.

To find out more about CSIRO's Central Processing Unit (CPU)-GPU supercomputer cluster and its capabilities contact:

Dr John A Taylor

Leader: Computational and Simulation Sciences

Phone: +61 2 6216 7077

Alt Phone: +61 2 6216 7000

Email: John.A.Taylor@csiro.au

A NVIDIA Cuda Research Center logo.

NVIDIA Cuda Research Centre.