People in remote-operations centres in large cities have been controlling locomotives and haulage trucks at major Australian mining operations for some years, with automation and robotics moving humans from dirty and dull tasks in dangerous environments to safer and more desirable locations where they can do their jobs more effectively.
Human-in-the-loop tech making remote operations more immersive
CSIRO's Integrated Mining team is working on solutions for yet another 'D' problem, distance, using mixed-reality and digital-twin technology to enhance non-line-of-sight control by making remote operations more immersive, giving operators a stronger sense of 'being there'.
"We call remote operations 'human-in-the loop-automation', because even though automation is changing their roles, people are essential to operations as they’re a lot more creative and can respond to exceptional circumstances more effectively," explains Craig James, Senior Research Projects Officer for the Integrated Mining team in Mine Processing Technologies (part of the Sustainable Mining Technologies program of CSIRO Minerals Resources).
Enhancing situational awareness using digital twin
"Situational awareness – knowing what's happening around you when you are controlling something - is crucial to remote operations – that's what we’re working to create," says James, who operates CSIRO’s Remote Management Centre [RMC]. Typical remote-operations centres – which most big mining companies now have – see people working with multiple screens, keyboard, mouse and perhaps a joystick or steering wheel.
However, that set-up, where people are somewhat disconnected from the process is limited in its ability to put people into a 'flow' state, which helps them to use a wide range of incoming information more instinctively and effectively.
"Immersing people deeper in virtual environments, modelled with actual data that replicate mining processes in real time, gives a much more powerful perspective, letting them step in rapidly as required," says James.
"This is the essence of the digital twin, and it helps you to focus on what is important in the environment around you."
The digital twin merges reality into a virtual environment
"Mixed reality is a term that covers a range of technologies from augmented reality [AR] to virtual reality [VR]. In between AR and VR is augmented virtuality [AV], also known as the digital twin," explains James.
"It involves bringing real-world data into virtual environments, and it's where we do most of our work at the RMC."
The RMC, in Pullenvale, Queensland, is equipped with cutting-edge hardware – including a 6-metre cylindrical display system, a multi-display wall, and wearable computing gear, including a wide range of VR and AR headsets.
It's an impressive display for the industry, government and academic visitors who tour the RMC each year, but it's the work developing the systems to bring it all to life which is critical, as more mining companies and more areas of the mining process move to being remotely controlled.
Solving the distance problem though digitally enabled technologies
"People want to work in the cities," says James of the move to reduce FIFO and DIDO roles in the sector in favour of remote operations.
Early on in the remote-ops era, many workers previously operated heavy machinery on a mine site, so even when they switched to remote control, they had "experienced it first-hand and could translate those skills to the new [remote] interface" explains James.
But as those people retire or move on, "you lose a lot of that first-hand skills and knowledge."
Initiating 'serious' gaming
Recently, the computer gaming industry has been reducing the cost and increasing the quality of technologies to create immersive, mixed-reality environments. These gaming technologies can be applied to real-world problems in a field called 'serious gaming'.
"Not so long ago, a flight simulator would cost a couple of million dollars," explains James.
"That kind of technology is still critical in some cases, but now we see very good outcomes, across a wider range of roles, from people using serious gaming systems that cost more like $6,000."
Using holograms for immersive training
AR and VR headsets showing holograms of equipment or of a process – which can increasingly be controlled by natural gestures – are already being used to provide immersive training for mining employees who did not come into the industry via a traditional on-site job.
Remote-ops training for people who will never be on a mine site using "this immersive, mixed-reality technology can train new generations and give them ready access to high-quality, spatially accurate information," says James.
"For example, there are virtual-reality systems that can train people on hydraulic excavators and build up the muscle memory that you can’t get by watching videos or using simulators driven by keyboard and mouse."
"There's also growing research on how digital twins can be used to improve 'human in the loop' automation."
CSIRO teams are working in this area of serious gaming to advance from training systems to operational ones, enabling workers to interact in real-time with the digital twin, and thereby observe and control the physical world.
"We aim to help people do more with less and continually improve the whole process," says James.
Using digital twin to test technologies in challenging environments
"We have a lot of experience in applying digital twin technologies in the mining industry, particularly in challenging underground environments where it has been difficult to acquire the data to power the digital twin – for example our laser-based ExScan sensor, part of our LASC [Longwall Automation Steering Committee] suite of technologies, now enables us to faithfully recreate the snapshots of the dynamic underground mining environment ," says James.
The future of this technology isn't only earthbound.
As well as "the terrestrial side of things" adds James, the team he works in is also connected to the CSIRO Space Technology Future Science Platform.
Digital twin for space exploration
The vision is to build digital twins that will advance mining processes for future space exploration, where the distances are even more extreme.
Mining remote resources will be crucial for space operations to be sustainable.
"When you go to the Moon or to Mars, you need to generate oxygen, water, fuel and building materials," he says.
"There is a whole process of ultra-remote mining being investigated called In Situ Resource Utilisation [ISRU], and digital twins can help with the operational side. So far, we've only scratched the surface – CSIRO is working hand in hand with the Australian and international space agencies to take our terrestrial technologies and apply them to our burgeoning space industry."