How to choose the right technology platform for integrating big data

Article #3 in this series discussed how incorporating big data in resource modelling and estimation can help geologists better understand the overall mineral deposit and, as a result, produce higher-quality estimates.

But working with all that big data is challenging, and many mining companies today do not yet have the technology they need to integrate what can amount to many terabytes of complex information in a variety of formats.

Traditional technology

Traditionally, geologists have needed laptops or desktops with high single-core performance, a large amount of on-board memory, high-speed storage — typically 10,000/15,000 rpm hard disk drives or, more recently, solid-state disks — and high-end graphics cards to display the outputs of their resource modelling and estimation.

But this kind of hardware has its drawbacks: it’s expensive; only a limited amount of storage and memory can be installed within a given system; and the technology tends to date quickly.

There are issues on the software side, too. For example, a number of classic geostatistical programs and modules are only capable of using a single processor core or thread for processing, which means they can’t take advantage of the multi-core processors common today. In addition, because the workflows within general mining packages consist of proprietary processes that are often combined with separate customised (expensive to update and maintain) routines, we end up with input and output files in many different formats, from text to binary.

Traditional storage

Then there is storage. The block models or geostatistical simulations and the numerous interim files resulting from resource modelling and estimation processes are typically each multi-gigabytes in size, which can both make local storage problematic and data sharing/transfer difficult.

At the same time, traditional long-term storage solutions for project data — such as CD/DVD/Blu-ray discs, tape and hard disk drives — take a long time to back-up large data files, which means back-ups might occur less frequently than they should, and it’s often difficult to find original data or models required for governance and assurance purposes.

Today’s data centre servers are a better storage workflow solution, because they provide access to more computing power via the use of virtual machines. However, because many of the modelling and interpretation steps still need to take place locally, the reliance on a network connection makes sharing or transferring large files difficult, while also making it impractical to run software or processes via remote desktop connections.

New technology and new storage

A technology platform is a central hub where a mining company can locate, integrate and manage all its systems, software, applications, and operational processes.

A platform suitable for managing big data — like the soft data required for high-quality resource modelling and estimation — will include:

• sufficient and secure long-term storage, readily available on demand, and
• high-availability and/or disaster recovery options across multiple sites or zones.

It will also offer the ability to:

• store data in any format (i.e., the platform is data agnostic)
• easily access and share data with upstream and downstream processes, with full version control
• quickly scale up data processing requirements (cores/threads and addressable memory) to accommodate short-term computing demands
• easily update workflows to incorporate new/updated processes
• incorporate new computing technologies and environments as they become available, and
• require little or no installation of local software or rich clients (applications that store and process data locally rather than remotely).

The future of resource estimation is in the cloud

Most advanced computing platforms today are cloud-based, and cloud-based workflows are, we believe, the future of resource modelling and estimation.

A cloud-based platform:

• Removes storage limitations and allows for on-demand access to both data and processing power — which means the appropriate amount of processing power (computing power, memory and storage) is available only when required, potentially lowering costs compared to physical hardware.
• Ensures high availability, which can replace current back-up and disaster recovery processes, except for those that are time-sensitive and can affect mining/production. (For these processes, other methods are needed for the rare occasions a cloud-based platform is unavailable.)
• Provides a central location to store and share all data, with sufficient on-demand computing resources available to accommodate repeatable workflows rather than a collection of independent, difficult to back-up processes run on separate devices with limited processing power.
• Offers the option of using standardised workflows to capture deep specialist knowledge, which then becomes permanently retained, role-based knowledge — minimising the need to redevelop processes once a particular specialist leaves the organisation and helping to ensure the transfer of skills and experience to new hires.
• Makes it possible to quickly incorporate artificial intelligence and machine learning techniques in workflows to automate a number of time consuming and/or repetitive tasks, thereby allowing technical personnel to devote more of their time to analysis and interpretation of results rather than data manipulation and preparing graphs and reports.
• Enables processes that depend on access to powerful computing resources to be run more efficiently, both in time and cost, than using local machines with limited capabilities. For example, with a cloud-based computing platform, it becomes much more feasible to routinely undertake valuable studies when new data becomes available, such as simulating variations:
  • in the geology model and the resource estimates, and then building multiple mine plans based on these variations, or
  • in the beneficiation process when handling ore with differing chemical characteristics or ratios of ore types.

In short, by being able to store, process, integrate, share and display all available data types required for high-quality resource modelling and estimation, a cloud-based platform will contribute to an overall improved orebody knowledge and understanding of the controls on mineralisation.

This in turn will result in significant downstream benefits, including better blending of material for processing, more consistent plant throughput and ultimately, most importantly, higher product quality and increased profit.

A related technical series, called How to Use Machine Learning in Resource Estimation, starts soon. Topics in this second series include what machine learning is and how it works, along with how it can be used in automatic data domaining to provide geologists with the most suitable sub-datasets to use in estimating distinct volumes of the orebody.

Author

Michael Mattera is a Mining Industry Process Consultant at Dassault Systèmes GEOVIA with 30 years of experience in Industry. Michael holds an MSc (Engineering) in Mineral Economics from the University of the Witwatersrand. He has experience across a wide range of commodities and geographies leading to a broad understanding of multiple mining disciplines and associated technical systems. This experience includes resource modelling and estimation, multi-disciplinary project reviews focusing on Mineral Resources (PFS to Post Investment stages), public reporting of Mineral Resources and Ore Reserves (R&R) in multiple jurisdictions, associated governance and assurance processes and development of multiple R&R reporting systems.