Shell is building an integrated data platform to accelerate the energy transition

Petrochemical giant Shell says it is increasing the efficiency of its operations through better use of data across the enterprise.

This is made possible through its new platform, which, according to the company’s internal data team, enables hundreds of engineers, scientists and analysts to innovate together, thanks to the democratization of data analysis and AI.

This is important, as Dan Jeayants, Vice President of Computational Science and Digital Innovation at Shell, explains:

If you think about where we are now, in the energy sector, we have a major problem, which is that most energy resources are still very heavily dependent on hydrocarbon-based fuels.

To quote Al Gore, Jeavoirs jokes, it’s an inconvenient truth, but it’s something companies like his know they need to address quickly. It has been estimated by the World Economic Forum, for example, that the global carbon footprint could be reduced by up to 20% through better deployment of large-scale digital technologies.

However, it will take major CapEx investments over several years to change the infrastructure that companies like Shell rely on. However, we said:

And that’s going to take time, of course – but one of the things we can do now is take advantage of digital technology, which is much faster and easier to deploy at scale. This can then allow us to have a significant impact on our CO2 emissions.

Jeavoirs – who has just moved to Bangalore to better oversee its 300-person global data team – says the company is prioritizing better use of data from the new platform to accelerate the company’s transition to greener working methods.

This will happen, he predicts, through better use of digital to make Shell’s existing business more effective and efficient.

For example, using data in smarter ways to enable design process changes using advanced simulation and computer science – combining things like AI and physics, he says, which can then translate to greater efficiency in the field.

Finally, and data analytics will also be leveraged to optimize the management of the more complex, distributed and diverse energy systems that Shell expects to have to deploy in the future. Jeavoirs said:

Optimizing all of this at scale is a challenge, so digital technology plays a key role in ensuring that the investments we deploy perform optimally.

Unify large amounts of sensor data

I said that in his industry, the obvious – and best – starting point is with data. However, the company wants to achieve this by more efficiently processing large-scale global time-series data. He explains:

If you think of a company like Shell, everything we do is physical – from our retail stations to the pipelines we operate, the wind turbines we operate, the solar farms we operate, our processing refineries and also, of course, in the upstream business where you have existing oil and gas assets.

There is data all over this map, he says, in the form of enormous amounts of individual measurements, for example, weather conditions, or temperatures, pressures, rotational speeds, etc.

This translates to a “large amount” of data from a “vast range” of sensors, but historically Shell has found the only way to process this data is through highly localized on-site data warehouses.

To get something like an enterprise-wide central data warehouse to run queries at the desired level and complexity, Shell had to create aggregation mechanisms, which often leaked data. Jeavoirs said:

The real reason you localize is that you can’t handle scale, and so you’re effectively simplifying and removing information as you go through those different stacks. We did pretty well, but we still struggled to do it on a large scale. And of course, when machine learning and AI came along, we wanted to be able to bring all of that information in its rawest form to train the algorithms to help optimize the overall energy system.

We seek to integrate all of this data into an integrated cloud environment, and then run both reporting mechanisms, standardized APIs, and also machine learning tasks, from the same integrated data architecture.

This was ultimately achieved, he says, through the use of vendor Databricks’ “data lakehouse” technology, with initial conversations beginning with the vendor in 2019. data for reporting purposes – the warehouse – can be gathered. A data lake has also been activated, which can be used to train new machine learning models that Shell wants to exploit.

The result, the company says, is a scalable, fully managed platform that unifies Shell’s entire data analytics lifecycle. Jeavoirs opposes past and future, and says:

Historically, to run a global query, you probably would have had to interact, at a minimum, with at least 10-12 different systems. Now you can do it from a single platform at scale in a powerful way. It allows us to answer all kinds of questions that we couldn’t answer before and to compare all kinds of parts of the business that were very difficult to compare historically.

Now there is only one data product that Shell business people can interact with to develop their dashboards and powerful applications, without having to shop everywhere to integrate all kinds of sets data across the enterprise.

The equivalent of removing 28,000 American vehicles from the road

To put things into perspective, Shell says it now has at least 2.7 trillion rows of data on its analytics platform. A practical way to have all this data in one place, he says, includes a radical improvement in proactive engineering monitoring, where its engineers know better when things could go wrong with the equipment – ​​where Shell can then step in. proactively. 10,000 pieces of equipment are now monitored in this way, with machine learning processes directly driven by training data in the lake.

Another key objective, as noted, was performance optimization. The primary benefits of unified data analytics across Shell are the reduction of the CO2 impact of its LNG (liquefied natural gas) trains. Jeavoirs said:

We have shown that we can increase the production of LNG trains by approximately 1-2% of liquefied natural gas trains, using the data from this.

We have demonstrated that we can remove something like 130 kilotonnes of CO2 emissions associated with this process, which is roughly equivalent to removing the equivalent of 57,000 European vehicles or 28,000 American vehicles from the road.

Jeavoirs concludes that he is convinced that data integration is the best basis for Shell to accelerate the energy transition.