Every morning last week I listened to presentations from OPC Day–which was actually a week. Once again this year was a conference reporting on a vast amount of work done by volunteers from numerous companies that push forward the cause of interoperability in manufacturing/industrial data communications. Earlier this year, I visited the ODVA annual general meeting. This virtual conference by the OPC Foundation is well worth a listen.
There were two or three presentations on MQTT where speakers tried mightily to tread the line between simplifying and explaining how these are not competing technologies and yet evangelizing the benefits of OPC UA over MQTT versus Sparkplug B. The presentations were balanced for the most part. OPC UA is a substantial information model. MQTT is a lightweight transport protocol widely adopted by IT. Sparkplug B is a lightweight information model requiring some extra defining work by the integrators but keeps overhead low. Obviously, there is a place for each.
I’ve added a list of videos from the OPC Foundation YouTube channel for your viewing pleasure:
Day 1: https://youtu.be/2i54Q-2IvCQ Day 2: https://youtu.be/CsXagNmWWjY Day 3: https://youtu.be/8XuTAcG598o Day 4: https://youtu.be/ezSRRaG1fAE Day 5: https://youtu.be/ZzS7Z8a7c1I
I wrote yesterday about some new developments in DataOps. Then I found this news about a survey into users’ needs and lacks in that technology and use case area. This comes from a company called Unravel Data. It bills itself as “the only data operations platform providing full-stack visibility and AI-powered recommendations”. If you’re not sure about DataOps, perhaps the attached infographic from the company will help. Briefly, the survey reveals poor visibility and lack of perspective as the top challenges for DataOps this year.
Unravel Data released key findings from a survey administered to several hundred attendees at the most recent DataOps Unleashed event in order to gauge the priorities, challenges, and progress of leading data teams as they seek to modernize their big data management and analytics capabilities.
“Data is the lifeblood of the modern enterprise and those organizations who have dedicated the resources and budget to modernizing their data stacks are the ones who will be best positioned to drive innovations in the coming decade,” said Kunal Agarwal, co-founder and CEO of Unravel Data. “The results of this latest survey show just how complex the modern data stack has become and illustrates the many unanticipated challenges that come with efficiently managing and optimizing data pipelines across multiple public cloud providers and platforms. It also serves to validate that there is an obvious demand for a purpose-built solution that can help these teams gain the critical visibility they need to drive the most value from their data operations.”
Some of the key findings collected from the most recent survey, which benchmarks responses from the year prior, include:
DataOps as a practice is hitting an inflection point: There was an almost 80% increase from the year prior of respondents who said they are in the active stage of adopting a formal DataOps approach to manage and optimize their data pipelines. This year, more than 41% of attendees reported they are actively employing DataOps methodologies, compared to just less than a quarter (24%) in 2021.
Visibility into data pipelines remains the top challenge: For the second year in a row, when participants were asked what they viewed as the top challenge with operating their data stack, respondents cited the lack of visibility across their environment as their most significant obstacle. Whereas in the previous year respondents reported that “controlling runaway costs” was the second biggest, this year the “lack of proactive alerts” was noted as the second most challenging aspect.
Complexity of cloud migrations is more time consuming than previously thought: Sixty percent of respondents from this year’s event estimated that their cloud migration project would take between 12-24 months, representing a 150% increase over the prior year’s projection. The challenge of forecasting the duration of these cloud migration initiatives reveals the vast amount of complexity and uncertainty that data teams face when attempting to map out these critical projects.
Automation continues to be a key driver: When asked about the role of automation in managing their DataOps pipelines, three in four DataOps professionals in both years reported that the ability to “automatically test and verify before moving jobs/pipelines to production” was the most important automation priority when compared to other aspects such as automating troubleshooting of platform or pipeline issues.
Data teams spend more time building than deploying/managing pipelines: For both years, data professionals reported that they spent the majority of their day building their data pipelines (39% in 2021 and 43% in 2022). In 2022, respondents reported spending slightly less time maintaining or troubleshooting their pipelines (30%) than the year prior (34%) while the time spent deploying data pipelines remained the same at 27% for both years.
I just released a podcast that can also be seen on YouTube where I discuss whatever happened to IIoT. The reason for the whole IIoT frenzy came from management’s need for data. That was all more of an IT-driven project than OT. From the various IoT-related technologies, the industry has moved on to Data Orchestration and DataOps. One of the earlier companies I talked with about data ops was Hitachi Vantara. Additional news from that company follows. Note, also talking “Edge-to-Cloud” which has become the new IoT term in the IT world.
Hitachi Vantara, the digital infrastructure, data management, analytics, and digital solutions subsidiary of Hitachi Ltd., introduced new Lumada DataOps capabilities for automated, AI-driven data operations for all enterprise customers and Lumada Industrial DataOps, providing advanced analytics capabilities for industrial use cases.
Data sprawl and governance have become more difficult as data becomes increasingly distributed across the data center, edge, hybrid, and public cloud infrastructure. This complexity can hinder an organization’s ability to turn data into business value. In a recent DataOps Survey by 451 Research, data privacy, compliance, and data access and preparation are top priorities for data-driven organizations.
Today’s additions to the Lumada DataOps portfolio allow organizations to create a seamless data fabric governed by an enhanced data catalog for automated data quality improvements and governance. With the latest updates to Data Integration powered by Pentaho technology, customers can reduce time and complexity to discover, access, prepare and blend data across multiple data sources and locations. The new Lumada Industrial DataOps portfolio includes IoT analytics models for industrial environments that seamlessly merge IT and OT data to unlock transformational business insights.
Lumada DataOps lets you automate the daily tasks of collecting, integrating, governing, and analyzing data on an intelligent platform providing an open and composable foundation for all enterprise data, while providing self-service data access to their choice of tools and analytics.
Today’s updates to Lumada DataOps include:
Data Catalog – Accelerate business insights with Data Catalog v7.0 using trusted data built on IO-Tahoe technology including a powerful new user interface, data quality and Collibra connectivity.
Data Integration – Integrate data across hybrid cloud with Pentaho v9.3 through flexible cloud deployment and new connectors for cloud data stores like Snowflake, MongoDB Atlas, Teradata, Elastic Search7.x and IBM MQ 9.2.
IT and OT Data Convergence for Digital Industrial Operations
Hitachi Vantara’s new Lumada Industrial DataOps portfolio enables real-time insights and outcomes that power critical operations to be more predictable and manageable. It accelerates IT and OT data convergence by building a data fabric for analytic solutions from edge to multi-cloud. Lumada Industrial DataOps IIoT software automates data pipeline delivery across OT and IT sources, feeding industrial AI and ML models for predictive maintenance and operations optimization. Capabilities of the new Lumada Industrial DataOps portfolio include:
IIoT Core – Accelerate and scale operation application deployment with a complete IIoT data platform including Core, Gateway, Digital Twins, and Machine Learning Services.
IIoT Analytics – Simplify AI and ML solutions creation through toolkits that simplify delivery through packaged Digital Twins with pre-trained ML models
Analyst firms have made considerable publicity projecting the amount of data generated by Industrial Internet of Things. IIoT populates many manufacturing enterprise databases with the promise of better output from analytics applications for improved decision making. With real-time data, operations leaders can use analytics to assess both demand and cost-to-serve, and make informed decisions. Additionally, the customer experience must be personalized and differentiated to be relevant and competitive in today’s digital economy.
Striim, a company I’ve only just now learned about, collects data in real time from enterprise databases (using non-intrusive change data capture), log files, messaging systems, and sensors, and delivers it to virtually any target on-premises or in the cloud with sub-second latency enabling real-time operations and analytics.
Its latest press release is not bashful about promoting its new product. Striim, Inc. announced general availability of Strim Cloud, the fastest way for customers to deliver real-time data and insights to power business intelligence and decision-making to meet the needs of the digital economy. Striim Cloud is the industry’s first and only fully-managed software-as-a-service (SaaS) platform for real-time streaming data integration and analytics. With a few clicks, customers can easily build real-time data pipelines to stream trillions of events every day, backed by enterprise-grade operational, security and management features. Striim Cloud’s zero-maintenance, infinitely scalable platform enables customers to transform their businesses by adopting new cloud models, digitizing legacy systems, and modernizing their application infrastructure.
I, of course, cannot verify any of these claims. We all know that “easy” is a relative term for engineering. But this looks worth checking out.
“Handling and analyzing large-scale data for real-time decision-making and operations is an ongoing challenge for every enterprise; one that is only going to become more challenging as more data sources come online,” said Ali Kutay, founder and CEO of Striim, Inc. “These challenges are driving ‘digital transformation.’ Striim Cloud is a powerful, cloud-based, SaaS platform that gives enterprises worldwide an invaluable advantage in reaching this goal.“
Striim Cloud seamlessly integrates data into platforms such as Azure Synapse Analytics, Google Big Query, and Snowflake. By doing so, Striim Cloud enables businesses to power business intelligence and decision-making to meet the needs of the digital economy while delivering an unbeatable, data-driven customer experience.
Early in my career I reported to the head of product development who realized the critical importance of data that originated with product engineering and design. He appointed me to lead the data function. Little did I realize that the role has become even more critical in manufacturing (and other) enterprises today.
I recently heard from my Datadobi PR contact who shared a copy of an announcement made later via the Datadobi blog with deep thoughts on current trends and requirements for data management. The news relates to a recently published IDC report.
IDC says a Data Mobility Engine Can Serve as the Core of an Effective Data Management Strategy
Research firm IDC predicts that, over the next five years, more than 80% of the data collected by organizations will be unstructured data, and that will only continue to grow 40-50% per year for most enterprises.
In the analyst brief, Burgener urges organizations to implement a comprehensive data management strategy to confront this increasing influx of data, noting that a data mobility engine provides the foundation for an effective data management strategy and can drive significant benefits for the hybrid multicloud enterprise.
In his analysis, Burgener outlines the five main components of an effective data mobility engine, including the following:
1) Vendor-neutral interoperability
2) provide visibility into data metrics, access patterns, and usage activities
3) Orchestration and automation
4) Scan-optimize-copy capabilities
5) Integrity enforcement
Over the last several years at Datadobi, we’ve had more and more IT leaders come to us with concerns around data classification, data visibility, and organization-wide data accessibility, as well as how to handle aging data and the high costs that result from a fragmented data management strategy.
Finally, Burgener states in his report that “the benefits of an effective data management strategy include reduced IT costs, easier data sharing, better security, less legal exposure, and an improved ability to demonstrate governance and regulatory compliance.”
GE Digital has been one busy organization lately. This is the third release along with one interview in the past month or so. The company continues to build on its platform not looking (so far) to bring it all together. I’ve only seen a couple of companies so far who have built from the ground up. This new piece of application software is a step toward letting its customers bring its data together in order to enable improved decision making.
Here is the short take:
• Updated software delivers actionable information with a cross-business digital operations hub
• New portfolio-wide data flow editor saves time and increases visibility by automatically integrating and transforming data for IoT-fueled analysis and optimization
• Code-free development environment accelerates configuration of rich web-enabled dashboards and applications through a library of widgets
GE Digital announced updates to Proficy Operations Hub, the company’s centralized environment for building industrial applications for web-based intelligence.
Proficy Operations Hub allows both developers and non-developers to quickly assemble displays through a comprehensive library of widgets and arrange them to provide responsive operator and supervisor visualization. Companies can define data sources for connected devices and create queries to access and transform data into actionable information for operations. Drag-and-drop design allows for simple placement and configuration of visualization components on the display, then dragging the query or data source onto the component quickly enables the data connections.
Designed as an OT business intelligence (BI) tool for any industrial environment, Proficy Operations Hub is used in diverse industries including water/wastewater treatment, automotive manufacturing, food and beverage processing, power and energy, and consumer goods.
As an example, ENGIE, a global company in low-carbon energy and services, worked with Control & Protection Automation NV (CPA) to leverage Proficy Operations Hub to accelerate time to value. The team developed and delivered an expanded remote and local monitoring and control solution. CPA maximized Proficy Operation Hub’s Rapid Application Development (RAD) capabilities in conjunction with the Proficy software portfolio to create reusable objects and High Performance HMI operator screens, GIS functionality, dashboards, and more.
New features in this latest update include increasing Rapid Application Development using cloud infrastructure with Microsoft Azure and Amazon Web Services (AWS), an expanded widget library for data display and analysis, and third-party systems integration enabled by OPC UA. The software also enhances OT BI with new visualization widgets for supervisory dashboards and a pivot grid for ad hoc multi-dimensional data analysis.
“Access to data across the organization provides faster response and better decision-making with centralized visualization, digitized processes, and data analysis in context. Ultimately that leads to decreased costs and time to market as well as lower maintenance costs,” said Richard Kenedi, General Manager for GE Digital’s Manufacturing and Digital Plant business. “These outcomes are the result of improved collaboration and continuous improvement programs that are key performance indicators in the industrial environment today.”
