DataOps began popping onto my radar last fall. First there was a startup of former Kepware people developing DataOps for manufacturing enterprises, and then it had a featured role at an IT conference.
I have mentioned the two previously, which attracted the attention of Kevin E. Kline, who is working with Sentry One. He has a heck of a bio—Principal Program Manager, Bestselling author of SQL in a Nutshell, Founder & President Emeritus, PASS.Org, and a Microsoft MVP since 2003. He pointed me to a blog he had written that explains much about the topic.
These passages are lifted from that blog to give you a taste. Check out the entire post for more details. Here is a description.
DataOps is a collaborative practice that improves integration, reliability, and delivery of data across the enterprise. It builds on the foundation of strong DevOps processes. Like DevOps, DataOps fosters communication between business functions like data platform, IT operations, business analytics, engineering, and data science. It focuses on streamlining and automating the data pipeline throughout the data lifecycle:
- Data integration—simplifying the process of connecting to disparate data sources
- Data validation—testing data to ensure that business decisions are supported by accurate information
- Metadata management—maintaining a clear understanding of the topography of the data estate, origin, dependencies, and how the data has changes over time
- Observability—capturing granular insights about data systems along with rich context to help DataOps teams better understand system behavior and performance
DataOps paves the way for effective data operations and a reliable data pipeline, delivering information that people trust with shorter development and delivery cycles.
This part discusses benefits. Later he discusses obstacles.
4 Benefits of DataOps Maturity
Terms that refer to effective collaboration are alignment, tearing down silos, “synergy,” and a newer term—interlock. These terms are prevalent in business because getting them right creates a force multiplier across departments. Imagine being in a rowboat with 10 other people, and none of them are rowing in the same direction. You might never get to where you’re trying to go.
A mature DataOps practice promotes up-front planning and construction, then automated ongoing execution. In other words, teams work together to define what will happen, and various software tools ensure that it happens the same way every time.
Similar to the benefit of collaboration, the automation of data and analytics operations removes a potential element of human unpredictability. We, as human beings, are capable of great things like free thought and reason. These abilities serve us well in many situations. However, they can introduce problems when dealing with repetitive processes that must always follow the same steps.
With a mature, documented, and automated DataOps process, plans to introduce change require fewer hands, less time, and a lower probability of introducing errors. Using this approach also makes it easier to adapt testing procedures. This effectively reduces the time it takes to move from development to production for changes.
DevOps and DataOps have emerged from Agile project management practices. Because of those roots, agility becomes table stakes in DataOps processes. Data teams that already practice Agile methodologies will find it easier to define, implement, and mature their DataOps practice.
Many engineers are looking for better ways to move data with fewer programming hours and headaches. Whereas OPC solved many problems leading to interoperability and data exchange, it also brings with it a higher overhead and programming load. For those searching for a something lighter, and also open source, along comes Sparkplug.
Cirrus Link authored the Sparkplug specification and provided it to the Eclipse Foundation, and several other companies support the group as founding members including Chevron, Canary Labs, HiveMQ, Inductive Automation, and ORing. Now additional companies are developing their products using Sparkplug for interoperability.
I recently received a paper authored by Arlen Nipper, president and CTO of Cirrus Link “Sparkplug: Open Source Technology to Bridge the OT-IT Gap”. He begins:
One of the primary pain points in Industrial IoT (IIoT) is disparate systems with both modern and legacy assets. Companies in any industry ranging from oil and gas to manufacturing can hardly imagine a world where they can choose any vendor’s hardware, plug it into their network, and have the hardware 100 percent self-discovered by their SCADA system and every application in the enterprise. True vendor interoperability for both data producers and data consumers is the vision, and new open-source technology may be the answer.
These days, everything relates back to digital transformation. Nipper write, Digital transformation requires devices in the field to be connected, with data made available that can speak the language of both OT and IT for improved business intelligence. In order for this type of digital transformation to be successful, data must be decoupled from a single application so it can flow to enterprise applications in a one-to-many approach.
From the first time I met Nipper, he has evangelized MQTT—a protocol he helped write—as an IT-friendly messaging protocol. It is lightweight. It is a publish-subscribe network protocol allowing for multiple data consumers.
MQTT is a messaging protocol. It does not describe the data traversing the wire (or air). While it provides an excellent engine for delivering IIoT data, MQTT doesn’t make the data interoperable across the enterprise. Thus, a new open source standard has been created and the IIoT industry should understand its importance for bridging the gap from OT to IT.
Nipper explains the next step:
The Internet expanded rapidly thanks to two open technologies – first HTTP, a data exchange protocol, and then HTML, which was used to define the data sent by HTTP. Both were needed. MQTT has needed its “HTML” for years in order for IIoT to explode in growth and adoption. In order to solve this problem of OT-IT interoperability, the Eclipse Sparkplug working group was launched in February 2020 to bring device communications standardization to IIoT.
The Eclipse Foundation states, “The Sparkplug Working Group was established to ‘improve the interoperability and scalability of IIoT solutions, and provide an overall framework for supporting Industry 4.0 for oil and gas, energy, manufacturing, smart cities, and other related industries.’ ”
Sparkplug is an open source software specification that provides MQTT clients with a framework to integrate data. The specification articulates three goals:
1. Define an MQTT Topic Namespace optimized for IIoT.
2. Define MQTT State Management to take advantage of continuous session awareness.
3. Define the MQTT Payload.
Sparkplug adds features including birth certificate and death certificate (session awareness) to help with contextualization of data.
Sparkplug makes this process fast, secure, and open standard so anyone can make use of the framework for MQTT interoperability. Many device manufacturers are supporting Sparkplug, which means it is built in natively on the device on the OT floor.
With Sparkplug, machine learning and artificial intelligence applications can utilize the same standard interface for data without having to know and understand the entire OT environment. They can subscribe to the OT data, and use it immediately for IT functions.
Antonio Neri, a scant two years into the position of President and CEO of Hewlett Packard Enterprise (HPE), laid out a vision and roadmap last year to make the company “the global edge-to-cloud platform as-a-service company” which develops and delivers “enterprise technology solutions and services that help organizations accelerate outcomes by unlocking value from all of their data, everywhere.”
This year he unveiled new executive leadership–one heading GreenLake, the as-a-Service portion of the vision, and the other heading Ezmeral, the newly defined software portfolio and direction–and showed how far the vision has come in a year.
I am impressed with the speed and consistency of this pivot.
HPE Discover, the annual conference usually held in Las Vegas, went totally online this year. This was my sixth or seventh of these within four weeks, and by far the best. Kudos to the communications team for pulling this all together. I attended as an Influencer, and the global leader assembled an outstanding group (except for me) and a good program for us to meet as much of the leadership as possible.
I will write indepth specifically about GreenLake and Ezmeral and sustainability and helping restart work following COVID-19–and perhaps most important for you, my industrial audience–Aruba Edge Orchestrator.
Why do I take all the time and effort to attend these enterprise and IT events?
Let me share one example. I sat in two sessions specifically talking industrial applications.
In the first, an engineering leader from Stanley Black & Decker spoke of his IoT project using HPE’s Edgeline compute platform at almost every level of the manufacturing stack. Edgeline is an industrialized version of HPE’s high performance compute platform. I have seen examples in previous Discover events from off-shore oil platforms and a refinery in Texas. This is one of the points where the reality of “IT/OT Convergence” hits home.
The other session featured HPE Fellow and VP of HPE Labs Dr. Tom Bradicich detailing the vision of data–from analog data (which he has been discussing with me since his days at NI several years ago) to the edge to the cloud.
You can visit HPE.com and register for the event for a while, yet. Click sessions and search for keywords such as IoT, edge, and so forth to watch the on-demand videos. Yet another benefit of the “virtual experience.”
[You can get these posts delivered to you via email by clicking the appropriate link and leaving your address. I prefer information pushed to me either by email or RSS feed, and I know many who do. I don’t sell or use your address. They remain buried somewhere in my WordPress CMS.]
Top Tens and Top Twenties of the past or future year have never been my favorites. However, one can perceive trends and strain out little nuggets of gold by scanning several. Especially industrial taken broadly along with Internet of Things (IoT) and other current digital trends. I just had an interesting chat with Sean Riley, Global Director of Manufacturing and Transportation for Software AG, who released his Top Ten for 2020.
Following are his ideas interspersed with a few of my comments.
Cost Management Becomes Exceptional
As uncertainty enters the global manufacturing outlook, enterprises will become myopically focused on cost reductions. This will drive organizations to find more efficient methods of providing IT support, leveraging supplier ecosystems and simplifying value chains. [GM-much of my early work was in cost management/reduction; this is a never-ending challenge in manufacturing; however, tools continue to evolve giving us more and better solutions.]
A Blurred Line Between Products & Services
Manufacturers continue their product innovation quest and more manufacturers will begin focusing on how to deliver products as a service. The Manufacturers that have already created smart products and have elevated service levels will now begin to work out the financing considerations needed to shift from a sales based to a usage based revenue model. [GM-This is a trend most likely still in its infancy, or maybe toddler-hood; we see new examples sprouting monthly.]
Moving To Redefine Cost Models To Match Future Revenue Streams
Anticipating the shift to continual revenue streams, manufacturers will seek to shift costs to be incurred in a similar manner. This will be initially seen as a continued push to subscription based IT applications. While much progress has already been made, a larger focus will occur. [GM-I like his idea here of balancing capital versus expense budgets, continually finding the best fund source for shifting costs.]
IT Focuses on Rapid Support for Growth
The lines between business and IT users become blurred as no-code applications allow for business users to create integration services. IT professionals will leverage DevOps & Agile methodologies alongside of microservices and containers to rapidly develop applications that are able to generate incremental growth as requested by business users. This will be critical to the near term success for manufacturers, especially with economic headwinds that seem to be growing stronger. [GM-I didn’t ask about DevOps, but this idea is springing into the industrial space; cloud and software-as-a-service provide scalability both up and down for IT to balance costs and services.]
Industrial Self-Service Analytics Become Mission Critical
Industrie 4.0 / Smart Manufacturing initiatives continue to receive greater amounts of investment but in the near term, manufacturers will focus on unleashing the power of the data they already have. Historians, LIMS, CMMS’ have valuable data going to and in them and enabling production engineers to leverage that data rapidly is critical. Industrial Self-Service Analytics that allow production and maintenance professionals to leverage predictive analytics without IT assistance will sought as a powerful differentiating factor. [GM-we are beginning to see some cool no-programming tools to help managers get data access more quickly.]
Industrie 4.0 / Smart Manufacturing Initiatives Continue to Draw Investment
It’s no surprise that Manufacturers will continue to invest in Industrie 4.0 as the promises are great however, the scaled returns have not been realized and won’t be realized in the near term. The difficult of implementing these initiatives has surpassed manufacturers expectations for several reasons. First, traditional OT companies were trusted to deliver exceptional, open platforms and that wasn’t delivered. Secondly, collaboration efforts between IT & OT professionals proved to be more convoluted and difficult than expected. [GM-I’m thinking these ideas became overblown and complex, and that is not a good thing; to swallow the whole enchilada causes stomach pain.]
Artificial Intelligence Enters the Mix
AI won’t allow for users to sit back and relax while AI handles all of their tasks for them but it will make an appearance in back office tasks. Freight payment auditing, invoice payment and, in some select areas, chatbots will be the initial main stream uses of AI and will be seen as not becoming an anomaly but be understood to be more mainstream this year. [GM-I think still an idea looking for a problem; however some AI ideas are finding homes a little at a time.]
3D Printing Find New Uses
While this technology has steadily crept into production lines, the push towards usage based product pricing will have the technology move into after market services. Slow moving parts will be the first target for this technology which will help to free up much needed working capital to support financial transformation. [GM-watch for better machines holding tighter tolerances making the technology more useful.]
5G & Edge Analytics Enable New Possibilities
As Industrie 4.0 is continued to be pursued, Manufacturers will implement new initiatives that could not previously be realized without the high speed data transmission promises of 5G or the ability to conduct advanced analytics at the edge where production occurs. This will also provide manufacturers with new methods to securely implement Smart Manufacturing initiatives and in new locations that were not previously feasible due to connectivity issues. [GM-5G is still pretty much a dream, but there is great potential for some day.]
Security Still Remains a Critical Focus
With the increasing rate of IoT sensors, IT-OT convergence, the usage of API’s and the interconnectivity of ecosystems ensuring data security remains a top priority for manufacturers. As more data becomes more available, the need to increase levels of security becomes ever greater. [GM-ah, yes, security–a never-ending problem.]
Salesforce recently began reaching out to me. I found a (to me) surprising connection to industrial / manufacturing applications beyond CRM and the like. In general, more and more applications are moving to the cloud. In Brief: New research finds The Salesforce Economy will create more than $1 trillion in new business revenues and 4.2 million jobs between 2019 and 2024. Salesforce ecosystem is on track to become nearly six times larger than Salesforce itself by 2024, earning $5.80 for every dollar Salesforce makes.
Financial services, manufacturing and retail industries will lead the way, creating $224 billion, $212 billion and $134 billion in new business revenue respectively by 2024.
Salesforce announced new research from IDC that finds Salesforce and its ecosystem of partners will create 4.2 million new jobs and $1.2 trillion in new business revenues worldwide between 2019 and 2024. The research also finds Salesforce is driving massive gains for its partner ecosystem, which will see $5.80 in gains for every $1 Salesforce makes by 2024.
Cloud computing is driving this growth and giving rise to a host of new technologies, including mobile, social, IoT and AI, that are creating new revenue streams and jobs that further fuel the growth of the cloud — creating an ongoing virtuous cycle of innovation and growth. According to IDC, by 2024 nearly 50 percent of cloud computing software spend will be tied to digital transformation and will account for nearly half of all software sales. Worldwide spending on cloud computing between now and 2024 will grow 19 percent annually, from $179 billion in 2019 to $418 billion in 2024.
“The Salesforce ecosystem is made possible by the amazing work of our customers and partners around the world, and because of our collaboration we’re able to generate the business and job growth that we see today,” said Tyler Prince, EVP, Industries and Partners at Salesforce. “Whether it’s through industry-specific extensions or business-aligned apps, the Salesforce Customer 360 platform helps accelerate the growth of our partner ecosystem, and most importantly, the growth of our customers.”
Because organizations that spend on cloud computing subscriptions also spend on ancillary products and services, the Salesforce ecosystem in 2019 is more than four times larger than Salesforce itself and will grow to almost six times larger by 2024. IDC estimates that from 2019 through 2024, Salesforce will drive the creation of 6.6 million indirect jobs, which are created from spending in the general economy by those people filling the 4.2 million jobs previously mentioned.
“The tech skills gap will become a major roadblock for economic growth if we don’t empower everyone – regardless of class, race or gender – to skill up for the Fourth Industrial Revolution,” said Sarah Franklin, EVP and GM of Platform, Developers and Trailhead at Salesforce. “With Trailhead, our free online learning platform, people don’t need to carry six figures in debt to land a top job; instead, anyone with an Internet connection can now have an equal pathway to landing a job in the Salesforce Economy.”
Industry Economic Benefits of the Salesforce Economy
Specifically, Manufacturing industry will gain $211.7 billion in new revenues and 765,800 new jobs will be created by 2024.
Salesforce’s multi-faceted ecosystem is the driving force behind the Salesforce Economy’s massive growth:
- The global ecosystem includes multiple stakeholders, all of which play an integral part in the Salesforce Economy. This includes the world’s top five consulting firms, all of whom have prominent Salesforce digital transformation practices; independent software vendors (ISVs) that build their businesses on the Salesforce Customer 360 Platform and bring Salesforce into new industries; more than 1,200 Community Groups, with different areas of focus and expertise; and more than 200 Salesforce MVPs, product experts and brand advocates.
- Launched in 2006, Salesforce AppExchange is the world’s largest enterprise cloud marketplace, and hosts more than 4,000 solutions including apps, templates, bots and components that have been downloaded more than 7 million times. Ninety-five percent of the Fortune 100, 81 percent of the Fortune 500, and 86 percent of Salesforce customers are using AppExchange apps.
- Trailhead is Salesforce’s free online learning platform that empowers anyone to skill up for the future, learn in-demand skills and land a top job in the Salesforce Economy. Since Trailhead launched in 2014, more than 1.7 million Trailblazers have earned over 17.5 million badges; a quarter of all learners on Trailhead have leveraged their newfound skills to jump-start their careers with new jobs. Indeed, the world’s #1 job site, included Salesforce Developer in its list of best jobs in the US for 2019, noting that the number of job postings for that position had increased 129 percent year-over-year.