Microsoft Azure had its day on Dec. 3 just as I was digesting the news from rival Amazon Web Services (AWS). The theme was “all about data and analytics.” The focus was on applications Microsoft has added to its Azure services. Anyone who ever thought that these services stopped at being convenient hosts for your cloud missed the entire business model.
Industrial software developers have been busily aligning with Microsoft Azure. Maybe that is why there was no direct assault on their businesses like there was with the AWS announcements. But… Microsoft’s themes of breaking silos of information and combining advanced analytics have the possibility of rendering moot some of the developers’ own tools—unless they just repackage those from Microsoft.
The heart of the meaning of the virtual event yesterday was summed up by Julia White, Corporate Vice President, Microsoft Azure, on a blog post.
Over the years, we have had a front-row seat to digital transformation occurring across all industries and regions around the world. And in 2020, we’ve seen that digitally transformed organizations have successfully adapted to sudden disruptions. What lies at the heart of digital transformation is also the underpinning of organizations who’ve proven most resilient during turbulent times—and that is data. Data is what enables both analytical power—analyzing the past and gaining new insights, and predictive power—predicting the future and planning ahead.
To harness the power of data, first we need to break down data silos. While not a new concept, achieving this has been a constant challenge in the history of data and analytics as its ecosystem continues to be complex and heterogeneous. We must expand beyond the traditional view that data silos are the core of the problem. The truth is, too many businesses also have silos of skills and silos of technologies, not just silos of data. And, this must be addressed holistically.
For decades, specialized technologies like data warehouses and data lakes have helped us collect and analyze data of all sizes and formats. But in doing so, they often created niches of expertise and specialized technology in the process. This is the paradox of analytics: the more we apply new technology to integrate and analyze data, the more silos we can create.
To break this cycle, a new approach is needed. Organizations must break down all silos to achieve analytical power and predictive power, in a unified, secure, and compliant manner. Your organizational success over the next decade will increasingly depend on your ability to accomplish this goal.
This is why we stepped back and took a new approach to analytics in Azure. We rearchitected our operational and analytics data stores to take full advantage of a new, cloud-native architecture. This fundamental shift, while maintaining consistent tools and languages, is what enables the long-held silos to be eliminated across skills, technology, and data. At the core of this is Azure Synapse Analytics—a limitless analytics service that brings together data integration, enterprise data warehousing, and Big Data analytics into a single service offering unmatched time to insights. With Azure Synapse, organizations can run the full gamut of analytics projects and put data to work much more quickly, productively, and securely, generating insights from all data sources. And, importantly, Azure Synapse combines capabilities spanning the needs of data engineering, machine learning, and BI without creating silos in processes and tools. Customers such as Walgreens, Myntra, and P&G have achieved tremendous success with Azure Synapse, and today we move to the global generally availability, so every customer can now get access.
But, just breaking down silos is not sufficient. A comprehensive data governance solution is needed to know where all data resides across an organization. An organization that does not know where its data is, does not know what its future will be. To empower this solution, we are proud to deliver Azure Purview—a unified data governance service that helps organizations achieve a complete understanding of their data.
Azure Purview helps discover all data across your organization, track lineage of data, and create a business glossary wherever it is stored: on-premises, across clouds, in SaaS applications, and in Microsoft Power BI. It also helps you understand your data exposures by using over 100 AI classifiers that automatically look for personally identifiable information (PII), sensitive data, and pinpoint out-of-compliance data. Azure Purview is integrated with Microsoft Information Protection which means you can apply the same sensitivity labels defined in Microsoft 365 Compliance Center. With Azure Purview, you can view your data estate pivoting on classifications and labeling and drill into assets containing sensitive data across on-premises, multi-cloud, and multi-edge locations.
visit us here
Yesterday, Microsoft announced that the latest version of Azure Synapse is generally available, and the company also unveiled a new data governance solution, Azure Purview.
In the year since Azure Synapse was announced, Microsoft says the number of Azure customers running petabyte-scale workloads – or the equivalent of 500 billion pages of standard printed text – has increased fivefold.
Azure Purview, now available in public preview, will initially enable customers to understand exactly what data they have, manage the data’s compliance with privacy regulations and derive valuable insights more quickly.
Just as Azure Synapse represented the evolution of the traditional data warehouse, Azure Purview is the next generation of the data catalog, Microsoft says. It builds on the existing data search capabilities, adding enhancements to help customers comply with data handling laws and incorporate security controls.
The service includes three main components:
- Data discovery, classification and mapping: Azure Purview will automatically find all of an organization’s data on premises or in the cloud and evaluate the characteristics and sensitivity of the data. Beginning in February, the capability will also be available for data managed by other storage providers.
- Data catalog: Azure Purview enables all users to search for trusted data using a simple web-based experience. Visual graphs let users quickly see if data of interest is from a trusted source.
- Data governance: Azure Purview provides a bird’s-eye view of a company’s data landscape, enabling data officers to efficiently govern data use. This enables key insights such as the distribution of data across environments, how data is moving and where sensitive data is stored.
Microsoft says these improvements will help break down the internal barriers that have traditionally complicated and slowed data governance.
You will find references to data often in this blog. Perhaps I’ve even been guilty of a phrase, “It’s all about the data.” Back in 2016, I wrote a post where the title included both “data” and “engineering.”
Marketing managers have been pinging me this year evidently after doing web searches for key words. They get a match on one of my blog posts and write trying to get a link added or an article published. They usually don’t know my focus or even what type of media this is. Many think I’m traditional traded press.
It must have been in such a manner that the marketing manager for Jelvix, which looks to be a Ukrainian software development and IT services company, wrote to me referencing this post I did in 2016. She referenced an article on the company web site by Python developer Vitaliy Naumenko regarding whether or when do you need a Data Scientist or a Data Engineer.
That is an interesting question–one which I have not run across in either my IT or my OT travels.
According to IBM’s CTO report, 87% of data science projects are never really executed. 80% of all data science projects end up failing. Mainly, this happens due to the market’s inability to distinguish data scientists and engineers.
Even now, it’s surprisingly common to find articles online about data scientists’ responsibilities when some of them belong to the data engineer job description. A lack of understanding of what data scientists can and cannot do leads to a high failure percentage and common burn-out.
The thing is, neither data scientists nor engineers can act on their own. Scientists hugely depend on engineers to provide infrastructure. If it’s not set up correctly, even the most skilled scientists with excellent knowledge of complex computational formulas will not execute the project properly.
The data development and management field include many specialties. Data engineers and scientists are only some of the roles necessary in the field. These positions, however, are intertwined – team members can step in and perform tasks that technically belong to another role.
Check out this image, for example. I like the addition of business as well as technology.
Check out his entire article if you are involved with doing something with all the data you are collecting. He suggests organizations for small, medium, and larger organizations. Unfortunately for me, industrial or manufacturing markets are not listed as specialties of the company. But the company has some good ideas to share.
Rockwell Automation introduced the first capability of its edge strategy with FactoryTalk Edge Gateway, to simplify and accelerate IT/OT convergence.
I have been working in the area of edge gateways for about six years as a marketing consultant, writer, influencer, analyst. I thought I knew what one was when I saw it. After reading this press release several time and visiting the Website (not easy to find on the RA Web, by the way), I see that Rockwell Automation has once again redefined what I thought I knew.
This edge gateway is not a hardware product with compute, storage, networking, and the like. It is a software product designed to connect IIoT data to Rockwell’s analytics engine. Therefore, it “helps converge IT and OT”.
Explaining the reason for the product (and editing out all the superlative adverbs lacing the text), Rockwell Automation writes, “Industrial enterprises struggle to aggregate operational data from heterogeneous sources and add relevant context from the source—such as process conditions, time stamps, machine states and other production states—to the IT layer. This prevents them from uncovering insights at the enterprise level. While traditional gateway solutions fail to address these important challenges, FactoryTalk Edge Gateway goes a step above to not only enrich OT data with critical context where it matters the most—at the edge—but also delivers it in a flexible common information model to IT applications, so that industrial enterprises can derive critical insights for a competitive edge.”
FactoryTalk Edge Gateway is an important foundation of a broader edge platform offering that will include elements of pre-built data analytics models, machine learning, tailored applications, and scalable compute. It is the latest addition to the highly recognized and widely adopted FactoryTalk Analytics suite offering that addresses diverse industries and Industrial IoT (IIoT) use cases. FactoryTalk Edge Gateway is also a key pillar in Rockwell Automation’s strategy to accelerate digital agility across the industrial devices to cloud spectrum with partnerships, including PTC and Microsoft.
A leading Fortune 100 pharmaceuticals manufacturer says, “Legacy systems today are not IIoT enabled and need many different software programs to collect and organize data. With existing methods, the operational attribute values collected have different timestamps, so it is impossible to synchronize the data together. Compared to that, FactoryTalk Edge Gateway automatically stitches the data together and puts it in a payload using pre-configured information model hierarchy. It gives one solution to collect and organize the relevant data.”
FactoryTalk Edge Gateway’s data management capabilities result in up to a 70% reduction in analytics data preparation efforts for data scientists or analysts, while providing quality OT data. The underlying common information data model is orchestrated by the Rockwell Automation FactoryTalk Smart Object capability and can be mapped to on-premises or cloud applications to generate predictive insights across the enterprise. FactoryTalk Edge Gateway is designed to integrate with best-in-class ecosystem solutions like Microsoft Azure and FactoryTalk InnovationSuite, powered by PTC, and a variety of big data, IIoT, and cloud applications.
“Industrial businesses need actionable enterprise-level insights to achieve their goals. As customers continue to drive IT/OT integration and leverage operational data to drive insights, they are realizing that having the right OT data context is critical to scale their digital transformation initiatives.,” said Arvind Rao, Director, Product Management for Information Systems at Rockwell Automation. “With FactoryTalk Edge Gateway, we are dramatically reducing the time and effort required to build, maintain and enrich this critical OT context. This provides our customers with the opportunity to realize double-digit operational improvements through analytics.”
I had been talking Open Source with Bill Lydon, who, like me, has been around many places in manufacturing and media in his career most recently as editor of InTech. He referred me to an article he wrote on automation.com.
This release actually points to two important technologies that will combine to improve management’s ability to operate a more profitable and efficient plant. One is the open source. The other is DataOps. I had begun hearing about that from IT companies and was just beginning to wonder about applications specifically to industry when I was approached by John Harrington, one of the founders, with the initial story of HighByte.
I have written several things about DataOps. Here is one from my visit last year to the Hitachi Vantara conference, and the other reports on lack of leveraging data.
On to the actual news:
HighByte announced the release of HighByte Intelligence Hub version 1.2. The release expands the software’s support for open standards and open platforms, including MQTT, Sparkplug, and OpenJDK. Open standards and platforms simplify industrial data integrations and accelerate the time-to-value for Industry 4.0 solutions.
“The MQTT Sparkplug specification is critical to ensuring information interoperability and governance in the plant and throughout the enterprise,” said HighByte CEO Tony Paine. “By deploying HighByte Intelligence Hub with Sparkplug support, our customers are able to standardize and streamline their data infrastructures from factory to cloud.”
HighByte Intelligence Hub is the first DataOps solution purpose-built for industrial environments. DataOps is a new approach to data integration and security that aims to improve data quality, reduce time spent preparing data for analysis, and encourage cross-functional collaboration within data-driven organizations.
In addition to MQTT Sparkplug and Open JDK support, HighByte Intelligence Hub version 1.2 includes model and instance categorization, enhanced security, and more flexible presentation of information models and output formats.
The software is available as an annual subscription and is priced per instance or per site. highbyte.com
From The HighByte Blog
The future of Industry 4.0 is open: open standards, open platforms, and open thinking. In today’s ecosystem, realizing the full potential of Industry 4.0 requires a mesh of products working together to fulfill each layer of the technology stack. Open standards and platforms simplify these integrations and speed up the time-to-value for Industry 4.0 solutions.
Open Standards. This release adds Sparkplug support over MQTT. Sparkplug is an open standard built on top of MQTT that defines the message format for Sparkplug-enabled applications. Many industrial sensors and systems have adopted the Sparkplug specification with MQTT as a means of integrating systems due to Sparkplug’s prescriptive topic namespace, payload definition, and state management. Using Sparkplug, customers can instantly consume and publish data models to and from other Sparkplug-enabled systems.
Open Platform. This release also supports OpenJDK v14, a free and open-source implementation of Java, extending the reach of HighByte Intelligence Hub to any OpenJDK-enabled platform. OpenJDK support for the underlying JAVA virtual machine ensures the longevity of the solution and reduces the cost of ownership of the solution for our customers.
HighByte Intelligence Hub version 1.2 offers the following new features:
- Enhanced security. Securely connect and configure HighByte Intelligence Hub using HTTPS.
- More flexible output formats. JSON output can now be further customized, allowing users to flatten or expand the hierarchy. Flexible presentation of information models is essential when supporting multiple use cases and target applications. While MQTT is becoming the de facto protocol for IoT data and many applications support it, each application has nuances in how they expect JSON to be structured. Applications and data storage engines also have unique needs regarding update frequency and how much information is included in the update. Flexible presentation of information models addresses these interoperability challenges.
- Publish only data that changes. Enable a flow to only publish model data that has changed, reducing the amount of data sent to applications.
- Easily organize models and instances. Models and model instances can now be organized into categories, making it easier to manage hundreds or thousands of models across your enterprise. The organization of models and instances is critical as companies scale the size of their deployments.
Tony Paine Blog Post
Communication within a start-up is pretty straightforward. If you have a question about a new product launch, you go directly to the owner or CEO. Problems with a design flaw? Talk to your lead engineer. As that business scales, your lines of communication become more complex. You may need to send information through multiple channels to get an answer. Without an easy way to send or retrieve information, it might get lost or misinterpreted or you may wait days for an answer. Anyone who has worked in that environment knows the inherent challenges.
Similarly, when organizations implement new industrial IoT solutions, they may work fine at first but become less effective as the project or company scales. The more capabilities you add, the more connections you create throughout your data systems.
For example, today you might need one or two pieces of production data, such as downtime or line speed, from a machine that feeds information into a business intelligence system and an analytics software package from different vendors. As your organization grows, accessing this information becomes more complex because you now have thousands of connection points. Each time you add an application to the system, you need to build connections between the new software and the other systems with which it must communicate. This dramatically increases integration costs and slows deployments.
To scale your industrial IoT implementation, you need a unified namespace. A unified namespace is a software solution that acts as a centralized repository of data, information, and context where any application or device can consume or publish data needed for a specific action. Without a centralized data repository, it could take months to deploy a new analytics application across the entire enterprise versus hours with a unified namespace.
For nearly two decades, MQTT has served as an effective messaging protocol that allows any program or device to publish data but doesn’t offer interoperability between third-party devices and applications. Technology companies have brought data interoperability to MQTT devices and applications through the development of the Sparkplug specification.
At HighByte, we view our unified namespace as a middleware solution that allows users to collect data from various sources, add context so there’s meaning to it, and transform it to a format that other systems can understand. That is why we are adding support for Sparkplug in the upcoming October release of HighByte Intelligence Hub.
This is where you begin to unlock the real value of machine learning because you now have the connectivity you need to optimize your systems, devices, and processes in real time and scale your IoT capabilities without costly, time-consuming implementations.
Just to keep my perspective in balance, I attended a national soccer referee instructor virtual training. It was terrible. The presenters were not familiar with the technology and the presentation was incoherent.
Therefore, what a joy to attend another well done industrial technology conference. The Ignition Community Conference sponsored by Inductive Automation was Tuesday Sept. 15, but you can click the link and see the presentations for a while. I attended two other conferences (unfortunately not the Apple one) and therefore ran out of time to watch more today. But, I’ll be back to catch other presentations.
Chief Marketing Officer Don Pearson always has a relevant quote to serve as a theme for the conference. “Let’s stop trying to predict the future; let’s build it!” Stop and ponder.
The Ignition platform has greatly expanded over the years without losing the core ideas founder Steve Hechtman first explained to me in 2003. Built upon IT-friendly technology with a core strategy of unlimited licensing, the result is a robust HMI/SCADA offering that is affordable. [Note: Inductive Automation is a sponsor, but I’ve been a fan since long before that happened.]
They call it a “community conference” because they consider themselves, their customers, and their partners as a community. And when they gather physically (and even virtually), the people all act like a community. I always enjoy the conference feeling.
Hechtman discussed how the executive team met at the very beginning of hearing about the pandemic to begin preparing for remote work. Among other things they bought a lot of laptop computers. As things went along, they discovered that overall everyone was more productive working remotely. This will become a new way of life for most at the company.
Preparing for business in the Covid environment, they improved the support function and added more people. They improved remote training–and a significant number of customers would prefer to continue remote training although many still wish to return to in-person classes when it’s possible.
Speakers extolled the stories returning about how people are using the Maker edition unveiled in June. Ignition Perspective is another key new product that serves as a base for the main product Ignition 8.1.
Key themes included filling out the promise of IoT broached back in 2016, mobility, remote control, large enterprise solutions, and working with AWS and Azure. Inductive has come a long way.
Ignition 8.1 is a release that will be supported five years. Its development them was refining version 8.0 released last year. It’s vision is based on Perspective. They worked to make it the best visualization platform. Perspective makes it easier to create in a variety of platforms–including the new Workstation edition. It’s now not only Web-based, but also has this Workstation edition to enhance speed. They have developed an easier-to-use and improved symbol library. They’ve added first-party Docker support. Plus a Quick Start application to help people new to the platform to get configured and ready to use.
Be sure to visit the ICC website and examine all the use cases and partners.
Helps enable seamless test operations and data management across an entire organization
It begins with data and it’s all about data. “It” being improving production and profitability and safety. It doesn’t end with data, though. A system is required. Part of the system is software that gathers, analyzes, visualizes data.
NI began with data—test and measurement. It just kept growing. This week it announced the enterprise version of SystemLink software. By standardizing the way data is shared and analyzed, the new enterprise version enables increased visibility and control of test systems across an entire organization. In this way, SystemLink software serves as an important bridge between engineering and manufacturing departments in their efforts to improve overall operational efficiencies and drive digital transformation initiatives.
NI, in its release information, acknowledges data can be an organization’s greatest asset when used to make more informed decisions. It can also be a drain on time and resources when it creates incompatible silos. SystemLink software connects test workflows to business performance, linking people, processes and technology across the enterprise, from engineering to production to the field.
Engineers save time through focus on quickly spotting patterns and proactively addressing issues before they become a problem. “Freeing up engineers to focus on the work that has the largest impact for their organization is smart business,” said Josh Mueller, VP of Experience at NI. “But it is also one of the core components of our company mission — elevating and empowering the engineer.”
Cree Lighting, a leading manufacturer of indoor, outdoor and consumer lighting, has implemented real-time data monitoring and display, post-test analysis and factory management tools using SystemLink software. “SystemLink enables our production floor to step into the future. It equips us with the visibility to respond to market conditions more quickly while optimizing our team’s production efforts all around the world,” said Ian Yeager, test engineering manager at Cree Lighting. “Now, I spend less time managing deployments and post-processing data and more time using the built-in tools to take care of low-hanging opportunities and improve efficiency for my team.”
The new version of SystemLink software is NI’s first hardware-agnostic systems and data management tool. The announcement of enterprise support underscores NI’s enterprise software strategy to help customers accelerate digital transformation efforts by coupling test operations with advanced product analytics enabled through its recent acquisition of OptimalPlus. By unlocking the value of test data and allowing more groups across an enterprise to work together, NI is helping connect the bold people, ideas and technologies required to push our world forward.