DataOps began popping onto my radar last fall. First there was a startup of former Kepware people developing DataOps for manufacturing enterprises, and then it had a featured role at an IT conference.
I have mentioned the two previously, which attracted the attention of Kevin E. Kline, who is working with Sentry One. He has a heck of a bio—Principal Program Manager, Bestselling author of SQL in a Nutshell, Founder & President Emeritus, PASS.Org, and a Microsoft MVP since 2003. He pointed me to a blog he had written that explains much about the topic.
These passages are lifted from that blog to give you a taste. Check out the entire post for more details. Here is a description.
DataOps is a collaborative practice that improves integration, reliability, and delivery of data across the enterprise. It builds on the foundation of strong DevOps processes. Like DevOps, DataOps fosters communication between business functions like data platform, IT operations, business analytics, engineering, and data science. It focuses on streamlining and automating the data pipeline throughout the data lifecycle:
- Data integration—simplifying the process of connecting to disparate data sources
- Data validation—testing data to ensure that business decisions are supported by accurate information
- Metadata management—maintaining a clear understanding of the topography of the data estate, origin, dependencies, and how the data has changes over time
- Observability—capturing granular insights about data systems along with rich context to help DataOps teams better understand system behavior and performance
DataOps paves the way for effective data operations and a reliable data pipeline, delivering information that people trust with shorter development and delivery cycles.
This part discusses benefits. Later he discusses obstacles.
4 Benefits of DataOps Maturity
Terms that refer to effective collaboration are alignment, tearing down silos, “synergy,” and a newer term—interlock. These terms are prevalent in business because getting them right creates a force multiplier across departments. Imagine being in a rowboat with 10 other people, and none of them are rowing in the same direction. You might never get to where you’re trying to go.
A mature DataOps practice promotes up-front planning and construction, then automated ongoing execution. In other words, teams work together to define what will happen, and various software tools ensure that it happens the same way every time.
Similar to the benefit of collaboration, the automation of data and analytics operations removes a potential element of human unpredictability. We, as human beings, are capable of great things like free thought and reason. These abilities serve us well in many situations. However, they can introduce problems when dealing with repetitive processes that must always follow the same steps.
With a mature, documented, and automated DataOps process, plans to introduce change require fewer hands, less time, and a lower probability of introducing errors. Using this approach also makes it easier to adapt testing procedures. This effectively reduces the time it takes to move from development to production for changes.
DevOps and DataOps have emerged from Agile project management practices. Because of those roots, agility becomes table stakes in DataOps processes. Data teams that already practice Agile methodologies will find it easier to define, implement, and mature their DataOps practice.
20 METATRENDS FOR THE ROARING 20S
Everybody it seems likes metatrends, megatrends, any-kind-of-trends, especially at the beginning of a calendar year. I think that many of these are good idea stretchers. Whether or not they serve as accurate predictors does not matter. People are working on many projects and ideas that will yield something in the future.
Peter Diamandis publishes an Abundance newsletter, preaches Abundance thinking, did the X-prize, and many more futuristic stretch-the-mind ideas.
I lifted the following introduction to his latest newsletter “20 Metatrends For the Roaring 20s.” I recommend visiting the website and thinking through these ideas. He is an abundant optimist about technology. I’m afraid that I’ve been around too many MBAs and marketers. So his idea that someday the Alexa’s and Siri’s of the world will be our loyal servants freeing us from advertising influence pushes aside the factor that these technologies are being developed by companies who survive on advertising. It will be interesting to see how this one plays out.
In the decade ahead, waves of exponential technological advancements are stacking atop one another, eclipsing decades of breakthroughs in scale and impact.
Emerging from these waves are 20 “Metatrends,” likely to revolutionize entire industries (old and new), redefine tomorrow’s generation of businesses and contemporary challenges, and transform our livelihoods from the bottom-up.
Among these metatrends are augmented human longevity, the surging smart economy, AI-human collaboration, urbanized cellular agriculture, and high-bandwidth brain-computer interfaces, just to name a few.
It is here that master entrepreneurs and their teams must see beyond the immediate implications of a given technology, capturing second-order, Google-sized business opportunities on the horizon.
Welcome to a new decade of runaway technological booms, historic watershed moments, and extraordinary abundance.
Top Tens and Top Twenties of the past or future year have never been my favorites. However, one can perceive trends and strain out little nuggets of gold by scanning several. Especially industrial taken broadly along with Internet of Things (IoT) and other current digital trends. I just had an interesting chat with Sean Riley, Global Director of Manufacturing and Transportation for Software AG, who released his Top Ten for 2020.
Following are his ideas interspersed with a few of my comments.
Cost Management Becomes Exceptional
As uncertainty enters the global manufacturing outlook, enterprises will become myopically focused on cost reductions. This will drive organizations to find more efficient methods of providing IT support, leveraging supplier ecosystems and simplifying value chains. [GM-much of my early work was in cost management/reduction; this is a never-ending challenge in manufacturing; however, tools continue to evolve giving us more and better solutions.]
A Blurred Line Between Products & Services
Manufacturers continue their product innovation quest and more manufacturers will begin focusing on how to deliver products as a service. The Manufacturers that have already created smart products and have elevated service levels will now begin to work out the financing considerations needed to shift from a sales based to a usage based revenue model. [GM-This is a trend most likely still in its infancy, or maybe toddler-hood; we see new examples sprouting monthly.]
Moving To Redefine Cost Models To Match Future Revenue Streams
Anticipating the shift to continual revenue streams, manufacturers will seek to shift costs to be incurred in a similar manner. This will be initially seen as a continued push to subscription based IT applications. While much progress has already been made, a larger focus will occur. [GM-I like his idea here of balancing capital versus expense budgets, continually finding the best fund source for shifting costs.]
IT Focuses on Rapid Support for Growth
The lines between business and IT users become blurred as no-code applications allow for business users to create integration services. IT professionals will leverage DevOps & Agile methodologies alongside of microservices and containers to rapidly develop applications that are able to generate incremental growth as requested by business users. This will be critical to the near term success for manufacturers, especially with economic headwinds that seem to be growing stronger. [GM-I didn’t ask about DevOps, but this idea is springing into the industrial space; cloud and software-as-a-service provide scalability both up and down for IT to balance costs and services.]
Industrial Self-Service Analytics Become Mission Critical
Industrie 4.0 / Smart Manufacturing initiatives continue to receive greater amounts of investment but in the near term, manufacturers will focus on unleashing the power of the data they already have. Historians, LIMS, CMMS’ have valuable data going to and in them and enabling production engineers to leverage that data rapidly is critical. Industrial Self-Service Analytics that allow production and maintenance professionals to leverage predictive analytics without IT assistance will sought as a powerful differentiating factor. [GM-we are beginning to see some cool no-programming tools to help managers get data access more quickly.]
Industrie 4.0 / Smart Manufacturing Initiatives Continue to Draw Investment
It’s no surprise that Manufacturers will continue to invest in Industrie 4.0 as the promises are great however, the scaled returns have not been realized and won’t be realized in the near term. The difficult of implementing these initiatives has surpassed manufacturers expectations for several reasons. First, traditional OT companies were trusted to deliver exceptional, open platforms and that wasn’t delivered. Secondly, collaboration efforts between IT & OT professionals proved to be more convoluted and difficult than expected. [GM-I’m thinking these ideas became overblown and complex, and that is not a good thing; to swallow the whole enchilada causes stomach pain.]
Artificial Intelligence Enters the Mix
AI won’t allow for users to sit back and relax while AI handles all of their tasks for them but it will make an appearance in back office tasks. Freight payment auditing, invoice payment and, in some select areas, chatbots will be the initial main stream uses of AI and will be seen as not becoming an anomaly but be understood to be more mainstream this year. [GM-I think still an idea looking for a problem; however some AI ideas are finding homes a little at a time.]
3D Printing Find New Uses
While this technology has steadily crept into production lines, the push towards usage based product pricing will have the technology move into after market services. Slow moving parts will be the first target for this technology which will help to free up much needed working capital to support financial transformation. [GM-watch for better machines holding tighter tolerances making the technology more useful.]
5G & Edge Analytics Enable New Possibilities
As Industrie 4.0 is continued to be pursued, Manufacturers will implement new initiatives that could not previously be realized without the high speed data transmission promises of 5G or the ability to conduct advanced analytics at the edge where production occurs. This will also provide manufacturers with new methods to securely implement Smart Manufacturing initiatives and in new locations that were not previously feasible due to connectivity issues. [GM-5G is still pretty much a dream, but there is great potential for some day.]
Security Still Remains a Critical Focus
With the increasing rate of IoT sensors, IT-OT convergence, the usage of API’s and the interconnectivity of ecosystems ensuring data security remains a top priority for manufacturers. As more data becomes more available, the need to increase levels of security becomes ever greater. [GM-ah, yes, security–a never-ending problem.]
This announcement hits many trends and things you will eventually grow tired of hearing—partnerships, collaboration among companies, ecosystems, Kubernetes, containers, and, yes, 5G. The latter is coming. We just don’t know when and how, yet.
Wind River, a leader in delivering software for the intelligent edge, announced that it is collaborating with Dell EMC as a key hardware partner for distributed edge solutions. A combined software and hardware platform would integrate Wind River Cloud Platform, a Kubernetes-based software offering for managing edge cloud infrastructure, with Dell EMC PowerEdge server hardware. The initial target use case will be virtual RAN (vRAN) infrastructure for 5G networks.
“As telecom infrastructure continues to evolve, service providers are facing daunting challenges around deploying and managing a physically distributed, cloud native vRAN infrastructure,” said Paul Miller, vice president of Telecommunications at Wind River. “By working with Dell EMC to pre-integrate our technologies into a reference distributed cloud solution, we can cost-effectively deliver carrier grade performance, massive scalability, and rapid service instantiation to service providers as their foundation for 5G networks.”
“In a 5G world, new services and applications will not be driven by massively scaled, centralized data centers but by intelligently distributed systems built at the network edge,” said Kevin Shatzkamer, vice president of Enterprise and Service Provider Strategy and Solutions at Dell EMC. “The combination of Dell EMC and Wind River technology creates a foundation for a complete, pre-integrated distributed cloud solution that delivers unrivaled reliability and performance, massive scalability, and significant cost savings compared to conventional RAN architectures. The solution will provide CSPs with what they need to migrate to 5G vRAN and better realize a cloud computing future.”
Wind River Cloud Platform combines a fully cloud-native, Kubernetes and container-based architecture with the ability to manage a truly physically and geographically separated infrastructure for vRAN and core data center sites. Cloud Platform delivers single pane of glass, zero-touch automated management of thousands of nodes.
Dell EMC hardware delivers potent compute power, high performance and high capacity memory is well suited to low-latency applications.
A commercial implementation of the open source project StarlingX, Cloud Platform scales from a single compute node at the network edge, up to thousands of nodes in the core to meet the needs of high value applications. With deterministic low latency required by edge applications and tools that make the distributed edge manageable, Cloud Platform provides a container-based infrastructure for edge implementations in scalable solutions ready for production.
Many engineers and programmers like open source projects combined with open APIs. Some open source catches on and quietly becomes widely used. Others languish. The Linux Foundation’s Edge project, especially EdgeX Foundry, keeps quietly growing. What are the odds that this becomes a widely used Internet of Things tool?
Today’s news in brief:
- EdgeX’s fifth release offers more scalable solutions to move data from devices to cloud, enterprise and on-premises applications
- The first LF Edge project to achieve Stage 3 ratification, EdgeX hits widespread adoption and production-level maturity
- EdgeX and LF Edge onsite at IoT Solutions World Congress with demos from Dell Technologies, Home Edge, IOTech and Project EVE
EdgeX Foundry, a project under the LF Edge umbrella organization within the Linux Foundation that aims to establish an open, interoperable framework for IoT edge computing independent of connectivity protocol, hardware, operating system, applications or cloud, announced the availability of its “Fuji” release. This release offers additional security and testing features on top of the production-ready “Edinburgh” release launched this spring.
“EdgeX Foundry has experienced significant momentum in developing an open IoT platform for edge-related applications and shows no signs of slowing down,” said Arpit Joshipura, general manager, Networking, Edge and IoT, the Linux Foundation. “As the only Stage 3 project under LF Edge, EdgeX Foundry is a clear example of how open collaboration is the key to an active community dedicated to creating an interoperable open source framework across IoT, Enterprise, Cloud and Telco Edge.”
Launched in April 2017, and now part of the LF Edge umbrella, EdgeX Foundry is an open source, loosely-coupled microservices framework that provides the choice to plug and play from a growing ecosystem of available third-party offerings or to augment proprietary innovations. With a focus on the IoT Edge, EdgeX simplifies the process to design, develop and deploy solutions across industrial, enterprise, and consumer applications. As a Stage 3 project under LF Edge, EdgeX is a self-sustaining cycle of development, maintenance, and long-term support. As an example of the rapidly accelerating use of the code, EdgeX hit a milestone of 1 million platform container downloads, which almost half of these took place in the last few months.
“The 1M container download isn’t our only milestone,” said Keith Steele, EdgeX Foundry chair of the Technical Steering Committee and LF Edge Governing Board member. “The development team has expanded with more than 150 active contributors globally and the partner ecosystem of complementary products and services continues to increase. As a result, we’re seeing more end-user case studies that range from energy and utilities, building automation, industrial process control and factory automation, smart cities, retail stores and distribution and health monitoring.”
The Fuji Release
As the fifth release in the EdgeX Foundry roadmap, Fuji offers significant enhancements to the Edinburgh 1.0 release, which launched in July, including:
- New and improved security features to include PKI infrastructure for token/key generation.
- Application services that now offer full replacement capability to the older export services provided with previous EdgeX releases. These application services offer more scalable and easier to use solutions to get data from the EdgeX framework to cloud, enterprise and on-premises applications.
- Example application services are provided with this release to allow users to quickly move data from EdgeX to the Azure and AWS IoT platforms.
- A new applications function Software Development Kit (SDK) also provides the EdgeX user community with the ability to create new and customized solutions on top of EdgeX – for example, allowing EdgeX to move edge data to legacy and non-standard environments.
- Unit test coverage is considerably increased (in some services by more than 200 percent) across EdgeX core and supporting microservices.
- New device service connectors to BLE, BACNet, IP camera, OPC UA, GPS, and REST device services.
- Choices for commercially-supported EdgeX device connectors are also starting to blossom with offerings for CANopen, PROFINET, Zigbee, and EtherCat available through EdgeX community members.
Inaugural EdgeX Open
The EdgeX Foundry community recently kicked off a series of hackathons, titled the EdgeX Open. More than 70 attendees participated in the first event on October 7- 8, 2019, in Chicago. Hosted by LF Edge and the Retail Industry Leader Association (RILA), and sponsored by Canonical, Dell Technologies, Deep Vision, Intel, IOTech, IoTium and Zededa, the event featured five teams that competed in retail use case categories. More details on the event, including the winning use case from Volteo, are available in this blog post.
The next hackathon will coincide with the Geneva release, targeted for Spring 2020. It will be centered on the Manufacturing vertical and held in a location in Europe.