DataOps began popping onto my radar last fall. First there was a startup of former Kepware people developing DataOps for manufacturing enterprises, and then it had a featured role at an IT conference.
I have mentioned the two previously, which attracted the attention of Kevin E. Kline, who is working with Sentry One. He has a heck of a bio—Principal Program Manager, Bestselling author of SQL in a Nutshell, Founder & President Emeritus, PASS.Org, and a Microsoft MVP since 2003. He pointed me to a blog he had written that explains much about the topic.
These passages are lifted from that blog to give you a taste. Check out the entire post for more details. Here is a description.
DataOps is a collaborative practice that improves integration, reliability, and delivery of data across the enterprise. It builds on the foundation of strong DevOps processes. Like DevOps, DataOps fosters communication between business functions like data platform, IT operations, business analytics, engineering, and data science. It focuses on streamlining and automating the data pipeline throughout the data lifecycle:
- Data integration—simplifying the process of connecting to disparate data sources
- Data validation—testing data to ensure that business decisions are supported by accurate information
- Metadata management—maintaining a clear understanding of the topography of the data estate, origin, dependencies, and how the data has changes over time
- Observability—capturing granular insights about data systems along with rich context to help DataOps teams better understand system behavior and performance
DataOps paves the way for effective data operations and a reliable data pipeline, delivering information that people trust with shorter development and delivery cycles.
This part discusses benefits. Later he discusses obstacles.
4 Benefits of DataOps Maturity
1. Collaboration
Terms that refer to effective collaboration are alignment, tearing down silos, “synergy,” and a newer term—interlock. These terms are prevalent in business because getting them right creates a force multiplier across departments. Imagine being in a rowboat with 10 other people, and none of them are rowing in the same direction. You might never get to where you’re trying to go.
A mature DataOps practice promotes up-front planning and construction, then automated ongoing execution. In other words, teams work together to define what will happen, and various software tools ensure that it happens the same way every time.
2. Reliability
Similar to the benefit of collaboration, the automation of data and analytics operations removes a potential element of human unpredictability. We, as human beings, are capable of great things like free thought and reason. These abilities serve us well in many situations. However, they can introduce problems when dealing with repetitive processes that must always follow the same steps.
3. Adaptability
With a mature, documented, and automated DataOps process, plans to introduce change require fewer hands, less time, and a lower probability of introducing errors. Using this approach also makes it easier to adapt testing procedures. This effectively reduces the time it takes to move from development to production for changes.
4. Agility
DevOps and DataOps have emerged from Agile project management practices. Because of those roots, agility becomes table stakes in DataOps processes. Data teams that already practice Agile methodologies will find it easier to define, implement, and mature their DataOps practice.
