I wrote yesterday about some new developments in DataOps. Then I found this news about a survey into users’ needs and lacks in that technology and use case area. This comes from a company called Unravel Data. It bills itself as “the only data operations platform providing full-stack visibility and AI-powered recommendations”. If you’re not sure about DataOps, perhaps the attached infographic from the company will help. Briefly, the survey reveals poor visibility and lack of perspective as the top challenges for DataOps this year.
Unravel Data released key findings from a survey administered to several hundred attendees at the most recent DataOps Unleashed event in order to gauge the priorities, challenges, and progress of leading data teams as they seek to modernize their big data management and analytics capabilities.
“Data is the lifeblood of the modern enterprise and those organizations who have dedicated the resources and budget to modernizing their data stacks are the ones who will be best positioned to drive innovations in the coming decade,” said Kunal Agarwal, co-founder and CEO of Unravel Data. “The results of this latest survey show just how complex the modern data stack has become and illustrates the many unanticipated challenges that come with efficiently managing and optimizing data pipelines across multiple public cloud providers and platforms. It also serves to validate that there is an obvious demand for a purpose-built solution that can help these teams gain the critical visibility they need to drive the most value from their data operations.”
Some of the key findings collected from the most recent survey, which benchmarks responses from the year prior, include:
DataOps as a practice is hitting an inflection point: There was an almost 80% increase from the year prior of respondents who said they are in the active stage of adopting a formal DataOps approach to manage and optimize their data pipelines. This year, more than 41% of attendees reported they are actively employing DataOps methodologies, compared to just less than a quarter (24%) in 2021.
- Visibility into data pipelines remains the top challenge: For the second year in a row, when participants were asked what they viewed as the top challenge with operating their data stack, respondents cited the lack of visibility across their environment as their most significant obstacle. Whereas in the previous year respondents reported that “controlling runaway costs” was the second biggest, this year the “lack of proactive alerts” was noted as the second most challenging aspect.
- Complexity of cloud migrations is more time consuming than previously thought: Sixty percent of respondents from this year’s event estimated that their cloud migration project would take between 12-24 months, representing a 150% increase over the prior year’s projection. The challenge of forecasting the duration of these cloud migration initiatives reveals the vast amount of complexity and uncertainty that data teams face when attempting to map out these critical projects.
- Automation continues to be a key driver: When asked about the role of automation in managing their DataOps pipelines, three in four DataOps professionals in both years reported that the ability to “automatically test and verify before moving jobs/pipelines to production” was the most important automation priority when compared to other aspects such as automating troubleshooting of platform or pipeline issues.
- Data teams spend more time building than deploying/managing pipelines: For both years, data professionals reported that they spent the majority of their day building their data pipelines (39% in 2021 and 43% in 2022). In 2022, respondents reported spending slightly less time maintaining or troubleshooting their pipelines (30%) than the year prior (34%) while the time spent deploying data pipelines remained the same at 27% for both years.
