Modernizing data operations changes the way organizations use data stacks. Industry experts share definitions for the new form of data stacks, what it includes and its benefits.
Enterprise executives continue to invest in projects to unlock value from the vast trove of data that their organizations accumulate. The modern data stack is one approach they're turning to.
The proof is in the numbers:
Sixty-two percent of CEOs see developing advanced analytics as their No. 1 strategy for achieving digital disruption, according to the "CEO Excellence Survey" from management consulting firm McKinsey & Co.
The global data analytics market is expected to hit $329.8 billion by 2030, growing from $31.8 billion in 2021 at a compound annual growth rate of 29.9%, according to Acumen Research and Consulting.
Much of the enterprise investments in this space are aimed at building a modern data stack, according to data professionals. This article examines what differentiates the modern data stack from traditional data stacks and the benefits possible for data management operations.
Defining the modern data stack
Organizations across industries are transforming their data environments to meet executive expectations. The goal is to shed legacy systems and modernize the technologies, tools and processes they use to ingest, move, store and process their data.
Many people define a modern data stack broadly as the collection of technologies and tools required to gather, store, process and analyze data in a manner that is cost-effective, efficient and scalable.
"A modern data stack is cloud-first, and it allows for scalability, elasticity, ease of integration with existing cloud infrastructure," said Bassel Haidar, AI and machine learning practice lead with consultancy Guidehouse.
Others have a more specific definition of the term, where modern data stack refers to suites of data software tools offered by technology vendors. "It is strictly a marketing term" to identify data products, said Zain Khan, director analyst with research firm Gartner.
Data is not defined by the tools, but the techniques and the approaches; they are what define a modern data architecture. The tools are simply a means to an end.
Zain KhanDirector analyst, Gartner
Khan described the modern data stack as a cloud-native, best-of-breed solution. Vendors offering modern data stack products often deliver both primary and secondary capabilities, he said. For example, a modern data stack vendor might position itself in the data integration space but offer data quality capabilities as well.
With this marketing definition of modern data stack in mind, organizations can create a modern data architecture that uses modern data stack solutions, but they don't necessarily need those solutions to achieve the architecture, Khan said.
"Data is not defined by the tools, but the techniques and the approaches; they are what define a modern data architecture. The tools are simply a means to an end," he said.
Data stack solutions include Amazon Redshift, Google BigQuery and Snowflake, Databricks, Dbt Cloud and Fivetran.
Features of a modern data stack vs. legacy data tooling
A legacy data stack typically features relational databases or data warehouses with extract, transform, load -- called ETL -- processes to move and transform data, said Kuruvilla Mathew, chief innovation architect and general manager for modernization at UST, a digital technology and transformation, IT and services provider.
In contrast, a modern data stack is more likely to feature a data lake or lakehouse, data fabric or data mesh. It uses the more flexible extract, load, transform -- called ELT -- sequence that can process structured and unstructured data.
A modern data stack involves configuring the various cloud-based components that the enterprise selects from the vendors. A legacy stack requires more development and customization by the enterprise IT team and its partners. Cloud hosting enables the modern data stack to easily scale its storage and processing capacity. Legacy technology does not provide that elasticity.
It also can be multi-cloud. "You can take workloads and spread them across multiple cloud providers, so workloads become really easy to manage," Mathew said.
Moreover, a modern data stack can process real-time and streaming data. A legacy one is generally limited to batch processing data.
Cloud-based components enable interoperability with various other applications. With a modern data stack, organizations can more easily move data between systems, and the data can be consumed and understood by all the systems within the stack.
Benefits of modernization
A modern data stack brings business benefits to the enterprise, in addition to being faster, more flexible and more scalable.
Cloud providers and cloud-based technologies have built-in tools for data governance, security and monitoring. This helps enterprise teams incorporate those critical elements into their data program, Haider said.
Another benefit is speed. The modern data stack is based on composable infrastructure that an IT team can configure more quickly than it could stand up or program a legacy data stack. The modern data environment gets initiatives moving more quickly, said Juan Tello, U.S. chief data officer at Deloitte and principal with Deloitte Consulting.
Organizations are also beginning to use more automation and AI within their data environments, both of which can enhance data programs.
The portability of data within a modern, cloud-based stack enables organizations to access and use data in multiple ways to gain insights that they might not have otherwise discovered.
Organizations that want to truly be data-driven can't stop at technological transformation. They need to modernize their culture and processes along with their data stack, Tello said. To do that, enterprise leaders must design and implement data literacy programs along with ways to measure the training's effectiveness.
They also need to have the data ready to use. This means data is high quality, complete and accurate, and it comes together for a single source of truth, Tello said. An organization must create and use data products based on quality data. The chief data officer and the business leaders seeking data-driven insights also should instill accountability for success.
"Everyone understands data is important," Tello said. "But I don't think everyone understands that it takes all of that to fully deliver on the value."
