Jul 9, 2025

The Snowflake vs. Databricks War and The Shift Towards Federated IT July 9, 2025 Snowflake Databricks Federated IT strategies prioritize open catalogs, interoperability, and vendor-agnostic architectures for scalable, secure data access. Organizations have been locked in an ongoing debate over whether Snowflake or Databricks provides the superior data platform. Both companies have developed impressive ecosystems, offering analytics, AI capabilities, and data engineering solutions. However, these platforms often require organizations to move or transform large volumes of data into proprietary storage or specific formats, which leads to concerns about vendor lock-in and operational rigidity. Both Snowflake and Databricks have acknowledged these concerns and have taken steps to offer more open architectures. Snowflake introduced support for Apache Iceberg, allowing users to store data outside its proprietary format while maintaining the advantages of Snowflake’s analytical engine. Meanwhile, Databricks launched its Uniform Format (UniForm) feature, enabling Delta Lake tables to be read as Apache Iceberg or Apache Hudi tables. While these efforts represent a move toward more openness, they do not fully resolve the complexities of modern data architectures that frustrate organizations by making them  choose between the two.. The Reality of Enterprise Data: It’s Everywhere No matter how compelling Snowflake and Databricks’ offerings may be, organizations will always have data residing in disparate locations. This is because enterprise data exists across on-premises databases and legacy data warehouses,  cloud object storage in multiple formats (Parquet, Avro, ORC, etc. ), modern data lakes leveraging open standards and third party data sources shared via APIs or external storage. Given this reality, the idea that a single platform can act as the universal hub for all data is unrealistic. Different teams within an organization use different tools that fit their specific needs. Data science teams may leverage Databricks for AI workloads, while business intelligence (BI) teams prefer Snowflake for analytics, and operational teams rely on transactional databases. The challenge, then, is not choosing between Snowflake and Databricks but finding a way to unify and govern all data without unnecessary friction. The Future: Federated IT and Portable Data Catalogs Rather than engaging in the Snowflake vs. Databricks debate, forward-thinking organizations want  a federated IT strategy. This approach embraces the diversity of data storage and processing solutions while ensuring seamless access and governance across the enterprise. At the heart of this transformation is the evolution of open and portable data catalogs. A modern data catalog is not just a metadata repository; it serves as the backbone of federated data access. It provides a unified view of data in the data lakehouse and governance and security policies that apply consistently regardless of where the data resides and is processed. Better yet, it offers interoperability with multiple query engines, enabling teams to use the best tool for their needs without migrating data. Projects like Apache Polaris, which powers both Snowflake’s Open Catalog and Dremio’s own built-in catalog, and Unity Catalog which serves as Databricks’ governance layer, are critical to enabling this shift. These catalogs establish a consistent layer for governance, curation, and query ability, ensuring data remains accessible and secure regardless of which tools different teams prefer. Delivering Data for BI and AI with Flexibility Interoperability across cloud and on-premises environments. A single, portable catalog layer that abstracts the complexity of multiple storage formats and query engines. Self-service capabilities for BI and AI teams, ensuring they can access trusted data without relying on centralized IT bottlenecks. By adopting a federated IT approach, organizations can empower their teams with the tools and platforms they prefer while maintaining a cohesive data strategy. This not only reduces vendor lock-in but also allows businesses to adapt to future data needs with agility and confidence. The Role of Data Products in a Federated IT Platform Beyond simply unifying data, a federated IT platform enables organizations to transform unified data into data products. Data products are governed, curated, and accessible datasets – or groups of datasets – managed with clear accountability, similar to how a product manager oversees traditional projects. These data products ensure that teams across the organization can rely on consistent, high-quality, and well-governed data for their specific use cases, whether for BI, AI, or operational analytics. A federated approach to IT should facilitate the creation and management of data products by enforcing governance policies that ensure data integrity, security, and compliance and by providing self-service data access with clear ownership and documentation. It also supports multiple storage formats and processing engines to cater to diverse needs and delivers an abstraction layer that enables data products to be consumed in a unified manner across different tools and platforms, By moving beyond just unifying data and focusing on creating consumable data products, organizations can extract greater value from their data and ensure that it meets the needs of various stakeholders. The Unanswered Question: How Will Federated Data Product Delivery Work? While the industry is moving toward a federated model for IT and data product management, an open and critical question remains: how can this be implemented at scale? Over the next few years, organizations and vendors will grapple with key challenges to address this by:: Standardizing data product definitions to ensure interoperability across tools. Implementing federated governance frameworks that work across cloud and on-prem environments. Developing automated pipelines for data product creation, maintenance, and distribution. Balancing flexibility with control to meet the needs of different teams while maintaining security and compliance. Companies are actively exploring different solutions that address these challenges, each offering their vision for the future of federated data product delivery. Open Table Formats & Lakehouse Catalogs: The Building Blocks of Federated Data Despite the uncertainties around implementation, the building blocks of federated data product delivery are beginning to take shape. Primitives like open table formats such as Apache Iceberg, Delta Lake, Apache Hudi  and lakehouse catalogs like Apache Polaris, Unity Catalog provide the foundational pieces for a future in which federated IT enables seamless data product creation and management. Organizations that adopt these primitives can start building a more open, flexible, and interoperable data ecosystem, allowing them to ensure longevity and portability of dataacross different tools and platforms, and reduce reliance on proprietary storage formatsthat create vendor lock-in. The approach will also allow them to simplify governance and security policiesthrough a centralized catalog layer while enabling teams to query, analyze, and transform datawithout unnecessary complexity. The companies developing and pitching their version of this future are making their play for dominance in the evolving data architecture landscape. Organizations must carefully evaluate these offerings and prioritize open, flexible solutions that align with their long-term data strategy. Moving on from the Snowflake/Databricks dilemma: Embrace Federated IT & It Won’t Matter Rather than getting caught up in the battle between Snowflake and Databricks, organizations should recognize that the future of data architecture is neither platform-specific nor proprietary. The real opportunity lies in federated governance, interoperability, and a unified catalog layer that enables seamless data access and management across diverse environments. By embracing open standards and leveraging portable catalogs like Apache Polaris and Unity Catalog, businesses can future-proof their data strategies. This shift also allows for greater flexibility, improved collaboration, and an architecture that supports BI, AI, and other emerging workloads without compromising governance or control. In the end, organizations don’t need to choose a winner in the Snowflake vs. Databricks war—they need a data strategy that transcends it. Alex Merced Alex Merced is a Senior Technology Evangelist for Dremio, the unified lakehouse platform for self-service analytics and AI. He has worked as a developer and instructor for companies like GenEd Systems, Crossfield Digital, CampusGuard and General Assembly. Alex is passionate about technology and has put out tech content on outlets such as blogs, videos and his podcasts Datanation and Web Dev 101. Alex Merced has contributed a variety of libraries in the Javascript & Python worlds including SencilloDB, CoquitoJS, Dremio-simple-query and more. previous post