Jason Risch | The Next Cloud Data Platform | Summary and Q&A

TL;DR

This essay analyzes the potential for startups to compete with the big three cloud providers by creating a platform combining data warehousing, lake houses, and semantic layers.

Key Insights

• 😃 Startups like Snowflake have shown that innovative approaches to data storage, structure, and usage can compete with the big cloud providers.
• 😶‍🌫️ The cloud data warehouse ecosystem is thriving, with various companies specializing in different aspects of data transformation and analytics.
• 😃 The future of cloud data platforms lies in building data apps on top of data warehouses, challenging the dominance of the big three cloud providers.
• 🤗 The lake house architecture offers cost savings, unified platform capabilities, and an open ecosystem, enabling multiple query engines and preventing vendor lock-in.

Transcript

Read and summarize the transcript of this video on Glasp Reader (beta).

Questions & Answers

Q: How did Snowflake manage to succeed against AWS Redshift and Google's BigQuery?

Snowflake differentiated itself by rethinking the data warehousing process, separating storage from compute and leveraging the elastic nature of cloud computing to build a better database. This approach allowed them to excel in operational excellence and IP development.

Q: What are the potential competitors and challenges for Snowflake in the future?

Companies like ClickHouse, Rockset, and Materialize are vying to become the next Snowflake in the cloud data warehouse space. Snowflake's expansion into data applications and the possibility of a Snowflake app store represent opportunities for growth. However, they face competition from the big three cloud providers in terms of core capacities.

Q: What is a lake house and how does it improve upon data lakes and data warehouses?

A lake house combines the best elements of data lakes and data warehouses by providing the structure and guarantees of the latter while leveraging cost-effective storage like S3 or blob storage. It supports acid transactions, schema enforcement, and governance mechanisms, making it more suitable for business analytics use cases.

Q: What is the semantic layer and how does it enhance the cloud data platform?

The semantic layer standardizes metric definitions and business logic independently from downstream systems, creating consistency across a company. It enables direct engagement with business objects and the development of applications that can write back to the database. DBT seems well-positioned to create this layer.

Summary

In this episode of the Gray Matter podcast, Greylock investor Jason Risch discusses the opportunities for startups in cloud data platforms. He analyzes the success of companies like Snowflake and explores the potential of data warehouses, lake houses, and the semantic layer to challenge the dominance of the big three cloud providers.

Questions & Answers

Q: How did Snowflake succeed against AWS and Google Cloud?

Snowflake was able to succeed by rethinking the data warehousing process and separating storage from compute. This allowed them to leverage the elastic resources of cloud computing and build a better database with deep IP and operational excellence.

Q: Are there other startups looking to replicate Snowflake's success?

Yes, venture capitalists are actively looking for the next Snowflake. Companies like ClickHouse, Rockset, and Materialize are competing to be the next streaming snowflake and solidify their staying power in the cloud data warehouse ecosystem.

Q: How can data warehouses be used as dynamic brains for applications?

Data warehouses can function as more than just static data stores. They can serve as the central nervous system for applications, providing real-time data and insights. Snowflake, for example, has a developer page showcasing examples of internal and customer-facing apps built on top of their platform.

Q: What are the key players in the cloud data warehouse ecosystem?

The cloud data warehouse ecosystem has already exploded with companies like FiveTran and Airbyte for moving data, dbt for transforming data, and tools like Looker/Mode, Census, and Monte Carlo for business intelligence and data observability.

Q: How do lake houses improve upon data lakes and warehouses?

Lake houses provide the structure and guarantees of data warehouses while upgrading data lakes. They support ACID transactions, enforce data quality, and provide consistency and isolation. Open table formats like Delta (from Apache Hoodie) and Tabular (from Apache Iceberg) form the core of the lake house architecture.

Q: What are the benefits of using a lake house over a data warehouse?

Lake houses offer significant cost savings as they reside on cheap storage. They also enable a combined platform across various data workloads and use cases, simplifying data management. Additionally, the lake house architecture allows users to leverage multiple query engines, preventing vendor lock-in.

Q: What is the role of the semantic layer in the cloud data platform vision?

The semantic layer is an evolution of the metric store and headless BI concepts. It standardizes metric definitions and business logic in a layer independent from downstream systems, ensuring consistency and stability. It creates an API for people and applications to access business objects directly, enabling the development of data applications.

Q: Who is best positioned to create the semantic layer?

DBT (Data Build Tool) seems to be best positioned to create the semantic layer, given their developer community and transformation capabilities. They have already discussed the concept as the next layer in the modern data stack and have called for third-party development within the ecosystem.

Q: What challenges do startups face in building within ecosystems?

Startups building within ecosystems need to be mindful of shifting product boundaries. Companies like Transform Data, which have been working on independent metric stores, now face competition from well-funded players. However, independence can still win out over platforms with diluted focus.

Q: What can we expect in the future of the cloud data platform?

The future of the cloud data platform lies in the convergence of data warehouses, lake houses, and the semantic layer. Startups across the Snowflake, lake house, and DBT ecosystems will play a crucial role in realizing the vision of the cloud data platform and challenging the dominance of the big three cloud providers.

Takeaways

In this episode, Jason Risch explores the opportunities for startups in cloud data platforms. He highlights Snowflake's success and the potential for other companies to replicate it. The cloud data warehouse ecosystem has exploded, with numerous players addressing various data needs. Lake houses improve upon data lakes and warehouses, offering significant cost savings and a unified platform. The semantic layer complements this by standardizing metrics and providing an API for direct access to business objects. Overall, startups that can navigate these ecosystems and contribute to the cloud data platform vision will have a significant impact in challenging the dominance of AWS, Google Cloud, and Microsoft Azure.

Summary & Key Takeaways

• Bold startups, like Snowflake, have succeeded by reimagining data storage, structure, and usage in the cloud.

• The cloud data warehouse ecosystem has exploded, with companies like Five Tran and Airbyte fueling its growth.

• The next chapter for cloud data warehouses involves expanding as a platform for data apps, challenging the dominance of the big three cloud providers.