How Facebook could have built Threads 30X faster & cheaper

TL;DR

Facebook could have built its threads feature 30 times faster and cheaper using Rama, a new data back-end platform developed by Nathan Mars, the creator of Apache Storm.

Transcript

Facebook spent 25 person years to create threads 25 to 50 Engineers spent seven months straight working on it but what if I told you they could have made threads with one engineer with under 10 000 lines of code at least 30 times faster and cheaper if they were using Rama by Red Planet labs they could have this new data back-end platform just dropp... Read More

Key Insights

• 👶 Rama is a new data back-end platform that offers a revolutionary approach to building scalable applications.
• ⏬ By integrating and generalizing different aspects of back-end system development, Rama reduces code complexity and enables faster development.
• 🛺 Rama's unique abstractions allow for auto-scaling and handling complex operations efficiently, leading to significant cost and time savings.
• 🔨 Traditional data systems often require separate tools and code bases, leading to redundancy and maintenance challenges, whereas Rama consolidates functionality into a single tool.
• ⚖️ Rama's ability to handle massive amounts of data and scale easily makes it a game-changer for applications that deal with social media scale.
• 💄 The learning curve associated with Rama is outweighed by its benefits, making it a valuable investment for developers.
• ❓ Red Planet Labs provides comprehensive documentation and tutorials to facilitate the adoption of Rama.

Questions & Answers

Q: How does Rama compare to traditional data systems like MySQL and Redis?

Rama offers a fundamentally different approach to building back-end systems by leveraging a general-purpose system. Unlike traditional data systems that require separate code bases and tools for different purposes, Rama allows developers to use a single tool with significantly less code.

Q: What are the key features of Rama?

Rama integrates distributed durable and replicated logs, extract-transform-load topologies, partition states for indexing data, and flexible query options. It enables auto-scaling and extreme expressiveness, making it easier to scale and handle complex operations efficiently.

Q: How does Rama compare to existing social media platforms like Twitter?

Rama's scalability and efficient codebase allow it to handle massive amounts of data and perform at Twitter scale. By building a Twitter-like social network, Mastodon, using Rama, the developers achieved the same functionality with only 10,000 lines of code and significantly fewer person years compared to the original implementation.

Q: Is there a learning curve associated with using Rama?

Yes, Rama has a learning curve, but the benefits it offers in terms of efficiency and productivity make it worthwhile for developers to invest time in learning it. The documentation provided by Red Planet Labs contains tutorials and in-depth guides to help developers get started.

Key Insights:

• Rama is a new data back-end platform that offers a revolutionary approach to building scalable applications.
• By integrating and generalizing different aspects of back-end system development, Rama reduces code complexity and enables faster development.
• Rama's unique abstractions allow for auto-scaling and handling complex operations efficiently, leading to significant cost and time savings.
• Traditional data systems often require separate tools and code bases, leading to redundancy and maintenance challenges, whereas Rama consolidates functionality into a single tool.
• Rama's ability to handle massive amounts of data and scale easily makes it a game-changer for applications that deal with social media scale.
• The learning curve associated with Rama is outweighed by its benefits, making it a valuable investment for developers.
• Red Planet Labs provides comprehensive documentation and tutorials to facilitate the adoption of Rama.
• The use of Rama could have potentially saved startups, like Posterous, from scaling challenges and subsequent failure.

Summary

In this video, the speaker discusses Rama by Red Planet Labs, a new data backend platform that has the potential to revolutionize software engineering. Rama is designed to be a general-purpose system that integrates and generalizes all aspects of building a backend system at scale. It aims to eliminate the need for developers to use multiple data stacks and repeat themselves in the codebase. The speaker explains how Rama can greatly reduce the amount of code required to build applications and enable auto-scaling and extreme expressiveness. The video also highlights the scalability and power of Rama compared to other systems like Mastodon and traditional databases. The speaker emphasizes that Rama provides a new abstraction for data structures, allowing developers to create data structures that mirror the underlying information they are trying to represent. Rama's architecture, including Depots, ETLs, P States, and queries, is explained, showcasing its flexibility and scalability. The video concludes by encouraging viewers to try Rama, acknowledging the learning curve but highlighting the significant benefits it offers to software development.

Questions & Answers

Q: How did Facebook create threads, and how does Rama propose to make it faster and cheaper?

Facebook spent 25 person years and had 25 to 50 engineers working for seven months to create threads. However, if they had used Rama, it could have been done with just one engineer and under 10,000 lines of code, making it 30 times faster and cheaper.

Q: What are the limitations of the current approach to building back-end systems with databases?

Currently, developers use a mix of different databases like MySQL, PostgreSQL, Oracle, MongoDB, Redis, and Snowflake. Each of these systems is a separate codebase with its own set of servers and devops needs. This repetition of code and tools in the data stack leads to more surface area for bugs and makes the development process inefficient.

Q: How does Rama address the limitations of traditional databases?

Rama integrates and generalizes every aspect of building a backend system, enabling developers to build applications in fundamentally different ways. It eliminates the need to combine different databases and tools, offering a single, general-purpose system. This allows for a significant reduction in code and provides greater flexibility, scalability, and auto-scaling capabilities.

Q: Can Rama scale and handle large volumes of data?

Yes, Rama is designed to handle large-scale applications. It is demonstrated through the example of a scalable clone of Mastodon, which can handle 100 million bots posting 3,500 statuses per second with an average fan-out of 408. The back-end of this implementation is 44 times less code than the open-source Mastodon, providing the same functionality at a fraction of the cost.

Q: How does Rama's indexing model compare to traditional databases?

Traditional databases have fixed indexing models, such as key-value, document, column-oriented, and graph. These models only support a limited number of data structure combinations, restricting flexibility. In contrast, Rama allows developers to create their own data structure combinations by using Partition States (P States). This ability to define custom data structures offers a more powerful and adaptable approach to indexing.

Q: How does Rama handle scalability and deployment across a cluster?

Rama is inherently distributed and scalable. It deploys applications across clusters by specifying the number of partitions and workers. It also offers built-in monitoring that alerts when more resources are required. Scaling an application in Rama is as simple as entering a one-liner command, which increases the number of workers, and Rama handles the transition seamlessly.

Q: Why is traditional database development wasteful in a modern data stack?

Traditional database development involves deploying and maintaining multiple databases with different data models. This approach becomes expensive as it requires managing and scaling multiple tech stacks, resulting in duplicated efforts and increased complexities. Rama's integrated approach removes the need for multiple databases and allows developers to create and manage data structures efficiently in one system.

Q: What are the main components and concepts in Rama's architecture?

Rama consists of several key components and concepts. Depots are distributed, durable, and replicated logs of data. ETLs (Extract Transform Load) consume data from Depots to produce indexes called Partition States (P States). These P States are the equivalent of databases but offer more flexibility. Finally, Rama utilizes queries, including ad hoc queries and query topologies, to retrieve and aggregate data efficiently.

Q: How does Rama enable developers to create more powerful and expressive applications?

Rama provides auto-scaling capabilities and extreme expressiveness through its integrated and generalized approach. The abstractions and architecture in Rama allow complex scaling operations to be brought down into the data layer. This eliminates the need for developers to write additional code for scalability, making applications more efficient and expressive.

Q: What resources are available to learn and try Rama?

Rama has a learning curve but offers significant benefits. The documentation on the Red Planet Labs website includes a six-part tutorial that gently introduces the concepts and helps build a complete social network in just 180 lines of code. The rest of the documentation provides in-depth guidance on Rama's various aspects. Interested individuals can try out Rama by accessing the build-a-rama tool available on the website.

Takeaways

Rama by Red Planet Labs introduces a new data backend platform that addresses the limitations of traditional database systems. By providing a more general-purpose and integrated approach, Rama allows developers to build applications with significantly less code, improved scalability, and extreme expressiveness. The architecture of Rama, including Depots, ETLs, P States, and queries, provides a flexible and powerful foundation for creating data structures that mirror the underlying information. By reducing the need for multiple databases and offering auto-scaling capabilities, Rama simplifies the development and maintenance of backend systems in a distributed and scalable manner. Interested developers can access the Rama documentation and try out the build-a-rama tool to get started on unleashing the potential of Rama in their own projects.

Summary & Key Takeaways

• Rama is a new data back-end platform that offers a more efficient and cost-effective approach to building back-end systems at scale.

• It integrates and generalizes every aspect of back-end system development, allowing developers to build applications with 100x less code.

• Rama's powerful abstractions enable auto-scaling, extreme expressiveness, and faster development time.