
Sign up to save your podcasts
Or


This transcript from an AWS re:Invent 2024 session details Amazon S3's architecture and engineering principles. Two senior engineers explain how S3's massive scale enables efficient data management, utilising techniques like shuffle sharding to distribute workloads across millions of drives. They discuss the physics of data storage, showcasing how S3's design improves performance and reliability by mitigating the impact of hardware limitations and individual workload bursts. Erasure coding is highlighted as a key technology that ensures data durability and facilitates faster software deployment. Finally, the presentation emphasises how S3's fault-tolerant design, built on principles of decorrelation, ultimately benefits customers by providing a consistently high-performing and reliable service.
This blog post by Marc Brooker, an AWS engineer, explains the write operations within Aurora DSQL, a scalable SQL database. It details how Aurora DSQL uses optimistic concurrency control (OCC) combined with multiversion concurrency control (MVCC) to achieve strong consistency and snapshot isolation. The system uses an "adjudicator" service to manage write conflicts and a "Journal" for durable, ordered data replication. The author highlights the benefits of this approach for scalability, availability, and performance, emphasising the avoidance of locking. Finally, the post touches upon the system's consistency guarantees and the importance of schema design to minimise write conflicts.
This research article details the development and benchmarking of a high-performance URL parser compliant with the WHATWG standard. The authors created a C++ implementation leveraging vectorisation techniques, resulting in a parser significantly faster than existing solutions like curl and rust-url. Their parser was integrated into Node.js, leading to substantial performance improvements in URL processing within that environment. Extensive benchmarks across various datasets and platforms demonstrated the superior speed and efficiency of their new parser. The authors also provide open-source access to their code and datasets.
This article presents technology predictions for 2025 and beyond, focusing on several key themes. Firstly, it highlights a growing mission-driven workforce prioritising positive societal impact over solely financial gain. Secondly, it discusses the crucial role of technological innovation in addressing the global energy crisis, advocating for a blend of renewable and nuclear solutions alongside improved energy consumption practices. Thirdly, the text explores the use of AI-powered tools to combat the spread of misinformation and disinformation. Finally, it examines the emerging trend of intention-driven consumer technology, emphasising mindful usage and a reduction in constant digital distraction.
This paper details the architecture and design of Amazon Aurora, a cloud-native relational database service. Key design choices focus on mitigating network bottlenecks inherent in high-throughput cloud systems by offloading redo processing to a separate, distributed storage service. This approach enhances durability and availability through a novel quorum model and segmented storage, significantly improving performance and reducing recovery times. The authors present performance benchmarks demonstrating Aurora's superior scalability and efficiency compared to traditional MySQL configurations, along with lessons learned from real-world customer deployments highlighting the system's suitability for modern cloud applications. Finally, the paper compares Aurora's design to related work in distributed database systems.
This research paper investigates the feasibility and limitations of using Amazon S3, a cloud storage service, as a database system for web applications. The authors present protocols for managing reads, writes, and commits to S3, addressing issues of concurrency and consistency. They explore different consistency levels, including eventual consistency and stronger guarantees like atomicity and monotonic reads/writes, analysing their performance and cost implications using the TPC-W benchmark. The study highlights the trade-offs between consistency, availability, and scalability inherent in utilising S3 for database applications and proposes solutions to enhance transactional properties while retaining S3’s inherent advantages. The paper concludes by discussing related work and future research directions.
This 2003 USENIX paper challenges the prevailing belief that event-based programming is superior to thread-based programming for high-concurrency servers. The authors argue that the perceived weaknesses of threads stem from flawed implementations rather than inherent limitations. They present a high-performance user-level thread package as evidence, supporting their claim that threads offer a simpler, more natural programming model. Furthermore, they propose compiler enhancements to further improve thread performance and safety. The paper compares and contrasts the two approaches, revisiting earlier work on their duality, and concludes that with proper compiler support, threads provide a superior solution for building scalable servers.
This Medium article by Shivam Bhadani provides a comprehensive guide to system design for beginners. It covers fundamental concepts like servers, latency, and throughput, progressing to advanced topics such as scaling strategies (vertical and horizontal), database scaling, microservices, caching, and message brokers. The author emphasises practical implementation alongside theory, using real-world examples and providing exercises to reinforce learning. The article concludes with advice on approaching system design problems and includes numerous illustrations. A strong focus is placed on distributed systems and their associated challenges, including consistency and leader election.
Article: https://medium.com/@shivambhadani_/system-design-for-beginners-everything-you-need-in-one-article-c74eb702540b
This 1995 paper argues that threads, while powerful for achieving true CPU concurrency, are overly complex for most programming tasks. The author, John Ousterhout, contends that event-driven programming offers a simpler, more reliable alternative for applications such as GUIs and distributed systems. He highlights the difficulties of thread synchronisation, debugging, and performance optimisation, contrasting them with the relative ease of event handling. Ousterhout advocates using threads only when genuine parallel processing across multiple CPUs is essential, suggesting that even then, they should be confined to a core kernel within a predominantly single-threaded application. Ultimately, the paper promotes a pragmatic approach, prioritising simplicity and maintainability unless high-performance concurrency is absolutely necessary.
This paper examines SQLite, the world's most widely deployed database engine, exploring its history, architecture, and performance characteristics. The authors benchmark SQLite against DuckDB, an analytics-focused database, across various workloads (OLTP, OLAP, and blob processing). Key performance bottlenecks in SQLite's OLAP capabilities are identified and addressed through optimisation, resulting in significant speed improvements. The study also considers the resource footprint of both databases, comparing their compilation times, library sizes, and data storage efficiency. Finally, the authors discuss future development directions for SQLite, balancing performance enhancements with its established strengths of portability, compactness, and reliability.