Link to Paper This paper, published in July 1974 is remarkable because the design decisions that were made back then by these guys working at Bell Labs on an operating system for the PDP-11 are still relevant. I am still struggling to create a mental model of the unix file system, the fact that it looks like a single tree with the root at the top while simultaneously you can have multiple devices mounted dates back to these guys at Bell Labs. ...

Link to Paper From the abstract: “This paper analyzes the scalability of seven system applications running on Linux on a 48-core computer…using mostly standard parallel programming techniques -this paper introduces one new technique sloppy counters these bottlencek can be removed from the kernl or avoided by changing the application slightly” This paper has an excellent system level tutorial on scalability. They explain that you don’t get linear increase in performance because in real life applications parallel tasks usually interact, an interaction forces serial execution. Then they list the common causes with common solutions. This paper is throughly written and researched. Writing truly parallel code is difficult and even then applications still compete for some shared resouce, be it a local cache, network access or disk I/O. ...

Go I have been writing go since a little after last year. I actually remember hearing about go when it first came out, back then I honestly never thought I’d be getting paid to work in it. Even though I’ve been writing code in go for a while, I don’t think I know the language in enough depth to consider myself a go expert. I want to change that. So I am going to start writing about go here as a way to “learn in public” Expect posts on the following topics: ...

Omega Link to paper Omega was the second cluster manager system built by Google. It is Borg’s sucessor and it was designed as a happy medium between Borg’s centralized scheduler architecture and Mesos’s two-level approach where the placement is delegated to the running framework. Omega shares the state of the cluster among leaders and uses optimistic concurrency control (detect when different cluster schedulers are competing for the same resource) The premise of the whole paper is that a centralized scheduler does not scale well, so there must be a better way to handle scheduling different types of workloads in a fast and conrrect manner. The two main types of workloads, services and batches have different requriements and present their unique challenges. The paper explains the type of simulations the engineer at Google used to determine that conflicts among different scheduler is not that common and that Omega manages to fit more tasks in the clusters than Mesos. ...

Borg Link to Paper Borg is the cluster management system that runs hundreds of thousands of jobs at Google, it is the original system, it’s sucessor Omega was written as a reaction to the lessons learned from it. Kubernetes is the third system written with the lessons from those two. This paper helped me understand a few things about my own system since we have our own cluster managenet and scheduler system that work a little different but in general do the same job. ...

Link to Paper This paper was published originally in 1983 by the legendary folks from the Xerox Palo Alto Research Center. The hints and tips should sound familiar but it’s interesting to notice the layer the author is talking about, these guys were designing at very low level. The fact that the same rules apply now it’s remarkable. It turns out breaking up a system into the right abstraction with a good interface it’s rather hard. ...

Link to Paper This paper is a list of recommendations for running a large-scale system that aims to keep quality high and costs low. The author comes from the Windows Live Services Platform but this might as well be read as an Amazon internal guide since I didn’t see a thing in this list that we don’t do (or aim to do) in AWS. EDIT: Of course all these things sound familiar! The author is James Hamilton! He’s a VP and distinguished engineer here at AWS, I know him from leading the weekly AWS Ops Review meeting. ...

Link to paper Today’s paper comes thanks to Will Larson this is a recommended paper in his book Elegant Puzzle This paper builds on the concepts from the the CAP Theorem which essentially says that when it comes to distributed systems you can only have 2 out of these 3 qualities: Consistency Avalaibility Partition Tolerance Then it introduces two concepts harvest and yield which is interesting because it’s not something you usually hear in distributed systems (this is a paper from the ’90s) and yet I think it’s import to know and to think in these terms. ...

Day three: Design patterns for container-based distributed systems Link to paper Containers provide the ability to package, deploy and reuse applications using a natural isolation boundary. Developers can expose application-specific functionality as interfaces as well as more generic hooks for many systems like metrics, health, etc. In this paper the authors present the readers with two types of container design patterns: Patterns for container in the same machine (Single node) and patterns where the containers are spread out in different machines (multi node) ...

Welcome to day 2 of my paper-every-day journey. Today we’re going to cover Borg, Omega and Kubernetes This paper goes over how Kubernetes, the de-facto, open-source orchestrator of containers was developed using the lessons learned from building Borg and Omega, Google’s internal job orchestrator. Kubernetes is a container orchestration system that aims to make developing and deploying complex distributed systems easier. The Borg and Omega papers are excellent papers on their own right, but this explanation original published in ACM Queue is very insightful. These are the lessons I gathered: ...