Software Engineering Daily

On January 31st 2017, GitLab experienced a major outage of their online repository hosting service. The primary database server experienced data loss due to a combination of malicious spam attacks and engineering mistakes that occurred while trying to respond to those spam attacks. GitLab responded to the event transparently. The company put up a postmortem describing the event in detail. In subsequent posts, GitLab expressed sympathy for the employee whoContinue reading...

Different databases have different access patterns. Key-value, document, graph, and columnar databases are useful under different circumstances. For example, if you are a bank, and you have a database of customers and the transactions they have performed, the ideal access pattern for aggregating the total amount of all transactions might be a columnar store. If the transaction amounts are all in one column, it helps to have all of theContinue reading...

With a fast-growing field like data science, it is important to keep some amount of skepticism. Tools can be overhyped, buzzwords can be overemphasized, and people can forget the fundamentals. If you have bad data, you will get bad results in your experimentation. If you don’t know what statistical approach you want to take to your data, it doesn’t matter how well you know Spark or TensorFlow. And if youContinue reading...

Apple controls the iOS ecosystem. As an accident of history, Apple also controls the podcasting ecosystem. Unlike most ecosystems within Apple’s dominion, podcasts remain open. A podcaster merely has to record an mp3, distribute it via RSS feed, and submit that RSS feed to the iTunes podcast portal. Podcasting has thrived in recent years, but very few technology companies have managed to take advantage of that growth. Libsyn is theContinue reading...

Social networks like Facebook and Twitter facilitate interactions between individuals. Every message I send to you on Facebook goes through Facebook’s servers before reaching you. This is known as the client-server model. Since the early days of the internet, engineers have always envisioned a peer-to-peer model, where I could communicate to you directly, without a company brokering that relationship. Andre Staltz works on Scuttlebutt, a peer-to-peer system for social graphs,Continue reading...

Everyone in the world should have some basic level of guaranteed healthcare. This is not controversial. But what should that basic level of healthcare be? Should it extend into the later years of your life, when the majority of your health costs are incurred? And how much has modern technology driven down the cost of what it should cost to treat a patient? Healthcare today has lots of problems withContinue reading...

Relay is a JavaScript framework for building data-driven React applications. Facebook open sourced Relay around the same time they open sourced GraphQL, and Facebook expected Relay to be the more popular of the two projects. However, the reality was reversed. Open source companies like Meteor quickly began to build GraphQL tools and a few businesses were started around GraphQL. One year later, the excitement for GraphQL had completely surpassed theContinue reading...

Healthcare is a complex business. Oscar is a company that wanted to build a new insurance provider–but realized that healthcare is so interconnected that in order to build a new insurance provider, realized it actually needed to build an entire healthcare business too, complete with patient management and facilities. Since Oscar is a modern technology company, the focus on customer service, engineering, and data management offers an optimistic view intoContinue reading...

Many companies are transitioning from a monolith to microservices architecture. Tools for cloud computing, containerization, and continuous delivery are making this easier. But there are still technological and organizational challenges that a company will encounter while making this transition. Cassandra Shum is an engineer with ThoughtWorks. She has worked with major financial institutions and other large companies to architect their migrations from monolith to microservices. Also, she regularly puts onContinue reading...

Cloud computing changed how we develop applications for the web. Over the last decade, engineers have been learning how to build software in this new paradigm. The costs have gone down, but our nodes can fail at any time. We no longer have to manage individual servers, but the layers of virtualization and containerization require new strategies for communicating between services. As we have adjusted to this new way ofContinue reading...