Software Engineering Daily

When an application is using all of its available resources, that application needs to be scaled. Scaling an application means giving it more resources–typically servers. Autoscaling is an engineering practice where an application is automatically given more or less resources based on how healthy the application performance is at a given time.

Applications on Heroku have access to autoscaling. Heroku users don’t need to worry about provisioning new servers manually because the platform does it for them. In this episode, we explore how Heroku built autoscaling.

Andrew Gwozdziewycz [@apgwoz] is an operational experience engineer with Heroku. As he describes, autoscaling requires frequent health checks of an application. Since thousands of applications are running on Heroku, a metrics pipeline using Kafka and Cassandra supports the high volume of health check data. That data feeds into the decision process for when an application needs to scale.

Full disclosure: Heroku is a sponsor of Software Engineering Daily.

The post Heroku Autoscaling with Andrew Gwozdziewycz appeared first on Software Engineering Daily.

In the mid 90s, data warehousing might have meant “using an Oracle database.” Today, it means a wide variety of things. You could be stitching together a big data pipeline using Kafka, Hadoop, and Spark. You could be using managed tools like BigQuery from Google.

How did we get from the simple days of Oracle databases to the wealth of options available today? Mark Rittman writes and podcasts about data engineering and data warehousing on his site Drill to Detail. Today, we explore the past, present, and future of data warehousing and touch on many of the trends that have been explored in recent episodes of Software Engineering Daily.

Google BigQuery, and Why Big Data is About to Have Its Gmail Moment

The post Data Warehousing with Mark Rittman appeared first on Software Engineering Daily.

Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy.

At the ridesharing company Lyft, every internal service runs a tool called Envoy. Envoy is a service proxy. Whenever a service sends or receives a request, that request goes through Envoy before meeting its destination.

Matt Klein started Envoy, and he joins the show to explain why it is useful to have this layer of standardization between services. He also gives some historical context for why Envoy was so helpful to Lyft.

The post Service Proxying with Matt Klein appeared first on Software Engineering Daily.

Infrastructure is a term that can mean many different things: your physical computer, the data center of your Amazon EC2 cluster, the virtualization layer, the container layer–on and on. In today’s episode, podcasters Chris Wahl and Ethan Banks discuss the past, present, and future of infrastructure with me.

Ethan and Chris host Datanauts, a podcast about infrastructure. In each episode, Datanauts goes deep on a topic such as networking, serverless, or OpenStack. As someone who hosts a similar podcast, I find it entertaining and educational to hear their points of view on a regular basis. If you like Software Engineering Daily, you might like Datanauts. And if you like Datanauts, you will love this episode of Software Engineering Daily.

The post Infrastructure with Datanauts’ Chris Wahl and Ethan Banks appeared first on Software Engineering Daily.

Giphy is a search engine for gifs, the short animated graphics that we see around the Internet. Giphy is also a creative platform where people create new gifs.

Every search engine requires the construction of a search index, which is a data structure that responds to search queries efficiently. Since Giphy is a search engine for graphics, there is almost no text inherently associated with the each document. Giphy uses a pipeline of different labeling techniques in order to make a gif indexable by the search engine.

In my conversation with Giphy CTO Anthony Johnson, we discuss how to scale a search engine, why Giphy needs to build new techniques for image processing, how human labeling for machine learning is evolving, and the future of Giphy–both as a creative medium and an advertising platform.

This was an exciting and wide-reaching interview.

The post Giphy Engineering with Anthony Johnson appeared first on Software Engineering Daily.

Back in 2008, the range of tools that engineers could use to connect computer systems together were getting quite good. Cloud computing was democratizing access to servers. But the telephony ecosystem was still inaccessible to the average developer. If you needed your program to make a phone call and connect a user to a customer service representative, there was no easy way to do that.

Twilio was started to make it easy for developers to connect to telephone systems using simple API calls. This has unlocked many important use cases: from Uber’s communication systems to the widespread adoption of 2-factor authentication.

In this episode, Twilio VP of product management Pat Malatack joins the show to explain how the company builds and scales the telephony systems that underpin applications which we use every day. We also talked about how Twilio’s culture shapes how engineering proceeds at the company.

Full disclosure: Twilio is a sponsor of Software Engineering Daily.

The post Twilio Engineering with Pat Malatack appeared first on Software Engineering Daily.