Software Daily

Software Daily

By SoftwareDaily.comNewsTech News
Download on the App Store

Software Daily episodes

  • Incident Response with Emil Storlarsky

    As a system becomes more complex, the chance of failure increases. At a large enough scale, failures are inevitable. Incident response is the practice of preparing for and effectively recovering from these failures.

    An engineering team can use checklists and runbooks to minimize failures. They can put a plan in place for responding to failures. And they can use the process of post mortems to reflect on a failure and take full advantage of the lessons of that failure.

    Emil Storlarsky is a production engineer at Shopify where his role shares many similarities with that of Google’s site reliability engineers. In this episode, Emil argues that the academic study of emergency management and industries such as aerospace and transportation have a lot to teach software engineers about responding to production problems.  

    In this interview with guest host Adam Bell, Emil argues that we need to move beyond tribal knowledge and incorporate practices such as an incident command system and rigorous use of checklists. Emil suggests that we need to move beyond a mindset of “move fast and break things” and toward a place of more deliberate preparation.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    Incident Response Insights Talk

    The Human Side Of Post Mortems

    0 min
  • Run Less Software with Rich Archbold

    There is a quote from Jeff Bezos: “70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.”

    That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had no choice but to manage your own database, data warehouse, and search cluster. If your server crashed in the middle of the night, you had to wake up and fix it. And you had to deal with these engineering problems in addition to building your business.

    Technology today evolves much faster than in 2006. That is partly because managed cloud services make operating a software company so much smoother. You can build faster, iterate faster, and there are fewer outages.

    If you are an insurance company or a t-shirt manufacturing company or an online education platform, software engineering is undifferentiated heavy lifting. Your customers are not paying you for your expertise in databases or your ability to configure load balancers. As a business, you should be focused on what the customers are paying you for, and spending the minimal amount of time on rebuilding software that is available as a commodity cloud service.

    Rich Archbold is the director of engineering at Intercom, a rapidly growing software company that allows for communication between customers and businesses. At Intercom, the engineering teams have adopted a philosophy called Run Less Software.

    Running less software means reducing choices among engineering teams, and standardizing on technologies wherever possible.

    When Intercom was in its early days, the systems were more heterogeneous. Different teams could choose whatever relational database they wanted–MySQL or Postgres. They could choose whatever key/value store they were most comfortable with.

    The downside of all this choice was that engineers who moved from one team to another team might not know how to use the tools at the new team they were moving to. After switching teams, you would have to figure out how to onboard with those new tools, and that onboarding process was time that was not spent on effort that impacted the business.

    By reducing the number of different choices that engineering teams have, and opting for managed services wherever possible, Intercom ships code at an extremely fast pace with very few outages. In our conversation, Rich contrasts his experience at Intercom with his experiences working at Amazon Web Services and Facebook.

    Amazon and Facebook were built in a time where there was not a wealth of managed services to choose from, and this discussion was a reminder of how much software engineering has changed because of cloud computing.

    To learn more about Intercom, you can check out the Inside Intercom podcast.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • Training the Machines with Russell Smith

    Automation is changing the labor market.

    To automate a task, someone needs to put in the work to describe the task correctly to a computer. For some tasks, the reward for automating a task is tremendous–for example, putting together mobile phones. In China, companies like FOXCONN are investing time and money into programming the instructions for how to assemble your phone. Robots execute those instructions.

    FOXCONN spends millions of dollars deploying these robots, but it is a worthwhile expense. Once FOXCONN pays off the capital investment in those robots, they have a tireless workforce that can build phones all day long. Humans require training, rest, and psychological considerations. And with robots, the error rate is lower. Your smart phone runs your life, and you do not want the liability of human imperfection involved in constructing that phone.

    As we race towards an automated future, the manual tasks that get automated first depend on their economic value. The manual labor costs of smartphone construction is a massive expense for corporations. This is also true for truck driving, food service, and package delivery. The savings that will be reaped from automating these tasks are tremendous–regardless of how we automate them.

    There two ways of building automated systems: rule-based systems and machine learning.

    With rule-based systems, we can describe to the computer exactly what we want it to do–like following a recipe. With machine learning, we can train the computer by giving it examples and let the computer derive its own understanding of how to automate a task.

    Both approaches to automation have difficulties. A rule-based approach requires us to enumerate every single detail to the machine. This might work well in a highly controlled environment like a manufacturing facility. But rule-based systems don’t work well in the real world, where there are so many unexpected events, like snowstorms.

    As we reported in a previous episode about how to build self-driving cars, engineers still don’t quite know what the right mix of rule-based systems and machine learning techniques are for autonomous vehicles. But we will continue to pour money into solving this problem, because the investment is worth figuring out how to train the machine.

    The routine tasks in our world will be automated given enough time. How soon something will be automated depends on how expensive that task is when it is performed by a human, and how hard it is to design an artificial narrow intelligence to perform the task instead of a human.

    Manual software testing is another type of work that is being automated today.

    If I am building a mobile app to play podcast episodes, and I make a change to the user interface, I want to have manual quality assurance (QA) testers run through tests that I describe to them, to make sure my change did not break anything. QA tests describe high level application functionality. Can the user register and log in? Can the user press the play button and listen to a podcast episode on my app?

    Unit tests are not good enough, because unit tests only verify the logic and the application state from the point of view of the computer itself. Manual QA tests ensure that the quality of the user experience was not impacted.

    With so many different device types, operating systems, and browsers, I need my QA test to be executed in all of the different target QA environments. This requires lots of manual testers. If I want manual testing for every deployment I push, that manual testing can get expensive.

    RainforestQA is a platform for QA testing that turns manual testing into automated testing. The manual test procedures are recorded, processed by computer vision, and turned into automated tests. RainforestQA hires human workers from Amazon Mechanical Turk to execute the well-defined manual tests, and the recorded manual procedure is used to train the machines that can execute the same task in the future.

    Russell Smith is the CTO and co-founder of RainforestQA, and he joins the show to explain how RainforestQA works: the engineering infrastructure, the process of recruiting workers from mechanical turk, and the machine learning system for taking manual tasks and automating them.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • High Volume Event Processing with John-Daniel Trask

    A popular software application serves billions of user requests. These requests could be for many different things. These requests need to be routed to the correct destination, load balanced across different instances of a service, and queued for processing. Processing a request might require generating a detailed response to the user, or making a write to a database, or the creation of a new file on a file system.

    As a software product grows in popularity, it will need to scale these different parts of infrastructure at different rates. You many not need to grow your database cluster at the same pace that you grow the number of load balancers at the front of your infrastructure. Your users might start making 70% of their requests to one specific part of your application, and you might need to scale up the services that power that portion of the infrastructure.

    Today’s episode is a case study of a high-volume application: a monitoring platform called Raygun.

    Raygun’s software runs on client applications and delivers monitoring data and crash reports back to Raygun’s servers. If I have a podcast player application on my iPhone that runs the Raygun software, and that application crashes, Raygun takes a snapshot of the system state and reports that information along with the exception, so that the developer of that podcast player application can see the full picture of what was going on in the user’s device, along with the exception that triggered the application crash.

    Throughout the day, applications all around the world are crashing and sending requests to Rayguns servers. Even when crashes are not occurring, Raygun is receiving monitoring and health data from those applications. Raygun’s infrastructure routes those different types of requests to different services, queues them up, and writes the data to multiple storage layers–ElasticSearch, a relational SQL database, and a custom file server built on top of S3.

    John-Daniel Trask is the CEO of Raygun and he joins the show to describe the end-to-end architecture of Raygun’s request processing and storage system. We also explore specific refactoring changes that were made to save costs at the worker layer of the architecture. This is useful memory management strategy for anyone working in a garbage collected language. If you would like to see diagrams that explain the architecture and other technical decisions, the show notes have a video that explains what we talk about in this show. Full disclosure: Raygun is a sponsor of Software Engineering Daily.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • Fiverr Engineering with Gil Scheinfeld

    As the gig economy grows, that growth necessitates innovations in the online infrastructure powering these new labor markets.

    In our previous episodes about Uber, we explored the systems that balance server load and gather geospacial data. In our coverage of Lyft, we studied Envoy, the service proxy that standardizes communications and load balancing among services. In shows about Airbnb, we talked about the data engineering pipeline that powers economic calculations, user studies, and everything else that requires a MapReduce.

    In today’s episode, we explore the business and engineering behind another online labor platform: Fiverr.

    Fiverr is a marketplace for digital services. On Fiverr, I have purchased podcast editing, logo creation, music lyrics, videos, and sales leads. I have found people who will work for cheap, and quickly finish a job to my exact specification. I have discovered visual artists who worked with me to craft a music video for a song I wrote.

    Workers on Fiverr post “gigs”–jobs that they can perform. Most of the workers on Fiverr specialize in knowledge work, like proofreading or gathering sales leads. The workers are all over the world. I have worked with people from Germany, the Philippines, and Africa through Fiverr.

    Fiverr has become the leader in digital freelancing. The staggering growth of Fiverr’s marketplace has put the company in a position similar to an early Amazon. There is room for strategic expansion, but there is also an urgency to improve the infrastructure and secure the market lead.

    Gil Scheinfeld is the CTO at Fiverr, and he joins the show to explain how the teams at Fiverr are organized to fulfill the two goals of strategic, creative growth and continuous improvement to the platform.

    One engineering topic we discussed at length was event sourcing. Event sourcing is a pattern for modeling each change to your application as an event. Each event is placed on a pub/sub messaging queue, and made available to the different systems within your company. Event sourcing creates a centralized place to listen to all of the changes that are occurring within your company.

    For example, you might be working on a service that allows a customer to make a payment to a worker. The payment becomes an event. Several different systems might want to listen for that event. Fiverr needs to call out to a credit card processing system. Fiverr also needs to send an email to the worker, to let them know they have been paid. Fiverr ALSO needs to update internal accounting records.

    Event sourcing is useful because the creator of the event is decoupled from all of the downstream consumers. As the platform engineering team works to build out event sourcing, communications between different service owners will become more efficient.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • BigQuery with Jordan Tigani

    Large-scale data analysis was pioneered by Google, with the MapReduce paper. Since then, Google’s approach to analytics has evolved rapidly, marked by papers such as Dataflow and Dremel.

    Dremel combined a column-oriented, distributed file system with a novel way of processing queries. A single Dremel query is distributed into a tree of servers, starting with the root server, splitting into the intermediate servers, and ending with the leaf servers talking to the file system. Once the data is pulled from the file system into the leaves, the data propagates back to the root server, and is shuffled along the way so that the root server receives a sorted response.

    When Google started turning its internal services into customer-facing cloud products, the effort to productize Dremel began, and BigQuery was born. Jordan Tigani is an engineering lead who works on BigQuery, and he joins the show to discuss the evolution of the data warehouse.

    Large scale distributed queries still can take a long time–but queries get faster every year. Queries that required a nightly Hadoop job 10 years ago can be viewed in a frequently updated user-facing dashboard. Power users of BigQuery talk about the speed and the query interface as being two of its most valuable differentiating features. As the job of a large scale data analyst becomes less technically intensive, tools like BigQuery will continue to rise in popularity.

    We have done some great shows about Google papers like Spanner, Dremel, and Dataflow. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help.

    Shout out to today’s featured contributor Shreyans Sheth. Shreyans has worked on the Software Engineering Daily search API, and has also helped us understand open source best practices, which we are still learning. Thanks again Shreyans for your work.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • Netflix Serverless-like Platform with Vasanth Asokan

    The Netflix API is accessed by developers who build for over 1000 device types: TVs, smartphontes, VR headsets, laptops. If it has a screen, it can probably run Netflix. On each of these different devices, the Netflix experience is different. Different screen sizes mean there is variable space to display the content.

    When you open up Netflix, you want to efficiently browse through movies. The frontend engineers who are building different experiences for different device types need to make different requests to the backend to fetch the right amount of data. This was the engineering problem that Vasanth Asokan and his team at Netflix was tasked with solving: how do you enable lots of different frontend engineers to get whatever they need from the backend?

    This problem led to the development of a “serverless-like platform” within Netflix, which Vasanth wrote about in a few popular articles on Medium. This platform enables frontend developers to write and deploy backend scripts to fetch data, decoupling the responsibilities of frontend engineers and backend engineers.

    The tight coupling of frontend and backend engineering was problematic to the development velocity of Netflix.

    We have done many shows about Netflix engineering, covering topics like data engineering, user interface design, and performance monitoring. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • Serverless Authentication with Bobby Johnson

    Serverless architecture is software that runs without an addressable server. Serverless is made possible by two types of technology: platform as a service providers like Auth0, and functions as a service like AWS Lambda.

    With both of these technologies, we can program logic that runs without being deployed to a server.

    Functions as a service are cheap and scalable. Write your code for a Serverless function, and the cloud provider will cheaply deploy and execute that function on some server somewhere. The difficult part is maintaining state.

    Since Serverless compute instances are ephemeral, you aren’t dealing with a system that will keep track of your state—it is going to disappear eventually.

    The ephemeral nature of Serverless code requires us to shift our thinking—but the dramatic cost and simplified scalability make it well worth the effort.

    Serverless functions can add complexity in exchange for lower price. Serverless “platform as a service” often lowers complexity at a slightly higher price.

    A Serverless database like Firebase handles database scaling and gives you a nice web interface. A Serverless machine learning platform like Google CloudML gives your models scalability and controlled deployment. A Serverless authentication service like Auth0 manages your authentication.

    In addition to authentication, Auth0 has built a set of tools to allow SaaS companies to extend their platforms into a sandboxed code execution environment. Bobby Johnson is an engineer at Auth0, and he joins the show to describe the toolbox that Auth0 has developed: authentication, webtasks, and extensibility–and how the world of “serverless” architecture is evolving. Full disclosure, Auth0 is a sponsor of Software Engineering Daily.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • Parlaying Failure to Fortune with Paul Martino

    In 2003, Paul Martino co-founded Tribe.net, one of the earliest social networking sites.  Tribe had significant traction, with hundreds of thousands of users.

    In the early 2000s, hundreds of thousands of users was enough traffic to pose a company with engineering challenges. Paul had studied computer science, and was able to use his knowledge of high-performance computing to write an efficient graph database, and solve the other technical puzzles that the company faced–but the business did not ultimately work out.

    The failure of Tribe made the founders even hungrier for success–and it taught them lessons that they carried into subsequent businesses.

    Paul went on to start Aggregate Knowledge, a marketing technology company that sold for $119 million. His Tribe co-founder Mark Pincus went on to start Zynga, the multi-billion dollar gaming company. Another Tribe employee co-founded Yammer, which sold to Microsoft for a billion dollars.

    Since his exit from Aggregate Knowledge, Paul Martino started Bullpen Capital, which makes post-seed investments. The Bullpen Capital portfolio is appealing to me–partly because of the number of Internet gambling companies. Paul and I talked about gambling and other taboo business sectors–as well as what makes a good investment in the “post-seed” category.

    I enjoyed speaking to Paul because he has a straightforward, no-nonsense way of talking about things–it’s very charismatic and uncommon.

    We have done some great shows with other engineering investors like Chris Dixon and Adrian Colyer. To find these old episodes, you can download the Software Engineering Daily app for iOS and for Android. In other podcast players, you can only access the most recent 100 episodes. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help.

    Shout out to today’s featured contributor Kurian Vithayathil. He has made significant contributions to the Software Engineering Daily Android app. Thanks again Kurian for your work.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min
  • Bad Men with Bob Hoffman

    In the 1960s, advertising agencies were high-dollar creative producers. A client would come to an ad agency and pay millions of dollars for artistic messaging that would convince a consumer to buy a product.

    How could you measure the success of these advertising campaigns? Maybe you could see success in the sales data. Maybe people were starting to talk about the product. Ultimately, success was defined by how satisfied the client was.

    When it comes to measuring outcomes, advertising has always been a messy business.

    Bob Hoffman’s long career in advertising included CEO positions at three different agencies. He helped huge brands craft their messaging, and grab consumer attention. In Bob’s world of advertising, lots of money was spent on creativity. Were the campaigns successful? That depends who you ask. In the old world of advertising, everyone acknowledged that success was subjective.

    As human attention moved online, the world of advertising changed. Advertising began to move from TV and magazines to websites. Technology companies were formed to enable this new type of advertising–known as adtech. These companies claimed to bring scientific accuracy to advertising campaigns. The biggest adtech player was Google, who perfected search advertising.

    If you could imagine the opposite of what Bob Hoffman built his career doing, it might look like search advertising.

    Bob’s campaigns were about creating a brand’s voice, with colorful art and subtlety and ambient messaging. Advertising was about turning a brand into an entity you recognize, teaching the consumer to associate Nike with fitness, or Dove soap with clean hands, or Cheetos with cheesy, salty attitude.

    Search advertising, on the other hand, is just text. You enter a search query, you are looking for black socks, and the top link that comes back is a line of text that says “cheap black socks.” Search advertising catches people who have an intent to do something. They have stated their intent by typing into a box. With search advertising, a brand might not even need a sexy, flashy creative.

    The idea of intent-based advertising was expanded with retargeting. As you navigate through the internet, adtech companies are watching you, gathering data on your intent. Maybe you aren’t typing your desires explicitly into a search box–but you are clicking on articles and blog posts and tweets. With all your online interactions, adtech companies can figure out that you are looking for black socks whether you say so or not.

    Money poured into adtech for very good reasons: intent-based marketing works. The shift to adtech put agencies in an uncomfortable position. If they couldn’t capture the advertising market by selling highly produced, unmeasurable creativity, they would have to make their money doing something else.

    The situation was this: big brands like Procter and Gamble were buying most of their advertising through agencies. Procter and Gamble decided it wanted digital advertising. The adtech companies were the ones who knew how to produce and distribute digital advertising. Since agencies had the relationships with the big brands, and adtech companies had the technology, agencies began to partner with adtech companies.

    This is actually a simplified version of what happens. Agencies subcontract advertising deals to digital agencies. A digital agency buys technology from a slew of adtech companies: some technology tracks users around the internet, some technology places bids on advertising spots that will land in front of users. Because of all the middlemen, the incentives are aligned against the brands.

    Contrast this world of agencies and adtech companies with the world of Google and Facebook. They are a duopoly because they earned that position. By creating a single monolithic purchasing process, they have removed much of the risk that comes from a purchasing process stocked with middlemen.

    But back to Bob Hoffman.

    As money poured into adtech, user tracking, and Google, brands started to care more about metrics. When Bob met with a brand, the brand wouldn’t be asking about the cool new advertising campaign featuring a young actress drinking a Coca Cola. The brand would be asking about the click-through rate of a display advertising campaign.

    Brands moved their focus to statistics, and away from creativity. And technology companies were happy to provide them with statistics. Whether those statistics were true or not is another story altogether. The industry was moving from creative BS to outright lying, and Bob decided to leave.

    In today’s episode, Bob explains how the state of advertising became so problematic, and the ways in which it harms us Internet users.

    We have done lots of reporting about advertising fraud for the last year, and it is a popular topic because people are often shocked to find that online advertising is inextricably linked to organized crime, surveillance, and Twitter botnets. That’s not to say that online advertising doesn’t work–it certainly does! But understanding the dark underbelly of the Internet’s cash cow is a necessary precondition to finding a solution.

    To find all of our old episodes about ad fraud, you can download the Software Engineering Daily app for iOS and for Android. With these apps, we are building a new way to consume content about software engineering. They are open-sourced at github.com/softwareengineeringdaily. If you are looking for an open source project to get involved with, we would love to get your help.

    Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

    0 min

About Software Daily

From the publisher's feed

Technical interviews about software topics.