Software Engineering Daily

Software engineering is a deterministic field. We write lines of code, and feed data into that code, expecting to get a certain answer. Computing is deterministic because humans developed it–we understand computers from top to bottom. The same cannot be said about biology.

Matt Might is an associate professor with a PhD in computer science. When his son was diagnosed with an extremely rare illness, he was confronted with the uncertainties of human biology. In this episode, we discuss Matt’s quest to solve the puzzle of his son’s disease–computer science meets genetics, on this episode of Software Engineering Daily.

The post Using Software to Discover Rare Diseases with Matt Might appeared first on Software Engineering Daily.

Stack Overflow is used by developers to find out how to build software. Stack Overflow is both a tool and a community, and today’s guest Jeff Atwood has made a career out of building tools and communities. As the co-founder of Stack Exchange and Discourse.org, Jeff has been solving the problem of civilized online communication for seven years.

In today’s episode of Software Engineering Daily, Jeff Atwood talks about building online communities from the perspective of an engineer as well as a sociologist.

The post State of Programming with Jeff Atwood appeared first on Software Engineering Daily.

This episode is a republication from my interview with Leslie Lamport on Software Engineering Radio.

Leslie Lamport won a Turing Award in 2013 for his work in distributed and concurrent systems. He also designed the document preparation tool LaTex. Leslie is employed by Microsoft Research, and has recently been working with TLA+, a language that is useful for specifying concurrent systems from a high level.

The interview begins with a definition: a distributed system is a multiprocessor system in which the time required for interprocess communication is large compared to the time for events within a single processor–in other words, it takes longer for interprocess communication than it does for a process to look at its own memory.

Alternatively, a distributed system is one in which processors communicate by sending messages. Leslie goes on to talk about how he became interested in distributed systems, and describes the story behind his paper about the Paxos algorithm. The goal of Paxos is to maintain consensus in an environment with unexpected faults (otherwise known as Byzantine faults). After the discussion of Paxos, Jeff asks Leslie about his recent talk “Thinking for Programmers,” which emphasizes the benefit of having a specification prior to writing actual code. “Specification” can mean a variety of things, but predicates and next-state relationships provide a mathematical rigor that is well-suited to distributed and concurrent systems. The conversation concludes with Jeff asking Leslie about how a programmer can build the mental resolve to work through a difficult problem.

The post Distributed Systems with Leslie Lamport appeared first on Software Engineering Daily.

Internet Explorer, Google Chrome, Firefox–it’s easy to forget that these modern browsers descended from the war between Microsoft and Netscape. Today, we hear from a software engineer who was on the front lines of that war, back in 1992.

Eric Sink was one of the original developers of Spyglass, a browser that was licensed to both Microsoft AND Netscape. If you want to understand why that happened, and what the inside story of the browser wars is, and what the business of selling browser licenses was like in 1992, check out this episode of Software Engineering Daily.

The post Browser Wars with Eric Sink appeared first on Software Engineering Daily.

Following the successful experiment of History of Hadoop, we are doing another Saturday experiment: an editorial podcast. Let us know your thoughts via Slack, Twitter, or email!

Our podcast errs on the side of technical rigor.

Whether the topic is distributed databases, microservices, Soylent, Uber, or Dwarf Fortress, we try to separate hype from substance, deferring the narrative to the guest. With that deference there is editorial objectivity on the part of Software Engineering Daily.

One Slack channel member said “when Software Engineering Daily’s opinions are announced, that takes time away from the guest.”

Fair enough.

However, as with any journalistic organization, we have opinions. On SE Daily, we stream objectivity and batch subjectivity.

This episode is willfully subjective. It has no guest. It is a monologue editorial inspired by Developer Tea.

Immutable laws are rare in software engineering, and when an engineer claims to have found one, that engineer is usually regarded with skepticism.

General principles are more welcome.

In this post and podcast episode, I convey some loose philosophies about modern software engineering. These are strong opinions weakly held. I welcome debate and discussion.

Software is a new field and nobody knows how to do it. If someone says you are unqualified and therefore you must do maintenance work, you should question that person. We have an upside down system where the people who are paid the least do the crappiest work. They tend to be young and naive.

This is not an axiom.

The narrative that is sold to young engineers by giant companies is the following: take your $80k/yr job, do the software maintenance which makes the company $1 million, and hate your life.

After you have spent enough time in the first tier of the intellectual strip mine, we will make you an SDE 2, where you can do slightly higher level refactoring for $150k a year, which will make the giant company $5 million. This is what is called an arbitrage.

We have an assembly line mindset left over from the industrial age. Software is more of an artisanship. Don’t believe that you are a replaceable cog. Don’t believe the one-size-fits-all interview process with whiteboarding problems. These serve to grind away your individuality and make you feel like an assembly line worker.

The planning and design process is an art, but once the requirements are in place you can proceed more deterministically. The same is true for the other quantitative activities I have taken part in–poker, music, and writing. As Michael Rosenthal and I discussed, the question of art vs. science is the same question as strategy vs. tactics.

I learned this very early on playing poker when I had to leave that career, and I had tightly coupled poker to my identity. If you make your job the same thing as who you are, then your self-worth is defined by those who are judging you in your job.

Your job is a means to service your own higher purpose.

When you take an action on your smartphone, there is latency before that action is ingested.

Servers sometimes will lie to you, but servers tend towards eventual consistency. The world works the same way. In the short term, human systems lie to us all the time, but the world tends toward eventual consistency–the truth eventually presents itself.

A decent analogy is the efficient market hypothesis: slowly efficient markets are an eventually consistent process.

The world is a distributed system–what is the consequence of this? We have to do the arduous risk and reward calculations that are mandatory for every distributed systems programmer.

Long-tail failures can and do occur.

In a distributed system, we often prioritize safety over liveness. In a distributed operating system, the programmer takes all precaution to avoid data loss. Similarly, if a real life scenario presents a small probability of giant downside risk, you should take huge precaution. If someone offers you to roll a die with 1000 sides, and if you roll a 1-999, you get $100 million, but if you roll a 1000, you get your head chopped off, the upside is capped, but the downside is not.

I can estimate how significantly $100 million will improve my life, but it is impossible to quantify how bad it is to get my head chopped off. I would almost never take that offer, because the expected value is infinitely negative.

Coming back to the consequences of living in a distributed system, there is a huge range of potential outcomes in a given transaction, and assessing these long tail, massive risks takes a lot of deep thought and calculation.

This is a quote from Peter Thiel. He argues that we should think in terms of calculus rather than statistics because if you believe the world is a lottery, then you will give yourself permission to lose.

Example: a software engineer might say “I am applying to a job with 1000 applicants, so the chances I will get the job are pretty low.” That is looking at a game of skill as a game of luck, and if you don’t optimize with lots of preparation and sleep for that interview, then it’s your own fault.

If you play poker or Magic or Dominion, it is easy to blame a loss on luck, but the best players are the ones who optimize in spite of bad luck. Personally, I like playing games under circumstances when I get unlucky because it makes for more of an interesting challenge and a better learning experience.

“Shallow men believe in luck or in circumstance. Strong men believe in cause and effect.”

Luck is an idea that rescues us when we don’t do enough due diligence.

We live in a society that loves to talk about luck because it gives us an excuse when we lose. That is not to say that there isn’t luck in the world–plenty of people are unlucky. But if you are reading this, chances are you are in the luckiest .01%.

There is a long-running philosophical argument about whether or not we control our own fate. In actuality, it doesn’t really matter.

We should assume that we are in control of our own fate.

If we aren’t in control, it doesn’t change anything whether or not we assume that we are in control. But if we are in control, it would be very hazardous to our well being to assume that we are not in control.

Talk is cheap and execution is scarce.

This is why nobody cares about your ideas, they care about seeing your prototype. John Mayer said that all he had to do to become successful is finish his songs.

At Amazon, they called this “bias for action”. At Facebook, they “move fast and break things”.

That said, usually you can choose both. Long-term planning is underrated because few people take the aggressive short term actions necessary to iterate towards it. If you have never seen long-term planning bear fruits, it’s hard to see the point.

You have to figure out what to believe for yourself.

There are also lots of great people, but it is up to you to develop intuition and strong reasoning.

There is a scene from The Big Short where Mark Baum, played by Steve Carrell, is standing in front of bankers saying “the world is full of fraud, our food is fraudulent, our banking system is fraudulent, our medicine is fraudulent”, and if you look around and investigate each of these things, you do find that these systems have layers of deception and lies–software engineering is the same.

For example, if you get to a place that gives you lots of perks like free lunch or back massages, understand that they are doing that because they are underpaying you.

If you are getting $130k salary, $70k in stock options, free lunch, and free laundry, that may sound like a pretty good deal, but what does that say about how much money the company is making off of you?

According to Business Insider:

This isn’t even skewed towards how much they make off of engineers.

Software economics are unintuitive.

What is crazy about software is that it takes $0 to make new copies of a commodity that only has to be written once. This contrasts with the assembly line where each new unit takes new effort and human friction to reproduce. With software, only the first unit takes significant labor to produce.

As software engineers, we have to rethink our value as workers in a marketplace with the insane leverage of software economics.

Giant tech companies can pull off this arbitrage of underpaying engineers because engineers let themselves be seduced into a set of myths.

“The intellectual tradition is one of servility to power, and if I didn’t betray it I’d be ashamed of myself.”

-Noam Chomsky

There is a narrative of a programmer who is incapable of doing anything except programming. Some programmers talk about this with pride, saying things like “I am just an engineer, I don’t want to think about the business side of things, I don’t understand the business side of things”.

Engineers have been seduced by the industrialist’s perspective that we cannot lead ourselves, we cannot evaluate opportunity cost, and we don’t understand the market as a whole.

All of these are lies, and the world will be more efficient and utilitarian if engineers take control of their careers and start evaluating the options outside of their immediate, narrow context.

If you started in sales, you can learn to become an engineer.

You don’t need a degree–if you can do the work, you can get a job as an engineer.

In the interview with my brother Michael Rosenthal, he talked about dropping out of school and doing freelance programming, and how he has learned faster since getting out of the highly tracked education process.

In Seth Godin’s episode, he talked about the deprecated education system, and the declining worth of a degree.

Don’t let yourself be defined by your past, and the labels and messaging that society applies to you. You can take control of your life and define your future.

You should be skipping to work. We spend most of our time at work.

The job market is really good right now because our economy is being completely reshaped by software. Massive economic opportunity is being generated:

These are economic fundamentals, not spurious technical signals.

Don’t believe the ranting and raving venture capitalists arguing for lower valuations and screaming about a bubble. Don’t believe the luddite anti-technologists who sneer into their lattes.

By any logical projection, the high demand for engineers isn’t dissipating any time soon…unless the entire economy collapses in some ghastly black swan catastrophe–in which case your current job will likely vaporize anyway.

As David Heinemeier Hansson said, it is worth considering how often your job puts you in a state of flow and tranquility. If your work is stressful and entirely unpleasant, finding a new job should be a huge priority.

Thanks for reading this far. I hope this post inspires some conversation, either negative or positive and I would love to know your thoughts.

Software Engineering Daily wants to make content that is useful to engineers. Whether or not this was useful, please let us know via Slack, Twitter, or email.

The post 10 Philosophies for Engineers appeared first on Software Engineering Daily.

This episode is different from the traditional interview format of Software Engineering Daily, and focuses on the history of Hadoop. Thanks to Marco Bonaci for allowing us to republish this in audio format.

You can find the original post here: History of Hadoop

The post The History of Hadoop appeared first on Software Engineering Daily.

Ruby on Rails is a web app framework released 10 years ago that influenced the way websites were being built. Rails skyrocketed to popularity in the late 2000’s and empowered many small companies to quickly build and maintain robust web apps.

While it is still a mainstay in web development, it has been overshadowed as of late by JavaScript, and Node.js on the backend. David joins Software Engineering Daily to discuss how Ruby on Rails has evolved to keep up with newer programming paradigms and why it is still an excellent choice to build web apps with nowadays.

David Heinemeier Hansson is the creator of Ruby on Rails and a partner at Basecamp. Rails was extracted from the work done at Basecamp, and open-sourced for the public to commit to and use for their applications.

The post The Evolution of Rails with David Heinemeier Hansson appeared first on Software Engineering Daily.

Brian Kernighan is a professor of computer science at Princeton University and the author of several books, including “The Go Programming Language” and “The C Programming Language”, a book more commonly referred to as K&R. Professor Kernighan also worked at Bell Labs alongside Unix creators Ken Thompson and Dennis Ritchie and contributed to the development of Unix.

The post Language Design with Brian Kernighan appeared first on Software Engineering Daily.

This episode is republished from The Quoracast.

Questions:

The post Internet Future with Vint Cerf appeared first on Software Engineering Daily.

“It’s a classic case where you have to be contrarian. It seems like the worst idea in the world to start a cloud hosting business. We didn’t know any better.”

Moisey Uretsky is the cofounder of Digital Ocean, a leading cloud hosting provider based in New York.

“It’s the usual immigrant story. My parents moved to America when we were four or five. We were poor and didn’t have much money so our parents focused on education.”

Moisey told me about his early years, and how he started working on technology with his brother Ben.

“I was bored during my first summer break and my brother had a job at a data center company and I just started reading books about it. Two weeks later he told me to take over as a sysadmin. From there we started our first company.”

Prior to Digital Ocean, Moisey worked on ServerStack, a hosting business similar to Digital Ocean. He also tried to start a hedge fund analytics company, which he had difficulty making successful.

“I did make one sale. I had some sales experience from ServerStack. The first year was just me working and pretending to be four people. But ultimately, product-market fit in that category is extremely complicated. It’s pretty much a nightmare. I don’t envy any financial services startup.”

Moisey has built on the learnings from ServerStack and his financial services company.

“It was tremendously formative. We made every single classic business mistake. This was a decade ago when there wasn’t much of a New York City startup community. We didn’t understand how to attract customers or market properly. We weren’t able to capture the success that was available in the space. It’s easy to start, but it’s hard to scale.”

Digital Ocean was founded in a time when the field was rife with competition. Amazon Web Services, Heroku, Google App Engine, Rackspace, and others had created a market that seemed saturated.

“It’s a classic case where you have to be contrarian. It seems like the worst idea in the world to start a cloud hosting business. We didn’t know any better.”

“What was missing from all the other providers was simplicity. I have a background as a developer and a sysadmin, and yet they were still too complicated for me.”

“With AWS, the learning curve is very steep. It could be a little daunting for people just starting out. There are a lot of things that are no longer relevant that you are forced to think about. These are carryovers from days when hard drives were counted in megabytes not gigabytes.”

Everyone else was putting engineering first, and not product first. “If you get a bunch of engineers together, often times you are going to get an engineering-focused solution. We always went backwards from the product side instead of the engineering side.”

Moisey found himself personally delighted by the product that Digital Ocean was building.

“It took us about six months to build Digital Ocean. When we spun up that first server, I was like ‘this is amazing, all I want to do is build a Rails app and spin up some more servers.’ We knew that if we could deliver that feeling to other customers we were on to something.”

He pointed out similarities and differences between Digital Ocean and Heroku. The two companies both allow easy onboarding for someone looking to deploy an application to a hosted environment. But Digital Ocean offers more options for simple yet advanced configuration–a huge upside for businesses that need to iron out small problems that magnify upon a business’s event of scale.

“When you get past the point of hitting product-market fit, you can experience really fast dramatic growth. At that point, you want to be more hands-on. All of those 1% errors that you could ignore with 100 users, they begin to creep into the picture.”

“With Heroku, it’s a bit more curated, but when it doesn’t fit your product, you might need to go find something else.”

This spectrum between ease of use and configurability parallels the spectrum from small business use cases to large business use cases.  “That’s why AWS is so large.” Large businesses require lots of knobs to turn, so those customers historically have had to turn to Amazon Web Services. Digital Ocean has found a sweet spot along that configurability/ease-of-use spectrum.

Digital Ocean was started as a spinoff project from ServerStack. This strategy leveraged both the hardware and the domain expertise which the Uretskys already had.

“If you look at start-ups that have hardware as well as software–it’s a lot harder. If you have never bled in a data center, you don’t know about the problems you are going to encounter.”

“We ran two companies at the same time. Eventually, Digital Ocean had to acquire Server Stack. Server Stack was like the incubator for Digital Ocean. We learned from the mistakes we made. Without the experience of Server Stack, we would not have been able to build Digital Ocean into what it is today.”

Digital Ocean’s customer service complements its easy-to-use product offerings. I asked if it was difficult to scale the support team.

“The way to scale anything is you get a good product. If you have a good product, you get less support questions. If you hire aggressively, you get less support questions. It’s a thing you put into your mission statement early on. When we didn’t have much funding, there was probably a 50% chance that I would be answering a customer’s support question.”

Moisey told me about his experience growing up with his eventual cofounder Ben.

“We came to America, and we were always passionate about computers, but we were really poor. We got a computer when we were 13 and used AOL and a dial-up modem. My mom was starting to use Visual Basic, and I stole her book and printed out the huge Windows API because I wanted to make my own app. I got about halfway through. It was doing stupid stuff that kids do. Cracking software and whatnot.”

“We always had a strong work ethic. One thing leads to another, I never had a resume, never worked a job. You never see things from the perspective of an employee if you’ve never been one. You always think about how you can push forward.”

“For me and Ben, it was a situation where he had a job and I stopped by to learn stuff and he gave me a job. The company we were working for was going out of business. Working with family is a unique dynamic. One thing that I can say for certain is that whatever the dynamic is with the brothers, that will be the dynamic within the office.”

Locating a tech company in New York has its perks as well as its hazards.

“Ten years ago, it would have been [difficult]. But today, you have a large number of local VCs and Silicon Valley VCs with local offices. The thing is, the best people are always hard to find. The benefit to New York is that there are people that get burnt out on other cities. And there all of these offices for Twitter and Facebook and Google,” and employees are sometimes willing to leave those companies for Digital Ocean.

“Technically speaking, we should be in San Francisco. But we love New York.”

I asked a question about what sort of economic boom we are in, which he answered with subtlety.

“The internet as a whole is still growing. Developers as a whole are growing more as a sector than any other degree. And we’ve been doing this for twelve years. Because we’ve already gone through a downturn, and it didn’t really affect what we were doing, we’re optimistic.”

“Until you can stream HD right to your Oculus Rift, we haven’t reached peak internet usage.  There’s still a lot of growth in the emerging markets. Globalization had a lot of negatives and a lot of positives. We can really be a part of that globalization of information.”

The post Founding Digital Ocean with Moisey Uretsky appeared first on Software Engineering Daily.