
Sign up to save your podcasts
Or


Software engineering is a deterministic field. We write lines of code, and feed data into that code, expecting to get a certain answer. Computing is deterministic because humans developed it–we understand computers from top to bottom. The same cannot be said about biology.
Matt Might is an associate professor with a PhD in computer science. When his son was diagnosed with an extremely rare illness, he was confronted with the uncertainties of human biology. In this episode, we discuss Matt’s quest to solve the puzzle of his son’s disease–computer science meets genetics, on this episode of Software Engineering Daily.
Big data is yesterday–fast data is now. FiloDB is a reactive columnar OLAP database that is built on Cassandra and Spark. Today’s guest is Evan Chan, creator of FiloDB.
In our discussion today, we talk about the use cases of an OLAP data store. Evan explains how to tackle the problem of video analytics–if you have ever found yourself asking how a company like YouTube or Netflix or Ooyala performs analytics on millions of users watching millions of videos, this episode is for you. By combining the database features of Cassandra with the data processing power of Spark, Evan created FiloDB to help solve this type of analytics problem. Evan will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.
Hadoop was created in 2003. In the early years, Hadoop provided large scale data processing with MapReduce, and distributed fault-tolerant storage with the Hadoop Distributed File System. Over the last decade, Hadoop has evolved rapidly, with the support of a big open-source community.
Today’s guest is Mike Cafarella, co-creator of Hadoop. Mike takes us on a journey from past to present. Hadoop was based on the Google File System and MapReduce papers, and so Mike and I talk about what it was like to work on a distributed file system in 2004, and the challenges of implementing real software systems based on white papers. We also discuss YARN, and the wave of innovation that YARN enabled within the Hadoop ecosystem. Mike will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.
Microsoft and Amazon are the tech giants of the Pacific Northwest. These two companies shape Seattle technology, and Todd Bishop has been writing about them for over a decade. Todd is the co-founder of GeekWire, an online media company focused on technology in the Pacific Northwest. We talk about the past and present of Microsoft and Amazon, what it means for an engineering company to be good or evil, and what it’s like to be a software journalist.
Technology careers are not as accessible to certain demographics, such as women. Sarah Allen is working to change that through education and outreach. Sarah Allen created RailsBridge and Bridge Foundry, a pair of organizations working to make technology more accessible to people who are underrepresented in tech.
RailsBridge workshops are a fun and free way to get started with Ruby, Rails or other technologies. RailsBridge was so successful that Sarah created Bridge Foundry, which has an even wider variety of free workshops, from Clojure to Go to Mobile. This is an inspirational episode of Software Engineering Daily, and is a great learning opportunity for anyone interested in building a community.
University research produces numerous papers about software engineering. Unfortunately, many of the problems explored by these software engineering researchers have no actual application. These papers fall into The Black Hole of Software Engineering Research.
Andy Ko is an associate professor at the University of Washington. When he worked as a CTO of AnswerDash, he experimented with some of the software engineering research coming out of academia and industry. Today on Software Engineering Daily, we discuss Andy’s results. Our conversation spans the topics of education, big companies, and startups.
JavaScript is everywhere, on the browser, the server, and now on hardware. Finally–the Internet of Things is upon us — and it is powered by JavaScript. Even if you consider yourself a web developer, and you have no interest in “maker” culture, keep your eye on this space. Factories in Shenzhen are making better, more modular parts, and the same supply chain economics that are driving down the cost of smart phones and smart cars, will give hobbyist engineers all the simple hardware components they could ever need. This will power a future of proliferated devices that have internet connectivity and “talk” to each other to enable user experiences we wouldn’t dream of today. And naturally, with this will come problems that we cannot anticipate as well.
On today’s episode, Andrew discusses how JavaScript can be used on hardware. Jeff and Andrew talk about the players in the hardware space, as well as how the big cloud players like AWS and Azure are positioning themselves to be the data lakes for the internet of things. Andrew is giving a talk about this at the 2016 O’Reilly Fluent Conference. If you want to win a free ticket to Fluent, tweet about your favorite episode of Software Engineering Daily and tag @fluentconf and @software_daily.
Andrew Chalkley is a hardware hobbyist, software engineer and a full-time teacher at Treehouse.
Following the successful experiment of History of Hadoop, we are doing another Saturday experiment: an editorial podcast. Let us know your thoughts via Slack, Twitter, or email!
Our podcast errs on the side of technical rigor.
Whether the topic is distributed databases, microservices, Soylent, Uber, or Dwarf Fortress, we try to separate hype from substance, deferring the narrative to the guest. With that deference there is editorial objectivity on the part of Software Engineering Daily.
One Slack channel member said “when Software Engineering Daily’s opinions are announced, that takes time away from the guest.”
Fair enough.
However, as with any journalistic organization, we have opinions. On SE Daily, we stream objectivity and batch subjectivity.
This episode is willfully subjective. It has no guest. It is a monologue editorial inspired by Developer Tea.
Immutable laws are rare in software engineering, and when an engineer claims to have found one, that engineer is usually regarded with skepticism.
General principles are more welcome.
In this post and podcast episode, I convey some loose philosophies about modern software engineering. These are strong opinions weakly held. I welcome debate and discussion.
Software is a new field and nobody knows how to do it. If someone says you are unqualified and therefore you must do maintenance work, you should question that person. We have an upside down system where the people who are paid the least do the crappiest work. They tend to be young and naive.
This is not an axiom.
The narrative that is sold to young engineers by giant companies is the following: take your $80k/yr job, do the software maintenance which makes the company $1 million, and hate your life.
After you have spent enough time in the first tier of the intellectual strip mine, we will make you an SDE 2, where you can do slightly higher level refactoring for $150k a year, which will make the giant company $5 million. This is what is called an arbitrage.
We have an assembly line mindset left over from the industrial age. Software is more of an artisanship. Don’t believe that you are a replaceable cog. Don’t believe the one-size-fits-all interview process with whiteboarding problems. These serve to grind away your individuality and make you feel like an assembly line worker.
The planning and design process is an art, but once the requirements are in place you can proceed more deterministically. The same is true for the other quantitative activities I have taken part in–poker, music, and writing. As Michael Rosenthal and I discussed, the question of art vs. science is the same question as strategy vs. tactics.
I learned this very early on playing poker when I had to leave that career, and I had tightly coupled poker to my identity. If you make your job the same thing as who you are, then your self-worth is defined by those who are judging you in your job.
Your job is a means to service your own higher purpose.
When you take an action on your smartphone, there is latency before that action is ingested.
Servers sometimes will lie to you, but servers tend towards eventual consistency. The world works the same way. In the short term, human systems lie to us all the time, but the world tends toward eventual consistency–the truth eventually presents itself.
A decent analogy is the efficient market hypothesis: slowly efficient markets are an eventually consistent process.
The world is a distributed system–what is the consequence of this? We have to do the arduous risk and reward calculations that are mandatory for every distributed systems programmer.
Long-tail failures can and do occur.
In a distributed system, we often prioritize safety over liveness. In a distributed operating system, the programmer takes all precaution to avoid data loss. Similarly, if a real life scenario presents a small probability of giant downside risk, you should take huge precaution. If someone offers you to roll a die with 1000 sides, and if you roll a 1-999, you get $100 million, but if you roll a 1000, you get your head chopped off, the upside is capped, but the downside is not.
I can estimate how significantly $100 million will improve my life, but it is impossible to quantify how bad it is to get my head chopped off. I would almost never take that offer, because the expected value is infinitely negative.
Coming back to the consequences of living in a distributed system, there is a huge range of potential outcomes in a given transaction, and assessing these long tail, massive risks takes a lot of deep thought and calculation.
This is a quote from Peter Thiel. He argues that we should think in terms of calculus rather than statistics because if you believe the world is a lottery, then you will give yourself permission to lose.
Example: a software engineer might say “I am applying to a job with 1000 applicants, so the chances I will get the job are pretty low.” That is looking at a game of skill as a game of luck, and if you don’t optimize with lots of preparation and sleep for that interview, then it’s your own fault.
If you play poker or Magic or Dominion, it is easy to blame a loss on luck, but the best players are the ones who optimize in spite of bad luck. Personally, I like playing games under circumstances when I get unlucky because it makes for more of an interesting challenge and a better learning experience.
Luck is an idea that rescues us when we don’t do enough due diligence.
We live in a society that loves to talk about luck because it gives us an excuse when we lose. That is not to say that there isn’t luck in the world–plenty of people are unlucky. But if you are reading this, chances are you are in the luckiest .01%.
There is a long-running philosophical argument about whether or not we control our own fate. In actuality, it doesn’t really matter.
We should assume that we are in control of our own fate.
If we aren’t in control, it doesn’t change anything whether or not we assume that we are in control. But if we are in control, it would be very hazardous to our well being to assume that we are not in control.
Talk is cheap and execution is scarce.
This is why nobody cares about your ideas, they care about seeing your prototype. John Mayer said that all he had to do to become successful is finish his songs.
At Amazon, they called this “bias for action”. At Facebook, they “move fast and break things”.
That said, usually you can choose both. Long-term planning is underrated because few people take the aggressive short term actions necessary to iterate towards it. If you have never seen long-term planning bear fruits, it’s hard to see the point.
You have to figure out what to believe for yourself.
There are also lots of great people, but it is up to you to develop intuition and strong reasoning.
There is a scene from The Big Short where Mark Baum, played by Steve Carrell, is standing in front of bankers saying “the world is full of fraud, our food is fraudulent, our banking system is fraudulent, our medicine is fraudulent”, and if you look around and investigate each of these things, you do find that these systems have layers of deception and lies–software engineering is the same.
For example, if you get to a place that gives you lots of perks like free lunch or back massages, understand that they are doing that because they are underpaying you.
If you are getting $130k salary, $70k in stock options, free lunch, and free laundry, that may sound like a pretty good deal, but what does that say about how much money the company is making off of you?
According to Business Insider:
This isn’t even skewed towards how much they make off of engineers.
Software economics are unintuitive.
What is crazy about software is that it takes $0 to make new copies of a commodity that only has to be written once. This contrasts with the assembly line where each new unit takes new effort and human friction to reproduce. With software, only the first unit takes significant labor to produce.
As software engineers, we have to rethink our value as workers in a marketplace with the insane leverage of software economics.
Giant tech companies can pull off this arbitrage of underpaying engineers because engineers let themselves be seduced into a set of myths.
There is a narrative of a programmer who is incapable of doing anything except programming. Some programmers talk about this with pride, saying things like “I am just an engineer, I don’t want to think about the business side of things, I don’t understand the business side of things”.
Engineers have been seduced by the industrialist’s perspective that we cannot lead ourselves, we cannot evaluate opportunity cost, and we don’t understand the market as a whole.
All of these are lies, and the world will be more efficient and utilitarian if engineers take control of their careers and start evaluating the options outside of their immediate, narrow context.
If you started in sales, you can learn to become an engineer.
You don’t need a degree–if you can do the work, you can get a job as an engineer.
In the interview with my brother Michael Rosenthal, he talked about dropping out of school and doing freelance programming, and how he has learned faster since getting out of the highly tracked education process.
In Seth Godin’s episode, he talked about the deprecated education system, and the declining worth of a degree.
Don’t let yourself be defined by your past, and the labels and messaging that society applies to you. You can take control of your life and define your future.
You should be skipping to work. We spend most of our time at work.
The job market is really good right now because our economy is being completely reshaped by software. Massive economic opportunity is being generated:
These are economic fundamentals, not spurious technical signals.
Don’t believe the ranting and raving venture capitalists arguing for lower valuations and screaming about a bubble. Don’t believe the luddite anti-technologists who sneer into their lattes.
By any logical projection, the high demand for engineers isn’t dissipating any time soon…unless the entire economy collapses in some ghastly black swan catastrophe–in which case your current job will likely vaporize anyway.
As David Heinemeier Hansson said, it is worth considering how often your job puts you in a state of flow and tranquility. If your work is stressful and entirely unpleasant, finding a new job should be a huge priority.
Thanks for reading this far. I hope this post inspires some conversation, either negative or positive and I would love to know your thoughts.
Software Engineering Daily wants to make content that is useful to engineers. Whether or not this was useful, please let us know via Slack, Twitter, or email.
Prepare for the post-browser world, says today’s guest Jonathan Stark. We are moving beyond the browser, into a world where the web is consumed by a variety of front ends.
Jonathan is a speaker at the upcoming O’Reilly Fluent Conference in San Francisco, and you can win a free ticket to the conference. To be entered into a random drawing for that ticket, send us a tweet about your favorite episode of Software Engineering Daily between now and February 22nd. Include the hashtag #fluentconf and and tag us @software_daily to make sure we can see your tweet.
Fluent is a conference about the web platform–which certainly includes the browser, but also mobile apps, VR experiences, and smart objects connected to the web. In today’s episode, Jeff and Jonathan explore what adjustments developers should make in this post-browser world.
Jonathan Stark is a mobile consultant who helps consumer brands thrive in the post-PC era.
You keep hearing about containers, and maybe you have even used Docker in production. Now, it’s time to move beyond Docker. In today’s episode of Software Engineering Daily, Steve Pousty talks about OpenShift, a platform from RedHat that helps engineers leverage the power of containers and the dev ops harmony of microservices.
To hear these buzzwords, you can always listen to Software Engineering Daily. But to see these buzzwords demonstrated in live presentations, the 2016 O’Reilly Fluent Conference is coming up March 8-10 in San Francisco, and you can get a chance to win a free ticket by tweeting at us about your favorite episode of Software Engineering Daily. Include the hashtag #fluentconf and and tag us @software_daily to make sure we can see your tweet.
Steve Pousty is a Developer Advocate (and PaaS Dust Spreader) at Red Hat.
From the publisher's feed