Software Architecture Insights

We often hear that being able to scale your application is important. But why is it important? Why do we need to be able to suddenly, and without notice, scale our application to handle double, triple, or even ten times the load it is currently experiencing?

Why is scaling important?

In this episode, I am going to talk about four basic reasons. Four reasons why scaling is important to the success of your business.

And then, what is the dynamic cloud?

This is Application Scaling, on Modern Digital Applications.

Links and More Information

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)

Why you must scale

We often hear that being able to scale your application is important. But why is it important? Why do we need to be able to suddenly, and without notice, scale our application to handle double, triple, or even ten times the load it is currently experiencing?

Why is scaling important?

There are many reasons why our applications must scale. A growing business need is certainly one important reason. But there are other reasons why architecting your application so it can scale is important for your business.

I am going to talk about four basic reasons. Four reasons why scaling is important to the success of your business.

Reason #1. Support your growing business

This is the first, and the most basic reason why your application has to scale. As your business grows, your application needs grow. But there is more to it than that. There are three aspects of a growing business that impact your application and require it to scale.

First, is the most obvious. As you get more customers, your customer’s make more use of your applications and they need more access to your website. This requires more capacity and more growth for the IT infrastructure for your sites.

But that’s not the only aspect.

As your application itself grows and matures, typically you will add more and more features and capabilities to the application. Each new feature and each new capability means customers will make more use of your application. As each customer uses more of your application, the application itself has to scale. Simply by your business maturing over time, even if the size of your customer base doesn’t grow, the computation needs for your application grow and your application must scale.

And finally, as your business grows and matures, and your application grows and matures, your more complex application will require more engineers to work on the application simultaneously, and they will work on more complex components. Your application might be rearchitected to be service based. It might add additional external dependencies and provisions. You will have to support more deployments and more updates. Your application and your application infrastructure will need to scale to support larger development teams and larger projects.

This means you need more mature processes and procedures to scale the speed at which your larger team can improve your application.

Reason #2. Handle surprise situations

The second reason you need to be able to scale your application is to handle surprise situations and conditions. All businesses have their biggest days. These are the days where traffic is at the heaviest. These are days like Black Friday in retail, or the day of the Super Bowl for companies that advertise during that event, or open enrollment periods, or start of travel season.

But your business may have unexpected business bumps. These are the traffic increases that occur not because of a known big event, but because of an unknown or unexpected event. When an event occurs that is favorable to your business, you need to be ready to handle the increased load that occurs to support the event. If you cannot handle the increased load, you risk losing the new business, and you risk disappointing your existing customers.

Sudden business success can kill you if you can’t scale to meet the need.

Just ask Robinhood Financial.

Robinhood Financial is an investment company that provides investment management services. On Monday, March 2nd, 2020, Robinhood faced a business crisis. They faced a sudden increase in business.

On that day, the United States stock market had a record-breaking day. This record-breaking day resulted in a record number of account signups and customer market transactions. This is good news for a company such as Robinhood.

The problem was that their traffic load was not only high, it was too high.

They needed to be able to respond to a huge spike in traffic to their application. Unfortunately, they were unable to keep up with the sudden demand.

The result was a failure of their systems…and their application.

The Robinhood Financial site was down…for a day and a half. This was during a peak stock market time, a time when their customer’s needed them the most.

As a result, they lost out on a huge amount of easy, new business; and they created hardship and disappointment for many of their existing customers. Potential new customers and existing customers alike, were disappointed. A potential opportunity for huge growth and huge upside for the company…instead turned into a major negative event for the company. An event their founders had to publicly apologize for.

All because they couldn’t scale to handle the surprise traffic load.

To be successful, companies must be able to scale to meet sudden and unexpected traffic demands.

Reason #3. Handle a partial outage

The third reason is a sneaky one. You need to be able scale in order to handle partial application outages. Partial outages can be a big problem for businesses. You have a large application, distributed across the globe in multiple data centers.or availability zones, if you are operating in the cloud.

You spread your application out like this for improved redundancy, availability, accessibility, and resiliency.

But now, one of your data centers goes down. Of course, since you are operating in more data centers, a single data center outage is far more likely. This means more chances for something to go wrong in any one of them.

But when a data center goes down, the traffic that would normally be sent to that data center has to be re-routed to other data centers. This results in a big uptick in traffic to those other data centers.

Can those data centers handle the increased traffic? If not, those data centers could go down as well. The result is your application can fail and become unavailable, due to excessive traffic.

This seems counter intuitive, but your plan to increase availability, just made your application less available.

Your plan for improved redundancy by increasing the number of data centers, actually made your application more fragile.

By increasing the number of data centers you were using, you increased the risk of a data center failure. And your application isn’t able to scale to handle the increased traffic needs of a data center failure. The result is an application melt down. A step to improve availability makes availability worse.

Can your other data centers accept the sudden challenge of handling the additional traffic that is sent to them from a failed data center? Can you respond to this sudden need for scale? You must, or your application is at risk.

Reason #4. Maintain availability

The fourth reason is to maintain availability. As your application gets more complex, it requires more interactions between many different components to work correctly.

If one of those components begins to act sluggishly, it can cause performance issues in downstream services. These downstream performance issues can become worse, and more critical problems can occur, such as transaction timeouts, corruption, data loss, and ultimately, upset customers.

A single service, slowing down for some simple reason, can cascade into a larger problem. And if your application can’t scale, the likelihood of individual components saturating and slowing down becomes a matter of when it will happen, not if if will happen.

Lack of scalability turns into lack of availability.

Lack of availability turns into failed customer expectations.

Failed customer expectations turns into a negative impact on your business.

Scaling is Critical

Scaling is critical to your business success. Whether your business is growing or not, you need to be able to handle the growing and spiky traffic needs of your customers…at anytime…or risk application failure, upset customers, and a business failure.

Scaling isn’t just important, it is a business necessity.

Tech Tapas — Dynamic Cloud

There are two ways that people utilize the cloud...the first is by taking an application that is designed to run anywhere, and run it in an infrastructure that was created in the cloud. This is typically called the static cloud, because you create resources, such as servers, that are long lived and use them to operate the application. The resource usage typically does not change much — or at all — over the long term as the application runs.

The other way is to only allocate the resources you absolutely need, when you need them. Given that it is very easy to allocate and free resources in a cloud — especially a public cloud — it’s relatively easy to build an application that allocates the resources it requires when it requires them, and frees the resources when they are no longer required.

This is called the dynamic cloud.

The dynamic cloud is where the true power of cloud computing exists, and where the true benefits of using the cloud can be unlocked. The ability to only consume the resources you absolutely require at the moment, coupled with the ability to quickly allocate the additional resources you require as your application needs increase — gives you incredible capabilities in building highly scalable applications that can meet your needs no matter the amount of traffic sent to them, yet conserve money when traffic is low.

When you perform a lift-n-shift migration of an application to the cloud, you typically move the application from a static data center to operating in a static cloud. You typically do not take advantage of the dynamic capabilities of cloud computing. Too often such application migrations end up being disappointments, because the application does not run any better in the cloud than it did in your own data center, yet the cloud resources may end up actually costing more money when used statically than equivalent resources in a static data center.

The only way to truly see the advantage of using cloud computing is to utilize the dynamic cloud to build dynamic applications. Then you only consume — and pay for — the resources you require at the time you require them, yet you can increase the resources available to your application very quickly to handle sudden increases in traffic.

Whether you are doing dynamic auto scaling, or using dynamic services such as Amazon DynamoDB, Google Big Table, AWS Lambda, or Azure Functions, using the cloud in a dynamic fashion — using the dynamic cloud — is the key to effectively utilizing the cloud to improve, and hence modernize, your application.

Ken Gavranovic was the Executive Vice President and GM for product at New Relic. In early 2019, Ken and I were in Boston together for an event, and we recorded an interview discussion about Risk Management in modern digital applications.

Both Ken and I have experience dealing with Risk Management issues in current and past assignments. I discuss Risk Management in my book, Architecting for Scale. Ken used a very similar risk management technique in his past corporate management gigs. In this interview, we compare notes and make recommendations on best practices for Risk Management that everyone can use.

Links and More Information

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)
• Risk Management with Ken Gavranovic Video (https://leeatchison.com/2019/02/06/managing-risk-in-modern-enterprise-applications/)
• Ken Gavranovic Twitter (https://twitter.com/kgavranovic)
• Ken Gavranovic LinkedIn (https://www.linkedin.com/in/gavranovic/)

Risk Management Interview

Ken: I know we both talk to a lot of customers. One of the questions is, where do I get started? What are some of the patterns we see in enterprises and our own experiences? We have an awesome opportunity to talk to a lot of companies doing digital transformation, but what is something that I can just go do tomorrow to get started?

Lee: One of the things I find it’s very easy to wrap your mind around is risk management. How do you build a risk matrix to track the issues and the risks you have within your system? I like to talk to companies about that because it gets people starting to think about what their system is doing, what problems they have, and how they deal with them. It gets them thinking beyond just the problem/resolution cycle, and more into a pro/con and risk assessment process. What is the benefit of fixing something versus the benefit of mitigating it versus the benefit of simply ignoring it? I like to talk about that because it gets conversations going within the company about the sorts of things that are important to them.

Creating a risk matrix is an important first step for anyone who is thinking about trying to improve their availability, trying to improve their scalability, or trying to modernize their application in many different ways. It helps get a grip on the issues that already exist in your system and what you are currently doing to manage those risks.

Ken: I 100% agree. I remember in a previous role, I had a couple hundred-million-dollar project, I had some challenges. We created a risk matrix which helped us solve those challenges. So I thought it might be helpful for people watching this video. Let’s double click and see what this might look like.

From my perspective, I think the key questions that need to be asked, those questions need to be asked in a bottoms-up way, not top down. Agreed?

Lee: Yes, definitely.

Ken: It’s not people at the top of the organization that are giving you the answers. It’s the team level that gives you the answers you need. Let me give you my shot and tell me where I miss.

First of all, the things that can go into the risk are the things that can go bump in the night.

Lee: Most people already have an idea of the things that keep them up at night. Things they think about, worry about. The things they think about on a regular basis, and that is a good place to start.

Ken: That makes sense. So, bottom up, by team, just create a list. Just list all the things that we think are some sort of risk to the project. These are things you know you should be resolving, but instead you have a habit of prioritizing feature development work over it instead.

Next, is to think about the likelihood that this risk will actually happen.

Lee: I tell people they need to think about two values for every risk item they come up with. Create a spreadsheet, and list all the risks as rows in the spreadsheet. Each individual risk, line by line. Then, for each risk, add two values in separate columns: likelihood and severity. That is, how likely is this risk to happen and if it does happen, how much negative impact will I have with it.

They should do this for every risk in the matrix, before they even begin to think about fixing or mitigation.

Ken: I think it’s important to share that this is what we’ve seen, not just from personal experience, but from a lot of companies that we work with.

Lee: Right.

Ken: What types of values should I use for likelihood and severity? Some people say I should score it from 1 to 10. I think that’s too granular. I like to keep it simple. Just use: Low, Medium, and High.

Lee: I agree with you. You do run into people that want to be highly analytic. They want to use numbers, say, from 1 to 100 and they end up arguing about whether a particular risk is a 35 or a 36. This is way too granular. Keep it simple.

Ken: Sometimes teams like to use their SPRINT approaches of throwing numbers, such as using cards.

Lee: Yeah, if you really want a more rigorous process, you can do something similar to the SPRINT throwing numbers approach, but just use three playing cards, say Ace, Five, and Ten. Then everyone can vote with a card and use that to determine High/Medium/Low.

But that sort of process is only for people that really want a truly analytic solution. It can be done much simpler than that. Often, items are clear to everyone that they are a high or a low or somewhere in between.

Ken: So, whether you use cards, or just use Low/Medium/High, or whatever. At the end of the day, the most important thing is to keep it simple. It’s not about a big debate.

Lee: Exactly.

Ken: At this stage, we are not trying to get into a great level of detail. Just a high-level description, likelihood, severity. Next thing for the matrix is, is this risk currently instrumented? Does it have observability? If this risk were to occur, would you know that it is occurring from a notification from an automated system, or would you find out from your customers telling you?

Lee: That’s a fantastic way to think about it. It’s one thing to know that if something goes wrong, what’s going to happen. It’s another thing to know that you’ll know when it happens.

Ken:** Agreed.

Lee: And, certainly when we talk later about mitigation, you absolutely need to know that knowing when a risk is occurring is a critical aspect of risk management. This is especially true for your high severity risks, whether or not they are high likelihood or low likelihood.

Ken: Kicking starting a program like this in an enterprise is obviously hard. You need top-down leadership to support this process that we are going to do.

Lee: Yes.

Ken: Risk matrix, containing lines with items, likelihood, severity, monitored or not monitored. Ok, what else, or is it just that simple?

Lee: Well, coming up with that list is going to get you 80% of the way to what you need. That’s because it gets you and your organization thinking about what’s going on. That’s the most important benefit of this process. You start thinking about risk and the impact risk has on your system. What’s going to happen during this risk discovery process is the engineers in the room, their minds are quickly going to go to the next thing, which is mitigation. They are going to start to think about how to handle the risk.

But, you are right. If you get nothing done but create that list of risks and put them in the matrix during the first meeting or two, that’s all you need and your world will be a whole lot better, just by simply having that matrix.

Ken: Right. Another point I want to throw out there and see if you agree, is around RCA and incident response processes. I think when you have an incident, during the RCA you should always check if this issue was already on the risk matrix. If it was not there, then it should be added, and some time should be spent on why it wasn’t added in the first place. Maybe a team wasn’t as aggressive, and they didn’t want to put everything in the matrix. Because, going back to no surprises, you want to understand why this incident was a surprise. One of my favorite phrases is, “surprises, not a fan of giving or receiving”. If you have a risk matrix and it’s done right, anything that goes bump in the night should have been known and on the risk matrix ahead of time.

Lee: Exactly, you know, every time you have an outage or an incident of any sort, you end up with some sort of post mortem whether it’s formalized or not. One of the key questions has to be, “did you know about this ahead of time?”, and that comes back to the risk matrix. Because, if you didn’t know about it, that’s a problem. It needs to be added to the risk matrix, so you understand that risk fully. But if you did know about it, you should also verify that the actual severity of the incident matches the severity you thought it should be on the risk matrix. Were you right or wrong in your estimates? You can gain a lot of knowledge when an incident occurs by answering questions like this with the help of a risk matrix.

Ken: So, let’s assume that as a leader, I’ve told my organization to build a risk matrix. They’ve done the process, I now have this risk matrix. From an execution point of view, I think there are two things that need to happen next.

First, you look at the high/highs – high likelihood, high severity. In some cases, removing these risks might involve rewriting. But the high/highs that you can fix, you should prioritize the work and get them fixed.

Second, you always have business partners. I’m a big believer that you should take that risk matrix and present it, at the executive level, to your business partners. You show the high/highs, the medium/mediums, or whatever they are. Now, as a company, think about one of two things. Should we focus on fixing these high/highs or should we all take a breath and say we are willing, for whatever reason, to take this risk on as a company. You go into that with open eyes, blameless culture, and state your willingness to take that risk together.

Lee: Yes, and that’s really critical too. Because no matter what, you are not going to remove all the risk from the system. You aren’t going to fix all the problems, nor is trying to do that necessarily the right investment for you. The right level of risk is whatever level your organization – your extended organization – is comfortable with. The business cost of the risk, the development cost of fixing it, all of these things have to fit together. But once you know what your risk is, you can evaluate whether you and the culture of your company, and your customers, and the business you provide, are comfortable with that level of risk.

Now, for the things that you are not comfortable with, you have to address these right away. You have to either mitigate these risks or remove them. But the other risks, the ones where you are comfortable with the level of risk, it’s not necessarily a good investment to work on resolving those things. Because there are going to be higher priority issues you want to work on.

Ken: Another important aspect is from the funding perspective. I look at the risk matrix as a living document. My thoughts are, you should run this exercise at least twice a year. Then, when you have incidents, you should update the risk matrix to match those incidents. The risk matrix should be accurate and maintained.

Lee: Absolutely.

Ken: Now, most companies fund on an annual basis. My perspective is a lot of times people forget about risk when it comes to funding. In some companies, what is funded are the “bright shiny objects”. That’s where the money is invested. So, for companies that are technology leaders, you should bring the risk matrix to the budgeting discussions. That way you can make sure everybody is clear and all discussions are open on what we are investing in and why we are investing in it. The risk matrix is part of the budgeting process.

Lee: Yes, it’s definitely a feedback into your budgeting process. But it’s also at a much lower level a feedback back into your SPRINT planning process.

Ken: Totally agree.

Lee: You use it to determine what you can accomplish this SPRINT, and how much you want to spend on doing risk management activity during this specific SPRINT versus doing new features or dealing with other problems.

Ken: I know many enterprises, if they are really focused on the customer experience say High/Highs must be done first, unless it involves a full rewrite. If you go into an organization that has a lot of technical debt, that may not be the case, you do as many as you can each SPRINT.

Lee: Yes, absolutely. But the one important thing to consider is that fixing does not have to mean removing it. It might be creating a mitigation for it that reduces its severity or likelihood to an acceptable level.

Ken: It might move from a High likelihood to a medium or might take it from a High impact to a medium or low.

Lee: And just by doing that you’ve brought it to down to within the comfort level of your organization. And once it’s in the comfort level of the organization, that’s a very successful place to be.

Ken: And you and I have seen this at hundreds of global companies. So, from a best practices’ standpoint, it really makes a lot of sense. Have a risk matrix, update it semi-annually and when incidents happen. Review it during the RCA process. Rinse and repeat. Then, take what you have and use that in the budgeting process.

Anything else we should add?

Lee: The only additional thing is that individual teams need to own their own risk matrices – remember, they are built bottom up. Individual teams need to have responsibility for their own risk matrices and be held accountable for the content. Then, they all need to bubble up to a high-level list that is known at the highest levels of the organization.

Ken: I agree. And the initiative and guidance to do it needs to come top-down, because it’s important to the entire organization.

Lee: Yes.

Ken: The actual work itself happens bottoms-up. Totally agree.

I’d like to thank Ken for his involvement in this interview and for the insights he provided to the very important topic of Risk Management. Additionally, thank you to Ken for providing the recording equipment he and I used for the interview.

Tech Tapas — History of the term Cloud

What is the history of the term “cloud” as it is used in Cloud Computing?

Well, that’s an interesting question and unfortunately there are probably as many answers as there are people who work in the cloud.

So, it’s very hard to answer.

But most people who have looked into this subject believe the term originally came in the 1980’s from the telephone companies of the day. Network engineers who drew diagrams of portions of their networks would often draw a big blob to indicate a portion of their network that they weren’t dealing with at the time. Rather than drawing these blobs as simple circles, squares, or rectangles, since those icons represented real entities in their diagrams, instead they drew the blob using rounded segments interconnected that made it look “nebulous” and nondescript in shape…it was, after all, suppose to represent a nebulous and nondescript part of the network.

In fact, this nebulous and nondescript shape looked like a cloud. So much so that these engineers started talking about the part of the network that they weren’t focusing on at the time — the part external to their area of concern — as being out in the cloud.

This usage expanded into software architects as they build their software diagrams and flow charts as well. They used the symbol for a similar purpose.

The term cloud computing, though, is much more recent. Some would argue that early server farms were really cloud computing, but the term really wasn’t popularly used back then. Some would say that Google Compute Platform (GCP) provided some of the earliest cloud computing technology to the industry. Others would say that Salesforce.com was a major early creator of cloud technology. But I think the real mainstream usage of the term cloud computing among technical professionals was popularized with the start of Amazon Web Services. AWS mainstreamed cloud computing — and hence mainstreamed the term cloud computing. This happened in the mid 2000’s.

But software running in the cloud was still something that was reserved for technical people to talk about. Mainstream non-techy people didn’t yet know what the term cloud was all about. This was certainly my personal experience. I was working at AWS in the early days, and I had a hard time telling my non-techy friends what I did for a living. They just didn’t know what I meant by “cloud computing”. They didn’t know what I meant when I said I worked “in the cloud”. They didn’t understand what the cloud was.

Like many of the changes in modern popular tech culture, that changed once Apple computer came into the picture. On October 12, 2011, Apple introduced iCloud to the Apple universe, and overnight, the word “cloud” was a part of mainstream culture. Those friends of mine who couldn’t understand what I was doing when I said I “worked in the cloud”, now understood, at least at some level, what the cloud was all about. Apple brought the term cloud to the mainstream.

I know I will likely get many people who disagree with my analysis on who invented the term cloud and who popularized its use. That’s because there really isn’t a single right answer. But I do believe that the biggest events in the history of the term “cloud computing”, were — first the network engineers, then AWS, then Apple computer. Each of those three groups played a role in bringing the world “cloud” into our everyday lives.

Modern applications require high availability. Our customers expect it, our customers demand it. But building a modern scalable application that has high availability is not easy and does not happen automatically. Problems happen. And when problems happen, availability suffers. Sometimes availability problems come from the simplest of places, but sometimes they can be highly complex.

In this episode, we will continue our discussion from last week with the remainder of the five strategies for keeping your modern application, highly available as well.

This is How to Improve Application Availability, on Modern Digital Applications.

Links and More Information

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)
• Robinhood Announcement (https://blog.robinhood.com/news/2020/3/3/an-update-from-robinhoods-founders)

How to Improve Availability, Part 2

Building a scalable application that has high availability is not easy and does not come automatically. Problems can crop up in unexpected ways that can cause your application to stop working for some or all of your customers. 

No one can anticipate where problems will come from, and no amount of testing will find all issues. Many of these are systemic problems, not merely code problems.

To find these availability problems, we need to step back and take a systemic look at our applications and how they works.

What follows are five things you can and should focus on when building a system to make sure that, as its use scales upwards, availability remains high. In part 1 of this series, we discussed two of these focuses. The first was building with failure in mind. The second was always think about scaling. In part 2 of this series, we conclude with the remaining three focuses.

Number 3 - Mitigate risk

Keeping a system highly available requires removing risk from the system. When a system fails, often the cause of the failure could have been identified as a risk before the failure actually occurred. Identifying risk is a key method of increasing availability.

All systems have risk in them. There is risk that: 

• A server will crash
• A database will become corrupted
• A returned answer will be incorrect
• A network connection will fail
• A newly deployed piece of software will fail

Keeping a system available requires removing risk. But as systems become more and more complicated, this becomes less and less possible. Keeping a large system available is more about managing what your risk is, how much risk is acceptable, and what you can do to mitigate that risk.

This is Risk management, and it is at the heart of building highly available systems. 

Part of risk management is risk mitigation. Risk mitigation is knowing what to do when a problem occurs in order to reduce the impact of the problem as much as possible. Mitigation is about making sure your application works as best and as completely as possible, even when services and resources fail. Risk mitigation requires thinking about the things that can go wrong, and putting a plan together now, to be able to handle the situation when it does happen.

For example, consider a typical online e-commerce store. Being able to search for product on the e-commerce store is critical to almost any online store. But what happens if search breaks?

To prepare for this, you need to have “Failed Search Engine” listed as a risk in your application risk plan. And in that risk, you need to specify a mitigation plan to execute if that risk ever triggers.

For example, we might know from history that 60 percent of people who search our site end up looking at and buying our famous red striped shirt. So, if our search service stops functioning, rather than simply failing, we could display an appropriate “I’m Sorry” page, followed by a list of our most popular T-Shirts, including our red striped shirts. For some number of customers, this would be a success. For the rest, it might create alternatives for them other than simply leaving in frustration. Combine this I’m sorry page with showing a coupon for 10% off their next visit, and you’ve turned a bad customer experience into an experience that might just create some return customers.

This is an example of a risk mitigation plan. It’s a plan that you build in advance of a potential but serious problem, and be able to implement it if that problem occurs. This is a great example of risk mitigation.

Other risk mitigation plans might be entirely technical. They might involve failover servers, or rapid response plans to resolve an issue. Whatever the plan is, risk mitigation is the process of creating and putting these plans into place.

Number 4 - Monitor availability

You can’t know if there is a problem in your application unless you can see the problem. Make sure your application is properly instrumented so that you can see how the application is performing.

Proper monitoring depends on the specifics of your application and needs, but usually entails some of the following capabilities some or all of the following:

• Server monitoring. Monitoring the health of the server infrastructure that your application is running on. This might be physical resources, or cloud-based virtual resources.
• Configuration change monitoring. Understanding when and how your system infrastructure changes and how those changes impact the operation of your application.
• Application performance monitoring. Look inside your application and services to make sure they are operating the way you expect them to operate.
• Synthetic testing. Monitor how your application works from an external perspective in order to catch problems as customers may see them before they actually see them.

Monitoring involves improving two aspects of modern application operation. MTTD and MTTR. That’s mean time to detection and mean time to resolution. Looking at key performance indicators for changes in patterns and alerting you of those changes will improve your mean time to detection. Giving you a wealth of data that can be used to diagnose the source of a problem will improve your mean time to resolution. Both of these are important measures for improving application availability.

Number 5 - Respond to issues in a predictable and well defined manner

Monitoring systems are useless unless you are prepared to act on the issues that arise. This means being alerted when problems occur so that you can take action. Additionally, you should establish processes and procedures that your team can follow to help diagnose issues and easily fix common failure scenarios. 

For example, if a service becomes unresponsive, you might have a set of remedies to try to make the service responsive. This might include tasks such as running a test to help diagnose where the problem is, restarting a daemon that is known to cause the service to become unresponsive, or rebooting a server if all else fails. Having standard processes in place for handling common failure scenarios will decrease the amount of time your system is unavailable. It will help with improving Mean Time to Resolution.

Additionally, they can provide useful followup diagnosis information to your engineering teams to help them deduce the root cause of common ailments, in order to reduce the likelihood of a reoccurrence of a problem, or the occurrence of a similar problem.

When an alert is triggered indicating that a service is or might be failing, the owners of that service must of course be alerted so they can deal with the issue in a timely manner.

However, other teams that are closely connected to the problem service may also want to be alerted. If you own a service that depends on the failing service, you might want to be informed of the problem even before it impacts your service, so that you could take preventative measures or institute actions from your risk management plan before they become critical. Additionally, if the failing service is a consumer of your service, you may want to be aware of the fact that traffic patterns from the failing service may change as the problem occurs and is being resolved. You may want to keep a close eye on your service to make sure any changes do not negatively impact you.

Documented processes and operations are an essential part of this process. Support artifacts should be well documented and available to all parties that require them. They should also be frequently reviewed and updated, and updating support artifacts should be a regular part of your process for adding new features and capabilities.

These processes and procedures are especially useful, after all, because outages often occur during inconvenient times — times such as the middle of the night or on weekends—times when your on-call team might not perform at peak mental efficiency.

These recommendations will assist your team in making smarter and safer moves toward restoring your system to operational status. 

Summary

No one can anticipate where and when availability issues will occur. But you can assume that they will occur, especially as your system scales to larger customer demands and more complex applications.

Preparation and planning are critical to improving availability and maintaining availability as your application scales.

That’s it for the five focuses to help improve your modern application availability. More information on these five focuses can be found in my book, Architecting for Scale, published by O’Reilly Media. A link can be found in the shownotes.

Tech Tapas — Can’t Scale? Time to go out of business

Why is it essential that your application scale? Well, why not ask Robinhood Financial. Robinhood is an investment company that provides investment management services for its tech-savvy clientele.

Robinhood learned the hard way the cost of success.

On Monday, March 2nd, 2020, the United States stock market had a record-breaking day. After previous significant drops due to virus pandemic scares, good news caused the stock market to rally. The result was record-breaking traffic in the stock market.

For a young investment company, like Robinhood Financial, this would typically be considered great news! The company thrives on new account signups and on customer market transactions. Both of these were available in record numbers to Robinhood on this day.

The problem? Their traffic was *too* high.

You see, companies like Robinhood need to be able to respond to variable loads, as spikes in traffic occur all the time.

Still, this record-breaking traffic spike and the volatile market conditions that went with it were more than Robinhood’s infrastructure could handle. The result? There systems started to fail. This failure created a “thundering herd” effect, as Robinhood founders described it, leading to a failure of their DNS system.

The result was that Robinhood systems were down for approximately one and a half days.

One and a half days.

This one and a half day outage occurred during a peak stock market time, a time when their customer’s needed them most.

During their most critical time, their systems were unavailable.

That’s often the problem, scaling related outages more often than not occur during good times, not the bad times. This can very quickly turn a significant business opportunity for success...into an utter failure.

Scaling and availability problems can take the moment of greatness you’ve been working towards all your life, and turn it into an event that can shutter your business.

This isn’t true just for Robinhood — all modern companies face problems like this. One of the most challenging things for an online business to handle...is success.

Success can be the killer of a business.

Success can shut you down.

Avoiding problems like this is why it is essential that you consider scaling and availability needs of your application well before your needs arise. The day when success is staring you in the face is too late to be planning for scaling in your system architecture.

In the shownotes, I have a link to the announcement of the outage by Robinhood in their blog, along with links to other useful information.

There are many resources you can use to help you build in scalability and availability into your business processes and your applications. My book, Architecting for Scale, published by O’Reilly Media, has lots of useful high level information on the systems and processes involved in building highly scaled, highly available applications. You can also listen to this podcast. I talk often about scaling and availability topics as it relates to building and operating your modern digital applications.

Modern applications require high availability. Our customers expect it, our customers demand it. But building a modern scalable application that has high availability is not easy and does not happen automatically. Problems happen. And when problems happen, availability suffers. Sometimes availability problems come from the simplest of places, but sometimes they can be highly complex.

In this episode, we will discuss five strategies for keeping your modern application, highly available as well.

This is How to Improve Application Availability, on Modern Digital Applications.

Links and More Information

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)

How to Improve Availability, Part 1

Building a scalable application that has high availability is not easy and does not come automatically. Problems can crop up in unexpected ways that can cause your application to stop working for some or all of your customers. 

These availability problems often arise from the areas you least expect, and some of the most serious availability problems can originate from extremely simple sources.

Let’s take a simple example from a real world application that I’ve worked on in the past. This problem really happened.

The software was a SaaS application. Customer’s could login to the application and they received a customized experience for their personal use. One of the ways that the customer could tell they were logged in is that an avatar of themselves appeared in the top right hand corner. It wasn’t a big deal, but it was a handy indicator that you were receiving a personalized environment. We’ve all seen this sort of thing, it’s pretty common in online software applications now-a-days.

Anyway, by default, when we showed the page, we read the avatar from a 3rd party avatar service that told us what avatar to display for the current user. One day, that third party system failed. Our application, which made the poor assumption that the avatar service would always be working, also failed. Simply because we were unable to display a picture of the user in the upper right hand corner, our entire application crashed and nobody could use it. It was, of course, a major problem for us. It was harder too because the avatar service was out of our control. Our business was directly tied to a 3rd party service we had no control over, and we weren’t even aware of the dependency.

A very minor feature crashed our entire business…Our business crashed because of an icon.

Obviously, that was unacceptable.

How could we have avoided this problem? There were a thousand solutions to the problem. By far the easiest would have been to notice and catch any failure of the 3rd party service in realtime, and if it did fail, show some default generic avatar instead. There was no need to bring down our entire application over this simple problem. A simple check, some error recovery logic, some fallback options, that’s all it would have taken to avoid crashing our entire business.

No one can anticipate where problems will come from, and no amount of testing will find all issues. Many of these are systemic problems, not merely code problems.

To find these availability problems, we need to step back and take a systemic look at our applications and how they works.

What follows are five things you can and should focus on when building a system to make sure that, as its use scales upwards, availability remains high.

Number 1 - Build with Failure in Mind

As Werner Vogels, CTO of Amazon, says:

“Everything fails all the time.”

You should plan on your applications and services failing.

It will happen.

Now, deal with it.

Assuming your application will fail, how will it fail? As you build your system, you need to consider availability during all aspects of your architecture, design, construction, and testing.

What design constructs and patterns have you considered or are you using that will help improve the availability of your software?

Are you using simple but effective methods for detecting problems with other services? This can be simply catching errors. It might mean appropriate retry logic. But it could also involve things like circuit breaker patterns in order to validate external systems are behaving as they should.

Circuit breaker patterns are particularly useful for handling dependency failures because they can reduce the impact a dependency failure has on your system as a whole, without having to continuously rediscover the problem and impact performance.

Once you detect a failure, what can you do? Is this a hard dependency that you cannot recover from if it fails? Or can you replace the call with a temporary result that may degrade the experience but allow other functions to continue? This is the case with the avatar service, you can show a default avatar as a degraded experience, yet still the main part of your service can continue operating.

Beside taking care of failures of systems that you depend on, you need to consider failures of systems that depend on you. What happens if a service that is calling your system behaves poorly? Can you handle excessive load from your consumers? Can you handle bad incoming requests and provide graceful responses?

Sometimes, denial-of-service attacks can come from “friendly” sources. A service calling you could have a bug that causes it to go into a tight infinite loop, calling you at an unacceptably high rate. Can you detect this and throttle traffic appropriately in a way that doesn’t impact valid requests that might also be coming at a high rate?

Number 2 - Always think about scaling

Speaking about scaling…

Always be thinking about scaling during all aspects of your system architecture, design, and construction. Thinking about scaling should be ingrained into your culture and part of every decision you make.

Just because your application works now does not mean it will work tomorrow. Most web applications have increasing traffic patterns. A website that generates a certain amount of traffic today might generate significantly more traffic sooner than you anticipate. As you build your system, don’t build it for today’s traffic; build it for tomorrow’s traffic. Build it for traffic spikes and bulges. Build it for your biggest days.

This might involve things as simple as increasing the size and capacity of your databases. But it might involve rethinking a design to reduce dependency on certain types of resources, or utilize caching and other scale-based-optimization techniques.

Think about what logical limits exist to your scaling. Where are your sensitive and vulnerable spots? What can you do today to reduce your impact on these vulnerable spots?

Can content you are currently generating dynamically, can it be generated statically and cached instead?

Be creative. You might be surprised what scaling can do to your thought process.

One simple example from my early days with Amazon. We were looking at how to scale an early version of the menu bar on the top of the amazon.com home page. That page was dynamically generated on every page load. It “had to be”, because the banner contained a “Hi Dave” personalized message for each customer. It had to be generated dynamically.

Until we discovered an interesting fact. A significantly large portion of our traffic came using only one of a hundred or so possible names. By creating static versions of the banner for each of these high traffic names and caching these, we would only have to dynamically create the banner for pages that were personalized using a different, less common name. The result was that a huge amount of our dynamic traffic could be cached.

Now, I don’t know if they ever implemented that approach or if there were other techniques that ultimately were used for improving scale, that’s all irrelevant now. But the point is that this was a big lesson for me and I’ve always remembered it. It can be surprising what scale can do to help you make things more optimized.

*** High scale is the friend of optimization, not the enemy.

That is worth repeating…

*** High scale is the friend of optimization, not the enemy.

We often think of optimization as a tool to fight the negative impact of increased scaling…negative impacts like resource depletion. But if you only take one lesson out of this discussion, take this. The higher scale an application operates at, the more options available to you to improve system optimization and efficiency. The more you scale, the more you can optimize for scale.

That’s it for part one of this story in this episode. We will continue with the remaining three focuses for improved availability in the next episode.

More information on these five focuses to improve availability can be found in my book, Architecting for Scale, published by O’Reilly Media. A link can be found in the shownotes.

Tech Tapas — What happened to HP cloud?

What happened to HP Cloud? HP Cloud never was at the scale of Amazon Web Services, nor Microsoft Azure, nor Google Cloud. Yet it was a player for awhile in the early cloud days. Like much of HP at the time, HP cloud was focused from the beginning on meeting the needs of enterprise customers.

HP had a great idea — focus on the enterprise — at the time before enterprise was a huge focus for any of the major cloud providers. While AWS was growing focusing on the small-to-medium sized businesses and individuals, HP was focusing on the enterprise.

But by the time HP was able to execute on their enterprise strategy, the big three public cloud providers — Amazon, Microsoft, Google — all had switched their focus to the enterprise customer as well. This left HP out in the cold.

HP is now virtually entirely out of the public cloud space. In fact, in 2015, HP announced it was shutting down its Helton public cloud offering and instead started focusing only on private and hybrid clouds. HP ceded the public cloud market to Amazon, Microsoft, and Google...

HP has been able to make some traction in this space...the private and hybrid cloud space... The reason? Their focus on using the OpenStack technology, a technology that has not shown much traction in the public cloud markets, but has gained significant traction in the private and hybrid cloud markets.

Maybe HP found a groove that will work for them. But in this space, they’ll have to compete with the likes of IBM, which is also focusing on the hybrid cloud space as well...

How long will HP remain a player in the private and hybrid cloud niche? We’ll have to wait to see, but as far as the public cloud is concerned, HP is nowhere to be found. That’s a shame, I would like to see HP — and IBM — both be larger players in the public cloud. We need more and better competition for AWS in the public cloud. But it looks like that competition isn’t going to come from HP.

In this episode, we know that using multiple availability zones helps increase your application availability and resiliency by distributing our application across multiple disperse data centers. But did you know that availability zones don’t necessarily give you the separation you expect? In fact, it is entirely possible to have two instances of a service running in two distinct availability zones, but actually have them running in the same data center, in the same physical rack, and possibly even on the same physical server! How can this be? And even more importantly, how can we avoid it? The answer involves understanding how availability zones work and how they are structured.

And then, one of the oddest cloud services created is also one of the first cloud services. Before AI and before machine learning, humans actually powered a part of the cloud.

This is, Life with Multiple AWS Accounts.

Links and More Information

The following are links mentioned in this episode, and links to related information:

• How to maintain availability when using multiple AWS accounts (https://www.infoworld.com/article/3444860/5-pain-points-of-modern-software-development-and-how-to-overcome-them.html)
• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)

Distributing Your Application

When building a modern, high-performant application at scale, it’s important to make sure the individual application instances are distributed across a variety of data centers in such a way that if any given data center goes offline, the application can continue to function relatively normally. This is an industry-wide best practice, and an important characteristic to architect into your applications in order to make them sufficiently resilient to data center problems.

The same philosophy occurs when you build your application in the cloud. Except, when you build a cloud-based application, you typically do not have visibility into which data center a particular server or cloud resource is located. This is part of the abstraction that gives the cloud its value.

Not having visibility into which data centers your application is operating in makes it difficult to build multi data center resiliency into your applications. 

To solve this problem, AWS created a cloud abstraction of the data center that allows you to build on this level of resiliency without being exposed to the details of data center location. The abstraction is the availability zone.

AWS availability zones

An AWS availability zone is an isolated set of cloud resources that allows specifying a certain level of isolation into your applications. Resources within a single availability zone may be physically or virtually near each other, to the extent that they can be dependent on each other and share subcomponents with each other. For example, two EC2 servers that are in the same availability zone may be in the same data center, in the same rack, or even on the same physical server.

However, cloud resources that are in different availability zones are guaranteed to be separated into distinct data centers. They cannot be in the same data center, they cannot be in the same rack, and they cannot be using the same physical servers. They are distinct and independent from each other.

Hence, the solution to the resiliency problem, you can build your application to live in multiple availability zones. If you construct your application so instances of your application are distributed across multiple availability zones, you can isolate yourself from hardware failures such as server failures, rack failures, and even entire data center failures. Using multiple availability zones allows you to build in application resiliency.

Availability zones as data centers

Loosely, availability zones can be thought of as data centers. At a first level of approximation, this is a roughly reasonable assumption to make. However, there is not a one-to-one mapping of availability zones to data centers. When you create your AWS account, your availability zone names are mapped to individual data centers in a dynamic fashion. This means one AWS account may have an availability zone named us-east-1a mapped to data center #4, and another account may have the same availability zone mapped to data center #2. Even worse, a given data center may map to different availability zones in different accounts. For example, data center #4 in account #1 may be used for availability zone us-east-1a, but the same data center in account #2 may be used for availability zone us-east-1b.

You can find out how your availability zones are mapped to specific data centers in a given account by looking in the AWS console in the Resource Access Manager (RAM). This shows a mapping of availability zone names to an AZ ID. An AZ ID is a unique identifier that can be effectively used as a data center identifier. It shows the mapping, for your current account, of each availability zone to its associated data center’s AZ ID. This mapping is shown for your currently selected region, but you can simply switch regions to show the mapping for any region in your account.

So, when you are using multiple accounts, don’t use availability zone to determine data center redundancy. Instead, you should use the AZ ID. If two availability zones in different accounts have different AZ IDs, you can be sure that those two availability zones are in distinct data centers. Using the AZ ID, rather than the availability zone name, is a safe way to ensure your applications live in distinct data centers across multiple accounts.

Tech Tapas — Mechanical Turk

One of the oddest cloud services created was also one of the first cloud services. In 2005, Amazon’s Jeff Bezos had an idea and created the Mechanical Turk service. Mechanical Turk, or MTurk, was a service that allowed one to programmatically request execution of micro tasks that could only be performed by a human being. The service allowed software applications to request the execution of a task, and on the back end a human was used to execute the task. In the days before high quality Artificial Intelligence, it was a way of automating many tasks that only a human could perform. One early use for mechanical Turk was to use it to compare product pictures to determine if pictures were of the same product or different products. A single micro task might be to compare one picture to another and respond back yes or no if they represented the same product. This technique was used to improve the quality of the Amazon catalog. Another common task was to examine a picture of a sales receipt and record what information was found on the receipt.

Workers on the back end were paid for each task they completed, typically making on average 10 cents for each micro task they completed.

I remember when I was working at Amazon the day that Jeff presented to the employees the Mechanical Turk service. It was highly innovative. Of course, most of what it could do is now, or soon will be, replaced by machine learning algorithms and Artificial Intelligence. But still, it was a fantastic innovation at the time.

This is a special edition of Modern Digital Applications.

July 9th, 2018 was the launch of a podcast episode. It was an episode of the “Modern Software Podcast”, a podcast sponsored by New Relic, and hosted by New Relic’s Fredric Paul and Tori Wieldt. This particular episode was titled “The Great Serverless Debate”. It was a debate between myself, and a good friend of mine, Clay Smith. Clay and I were guests on the show.

That episode was a huge success, and I still get asked questions about it today. It seemed to me that it was time for an update of that debate…a redux if you will. Since New Relic’s Modern Software Podcast isn’t active right now, I thought I would take on the challenge myself and host a redo of the great debate — based on what we know about serverless in 2020.

So, on February 21, 2020, Clay and I got together for an update to our views on the world of serverless.

This is The Great Serverless Debate, Redux. This is the second part of that interview.

Links and More Information

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)
• The Great Serverless Debate (https://leeatchison.com/2018/07/11/the-great-serverless-debate/)
• Clay Smith - Twitter (https://twitter.com/smithclay)
• Clay Smith - LinkedIn (https://www.linkedin.com/in/smithclay/)
• Clay Smith’s Monitoring Monitoring Newsletter (https://monitoring2.substack.com/)
• Modern Software Podcast - Fredric Paul @TheFreditor (https://twitter.com/TheFreditor)
• Modern Software Podcast - Tori Wieldt @ToriWieldt (https://twitter.com/ToriWieldt)

Lee’s Guest — Clay Smith

Clay Smith is a good friend of mine. He was a senior software engineer at several early-stage startup companies and has been building serverless solutions for many years now, from mobile backends to real-time APIs with Amazon Web Services. Clay was a senior Developer Evangelist at New Relic, which is where Clay and I met.

Clay’s newsletter is “Monitoring Monitoring”. You can subscribe to the newsletter at https://monitoring2.substack.com/.

Questions/Issues Discussed

• Is Lambda living up to the hype?
• Is there an end to the hype anytime soon?
• Has Fargate lived up to the hype?
• What is the role of containers vs FaaS?
• What is the role of Kubernetes?
• What types of problems are suited for FaaS and what kind of problems are not?
• How good was our guesses in 2018 for the state of serverless in 2020?
• What will be the state of serverless in 2022?
• We need a better term for FaaS than the generic term “serverless”
• Use of FaaS as a glue between external services

Quotes

• Lee: “One of the problems with Lambda is…you are not making your service boundaries based on functionality, you are making service boundaries based on the limitation of the technology.”
• Clay: “It’s been a great two years for event driven architectures in general.”
• Clay: “There’s a new class of startups giving a Heroku type experience that look very interesting…”
• Lee: “Lambda is very good…for the hook and glue use case” [system integration] Clay: “API Glue”
• Clay: “We both definitely agree that the glue aspect is still the killer use case…”
• Clay: “Greatest disappointment around FaaS the past two years is the DevOps…workflows. These sorts of solutions are still a total mess.”
• Lee: “When Lambda first came out, that was my number one concern…how do you deal with them operationally. We are still operating the same way we were [then] with the tooling available now. It isn’t yet to the level it needs to be to be a true partner in the operations or DevOps organizations. … No-one has figured out how to do it yet.”
• Lee: “When you move from a monolith to services, you aren’t decreasing the complexity, you are moving it out of the code and into the system of service interactions. With Lambda, there aren’t the tools yet to manage the complexity between interacting Lambda functions reasonable.”

Administrative

Clay appeared as a personal guest, and not a representative of his employer. All views expressed by Clay are his own and not of his employer.

This podcast, “Modern Digital Applications”, is not affiliated with the “Modern Software” podcast. All views expressed in “The Great Serverless Debate, Redux” are those of Clay and Lee personally and not related to the “Modern Software” podcast.

The original “Great Serverless Debate” was created by Fredric Paul on the “Modern Software" podcast.

The second edition of my book, Architecting for Scale, is now available!

• Book Website (https://architectingforscale.com)
• Amazon.com (https://www.amazon.com/Architecting-Scale-Maintain-Availability-Manage/dp/1492057177/)
• O’Reilly (http://shop.oreilly.com/product/0636920274308.do)

Hello everyone and welcome to Modern Digital Applications. This is just a very quick news break...

The second edition of my book, Architecting for Scale, is now available! This edition is dramatically updated and improved. It includes new content on cloud computing, microservices, and serverless computing. All the content has been updated and it has been significantly reorganized.

The book is now available for purchase on amazon.com, other technical bookstores, or directly from O’Reilly Media. It’s also available included as part of your O’Reilly Safari subscription.

Check it out! If you have any questions or comments, feel free to reach out to me. Links to the websites where you can find out more about the book or purchase it are contained in the shownotes for this episode.

Thank you and enjoy!

This is a special edition of Modern Digital Applications.

July 9th, 2018 was the launch of a podcast episode. It was an episode of the “Modern Software Podcast”, a podcast sponsored by New Relic, and hosted by New Relic’s Fredric Paul and Tori Wieldt. This particular episode was titled “The Great Serverless Debate”. It was a debate between myself, and a good friend of mine, Clay Smith. Clay and I were guests on the show.

That episode was a huge success, and I still get asked questions about it today. It seemed to me that it was time for an update of that debate…a redux if you will. Since New Relic’s Modern Software Podcast isn’t active right now, I thought I would take on the challenge myself and host a redo of the great debate — based on what we know about serverless in 2020.

So, on February 21, 2020, Clay and I got together for an update to our views on the world of serverless.

This is The Great Serverless Debate, Redux.

Links and More Information

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)
• The Great Serverless Debate (https://leeatchison.com/2018/07/11/the-great-serverless-debate/)
• Clay Smith - Twitter (https://twitter.com/smithclay)
• Clay Smith - LinkedIn (https://www.linkedin.com/in/smithclay/)
• Clay Smith’s Monitoring Monitoring Newsletter (https://monitoring2.substack.com/)
• Modern Software Podcast - Fredric Paul @TheFreditor (https://twitter.com/TheFreditor)
• Modern Software Podcast - Tori Wieldt @ToriWieldt (https://twitter.com/ToriWieldt)

Lee’s Guest — Clay Smith

Clay Smith is a good friend of mine. He was a senior software engineer at several early-stage startup companies and has been building serverless solutions for many years now, from mobile backends to real-time APIs with Amazon Web Services. Clay was a senior Developer Evangelist at New Relic, which is where Clay and I met.

Clay’s newsletter is “Monitoring Monitoring”. You can subscribe to the newsletter at https://monitoring2.substack.com/.

Questions/Issues Discussed

• Is Lambda living up to the hype?
• Is there an end to the hype anytime soon?
• Has Fargate lived up to the hype?
• What is the role of containers vs FaaS?
• What is the role of Kubernetes?
• What types of problems are suited for FaaS and what kind of problems are not?
• How good was our guesses in 2018 for the state of serverless in 2020?
• What will be the state of serverless in 2022?
• We need a better term for FaaS than the generic term “serverless”
• Use of FaaS as a glue between external services

Quotes

• Lee: “One of the problems with Lambda is…you are not making your service boundaries based on functionality, you are making service boundaries based on the limitation of the technology.”
• Clay: “It’s been a great two years for event driven architectures in general.”
• Clay: “There’s a new class of startups giving a Heroku type experience that look very interesting…”
• Lee: “Lambda is very good…for the hook and glue use case” [system integration] Clay: “API Glue”
• Clay: “We both definitely agree that the glue aspect is still the killer use case…”
• Clay: “Greatest disappointment around FaaS the past two years is the DevOps…workflows. These sorts of solutions are still a total mess.”
• Lee: “When Lambda first came out, that was my number one concern…how do you deal with them operationally. We are still operating the same way we were [then] with the tooling available now. It isn’t yet to the level it needs to be to be a true partner in the operations or DevOps organizations. … No-one has figured out how to do it yet.”
• Lee: “When you move from a monolith to services, you aren’t decreasing the complexity, you are moving it out of the code and into the system of service interactions. With Lambda, there aren’t the tools yet to manage the complexity between interacting Lambda functions reasonable.”

Administrative

Clay appeared as a personal guest, and not a representative of his employer. All views expressed by Clay are his own and not of his employer.

This podcast, “Modern Digital Applications”, is not affiliated with the “Modern Software” podcast. All views expressed in “The Great Serverless Debate, Redux” are those of Clay and Lee personally and not related to the “Modern Software” podcast.

The original “Great Serverless Debate” was created by Fredric Paul on the “Modern Software" podcast.

In this episode, Our customers depend on them, and our businesses depend on them. Without modern web apps, most businesses would not survive. This is the second in a two part series on the principles for modernizing your enterprise web application. 

And then, what is STOSA and how does it help with modern digital applications?

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)
• STOSA — Single Team Oriented Service Architecture (https://stosa.org)

Modern web applications have a lot riding on them. Our customers depend on them, and our business depends on them. Without modern web applications, many businesses would not survive. Modern web applications must scale to meet our biggest needs without suffering outages or other availability issues. In order to meet and exceed this high bar, we must build, manage, and monitor our applications using modern principles, processes, and procedures.

There are five guiding principles that a modern application requires in order to meet these expectations. 

In previous episodes, we discussed principle 1, using service-based architectures; principle 2, organizing teams around services; and principle 3, using DevOps processes and procedures.

In this episode, we will finish the remaining two principles. Starting with…

Customer traffic loads on our applications vary considerably. The upper bound or maximum amount of traffic your application needs to support can never be known accurately.

In the past, as application traffic grew, we simply threw additional hardware at the problem. But as traffic variations increase, and customer usage patterns become more complicated. This simple solution is no longer reasonable.

Simply adding hardware may be fine to handle expected peaks, but what do you do when an unexpected spike in usage arrives? When I was at Amazon, there was a term that was used. It was called the “Johnny Carson Effect”. It’s origin comes from the day that Johnny Carson died. On that day, there was a huge uptick in demand for any videotape of old episode’s of Johnny Carson’s The Tonight Show. Demand was huge, and given the suddenness of the event, it was all unexpected demand. Now-a-days, the term “going viral” is both a positive and a negative for most businesses. Going viral means huge attention given to you and your business. But it also means huge and unexpected traffic volume. The last thing you want is for your application to fail right at the point when everyone is watching.

How can you handle the unexpected surge from such an unexpected event? It is no longer possible to throw large quantities of hardware at an application. You never know how much hardware is enough hardware to handle your possible maximum need.

Additionally, when your application is not operating at a high volume, what do you do with the extra hardware? Typically, it’s sitting idle...which is a huge a waste of resources...and even more importantly...a waste of money. Especially if your traffic needs are highly variable or highly spiky, simply adding hardware to handle your peaks is not an effective use of resources.

Instead, you must add resources to your applications dynamically, as they are needed and when they are needed. These resources can be applied when your traffic needs are high, and they can be released when they are no longer needed. This is what dynamic infrastructures are all about.

The only way to implement a true dynamic infrastructure for an application with a highly variable load is to use the dynamic capabilities of the cloud. In the cloud, resources can be added on demand, and released when they are no longer needed, becoming available for other uses. To be successful, modern applications must make use of dynamic infrastructures to not only optimize their operational costs, but maintain availability during all types of traffic patterns.

I have many articles I’ve written, and I’ve given many talks on dynamic infrastructures and the dynamic cloud. For more information, go to leeatchison.com and click on “Articles”, or click on “Presentations”.

It is impossible to keep a modern application working if you do not have visibility into what your application is doing and how it is performing.

This is a basic truism of modern applications.

Without proper visibility into the operation of your application, you have no idea if the application is meeting the needs of your customers, or of your business.

Without proper visibility, you have no way of knowing if a ticking time bomb is ready to go off and make your application suddenly unavailable.

**With** proper visibility, you can understand what your application is doing, you can understand how it’s doing it, and you can understand if there are any lurking issues that might impact the application’s ability to perform those operations in the foreseeable future. 

Further, while using tools such as synthetic testing and infrastructure performance monitoring is important, this level of monitoring is not sufficient.

You cannot completely understand how your application is performing without an internal view of its operations. This point is worth restating. You cannot understand how your application is performing without an ***internal*** view of its operation.

This is only possible using application performance monitoring, log monitoring, and trace analysis. These three tools are critical to the ability to have proper visibility into the operation of your modern application.

With proper visibility, comes the ability to build and maintain your ever growing, ever more complex, modern applications.

That’s it, that’s the last of the five principles required to modernize your enterprise applications. In summary, the five principles are: Use service-based architectures, organize your teams around your service architecture, use DevOps principles methodologies, use dynamic infrastructures, and maintain high visibility.

Our applications *are* getting more complex, and becoming more intertwined with the fundamental operation of our business. As such, the expectations of our customers...expectations on reliability, functionality, and availability...continue to grow. As we modernize our applications to meet these growing needs, these five principles will you meet the needs of your customers and your business.

More information on these five principles is available in my article published in InfoWorld, “How to modernize enterprise web applications”. A link to this article is available in the show notes.

Single Team Owned Service Architecture ... STOSA.

STOSA is an operational model for how to structure your organization to effectively manage a multi service or micro service architected application. For your teams and organization to be STOSA compliant, they must:

Have an application that is constructed using a service-based architecture or a microservice-based architecture.

Have multiple development teams that are responsible for building and maintaining the application.

Each and every service in your application must be assigned to a development team, who owns that service. The team that owns each service should be well documented and readily available to everyone in the organization.

No service should be assigned to more than one development team, however individual development teams may own more than one service.

Teams are responsible for all aspects of managing the service, from service architecture and design, development, testing, deployment, monitoring, and incident resolution.

Services have strong boundaries between them, including well-documented APIs.

The service owns its own data. Data is part of the service. If a service needs to access data stored in a different service, it must use one of the well-documented APIs to access that data.

Services maintain internal service-level agreements (SLAs) between them that are monitored and violations reported to the owning team.

A STOSA-based application is an application for which all services follow these rules. A STOSA-based organization is one in which all service teams follow the preceding rules and all applications are STOSA applications.

I developed this model based on my experiences building and operating large scaled applications. The model first appears in my book Architecting for Scale, by O’Reilly Media.

You can learn more about the STOSA model by going to stosa.org. That’s s-t-o-s-a-dot-o-r-g.

Links are available in the show notes.

In this episode, Our customers depend on them, and our businesses depend on them. Without modern web apps, most businesses would not survive. I’ll present the first in a two part series on the principles for modernizing your enterprise web application. 

And then, what does any of this have to do with my son and his wage reporting application?

The following are links mentioned in this episode, and links to related information:

• Modern Digital Applications Website (https://mdacast.com)
• Lee Atchison Articles and Presentations (https://leeatchison.com)
• Architecting for Scale, published by O’Reilly Media (https://architectingforscale.com)

Modern web applications have a lot riding on them. Our customers depend on them, and our business depends on them. Without modern web applications, many businesses would not survive. Modern web applications must scale to meet our biggest needs without suffering outages or other availability issues. In order to meet and exceed this high bar, we must build, manage, and monitor our applications using modern principles, processes, and procedures.

There are five guiding principles that a modern application requires in order to meet these expectations. 

Modern applications are large and complex, too complex to be handled as a single entity by a single team. Instead, multiple teams are required to develop, test, support, and operate these applications. This complex task is impossible when the application is a single monolith.

To handle a large complex application, split the application into multiple independent services. Then, you can assign different pieces of the application to different teams, allowing parallel development and operation.

By keeping modules isolated, they can be built, tested, and operated in isolation. Application problems can be more easily correlated and isolated to individual services, making it easier to decide which team should be called on to work on an issue when a problem arises.

Scaling is not just about the amount of traffic an application receives, but about the size and complexity of the application itself. Using service-based architectures makes scaling the application easier and allows larger and more complex applications to be dealt with reasonably.

In future segments, we will talk about the remaining four guiding principles that are required for enterprise applications to modernize. In the next segment, principle #2 is about team organization.

Our applications *are* getting more complex, and becoming more intertwined with the fundamental operation of our business. As such, the expectations of our customers are growing, and the demands for reliability, scalability, and functionality from management are keeping pace. Only by modernizing our applications can we make our applications meet the needs of our customers and our business.

Architecting your application as a service-based application is only part of the answer to modernizing your enterprise systems. Once you have your application split up into services, it is essential that you structure your development teams around those services. It is critical that a single team owns each service.

And when I say “own”, I mean complete ownership...front to back...top to bottom.

This includes development, testing, operation, and support. All aspects of the development and operation of each service should be handled by one and only one team.

There is a model for application management that stresses these ownership values. STOSA, which stands for Single Team Owned Service Architecture, provides guiding principles on team-level ownership, providing clear boundaries between services and promoting clear understanding and expectations between teams. We will talk about STOSA and the operating principles behind the STOSA model in much greater detail in future segments. Until then, you can read more about STOSA at stosa.org.

Now that we have team organization and ownership taken care of, the next principle is to focus on the policies and practices that those teams use to build and operate their services. For modern applications, teams must make use of modern devops processes and procedures to build and maintain their services.

Many organizations claim that they “use devops,” but ultimately they do not operate their organizations in a true devops manner.

True devops techniques require a complete organizational flush of all processes and systems in order to incorporate the principles of devops within the entire organization. You cannot accomplish this transformation by simply “creating a DevOps team,” though many organizations try.

DevOps is a cultural transformation. And only by undergoing this cultural transformation can an organization truly build and operate modern applications at scale. Without DevOps, your application will tend to stale in some spots and the work you have done to separate and “service-ize” your application will be for naught. You will not be able to build and maintain your applications unless you adopt...and truly become...a true DevOps culture within your organization.

There are many good sources of information on DevOps. But, certainly, no DevOps bookshelf is complete without The Phoenix Project. You can find a link to The Phoenix Project in the shownotes.

In upcoming segments, we will talk about the remaining guiding principles that are required for enterprise applications to modernize.

More information on these five principles is available in my article published in InfoWorld, “How to modernize enterprise web applications”. A link to this article is available in the show notes.

My son has an app on his iPhone that he uses to record his income for some of the government benefits he gets. Once a month, he has to run this app to simply enter the amount of money he’s earned that month. He simply enters the number and his social security number into the app, presses submit, and he’s done. That’s it.

The problem is, the application on his iPhone only works between the hours of 8am and 5pm, Eastern Standard Time, Monday through Friday. If you try and use the application any other time of the day or night, the application pops up an error message and says the application is offline right now, and you can’t use it.

How many of you can operate applications like that? How many of you can make it so your customers only use your mobile applications during business hours, and prohibit them from using them at any other time?

I know the answer…none of you. It is simply not possible in today’s modern world. People expect our applications to function, and they expect them to function all the time. Anytime they attempt to use your application, they expect it to work, without exception. 24 hours a day, 7 days a week…always.

That is what’s required of modern applications. There is not acceptable downtime. I sometimes here people say they have “nearly 100% uptime!”. But then when you dig in deeper, you find out that they bring their application down for a “maintenance window” every Sunday evening. How is that 100% uptime? The answer is, it’s not. In the modern world, we have to perform all of our maintenance live, and we have to have our applications up and operational all the time.

In our customer’s minds, there is no excuse for anything other than that.