The Internet Report

Configuration changes played an outsized role 2024 outages. Tune in to hear more about this and other outage trends—and learn how ITOps teams should plan accordingly in the year ahead.

We’ll also share insights from recent incidents at OpenAI and Google Cloud’s Pub/Sub, and dive deeper into a degradation incident that Netflix experienced at the end of 2024.

Read on to learn more, or use the chapters below to jump to the sections that most interest you.

CHAPTERS
00:00 Intro
00:58 Cloud Service Provider (CSP) Outages Continue To Rise 
01:52 Accidental Misconfigurations Trending for Clouds and Apps
07:10 OpenAI Outage
09:55 Google Cloud’s Pub/Sub Disruption
14:47 Lessons From a Netflix Incident
18:57 Recent Outage Trends: By the Numbers
21:01 Get in Touch

———

For additional insights, check out the links below:

- The Internet Report blog: https://www.thousandeyes.com/blog/internet-report-configuration-change-outages?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep5_podcast

- 2024 Outage Trends Solidify; Plus OpenAI & Meta Outages: https://www.thousandeyes.com/blog/internet-report-2024-outage-trends-openai-meta-outages?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep5_podcast

- Netflix Broadcast Disruption: Lessons for Major Live Events: https://www.thousandeyes.com/blog/netflix-disruption-analysis-november-15-2024?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep5_podcast

- And join our upcoming webinar, “Top Outages of 2024, Explained: Lessons in Digital Resilience.” We’ll unpack notable outages and performance degradations of 2024 and share lessons IT Operations teams can take away from these incidents to strengthen their digital resilience: https://www.thousandeyes.com/webinars/na-top-outages-2024-lessons-in-digital-resilience?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep5_podcast 

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on LinkedIn or X at @thousandeyes

With nearly a year of data available, the topline outage trends for 2024 are coming into focus. Tune in to see what the numbers are showing.

The Internet Report team will discuss how Internet service provider (ISP) outage numbers are continuing to increase, while cloud service provider (CSP) outages are also becoming more frequent, indicating a changing landscape in service reliability. They’ll also unpack the recent OpenAI and Meta outages.

———

CHAPTERS:
00:00 Intro
00:49 Outage Trends Across 2024
07:37 OpenAI Outage
13:10 Meta Outage
18:48 Get in Touch 

———

For additional insights, check out The Internet Report's latest blog: https://www.thousandeyes.com/blog/internet-report-2024-outage-trends-openai-meta-outages?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep4_podcast

And join our upcoming webinar, “Top Outages of 2024, Explained: Lessons in Digital Resilience.” We’ll unpack notable outages and performance degradations of 2024 and share lessons IT Operations teams can take away from these incidents to strengthen their digital resilience: https://www.thousandeyes.com/webinars/na-top-outages-2024-lessons-in-digital-resilience?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep4_podcast 

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on LinkedIn and X at @thousandeyes

The past few weeks are somewhat of a representative sample of 2024 from an outage perspective, with connectivity issues and updates at the root of the four recent incidents.

Both DigitalOcean and real-time payments provider Worldline experienced connectivity issues to data centers that made services unreachable. Meanwhile, Microsoft and Reddit encountered problems following changes to their systems that appeared to have unexpected user impacts and had to be rolled back. 

Tune in to hear The Internet Report team unpack these incidents and discuss the latest outage trends.

———

CHAPTERS:
00:00 Intro
00:50 Reddit Server Errors
04:33 Issues for Microsoft 365
09:42 DigitalOcean’s Network Issues
10:52 Worldline’s Payment “Perturbations”
12:40 Outage Trends: By the Numbers
15:06 Get in Touch  

———

For additional insights, check out this week’s The Internet Report blog: https://www.thousandeyes.com/blog/internet-report-digitalocean-reddit-outages?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep3_podcast

And to learn more about how to deliver seamless digital experiences in a distributed IT landscape, read this eBook:  https://www.thousandeyes.com/resources/guide-to-next-generation-assurance-ebook?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep3_podcast  

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on LinkedIn or X at @thousandeyes

Powerful things happen when ITOps teams move beyond a break-fix approach and lean into proactive optimization. 

Instead of just responding to issues as they occur, when teams have independent visibility into their end-to-end service delivery chain, they can proactively identify possible areas for optimization and improvement. For example, streamlining one small part of a complex process could shave seconds off the total transaction time; do this for every part of the process, and the efficiency savings can quickly add up.

In recent weeks, it appeared OpenAI’s ChatGPT may have been undergoing this type of optimization, as we observed material improvements in page load times and some evidence suggesting possible configuration changes and re-architecture in the pursuit of performance improvements.

Read on to learn more about what we saw at ChatGPT, as well as insights from incidents at companies including Grammarly, Bluesky, and Netflix, or use the chapters below to jump to the sections that most interest you.

CHAPTERS
00:00 Intro
00:48 ChatGPT Disruptions
08:17 Grammarly Issue
10:59 Netflix Issues
12:09 Bluesky Disruptions
13:26 Salesforce Outage
14:50 Verizon Fios Disruption
16:14 Outage Trends: By the Numbers
18:36 Get in Touch

———

For additional insights on understanding and proactively optimizing digital experiences, read this eBook:  https://www.thousandeyes.com/resources/guide-to-next-generation-assurance-ebook?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep2_podcast  

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on X: @thousandeyes

The Digital Operational Resilience Act (DORA) goes into effect on January 17, 2025, and financial institutions serving the EU will need to meet an enhanced set of requirements related to risk management, network resilience, and incident reporting.

While DORA is directly applicable to EU financial institutions, it prompts important discussions about resilience and ensuring digital experiences that are relevant for all IT operations teams, regardless of industry or region.

Tune in to the podcast to hear The Internet Report team and special guest Bernie Clairmont, Product Solution Architect at ThousandEyes, dive deeper into DORA.

As usual, we’ll also unpack recent outages from the last few weeks, examining the DDoS attacks against the Internet Archive as well as some power and cooling problems at Google Cloud. We’ll also revisit a previous Azure outage and discuss the latest global outage numbers.

——— 

CHAPTERS:
00:00 Intro
01:20 DORA: What ITOps Teams Need to Know
18:38 BMO Disruption
21:16 Incident Retrospective: Azure Virtual Desktop Outage
25:00 Google Cloud Outage
27:47 Internet Archive Outage
29:35 Get in Touch 

———

For more insights, check out the links below:

- The Internet Report Blog: DORA & ITOps Best Practices; Plus BMO, Google Outages: https://www.thousandeyes.com/blog/internet-report-dora-itops-best-practices?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep1_podcast

- White Paper: ThousandEyes and the Digital Operational Resilience Act (DORA): https://www.thousandeyes.com/resources/digital-operational-resillience-act-dora-white-paper?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep1_podcast

- Blog: Introducing Adaptive API Monitoring: https://www.thousandeyes.com/blog/introducing-adaptive-api-monitoring?utm_source=transistor&utm_medium=referral&utm_campaign=fy25q2_internetreport_q2fy25ep1_podcast

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on X: @thousandeyes

A recent Salesforce outage highlighted the limitations of status pages and the importance of considering a variety of data points when identifying the source of an outage.

Tune in to hear The Internet Report team discuss what happened and why. They’ll also share insights from a recent Microsoft Outlook outage and cover the latest Internet outage trends.

Listen now or use the chapters below to jump to the sections that most interest you.

CHAPTERS
00:00 Intro
00:48 Salesforce Outage
10:00 Microsoft Outlook Outage
14:43 Outage Trends: By the Numbers
17:22 Get in Touch 

———

For additional insights, check out the latest Internet Report blog: https://www.thousandeyes.com/blog/internet-report-salesforce-microsoft-outage?utm_source=transistor&utm_medium=referral&utm_campaign=na_fy25q1_internetreport_q1fy25ep6_podcast

And for more on troubleshooting digital experiences across owned and unowned networks, watch this webinar featuring The Internet Report’s Mike Hicks: https://www.thousandeyes.com/resources/apjc-troubleshooting-digital-experiences-across-owned-unowned-networks-webinar?utm_source=transistor&utm_medium=referral&utm_campaign=na_fy25q1_internetreport_q1fy25ep6_podcast 

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on X: @thousandeyes

A recent certificate problem impacted ServiceNow, and other issues prevented users from accessing key cloud services including Microsoft 365, Azure Virtual Desktop, and Workday.

Tune in to hear what happened during these incidents and a separate data center fire that caused a Reliance Jio outage for customers across multiple areas of India.

Listen now or use the chapters below to jump to the sections that most interest you.

CHAPTERS
00:00 Intro
00:59 ServiceNow Outage
03:20 Microsoft 365 Outage
04:35 Azure Virtual Desktop Outage
05:50 Workday Outage
09:39 Reliance Jio Outage
13:06 Outage Trends: By the Numbers
15:16 Get in Touch 

———

For more insights, check out The Internet Report blog: https://www.thousandeyes.com/blog/internet-report-workday-outage-access-issues?utm_source=transistor&utm_medium=referral&utm_campaign=na_fy25q1_internetreport_q1fy25ep5_podcast

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on X: @thousandeyes

During high-traffic seasons like Black Friday or a much-anticipated product launch, maintaining good digital experiences for customers is vital. We’ve all heard tales of floods of eager shoppers crashing a website during a major sale—leaving them unable to make their coveted purchases. To guard against a breakdown like this during high-traffic periods, companies sometimes use various traffic management strategies such as digital waiting rooms.

In this episode, The Internet Report team discusses the pros and cons of traffic management and looks at the different techniques used by ticketing platforms for the upcoming Oasis reunion tour concerts. They also cover an AT&T issue that impacted Microsoft, as well as disruptions at Akamai, Alibaba Cloud, and Cloudflare.

Listen to the full episode now or use the chapters below to skip to the sections that most interest you.

CHAPTERS
00:00 Intro
00:54 Oasis Reunion Ticket Issues
10:16 Microsoft Outage
13:00 Akamai Outage
14:40 Alibaba Cloud Outage
16:28 Cloudflare Incident
17:40 Outage Trends: By the Numbers
18:40 Get in Touch

———

For more insights, check out The Internet Report blog: https://www.thousandeyes.com/blog/internet-report-traffic-management-microsoft-outage?utm_source=transistor&utm_medium=referral&utm_campaign=na_fy25q1_internetreport_q1fy25ep4_podcast

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on X: @thousandeyes

Let’s dive into the fascinating world of subsea cables. With special guest Murray Burling—Executive Director of Oceans and Environment at RPS—we’ll explore the current subsea cable ecosystem and chat about what the future might hold.

Tune in for insights on how important subsea cables are for today’s digital experiences, how decisions are made on where to place them, the consequences of cable cuts, and route diversity and Internet resilience.

CHAPTERS
00:00 Intro
02:29 Current Subsea Cable Ecosystem
07:16 Subsea Cable Cuts
15:15 Route Diversity & Internet Resilience
18:51 What’s Next
22:05 Get in Touch

ABOUT MURRAY BURLING
Murray Burling is the Executive Director of Oceans and Environment at RPS. Murray has extensive experience as a consulting coastal engineer, oceanographer, and marine modeler, undertaking multifaceted studies in Australia, New Zealand, South East Asia, and the Middle East. He has managed the implementation of large web-based data and model applications and his team supports data acquisition and analysis for a wide range of industries developing and operating in the world’s oceans. These sectors include conventional and renewable energies, shipping, ports, and communications.  He is also an expert in complex dataset analysis and visualization, and has authored and co-authored many technical reports and publications.

Connect with Murrary on LinkedIn: https://www.linkedin.com/in/murray-burling-a527839a/

———

For more insights, check out The Internet Report blog: https://www.thousandeyes.com/blog/internet-report-subsea-cables?utm_source=transistor&utm_medium=referral&utm_campaign=na_fy25q1_internetreport_q1fy25ep3_podcast

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on X: @thousandeyes

Explore the recent Google Cloud and GitHub outages, plus get insights from a network perspective into the August 12 X livestream event featuring Elon Musk and Donald Trump.

In the case of Google Cloud, a power issue in one of its European regions impacted connectivity and affected several services and networking equipment. The problems disrupted connectivity into the region as well as some Partner Interconnect connections and associated routes between other Google regions.

Traffic to and from GitHub.com encountered an issue when a database configuration change resulted in critical services unexpectedly losing connectivity.

And when access issues during the August 12 X “Spaces” livestream were attributed to a network load-related issue, traffic behavior observed during the event suggested a different technical explanation.

Tune in to learn more about these three events or use the chapters below to jump to the sections that most interest you.

CHAPTERS:
00:00 Intro
00:47 Google Cloud Outage
05:03 GitHub Outage
11:26 X Spaces Insights
14:16 Outage Trends: By the Numbers
15:57 Get in Touch 

———

For more insights, check out The Internet Report blog: https://www.thousandeyes.com/blog/internet-report-github-google-outages?utm_source=youtube&utm_medium=referral&utm_campaign=na_fy25q1_internetreport_q1fy25ep2_podcast

———

Want to get in touch?

If you have questions, feedback, or guests you would like to see featured on the show, send us a note at [email protected]. Or follow us on X: @thousandeyes