The Cloud Pod Puts a Hex-LLM on all these AI Announcements

Welcome to episode 270 of the Cloud Pod Podcast – where the forecast is always cloudy! Jonathan, Ryan, Matt and Justin are your hosts today as we sort through all of the cloud and AI news of the week, including updates to the Crowdstrike BSOD event, more info on that proposed Wiz takeover (spoiler alert: it’s toast) and some updates to Bedrock. All this and more news, right now on the Cloud Pod! 

01:33 In what feels suspiciously like an SNL skit, CrowdStrike sent its partners $10 Uber Eats gift cards as an apology for mass IT outage

04:37 Jonathan – “I think part of the blame was on the EU, wasn’t it, against Microsoft, in fact, for making Microsoft continue to give kernel level access to these types of integrations. Microsoft wanted to provide all this functionality through an API, which would have been safe. They wouldn’t have caused a blue screen if there had been an error. But in the EU, there were complaints from antivirus vendors. They wanted direct access to things in the kernel rather than going through an API.”

08:57 Delta hires David Boies to seek damages from CrowdStrike, Microsoft after outage 

12:23 Cyber-security firm rejects $23bn Google takeover  

13:33 Justin – “I mean, I don’t know why they’re not going public now. mean, at 500 million in ARR and the number of employees, their costs, their margins have to be really good unless they’re paying a ton of money for marketing. yeah, it’s something IPO I’ll be keeping an eye out for.”

14:18  Introducing Llama 3.1: Our most capable models to date

What Meta’s Largest Llama Model is Missing 

Meta’s Llama 3.1 is now available on Google Cloud 

A New Standard in Open Source AI: Meta Llama 3.1 on Databricks 

Meta Llama 3.1 generative AI models now available in Amazon SageMaker JumpStart

Meta Llama 3.1 generative AI models now available in Amazon Bedrock  

Announcing Llama 3.1 405B, 70B, and 8B models from Meta in Amazon Bedrock 

Meta’s Llama 3.1 405B Now Available for Enterprise App Development in Snowflake Cortex AI

Meta Llama 3.1 now available on Workers AI  

16:03 Snowflake Cortex AI Launches Cortex Guard to Implement LLM Safeguards  

17:56 Justin – “If I was going into production with an LLM of any kind on my website, I would definitely be looking to get some safeguards. Because like this type of stuff, you have to be thinking about how to protect against these types of attacks all the time. Because these prompt hijacking attacks are just, people are getting good at them, they’re getting to the point where they can break in raw data. Even Apple Intelligence, which is the new Apple AI, people were jailbreaking it already, now, in the beta where the people they were getting like releases of information of certain iPhone models that are coming out like not specific hardware information but like model numbers and Revision numbers for the hardware that shouldn’t be public information. So you got to be careful with these AI models.”

21:59 SearchGPT Prototype

22:56 Ryan – “This is kind of like when they were announced Bard, right, it felt very search heavy, like very opinionated. So it’s kind of funny to see it come kind of full circle, because Google had to pivot very quickly to something that wasn’t very search oriented, because that’s not what people wanted. And now to see OpenAI kind of go back the other way is fun.”

28:15 Justin – “I’m sort of intrigued by the idea of it. But one thing about most of these models, OpenAI, Claude, et cetera, they’re really not helpful for things that are happening very soon or occurred since the model was built. And most of them don’t stay up to date.”

28:35  Introducing AWS End User Messaging

29:26 Jonathan – “Anything’s better than Twilio.”

30:02 Mistral Large 2 is now available in Amazon Bedrock  

30:40 Jonathan – “So I think the best thing about Michel 2 is that it was specifically trained to know things that it didn’t know. So instead of hallucinating some answer that sounds plausible, it does a pretty good job of saying, I don’t know the answer to that question, which is awesome. Everyone should do the same thing.

33:44 How to migrate your AWS CodeCommit repository to another Git provider 

37:54 Justin – “…Code Commit in particular is in a lot of Amazon documentation as examples for using code stuff. And so like to kill Code Commit without much notice or to allow documentation to get updated, to leverage GitHub or GitLab or some other solution, that’s a, that’s a bit of a mistake, I think on Amazon’s part.”

40:02 AWS Graviton-based EC2 instances now support hibernation

40:32 Jonathan – “I think the coolest thing I learned about Hibernate support is that you can Hibernate EC2 instances using CloudFormation.”

41:17 Announcing VPC Service Controls with private IPs to extend data 

exfiltration protection 

42:02 Jonathan – “So the way that VPC service controls work is that you sort of add your GCP APIs and your resources within GCP to secure perimeters, and then you can sort of dictate the communication that’s allowed between those perimeters. And so what this does is allows you to put a boundary on communication from private IPs between those perimeters.”

44:38 Mistral AI’s Codestral launches as a service, first on Vertex AI

45:52 Jonathan – “Well, if you want to chat with it, then Gemini makes sense. But if you want to programmatically send a request to generate some code to an endpoint and have it return code in a known format… this is all going to be old news when we just realize that AIs can just replace the entire stack, the operating system, the applications running on them. We give the AI the instructions and say, OK, show me a user interface on my screen that does this and does this on the back end or does whatever else. And it just does it. It runs constantly. It’s constantly running inference to actually solve the problems that we have rather than generating code to run elsewhere.”

47:10 Hex-LLM: High-efficiency large language model serving on TPUs in Vertex AI Model Garden  

48:19 Justin – “Yeah, so basically it’s instead of using a generic third party serving stack on top of the TPUs that Google sells you, they now have a customized TPU serving stack that is optimized to use Google’s TPUs.”

49:57 Gemini’s big upgrade: Faster responses with 1.5 Flash, expanded access and more   

51:43 Announcing Phi-3 fine-tuning, new generative AI models, and other Azure AI 

updates to empower organizations to customize and scale AI applications 

52:47 Matthew – “I’ve tried Claude now. I like Claude quite a bit. I use open AI quite a bit. I like that as well. You know, just, on my LM studio, I use the Meta Lama 3 .1 and 3 .0. You know, it just depends on what you want. But you know, and which one do you like to have? Why do you bower? That’s really the question.”

53:36 Reintroducing the autonomous database, now with Oracle Database 23ai

And that is the week in the cloud! Visit  our website, the home of the Cloud Pod where you can join our newsletter, slack team, send feedback or ask questions at theCloud Pod.net or tweet at us with hashtag #theCloudPod

Joe Daly is a Director of Community for the FinOps Foundation, which is kind of like sitting at the largest lunch table in Middle School, but with less vaping.  He’s had illustrious careers as a CPA (the Statute of Limitations has past for all tax returns he prepared and he has let his CPA expire), Corporate Taxation, IT Finance & Accounting, IT Portfolio Management, a regrettable stint as Manager of Server Operations, and has started two teams that perform what has come to be known as FinOps.  He lives in Columbus, OH and enjoys copying off Rob. Go Captains!

Rob Martin is a FinOps Principal at the FinOps Foundation, which is kind of like being a Middle School Principal, but with less vaping. He’s had illustrious careers at Accenture, the US Department of Justice, Amazon Web Services, and Cloudability, and less lustrious jobs at a few other places. He now spends his time collecting, developing, and distributing FinOps content among the huge global community of people who deliver value from cloud. He lives in Leesburg, VA, and enjoys games (including the FinOps Boardgame!), hiking, and announcing for his son’s high school soccer team. Go Captains!

For this special edition of TCP Talks, Justin and Jonathan are joined by Travis Runty, CTO of Public Cloud with Rackspace Technology. In today’s interview, they discuss being accidentally multi cloud, public vs private cloud, and cloud migration, and best practices when assisting clients with their cloud journeys. 

Background

Rackspace Technology, commonly known as Rackspace, is a leading multi-cloud solutions provider headquartered in San Antonio, Texas, United States. Founded in 1998, Rackspace has established itself as a trusted partner for businesses seeking expertise in managing and optimizing their cloud environments.

The company offers a wide range of services aimed at helping organizations navigate the complexities of cloud computing, including cloud migration, managed hosting, security, data analytics, and application modernization. Rackspace supports various cloud platforms, including AWS, Azure, and GCP, among others. 

Rackspace prides itself on its “Fanatical Experience” approach, which emphasizes delivering exceptional customer support and service. This commitment to customer satisfaction has contributed to Rackspace’s reputation as a reliable and customer-centric provider in the cloud computing industry. 

Meet Travis Runty, CTO of Public Cloud for Rackspace Technology

Beginning his career with Rackspace as a Linux engineer, Travis has spent the last 15 years working his way through multiple divisions of the company, including 10 years in senior and director level positions. Most recently, Travis served as VP of Technical Support of Global Cloud Operations from 2020-2022. 

Travis is extremely passionate about building and leading high performance engineering teams and delivering innovative solutions. Most recently, as a member of their technology council, Travis wrote an article for Forbes – Building a Cloud-Savvy Workforce: Empowering Your Team for Success – where he discussed best practices for prioritizing workforce enablement, especially when it comes to training and transformation initiatives. 

Interview Notes:

In the main show, TCP has been talking a lot about Cloud / hybrid cloud / multi-cloud and repatriating data back to on prem, and today’s guest knows all about those topics. 

Rackspace has had quite a few phases in their journey to public cloud – including building a data center in an unused mall, introducing managed services, creating partnerships with VMware, an attempt to go head to head with the hyperscalers, and then ultimately focusing on public cloud and instead partnering with the hyperscalers. 

Rackspace has both a focus on private and public cloud; when it comes to private cloud they focus mainly on VMware and OpenStack, whereas in the public cloud side, Rackspace partners with the hyperscalers to assist clients with their cloud journey. 

Quotes from today’s show 

Travis: “We want to make sure that when a customer goes on their public cloud journey, that they actually have a robust strategy that is going to be effective. From there, we’re able to leverage our professional services teams to make sure that they can realize that transformation, and hopefully there *is* a transformation, and it’s not just a lift and shift.”

Travis: “A conflict that we continuously have to strike the balance of is when do we apply a cloud native solution, and where do we apply the Rackspace elements on top. The hyperscalers technology is the best there is, and we’re probably not going to create a better version of “x” than AWS does – nor do we want to.”

Travis: “We favor cloud native. Every single time we’re going to favor the platform’s native solution, unless the customer has a really really strong opinion about being vendor locked. Which sometimes they do. And if that’s the case we can establish a solution that gives them that portability. But for right now, the customers are generally preferring cloud native solutions.”

A bonus episode of The Cloud Pod may be just what the doctor ordered, and this week Justin and Jonathan are here to bring you an interview with Sandy Bird of Sonrai Security. There’s so much going on in the IAM space, and we’re really happy to have an expert in the studio with us this week to talk about some of the security least privilege specifics. 

Sonrai (pronounced Son-ree, which means data in Gaelic) was founded in 2017. Sonrai provides Cloud Data Control, and seeks to deliver a complete risk model of all identity and data relationships, which includes activity and movement across cloud accounts, providers, and third party data stores.

Start your free trial today

Sandy is the co-founder and CTO of Sonrai, and has a long career in the tech industry. He was the CTO and co-founder of Q1 Labs, which was acquired by IBM in 2011, and helped to drive IBM security growth as CTO for global business security there. 

One of the big questions we start the interview with is just how has IAM evolved – and what kind of effect have those changes had on the identity models?  Enterprise wants things to be least privilege, but it’s hard to find the logs. In cloud, however *most* things are logged – and so least privilege became an option. 

Sonrai offers the first cloud permissions firewall, which enables one click least privilege management, which is important in the current environment where the platforms operate so differently from each other. With this solution, you have better control of your cloud access, limit your permissions, attack surface, and automate least privilege – all without slowing down DevOps2. 

Is the perfect policy achievable? Sandy breaks it between human identities and workload identities; they’re definitely separate. He claims, in workload identities the perfect policy is probably possible. Human identity is hugely sporadic, however, it’s important to at least try to get to that perfect policy, especially when dealing with sensitive information. One of the more interesting data pieces they found was that less than 10% of identities with sensitive permissions actually used them – and you can use the information to balance out actually handing out permissions versus a one time use case. 

Sonrai spent a lot of time looking at new solutions to problems with permissions; part of this includes purpose-built integration, offering a flexible open GraphQL API with prebuilt integrations. 

Sonrai also offers continuous monitoring; providing ongoing intelligence on all the permission usage – including excess permissions – and enables the removal of unused permissions without any sort of disruptions. Policy automation automatically writes IAM policies tailored to access needs, and simplifies processes for teams. 

On demand access is another tool that gives on demand requests for permissions that are restricted with a quick and efficient process. 

Sandy: “The unbelievably powerful model in AWS can do amazing things, especially when you get into some of the advanced conditions – but man, for a human to understand what all this stuff is, is super hard. Then you go to the Azure model, which is very different. It’s an allow first model. If you have an allow anywhere in the tree, you can do whatever is asked, but there’s this hierarchy to the whole thing, and so when you think you want to remove something you may not even be removing it., because something above may have that permission anyway. It’s a whole different model to learn there.” 

Sandy: “Only like 8% of those identities actually use the sensitive parts of them; the other 92 just sit in the cloud, never being used, and so most likely during that break loss scenario in the middle of the night, somebody’s troubleshooting, they have to create some stuff, and overpermission it . If we control this centrally, the sprawl doesn’t happen.”

Sandy: There is this fear that if I remove this identity, I may not be able to put it back the way it was if it was supposed to be important… We came up with a secondary concept for the things that you were worried about… where we basically short circuit them, and say these things can’t log in and be used anymore, however we don’t delete the key material, we don’t delete the permissions. We leave those all intact.” 

In this episode, Andrew Krug talks about Datadog as a security observability tool, shedding light on some of its applications as well as its benefits to engineers.

Andrew is the lead in Datadog Security Advocacy and Datadog Security Labs. Also a Cloud Security consultant, he started the Threat Response Project, a toolkit for Amazon Web Services first responders. Andrew has also spoken at Black Hat USA, DEFCON, re:Invent, and other platforms..

Datadog is focused on bringing security to engineering teams, not just security people. One of the biggest advantages of Datadog or other vendors is how they ingest and normalize various log sources. It can be very challenging to maintain a reasonable data structure for logs ingested from cloud providers.

Vendors try to provide customers with enough signals that they feel they are getting value while trying not to flood them with unactionable alerts. Also, considering the cloud friendliness for the stack is crucial for clients evaluating a new product.

Datadog is active in the open-source community and gives back to groups like the Cloud native computing foundation. One of their popular open-source security tools created is Stratus-red-team which simulates the techniques of attackers in a clean room environment. The criticality of findings is becoming a major topic. It is necessary when evaluating that criticality is based on how much risk applies to the business, and what can be done.

One of the things that teams struggle with as high maturity DevOps is trying to automate incident handling or response to critical alerts as this can cause Configuration Drift which is why there is a lot of hesitation to fully automate things. Having someone to make hard choices is at the heart of incident handling processes.

Datadog Cloud SIEM was created to help customers who were already customers of logs. Datadog SIEM is also very easy to use such that without being a security expert, the UI is simple. It is quite difficult to deploy a SIEM on completely unstructured logs, hence being able to extract and normalize data to a set of security attributes is highly beneficial. Interestingly, the typical boring hygienic issues that are easy to detect still cause major problems for very large companies. This is where posture management comes in to address issues on time and prevent large breaches.

Generally, Datadog is inclined towards moving these detections closer to the data that they are securing, and examining the application run time in real-time to verify that there are no issues. Datadog would be helpful to solve IAM challenges through CSPM which evaluates policies. For engineering teams, the benefit is seen in how information surfaces in areas where they normally look, especially with Datadog Security products where Issues are sorted in order of importance.

Security Observability Day is coming up on the 18th of April when Datadog products will be highlighted; the link to sign up is available on the Datadog Twitter page and Datadog community Slack. To find out more, reach out to Andrew on Twitter @andrewkrug and on the Datadog Security Labs website.

Top Quotes

In this episode, Ravi Mayuram highlights the functionality of Couchbase as an evolutionary database platform, citing several simple day-to-day use cases and particular advantages of Couchbase.

Ravi Mayuram is CTO of Couchbase. He is an accomplished engineering executive with a passion for creating and delivering game-changing products for startups as well as Fortune-500 industry-leading companies.

Notes

Couchbase set out to build a next-generation database. Data has evolved greatly with IT advancements. The goal was to build a database that will connect people to the newer technologies, addressing problems that relational systems did not have to solve. The fundamental shift is that earlier systems were internally focused, built for trained users but now the systems are built directly for consumers. This shift also plays out in the vast difference in the number of consumers now interacting with these systems compared to the fewer trained users previously interacting with the systems.

One of the key factors that sets Couchbase apart is the No-SQL Database. It is a database that has evolved by combining five systems; a Cache and Key-value store, a Document store, a Relational document store, a Search system, and an Analytical system. Secondly, Couchbase performs well in the geo-distributed manner such that with one click, data is made available across availability zones. Lastly, all of this can be done at a large scale in seconds.

Regarding the global database concept that Google talks about, a globally consistent database may not be needed by most companies. The performance will be the biggest problem as transaction speed will be considerably low. Couchbase does these transactions locally within the data center and replicates them on the other side. The main issue of relational systems is that they make you pay the price of every transaction no matter how minor, but with Couchbase, it is possible to pay only the cost only with certain crucial transactions.

Edge has become a part of the enterprise architecture even such that people now have edge-based solutions. Two edges are emerging; the Network edge and the Tool edge where people are interfacing. Couchbase has built a mobile database available on devices, with sync capability.

As a consumer, the primary advantage of bringing data closer to the consumer is the latency issue. Often, data has to go through firewalls and multiple steps which delays it but this is the benefit of Couchbase. The user simply continues to have access to the data while Couchbase synchronizes the data in the back.

One of the applications of Couchbase in healthcare is insulin tracking. With many devices that monitor insulin which must work everywhere you go, Couchbase Lite does the insulin tracking, keeps the data even in the absence of a network, and later syncs it for review by healthcare professionals. This is also useful in operating rooms where the network is not accessible. The real benefit is seen when the data eventually gets back to the server and can be interpreted to make decisions on patient care.

The Couchbase Capella Service runs in the cloud and allows clients to specify what data should be sent to the edge and what should not be. This offers privacy and security measures, such that even in the loss or damage of a device, the data is secure and can be recovered. To effectively manage edge in devices, a lot of problems must be addressed to make it easier.

One of the concerns for anyone coming into Couchbase Capella is the expense of data extraction from the cloud, however, Couchbase is available on all three cloud providers. Also, with Couchbase, there is no need to keep replicating data as you can work on the data without moving it, which largely saves costs.

Other use cases for Couchbase include information for flight bookings, flight crew management systems, hotel reservations, and credit card payments. To learn more, visit the Couchbase website. There is also a free trial for the Couchbase Capella Service.  

Top Quotes

In this episode, Daniel explains a new strategy towards observability aimed at contextualizing large volumes of data to make it easier for users to identify the root cause of problems with their systems.

Daniel Kim is a Principal Developer Relations Engineer at New Relic and the founder of Bit Project, a 501(c)(3) nonprofit dedicated to making tech accessible to under-served communities. His job is basically to get developers excited about Observability, and he hopes to inspire students to maximize their potential in tech through inclusive, accessible developer education. He is passionate about diversity and inclusion in tech, good food, and dad jokes.

First, it is important to differentiate between monitoring and observability. Monitoring is basically when a code is instrumented to send data to a backend, to give answers to preconceived questions. With Observability, the goal is to monitor your system so as to later ask questions that were not in mind during the instrumentation of the system. Hence, if something new comes up you can find the root cause without modifying the code. There are so many levels of things to check when troubleshooting to find the cause of a problem, and this is where observability comes in.

There are different use cases for logs, metrics, and traces; Logs are files that record events, warnings, or errors however logs are ephemeral which means there is increased risk of losing a lot of data. A system needs to be in place to move logs to a central source. Another issue with logs is that it is poorly structured data. Logs are good to have as the last step of observability. Metrics and traces can however help to narrow down where to search in the logs to solve an issue.

Metrics are measurements that reflect the performance or health of your applications. They give an overview of how the systems are doing but tend to not be very specific in finding the root cause of a problem; other forms of data have to be adopted to get a clear picture. This is where Traces come in.

Traces are pieces of data that track a request as it goes through the system. Because of this, they can identify the root cause of an error or bottlenecks slowing down the system. However, they are very expensive and as such sampling is used when tracing but this reduces the accuracy of traces. Correlating information from logs, metrics, and traces gives a full clear picture for debugging to be carried out successfully. A lot of New Relic customers strive to get more pieces of data to get errors faster.

To balance the right data at the right time with the right cost, the first step when collecting large amounts of data is to find out how your organization is leveraging the data. A quick audit of the data to identify useful data is helpful. This can be done monthly or quarterly. Unstructured logs are difficult to aggregate

In the cloud native space, being able to be compatible with as many people as possible will determine the winners because there are many projects people use in production. Projects that are compatible with many other projects are the way forward.

APM is still very useful to understand application performance and in the future, data from all sources will be correlated to figure out the cause of a problem. Getting value very early from the system involves having a solid infrastructure and installing APM. The real power of full stack observability is getting data from different parts of your stack so you can diagnose what part of your system is going wrong. Leveraging AI to make sense of large amounts of data for engineers is going to be a huge plus.

A lot of vendors claim that their alert systems will automatically generate all alerts for you but this is not true because they would not know your team’s needs. It is ultimately up to your team to set up alerts that create an observability strategy. Those who invest time into setting this up get the most ROI from New Relic. Engineers need to figure out what metrics are important to them.

This was made to be a singular observability platform where people can correlate various pieces of data to get more context making the work easy for engineers. The goal was to help engineers to find the information they need as fast as possible, especially during a crisis.

This kind of third-party solution is much more applicable for processing millions of logs or larger data, compared to native tools. It also provides a large amount of expertise around observability and curated experiences around machine-generated data.

The future seems to have customers tilting towards open-source observability solutions. OpenTelemetry is one example of this, as it brings together all observability offerings in open source in a whole stack observability experience.

Visit the New Relic website to learn more about it. To learn more about ways to use New Relic, check out the New Relic Blogs.

In this episode, Peter sits with Rahul Thakkar to discuss the revolutionary AWS SimSpace Weaver, highlighting its unique function and applications across several industries.

Rahul Thakkar is the Director and General Manager of Simulation Technologies at Amazon Web Services. Before AWS, he held multiple executive roles at Boeing, Brivo, PIXIA, and DreamWorks Animation. He is an inventor, and global technology executive with a background in cloud computing, distributed and high-performance computing, media and entertainment, film, television, defense and intelligence, aerospace, and access control.

His film credits include Shrek, Antz, and Legend of Bagger Vance. In 2002, he was part of the team that won an Academy Award for Shrek as the Best Animated Feature. Again in 2016, at the 88th Annual Academy Awards, Thakkar received a Technical Achievement Award.

AWS SimSpace Weaver enables customers to run extremely large-scale spatial simulations without having to manage any of the underlying infrastructure. It also removes the complexity of state management of entities as they move about the simulation. Previously, carrying out such simulations would be done sequentially, in a cumbersome manner over years but now it can be done in parallel in weeks. Different organizations have tried out this functionality for several scenarios and the results have been amazing. This value was largely made possible due to the approach of working with customer feedback.

Rahul’s interest in the cloud came much later in his career which started initially in the R&D department of the Motion Picture industry where he created many of the complex graphics in movies. He later moved into a small start-up that was developing technologies for satellite imagery and mapping, and from here he moved to aerospace. Generally, he observed the problem that it is very expensive for companies to maintain their infrastructure when dealing with simulations. It also would drain resources and distract from the main focus of the company. Eventually, knew he had to use AWS, and now he works with them.

All the other primitive tools within AWS are being consumed to build the service. There is also the ability to write to S3 so that customers can write the simulations out. This helps customers to remember how the simulation played out.

Relating this new service to the metaverse, Rahul believes that when it comes to the metaverse, each organization has its vision of what it should be. However, AWS built the tools to empower these organizations to build their metaverses. Despite the possibility of having competition from Azure or GCP, the focus of AWS would remain on the customer and their needs, innovation on their behalf.

Identifying new problems that the service would be very applicable for is a great challenge that AWS relies on customers for, to help AWS envision where they want to go with the service. There are definitely many companies running simulations but it is hard to predict how many would migrate to the AWS SimSpace Weaver because it is still a new product. Nonetheless, a lot of industries are interested in this new service. These include smart cities, organizations ranging from local to federal or international, logistics and supply chains, large-scale event planning, or any situation where there is a need to simulate a large problem with digital replicas of the real world.