Sign up to save your podcasts
Or
Share 270: The Cloud Pod Puts a Hex-LLM on all these AI Announcements
Share to email
Share to Facebook
Share to X
Share to email
Share to Facebook
Share to X
Or
270: The Cloud Pod Puts a Hex-LLM on all these AI Announcements
The Cloud Pod Puts a Hex-LLM on all these AI Announcements
Welcome to episode 270 of the Cloud Pod Podcast – where the forecast is always cloudy! Jonathan, Ryan, Matt and Justin are your hosts today as we sort through all of the cloud and AI news of the week, including updates to the Crowdstrike BSOD event, more info on that proposed Wiz takeover (spoiler alert: it’s toast) and some updates to Bedrock. All this and more news, right now on the Cloud Pod!
Titles we almost went with this week:
A big thanks to this week’s sponsor:
Follow Up
01:33 In what feels suspiciously like an SNL skit, CrowdStrike sent its partners $10 Uber Eats gift cards as an apology for mass IT outage
04:37 Jonathan – “I think part of the blame was on the EU, wasn’t it, against Microsoft, in fact, for making Microsoft continue to give kernel level access to these types of integrations. Microsoft wanted to provide all this functionality through an API, which would have been safe. They wouldn’t have caused a blue screen if there had been an error. But in the EU, there were complaints from antivirus vendors. They wanted direct access to things in the kernel rather than going through an API.”
08:57 Delta hires David Boies to seek damages from CrowdStrike, Microsoft after outage
12:23 Cyber-security firm rejects $23bn Google takeover
13:33 Justin – “I mean, I don’t know why they’re not going public now. mean, at 500 million in ARR and the number of employees, their costs, their margins have to be really good unless they’re paying a ton of money for marketing. yeah, it’s something IPO I’ll be keeping an eye out for.”
AI Is Going Great – Or, How ML Makes All It’s Money
14:18 Introducing Llama 3.1: Our most capable models to date
What Meta’s Largest Llama Model is Missing
Meta’s Llama 3.1 is now available on Google Cloud
A New Standard in Open Source AI: Meta Llama 3.1 on Databricks
Meta Llama 3.1 generative AI models now available in Amazon SageMaker JumpStart
Meta Llama 3.1 generative AI models now available in Amazon Bedrock
Announcing Llama 3.1 405B, 70B, and 8B models from Meta in Amazon Bedrock
Meta’s Llama 3.1 405B Now Available for Enterprise App Development in Snowflake Cortex AI
Meta Llama 3.1 now available on Workers AI
16:03 Snowflake Cortex AI Launches Cortex Guard to Implement LLM Safeguards
17:56 Justin – “If I was going into production with an LLM of any kind on my website, I would definitely be looking to get some safeguards. Because like this type of stuff, you have to be thinking about how to protect against these types of attacks all the time. Because these prompt hijacking attacks are just, people are getting good at them, they’re getting to the point where they can break in raw data. Even Apple Intelligence, which is the new Apple AI, people were jailbreaking it already, now, in the beta where the people they were getting like releases of information of certain iPhone models that are coming out like not specific hardware information but like model numbers and Revision numbers for the hardware that shouldn’t be public information. So you got to be careful with these AI models.”
21:59 SearchGPT Prototype
22:56 Ryan – “This is kind of like when they were announced Bard, right, it felt very search heavy, like very opinionated. So it’s kind of funny to see it come kind of full circle, because Google had to pivot very quickly to something that wasn’t very search oriented, because that’s not what people wanted. And now to see OpenAI kind of go back the other way is fun.”
28:15 Justin – “I’m sort of intrigued by the idea of it. But one thing about most of these models, OpenAI, Claude, et cetera, they’re really not helpful for things that are happening very soon or occurred since the model was built. And most of them don’t stay up to date.”
AWS
28:35 Introducing AWS End User Messaging
29:26 Jonathan – “Anything’s better than Twilio.”
30:02 Mistral Large 2 is now available in Amazon Bedrock
30:40 Jonathan – “So I think the best thing about Michel 2 is that it was specifically trained to know things that it didn’t know. So instead of hallucinating some answer that sounds plausible, it does a pretty good job of saying, I don’t know the answer to that question, which is awesome. Everyone should do the same thing.
33:44 How to migrate your AWS CodeCommit repository to another Git provider
37:54 Justin – “…Code Commit in particular is in a lot of Amazon documentation as examples for using code stuff. And so like to kill Code Commit without much notice or to allow documentation to get updated, to leverage GitHub or GitLab or some other solution, that’s a, that’s a bit of a mistake, I think on Amazon’s part.”
40:02 AWS Graviton-based EC2 instances now support hibernation
40:32 Jonathan – “I think the coolest thing I learned about Hibernate support is that you can Hibernate EC2 instances using CloudFormation.”
GCP
41:17 Announcing VPC Service Controls with private IPs to extend data
exfiltration protection
42:02 Jonathan – “So the way that VPC service controls work is that you sort of add your GCP APIs and your resources within GCP to secure perimeters, and then you can sort of dictate the communication that’s allowed between those perimeters. And so what this does is allows you to put a boundary on communication from private IPs between those perimeters.”
44:38 Mistral AI’s Codestral launches as a service, first on Vertex AI
—Arthur Mensch, Co-Founder and CEO, Mistral AI
45:52 Jonathan – “Well, if you want to chat with it, then Gemini makes sense. But if you want to programmatically send a request to generate some code to an endpoint and have it return code in a known format… this is all going to be old news when we just realize that AIs can just replace the entire stack, the operating system, the applications running on them. We give the AI the instructions and say, OK, show me a user interface on my screen that does this and does this on the back end or does whatever else. And it just does it. It runs constantly. It’s constantly running inference to actually solve the problems that we have rather than generating code to run elsewhere.”
47:10 Hex-LLM: High-efficiency large language model serving on TPUs in Vertex AI Model Garden
48:19 Justin – “Yeah, so basically it’s instead of using a generic third party serving stack on top of the TPUs that Google sells you, they now have a customized TPU serving stack that is optimized to use Google’s TPUs.”
49:57 Gemini’s big upgrade: Faster responses with 1.5 Flash, expanded access and more
Azure
51:43 Announcing Phi-3 fine-tuning, new generative AI models, and other Azure AI
updates to empower organizations to customize and scale AI applications
52:47 Matthew – “I’ve tried Claude now. I like Claude quite a bit. I use open AI quite a bit. I like that as well. You know, just, on my LM studio, I use the Meta Lama 3 .1 and 3 .0. You know, it just depends on what you want. But you know, and which one do you like to have? Why do you bower? That’s really the question.”
OCI
53:36 Reintroducing the autonomous database, now with Oracle Database 23ai
Closing
And that is the week in the cloud! Visit our website, the home of the Cloud Pod where you can join our newsletter, slack team, send feedback or ask questions at theCloud Pod.net or tweet at us with hashtag #theCloudPod
View all episodes
By Justin Brodley & Jonathan BakerThe Cloud Pod Puts a Hex-LLM on all these AI Announcements
Welcome to episode 270 of the Cloud Pod Podcast – where the forecast is always cloudy! Jonathan, Ryan, Matt and Justin are your hosts today as we sort through all of the cloud and AI news of the week, including updates to the Crowdstrike BSOD event, more info on that proposed Wiz takeover (spoiler alert: it’s toast) and some updates to Bedrock. All this and more news, right now on the Cloud Pod!
Titles we almost went with this week:
A big thanks to this week’s sponsor:
Follow Up
01:33 In what feels suspiciously like an SNL skit, CrowdStrike sent its partners $10 Uber Eats gift cards as an apology for mass IT outage
04:37 Jonathan – “I think part of the blame was on the EU, wasn’t it, against Microsoft, in fact, for making Microsoft continue to give kernel level access to these types of integrations. Microsoft wanted to provide all this functionality through an API, which would have been safe. They wouldn’t have caused a blue screen if there had been an error. But in the EU, there were complaints from antivirus vendors. They wanted direct access to things in the kernel rather than going through an API.”
08:57 Delta hires David Boies to seek damages from CrowdStrike, Microsoft after outage
12:23 Cyber-security firm rejects $23bn Google takeover
13:33 Justin – “I mean, I don’t know why they’re not going public now. mean, at 500 million in ARR and the number of employees, their costs, their margins have to be really good unless they’re paying a ton of money for marketing. yeah, it’s something IPO I’ll be keeping an eye out for.”
AI Is Going Great – Or, How ML Makes All It’s Money
14:18 Introducing Llama 3.1: Our most capable models to date
What Meta’s Largest Llama Model is Missing
Meta’s Llama 3.1 is now available on Google Cloud
A New Standard in Open Source AI: Meta Llama 3.1 on Databricks
Meta Llama 3.1 generative AI models now available in Amazon SageMaker JumpStart
Meta Llama 3.1 generative AI models now available in Amazon Bedrock
Announcing Llama 3.1 405B, 70B, and 8B models from Meta in Amazon Bedrock
Meta’s Llama 3.1 405B Now Available for Enterprise App Development in Snowflake Cortex AI
Meta Llama 3.1 now available on Workers AI
16:03 Snowflake Cortex AI Launches Cortex Guard to Implement LLM Safeguards
17:56 Justin – “If I was going into production with an LLM of any kind on my website, I would definitely be looking to get some safeguards. Because like this type of stuff, you have to be thinking about how to protect against these types of attacks all the time. Because these prompt hijacking attacks are just, people are getting good at them, they’re getting to the point where they can break in raw data. Even Apple Intelligence, which is the new Apple AI, people were jailbreaking it already, now, in the beta where the people they were getting like releases of information of certain iPhone models that are coming out like not specific hardware information but like model numbers and Revision numbers for the hardware that shouldn’t be public information. So you got to be careful with these AI models.”
21:59 SearchGPT Prototype
22:56 Ryan – “This is kind of like when they were announced Bard, right, it felt very search heavy, like very opinionated. So it’s kind of funny to see it come kind of full circle, because Google had to pivot very quickly to something that wasn’t very search oriented, because that’s not what people wanted. And now to see OpenAI kind of go back the other way is fun.”
28:15 Justin – “I’m sort of intrigued by the idea of it. But one thing about most of these models, OpenAI, Claude, et cetera, they’re really not helpful for things that are happening very soon or occurred since the model was built. And most of them don’t stay up to date.”
AWS
28:35 Introducing AWS End User Messaging
29:26 Jonathan – “Anything’s better than Twilio.”
30:02 Mistral Large 2 is now available in Amazon Bedrock
30:40 Jonathan – “So I think the best thing about Michel 2 is that it was specifically trained to know things that it didn’t know. So instead of hallucinating some answer that sounds plausible, it does a pretty good job of saying, I don’t know the answer to that question, which is awesome. Everyone should do the same thing.
33:44 How to migrate your AWS CodeCommit repository to another Git provider
37:54 Justin – “…Code Commit in particular is in a lot of Amazon documentation as examples for using code stuff. And so like to kill Code Commit without much notice or to allow documentation to get updated, to leverage GitHub or GitLab or some other solution, that’s a, that’s a bit of a mistake, I think on Amazon’s part.”
40:02 AWS Graviton-based EC2 instances now support hibernation
40:32 Jonathan – “I think the coolest thing I learned about Hibernate support is that you can Hibernate EC2 instances using CloudFormation.”
GCP
41:17 Announcing VPC Service Controls with private IPs to extend data
exfiltration protection
42:02 Jonathan – “So the way that VPC service controls work is that you sort of add your GCP APIs and your resources within GCP to secure perimeters, and then you can sort of dictate the communication that’s allowed between those perimeters. And so what this does is allows you to put a boundary on communication from private IPs between those perimeters.”
44:38 Mistral AI’s Codestral launches as a service, first on Vertex AI
—Arthur Mensch, Co-Founder and CEO, Mistral AI
45:52 Jonathan – “Well, if you want to chat with it, then Gemini makes sense. But if you want to programmatically send a request to generate some code to an endpoint and have it return code in a known format… this is all going to be old news when we just realize that AIs can just replace the entire stack, the operating system, the applications running on them. We give the AI the instructions and say, OK, show me a user interface on my screen that does this and does this on the back end or does whatever else. And it just does it. It runs constantly. It’s constantly running inference to actually solve the problems that we have rather than generating code to run elsewhere.”
47:10 Hex-LLM: High-efficiency large language model serving on TPUs in Vertex AI Model Garden
48:19 Justin – “Yeah, so basically it’s instead of using a generic third party serving stack on top of the TPUs that Google sells you, they now have a customized TPU serving stack that is optimized to use Google’s TPUs.”
49:57 Gemini’s big upgrade: Faster responses with 1.5 Flash, expanded access and more
Azure
51:43 Announcing Phi-3 fine-tuning, new generative AI models, and other Azure AI
updates to empower organizations to customize and scale AI applications
52:47 Matthew – “I’ve tried Claude now. I like Claude quite a bit. I use open AI quite a bit. I like that as well. You know, just, on my LM studio, I use the Meta Lama 3 .1 and 3 .0. You know, it just depends on what you want. But you know, and which one do you like to have? Why do you bower? That’s really the question.”
OCI
53:36 Reintroducing the autonomous database, now with Oracle Database 23ai
Closing
And that is the week in the cloud! Visit our website, the home of the Cloud Pod where you can join our newsletter, slack team, send feedback or ask questions at theCloud Pod.net or tweet at us with hashtag #theCloudPod