Guest: Sander Schulhoff, CEO and Co-Founder, Learn Prompting [@learnprompting]
On LinkedIn | https://www.linkedin.com/in/sander-schulhoff/
____________________________
Host: Sean Martin, Co-Founder at ITSPmagazine [@ITSPmagazine] and Host of Redefining CyberSecurity Podcast [@RedefiningCyber]
On ITSPmagazine | https://www.itspmagazine.com/sean-martin
View This Show's Sponsors
___________________________
Episode Notes
In this episode of Redefining CyberSecurity, host Sean Martin engages with Sander Schulhoff, CEO and Co-Founder of Learn Prompting and a researcher at the University of Maryland. The discussion focuses on the critical intersection of artificial intelligence (AI) and cybersecurity, particularly the role of prompt engineering in the evolving AI landscape. Schulhoff's extensive work in natural language processing (NLP) and deep reinforcement learning provides a robust foundation for this insightful conversation.
Prompt engineering, a vital part of AI research and development, involves creating effective input prompts that guide AI models to produce desired outputs. Schulhoff explains that the diversity of prompt techniques is vast and includes methods like the chain of thought, which helps AI articulate its reasoning steps to solve complex problems. However, the conversation highlights that there are significant security concerns that accompany these techniques.
One such concern is the vulnerability of systems when they integrate user-generated prompts with AI models, especially those prompts that can execute code or interact with external databases. Security flaws can arise when these systems are not adequately sandboxed or otherwise protected, as demonstrated by Schulhoff through real-world examples like MathGPT, a tool that was exploited to run arbitrary code by injecting malicious prompts into the AI’s input.
Schulhoff's insights into the AI Village at DEF CON underline the community's nascent but growing focus on AI security. He notes an intriguing pattern: many participants in AI-specific red teaming events were beginners, which suggests a gap in traditional red teamer familiarity with AI systems. This gap necessitates targeted education and training, something Schulhoff is actively pursuing through initiatives at Learn Prompting.
The discussion also covers the importance of studying and understanding the potential risks posed by AI models in business applications. With AI increasingly integrated into various sectors, including security, the stakes for anticipating and mitigating risks are high. Schulhoff mentions that his team is working on Hack A Prompt, a global prompt injection competition aimed at crowdsourcing diverse attack strategies. This initiative not only helps model developers understand potential vulnerabilities but also furthers the collective knowledge base necessary for building more secure AI systems.
As AI continues to intersect with various business processes and applications, the role of security becomes paramount. This episode underscores the need for collaboration between prompt engineers, security professionals, and organizations at large to ensure that AI advancements are accompanied by robust, proactive security measures. By fostering awareness and education, and through collaborative competitions like Hack A Prompt, the community can better prepare for the multifaceted challenges that AI security presents.
Top Questions Addressed
___________________________
Sponsors
Imperva: https://itspm.ag/imperva277117988
LevelBlue: https://itspm.ag/attcybersecurity-3jdk3
___________________________
Watch this and other videos on ITSPmagazine's YouTube Channel
Redefining CyberSecurity Podcast with Sean Martin, CISSP playlist:
📺 https://www.youtube.com/playlist?list=PLnYu0psdcllS9aVGdiakVss9u7xgYDKYq
ITSPmagazine YouTube Channel:
📺 https://www.youtube.com/@itspmagazine
Be sure to share and subscribe!
___________________________
Resources
The Prompt Report: A Systematic Survey of Prompting Techniques: https://trigaten.github.io/Prompt_Survey_Site/
HackAPrompt competition: https://www.aicrowd.com/challenges/hackaprompt-2023
HackAPrompt results published in this paper "Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition EMNLP 2023": https://paper.hackaprompt.com/
___________________________
To see and hear more Redefining CyberSecurity content on ITSPmagazine, visit:
https://www.itspmagazine.com/redefining-cybersecurity-podcast
Are you interested in sponsoring this show with an ad placement in the podcast?
Learn More 👉 https://itspm.ag/podadplc
View all episodes
By Sean Martin, ITSPmagazine
5
33 ratings
Guest: Sander Schulhoff, CEO and Co-Founder, Learn Prompting [@learnprompting]
On LinkedIn | https://www.linkedin.com/in/sander-schulhoff/
____________________________
Host: Sean Martin, Co-Founder at ITSPmagazine [@ITSPmagazine] and Host of Redefining CyberSecurity Podcast [@RedefiningCyber]
On ITSPmagazine | https://www.itspmagazine.com/sean-martin
View This Show's Sponsors
___________________________
Episode Notes
In this episode of Redefining CyberSecurity, host Sean Martin engages with Sander Schulhoff, CEO and Co-Founder of Learn Prompting and a researcher at the University of Maryland. The discussion focuses on the critical intersection of artificial intelligence (AI) and cybersecurity, particularly the role of prompt engineering in the evolving AI landscape. Schulhoff's extensive work in natural language processing (NLP) and deep reinforcement learning provides a robust foundation for this insightful conversation.
Prompt engineering, a vital part of AI research and development, involves creating effective input prompts that guide AI models to produce desired outputs. Schulhoff explains that the diversity of prompt techniques is vast and includes methods like the chain of thought, which helps AI articulate its reasoning steps to solve complex problems. However, the conversation highlights that there are significant security concerns that accompany these techniques.
One such concern is the vulnerability of systems when they integrate user-generated prompts with AI models, especially those prompts that can execute code or interact with external databases. Security flaws can arise when these systems are not adequately sandboxed or otherwise protected, as demonstrated by Schulhoff through real-world examples like MathGPT, a tool that was exploited to run arbitrary code by injecting malicious prompts into the AI’s input.
Schulhoff's insights into the AI Village at DEF CON underline the community's nascent but growing focus on AI security. He notes an intriguing pattern: many participants in AI-specific red teaming events were beginners, which suggests a gap in traditional red teamer familiarity with AI systems. This gap necessitates targeted education and training, something Schulhoff is actively pursuing through initiatives at Learn Prompting.
The discussion also covers the importance of studying and understanding the potential risks posed by AI models in business applications. With AI increasingly integrated into various sectors, including security, the stakes for anticipating and mitigating risks are high. Schulhoff mentions that his team is working on Hack A Prompt, a global prompt injection competition aimed at crowdsourcing diverse attack strategies. This initiative not only helps model developers understand potential vulnerabilities but also furthers the collective knowledge base necessary for building more secure AI systems.
As AI continues to intersect with various business processes and applications, the role of security becomes paramount. This episode underscores the need for collaboration between prompt engineers, security professionals, and organizations at large to ensure that AI advancements are accompanied by robust, proactive security measures. By fostering awareness and education, and through collaborative competitions like Hack A Prompt, the community can better prepare for the multifaceted challenges that AI security presents.
Top Questions Addressed
___________________________
Sponsors
Imperva: https://itspm.ag/imperva277117988
LevelBlue: https://itspm.ag/attcybersecurity-3jdk3
___________________________
Watch this and other videos on ITSPmagazine's YouTube Channel
Redefining CyberSecurity Podcast with Sean Martin, CISSP playlist:
📺 https://www.youtube.com/playlist?list=PLnYu0psdcllS9aVGdiakVss9u7xgYDKYq
ITSPmagazine YouTube Channel:
📺 https://www.youtube.com/@itspmagazine
Be sure to share and subscribe!
___________________________
Resources
The Prompt Report: A Systematic Survey of Prompting Techniques: https://trigaten.github.io/Prompt_Survey_Site/
HackAPrompt competition: https://www.aicrowd.com/challenges/hackaprompt-2023
HackAPrompt results published in this paper "Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition EMNLP 2023": https://paper.hackaprompt.com/
___________________________
To see and hear more Redefining CyberSecurity content on ITSPmagazine, visit:
https://www.itspmagazine.com/redefining-cybersecurity-podcast
Are you interested in sponsoring this show with an ad placement in the podcast?
Learn More 👉 https://itspm.ag/podadplc
364 Listeners
370 Listeners
318 Listeners
31 Listeners
174 Listeners
190 Listeners
77 Listeners
772 Listeners
57,565 Listeners
128 Listeners
2 Listeners
2 Listeners
4 Listeners
0 Listeners
19,479 Listeners
3 Listeners
0 Listeners
1,027 Listeners
31 Listeners