This research paper describes a new method called "Best-of-N Jailbreaking," which is a way to trick AI systems into giving harmful responses. It works by slightly changing the way a question is asked, like changing the capitalization or adding background noise to an audio question. The researchers found that this method was very effective at getting harmful answers from different AI systems, including ones that are designed to be safe. They also found that the more they changed the questions, the more likely they were to get a harmful answer. The paper shows that even though AI systems are very advanced, they can still be tricked by simple methods, and it's important to find ways to protect them from these kinds of attacks. The researchers suggest that this method could be used to test the safety of AI systems and help developers make them more secure.

https://arxiv.org/pdf/2412.03556