A Trivial Jailbreak Against Llama 3

Zuck and Meta dropped the "OpenAI killer" Llama 3 on Thursday. It is no doubt a very impressive model.

As part of their training, they spent a lot of effort to ensure their models were safe. Here's what the Meta team did:

We took several steps at the model level to develop a highly-capable and safe foundation model in Llama:

• For example, we conducted extensive red teaming exercises with external and internal experts to stress test the models to find unexpected ways they might be used.

• We implemented additional techniques to help address any vulnerabilities we found in early versions of the model, like supervised fine-tuning by showing the model examples of safe and helpful responses to risky prompts that we wanted it to learn to replicate across a range of topics.

• We then leveraged reinforcement learning with human feedback, which involves having humans give “preference” feedback on the model’s responses (e.g., rating which response is better and safer).

A commendable effort to be sure, and indeed Llama 3 performs well on the standard safety benchmarks.

Priming our Way Around Safeguards

However, it turns out we can trivially get around these safety efforts by simply "priming" the model to produce a harmful response. First, let's consider what a classic dialog flow looks like, and how the safety training of Llama 3 works in this setting:

Figure 1: Standard dialog flow. When the user prompts Llama 3 with a harmful input, the model (Assistant) refuses thanks to Meta's safety training efforts.

However, if we simply prime the Llama 3 Assistant role with a harmful prefix (cf. the edited encode_dialog_prompt function in llama3_tokenizer.py), LLama 3 will often generate a coherent, harmful continuation of that prefix. Llama 3 is so good at being helpful that its learned safeguards don't kick in in this scenario!

Figure 2: A jailbroken Llama 3 generates harmful text. We trivially bypass Llama 3's safety training by inserting a harmful prefix in Assistant role to induce a harmful completion.

Conveniently, there's no need to handcraft these harmful prefixes. Indeed, we can simply just call a naive, helpful-only model (e.g. Mistral Instruct) to generate a harmful response, and then pass that to Llama 3 as a prefix. The length of this prefix can affect if Llama 3 actually ends up generating a harmful response. Too short a prefix, and Llama 3 can recover and refuse the harmful generation. Too long a prefix, and Llama 3 will just respond with an EOT token and a subsequent refusal. Here's the gradation of Attack Success Rate (ASR) at increasing harmful prefix max token lengths on the AdvBench subset:

Table 1: ASR at varying harmful assistant prefix lengths. Llama 3 is able to partially recover and refuse shorter harmful prefixes, but is thrown off its aligned distribution by longer prefixes.

A Lack of Self-Reflection?

Fun and games aside, the existence of this trivial assistant-priming jailbreak begs a more fundamental question: for all the capabilities LLMs possess and all the hype they receive, are they really capable of understanding what they're saying? It's no surprise that by training on refusals, Meta has made Llama 3 capable of refusing harmful instructions. But what this simple experiment demonstrates is that Llama 3 basically can't stop itself from spouting inane and abhorrent text if induced to do so. It lacks the ability to self-reflect, to analyze what it has said as it is saying it.

That seems like a pretty big issue.

Shoot us a message at contact@haizelabs.com if you have ideas on this or otherwise want to chat.

Contemporaneous Collaborators

Shoutout to Ben Lemkin (Princeton) and Jason Vega (UIUC) for working on a similar flavor of priming jailbreaks contemporaneously.

Citing This Work

If you find this work useful, please consider citing it in your research. Use the following BibTeX entry:

Citing This Work

If you find this work useful, please consider citing it in your research. Use the following BibTeX entry:

@misc{llama3jailbreak2024,
  title={A Trivial Jailbreak Against Llama 3},
  author={Leonard Tang},
  year={2024},
  doi={10.5281/zenodo.13187036},
  howpublished={\url{https://github.com/haizelabs/llama3-jailbreak}}
}