I’ve finally had an opportunity to gather the available information about Llama-2 and take an in-depth look at the system card.

My conclusion is that Llama-2 looks to score about 3.4 GPTs, with coding as its relative weak point. The system card tries to claim better performance than that in some places in rather misleading fashion, but in other places it does not make such claims.

For its intended purposes it is now the best open source model, while remaining well behind closed source models. There is substantial improvement over Llama-1 in capabilities, it comes with fine tuning, and also with an attempt at harmlessness.

That attempt at harmlessness appears even more ham-fisted than usual. The claims of a 0.05% (!?!) false refusal rate are clearly very false. Early public red teaming quickly revealed a number of problems, in a model that cannot be unreleased or fully patched.

Llama We Doing This Again?

Meta notices world not yet destroyed and people remain alive, so it has not open sourced enough models. Hence it released Llama 2. Here’s the paper, here’s the blog announcement, here is a download link to GitHub. Here’s Llama-70B on Replicate.

Simon Willison (re: Replicate): Here’s how [...]

---

First published:

Source:

Narrated by TYPE III AUDIO.