Anthropic Releases Claude Instant 1.2 with Improved Performance on Coding and Math Benchmarks

Since OpenAI released ChatGPT, many companies have attempted to build their own AI models, but only a few have stood out, with Anthropic being one of them.

The AI startup launched its Claude model in March, demonstrating performance comparable to OpenAI’s GPT‑3.5 and GPT‑4. At the same time, Anthropic released Claude Instant, a lighter, cheaper, and faster version, which is now being upgraded.

On Wednesday, Anthropic announced Claude Instant 1.2, an improved version that incorporates the capabilities of Claude 2.0, the latest release from July.

According to the announcement, the use of Claude 2.0’s advanced abilities gives Claude Instant 1.2 significant improvements in mathematics, coding, reasoning, and safety, and it generates longer, more structured responses.

To evaluate the model, Anthropic compared Claude Instant 1.1 and 1.2 on standard benchmark tests, including the Codex evaluation and the Grade‑school Math (GSM8k) benchmark, which are well‑known measures of coding and math proficiency.

In both cases, 1.2 scored 58.7 % on the Codex test versus 52.8 % for the original version, and 86.7 % on GSM8k versus 80.9 % for the original.

For other benchmark exams, the new model’s performance is either slightly lower or slightly higher than the previous version, with only minor differences.

The quality of the generated answers also improved, with fewer hallucinations and stronger resistance to jailbreak attempts; red‑team assessments found Claude 1.2 to be the safest model.

Enterprises can obtain the new model by filling out an interest form, while developers can access it via an API that is considerably cheaper than Claude 2.