Laya AI - Open Source System-1 Decision Model

Laya vs Jev benchmark explained

The scores disagree because they are not the same experiment. Mixing them into one ranking is what makes Laya vs Jev threads look contradictory.

• Official self-test versus a published Jev number. The README and BENCHMARKS.md measure Laya. Jev cells are third-party published figures. The authors say they had no TypeSafe API access, so sample size and prompts differ.
• Fine-tuned specialist versus the base checkpoint. 0.766 on typed-decisions is laya-typed-decisions. The base checkpoints in that file are 0.361 and 0.342, under the 0.461 majority-class line.
• Two files, two AG News cells. The README routed column is 0.950. The BENCHMARKS.md headline is 0.953, which matches the typed-decisions column in the dataset table. English MASSIVE intent is 0.783 in the summary and 0.820 in the 51-language grid.
• Brier is 0.062 on the README and the Hugging Face model card, and 0.061 in BENCHMARKS.md. The table below keeps 0.061 and names the other figure in its caption.
• Banking77 is not one protocol. The official comparison is Jev 0.870 on 72 labels against Laya 0.425 on 77 labels at the default token budget. A community fine-tune, Cahol/laya-banking77-v1, reports 85.55% on the 3,080-example 77-label test and 45.91% for the untuned base under that same protocol. That card says this is not a claim of beating Jev.
• Community remeasure versus the model card. Luni/laya-jev-benchmark says a model-card comparison of 83.8% with 67.8% uses two different benchmarks. On that card's own typed-decisions rerun, untuned Laya is 0.360 and a fine-tune is 0.767, next to Jev's published 0.727. The Jev row is marked published there too. Phishing on a held-out set is 0.505 raw for Laya, 0.611 after Platt scaling on half the emails, and 0.626 for the published Jev figure. Luni notes the 0.611 uses a calibration half and the 0.626 does not.

How to read these numbers

Laya figures in BENCHMARKS.md and the README are the project's own measurements. The report says every checkpoint answered byte-identical questions in each run, with a fixed seed. Reproduce from notebooks/laya_benchmark_colab.ipynb on a T4, and from the JSON files named in that report: research/results/t4_colab_benchmark.json, cpu_51_language_sweep.json, and app_benchmark.json.

Jev figures in the same files are third-party published numbers. The Laya authors say they had no TypeSafe API access, so sample sizes and prompts differ. Treat the Jev column as indicative, which is the wording in BENCHMARKS.md.

Two upstream tables do not use the same English MASSIVE number: the 51-language grid lists English laya at 0.820, and the “English vs the rest” summary lists 0.783. The README routed AG News cell is 0.950, while the BENCHMARKS.md headline AG News cell is 0.953 (the typed-decisions checkpoint in the dataset table). Both are shown under the caption of the table they came from.

Where the report says it fits

The fine-tuned laya-typed-decisions checkpoint is the one that reaches 0.766 accuracy on 2,000 typed decisions. The base checkpoints sit at 0.361 and 0.342, under the 0.461 majority-class baseline in that file. The report's own limit statement is that Laya is a fast base to specialise, not a zero-shot decision engine.

Small label sets are the published strong side: AG News has four labels. Email spam and phishing, both marked in training, are 0.993 in the themes table. Single-question T4 latency for the multilingual checkpoint is 32.8 ms. The README comparison also lists self-hosted cost as $0 against Jev's published $0.042 / 1M tokens, and Apache 2.0 weights against a closed API.

For languages, use the router. The English checkpoint collapses off English and stays confident: Khmer is 0.000 accuracy at 0.952 confidence in the per-language table. Confidence gating cannot catch that, which is why routing happens before the forward pass.

Where the report says Jev leads

Banking77 is the clear loss in the public-dataset table: Jev 0.870 on 72 labels, Laya 0.425 on 77 labels at the default token budget, and 0.492 on the typed-decisions checkpoint. The README calls this an architectural head_max_len limit and says Jev is currently better for 50+ options in one prompt without tuning.

On typed-decisions, Jev's published soft accuracy is 0.580 against Laya's 0.471, and Jev's published raw ECE is 0.144 against 0.213 before temperature fitting. Laya's 0.081 ECE is after a temperature refit. Held-out toxicity moderation is 0.530. Ordinal score is the weakest primitive in the README (SST-5 0.372 on the English task list).