Open navigation menu

The Enterprise LLM Landscape: Benchmarks, Training, Context, Inference Costs, and Model Selection

At the frontier, the best model changes with the month, the referee and the effort setting. Price and deployment route differ by orders of magnitude. Choose the route and build the evaluation capability; treat the model as a replaceable part.

By INVISOMOctober 1, 202618 min read

Updated 1 October 2026. Every model, price and score in this article is as of that date and comes from the sources listed, with vendor-reported results labelled as such. Prices are list prices in US dollars per million tokens. INVISOM has no commercial relationship with any provider named. This article was researched and drafted with AI assistance, including a model from one of the vendors compared, and was reviewed for factual accuracy and balance.

The frontier is a moving tie

September 2026 shows why a static ranking of large language models is of little use to an enterprise. In one month:

  • OpenAI released GPT-6 Astra on 3 September, followed by GPT-6 Sol and GPT-6 Luna, then GPT-6.1 Sol at its DevDay on 29 September. SiliconANGLE reported that OpenAI is withholding GPT-6.1 Astra over safety concerns.
  • Anthropic released Claude Fable 5.1, then Claude Opus 5.5 on 22 September and Claude Sonnet 5.5. A sibling model, Claude Mythos 5.1, is offered only by invitation through Anthropic's Project Glasswing.
  • Google shipped Gemini 3.8 Flash on 2 September, its third Flash release in six weeks, and on 30 September announced Gemini 4 Argon, which is rolling out first to selected cyber defenders.
  • Meta released Muse Spark 1.3, a hosted model from its Superintelligence Labs, while DeepSeek and Alibaba's Qwen team published new open-weight models in August and September.

Two patterns stand out. First, all three leading labs now hold their most capable model back from general release. Second, the gaps between the generally available frontier models are small and inconsistent. Each lab leads on some benchmarks and trails on others, and independent leaderboards disagree about who is first.

Our thesis: for most enterprise decisions, "which model is best?" is the wrong question. Differences of a few benchmark points are smaller than the differences created by who runs the test and at what effort setting. Price differences between model tiers, by contrast, run to two orders of magnitude, and the route through which you consume a model decides your data residency, security controls and even which features you get. The durable investments are a deliberate choice of deployment routes, an evaluation pipeline built on your own tasks, and an architecture that lets you swap models at each release.

Figure 1

The most capable models disclose the least

Current model families, where you can run them, and what their developers disclose, as of 1 October 2026

  • OpenAI

    Proprietary + open

    ModelsGPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna; open-weight gpt-oss-120b and 20b (August 2025)

    RoutesOpenAI API; Microsoft Foundry; Amazon Bedrock

    DisclosedGPT-6: parameters and training data not publicly disclosed; knowledge cutoff disclosed. gpt-oss-120b: 117B total, 5.1B active, Apache 2.0

  • Anthropic

    Proprietary

    ModelsClaude Fable 5.1, Opus 5.5, Sonnet 5.5, Haiku 4.5

    RoutesClaude API; Amazon Bedrock; Google Cloud; Microsoft Foundry

    DisclosedParameters and training data not publicly disclosed; knowledge and training-data cutoffs disclosed

  • Google

    Proprietary + open

    ModelsGemini 3.8 Flash, 3.1 Pro (preview), 3.1 Flash-Lite; Gemini 4 Argon (limited rollout); open Gemma 4

    RoutesGemini API; Vertex AI; Gemma also on Amazon Bedrock

    DisclosedGemini parameters not publicly disclosed; model card gives knowledge cutoff (March 2026 for 3.8 Flash)

  • Meta

    Proprietary + open

    ModelsMuse Spark 1.3 (hosted); open-weight Muse Glimmer 30B and Llama 4

    RoutesMeta Model API; Oracle; Google Cloud (private preview); Llama on Bedrock and Foundry

    DisclosedMuse Spark: not publicly disclosed. Muse Glimmer: 30B dense, Apache 2.0; training tokens not disclosed

  • Mistral AI

    Proprietary + open

    ModelsMistral Medium 3.5; open-weight Mistral Large 3, Small 4, Ministral

    RoutesMistral API; Amazon Bedrock; Microsoft Foundry; self-hosted

    DisclosedLarge 3, Small 4 and Ministral released under Apache 2.0

  • Cohere

    Proprietary

    ModelsCommand A+, Embed v5, Rerank v4

    RoutesCohere API; Microsoft Foundry; older Command and Embed models on Bedrock

    DisclosedCommand A+ described as a mixture-of-experts model; strong focus on embeddings and reranking

  • Amazon

    Proprietary

    ModelsNova 2 Lite, Nova 2 Sonic, Nova Multimodal Embeddings

    RoutesAmazon Bedrock

    DisclosedParameters not publicly disclosed; customization up to pre-training via Nova Forge

  • DeepSeek

    Open weights

    ModelsDeepSeek V4 Pro, V4.1 Flash

    RoutesDeepSeek API; open weights; V4 in Microsoft Foundry

    DisclosedV4.1 Flash card: MIT licence, mixture-of-experts design, 45 trillion pre-training tokens

  • Alibaba Qwen

    Proprietary + open

    ModelsQwen3.8 Max (API); open-weight Qwen3.8-27B and Qwen3.8-Flash-Next

    RoutesQwen API; open weights; earlier Qwen3 models on Bedrock

    Disclosed27B: Apache 2.0. Flash-Next: 125B total, 6B active, Qwen community licence. Training tokens not disclosed

  • Gulf regional

    Open weights

    ModelsJais 2, ALLaM, Falcon Arabic

    RoutesOpen weights; national and regional platforms

    DisclosedJais 2: 70B, trained from scratch. ALLaM and Falcon Arabic: continued pretraining of existing models

Sources: provider model documentation and model cards (OpenAI, Anthropic, Google, Meta, Mistral, Cohere, AWS, DeepSeek, Qwen on Hugging Face); Amazon Bedrock and Microsoft Foundry model catalogues. Accessed 1 October 2026. Routes listed are those verified, not exhaustive. Title paraphrases a finding of the Stanford HAI AI Index 2026.

Three ways to consume a model

Every option sits in one of three deployment models, and the boundaries have blurred.

Proprietary API, direct from the developer. OpenAI, Anthropic, Google, Meta (Muse Spark), Mistral, Cohere, DeepSeek and Qwen all sell API access. This is usually where new models and features appear first.

Managed cloud. The hyperscalers now resell most frontier families under their own contracts, identity and billing. Amazon Bedrock lists GPT-6 and Claude 5.x models alongside Amazon Nova, Mistral, Qwen, DeepSeek, Kimi, Grok and others, but only Google's open Gemma models rather than Gemini. Microsoft Foundry sells GPT-6, DeepSeek V4, Grok, Kimi, Mistral Large 3, Cohere Command A+, Llama and Microsoft's own MAI models, and offers Claude through Anthropic. Claude is available on all three major clouds; Gemini is offered through Google's own platforms. The same model can differ by route: in features, in regional endpoints and in price. Anthropic, for example, documents a 10% premium for regional over global endpoints on Bedrock and Google Cloud. Our comparison of Claude Code and OpenAI Codex shows how much feature parity can vary for the same product by cloud.

Open weights, self-hosted or on a managed endpoint. The open-weight tier is now credible for many enterprise tasks. Stanford's 2025 AI Index found the gap between open and closed models had narrowed from 8% to 1.7% on some benchmarks in a single year. Options include DeepSeek V4 (MIT licence), Qwen3.8-27B (Apache 2.0), OpenAI's gpt-oss (Apache 2.0), Meta's Muse Glimmer 30B (Apache 2.0), Mistral Large 3 and Small 4 (Apache 2.0) and Google's Gemma. Licences differ: Qwen3.8-Flash-Next, for instance, uses a Qwen community licence rather than Apache 2.0. Hardware differs even more. OpenAI states gpt-oss-120b fits on a single 80 GB GPU; DeepSeek V4 Pro is listed at 1.7 trillion parameters, which means a multi-GPU cluster. Running weights on your own infrastructure means prompts never reach the developer, but you take on patching, scaling, safety tuning and a supply-chain review of the weights and serving code, whatever the model's origin.

Figure 2

Choose the route before you choose the model

Proprietary API, managed cloud or open-weight self-hosting. Work down the questions; the first yes usually decides.

  1. 1

    Must inference or data stay in an environment that no provider endpoint covers (in-country, air-gapped, classified)?

    If yes: Open-weight model, self-hosted or on a sovereign cloud. Budget for GPUs, an inference stack, patching and your own evaluation team.
  2. 2

    Do you already buy compute through a hyperscaler, with residency, identity and spend controls in place?

    If yes: Managed cloud route (Bedrock, Vertex AI, Foundry). Confirm in writing which models, regions and features are actually available on that route.
  3. 3

    Does the use case need frontier capability: long-horizon agents, hard reasoning, complex coding?

    If yes: Proprietary frontier model, via managed cloud or direct API. Keep an abstraction layer so you can switch vendor at the next release.
  4. 4

    Is the task narrow, high-volume and stable, and does an open model pass your own evaluation set?

    If yes: Small proprietary tier or open-weight model on a managed endpoint. Self-host only when sustained utilization justifies it.

Most enterprises end up with more than one route: a managed-cloud default, a frontier model for the hardest work, and a small or open model for volume.

INVISOM framework. Feature parity, regional availability and pricing differ by route and change often; verify each against current provider documentation.

Context and modality: a million tokens is table stakes

At the frontier, a context window of about one million tokens is now standard: GPT-6 models accept 1.05 million (with a 922,000-token input cap), Claude Fable 5.1, Opus 5.5 and Sonnet 5.5 accept 1 million, as do Gemini 3.8 Flash, Gemini 3.1 Pro, DeepSeek V4 and Muse Spark 1.3. The differences that matter are elsewhere.

Figure 3

A million tokens is now the frontier default; modality is the real differentiator

Maximum context window, maximum output and accepted input types. All listed models output text.

  • GPT-6 Astra, 6.1 Sol, 6 Luna

    Context 1.05M (922K max input) · Output 128K

    TextImageVideoAudioPDF
  • Claude Fable 5.1, Opus 5.5, Sonnet 5.5

    Context 1M · Output 128K

    TextImageVideoAudioPDF
  • Claude Haiku 4.5

    Context 200K · Output 64K

    TextImageVideoAudioPDF
  • Gemini 3.8 Flash, 3.1 Pro (preview)

    Context 1M · Output 64K

    TextImageVideoAudioPDF
  • DeepSeek V4 Pro, V4.1 Flash

    Context 1M · Output 384K

    TextImageVideoAudioPDF
  • Meta Muse Spark 1.3

    Context 1M · Output Not stated

    TextImageVideoAudioPDF
  • Qwen3.8-27B (open)

    Context 262K native, extensible to 1M · Output Not stated

    TextImageVideoAudioPDF
  • Cohere Command A+

    Context 128K · Output 64K

    TextImageVideoAudioPDF
Sources: OpenAI, Anthropic, Google Gemini API, DeepSeek API, Meta developer, Cohere documentation; Qwen model card on Hugging Face. As of 1 October 2026. DeepSeek image input applies to the Flash model. Advertised context is not the same as reliable use of the whole window.
  • Input modalities. Gemini models accept text, images, video, audio and PDF natively. GPT-6 and Claude models accept text and images. If the workload is video or call audio, this narrows the shortlist before any benchmark.
  • Maximum output. It ranges from 64,000 tokens (Gemini, Claude Haiku 4.5) to 128,000 (GPT-6, Claude) and 384,000 (DeepSeek's API), which matters for long reports, code generation and agent transcripts.
  • Long-context pricing. OpenAI prices GPT-6.1 Sol prompts above 272,000 input tokens at twice the input rate and 1.5 times the output rate for the whole request. Google charges more for Gemini 3.1 Pro prompts above 200,000 tokens. Anthropic bills Claude 4.6 and later at standard rates across the full window.
  • Knowledge cutoff. Disclosed cutoffs are June 2026 for Claude 5.5 and Fable 5.1, 30 April 2026 for GPT-6 Astra and 6.1 Sol, and March 2026 for Gemini 3.8 Flash. Retrieval, not the cutoff, should carry anything that changes.

A large window is a capacity, not a guarantee that the model uses every part of it equally well. Our analysis of RAG versus fine-tuning covers when long context with caching beats retrieval, and when it does not.

What is disclosed about training, and what is not

For the proprietary frontier, the honest answer to "how big is it and what was it trained on?" is not publicly disclosed. None of OpenAI's GPT-6 models, Anthropic's Claude 5.x models, Google's Gemini 3.x models, Meta's Muse Spark or Amazon's Nova 2 models has a published parameter count, training-token count or training-compute figure in the documentation reviewed. Developers do disclose knowledge cutoffs and, in model cards, outline training approaches in general terms.

This is a documented trend. Stanford CRFM's Foundation Model Transparency Index fell to a mean score of 41 in December 2025, a 17-point drop on 2024, with training data still the most opaque area. The 2026 AI Index notes that the most capable models often disclose the least.

Open-weight developers disclose more, though not everything. DeepSeek's V4.1 Flash model card describes its mixture-of-experts design and pre-training on 45 trillion tokens. Qwen discloses 125 billion total and 6 billion active parameters for Qwen3.8-Flash-Next, and OpenAI 117 billion total and 5.1 billion active for gpt-oss-120b. Training-token counts for Qwen3.8 and Muse Glimmer are not disclosed.

Training-compute figures you see for closed models are estimates. Epoch AI, the most widely used source, labels its estimates as accurate within a factor of 3, 10 or 30, depending on confidence. Never put an estimated parameter count or training cost into a business case or risk assessment as if it were fact. For enterprises the useful disclosure questions are practical: licence terms, intellectual-property indemnities, data-use terms, and whether the provider will tell you when a model version changes.

Benchmarks: read them like a referee

Benchmarks are a reasonable way to build a shortlist and a poor way to make a final choice. Four problems recur.

Figure 4

Who is “best” depends on who runs the test

Top: the same benchmark, vendor-reported vs independently run. Bottom: three independent leaderboards, three different answers.

Terminal-Bench 4.0 task success (%)

Vendor launch postIndependent (Vals AI)
Claude Opus 5.5Anthropic
66.4
65.15
GPT-6 AstraOpenAI
57.9
59.6
Claude Fable 5.1Anthropic
55.8
58.08
50%60%70%

Ranked first overall

  • Artificial Analysis Intelligence Index v4.3.2

    Claude Opus 5.5 (max effort)

    Ten evaluations run by Artificial Analysis, scored per reasoning setting

  • Vals Index, updated 30 September 2026

    Gemini 4 Argon

    Agentic tasks in finance, coding, legal and tax; 41 models

  • LMArena text leaderboard, 30 September 2026

    Gemini 4 Argon (high)

    Human preference votes; leader marked preliminary; top ten within 31 points

Sources: Anthropic, Introducing Claude Opus 5.5 (22 September 2026), which also reports the GPT-6 Astra figure; Vals AI Terminal-Bench 4.0 leaderboard, mini-swe-agent harness, updated 29 September 2026; Artificial Analysis leaderboard; Vals AI Index; LMArena (arena.ai). All accessed 1 October 2026.

The referee changes the result. On Terminal-Bench 4.0, Anthropic reports 66.4% for Claude Opus 5.5 and Vals AI measures 65.15%. OpenAI's GPT-6 Astra is reported at 57.9% in Anthropic's launch table and measured at 59.6% by Vals AI. Claude Fable 5.1 scores 55.8% in Anthropic's table and 58.08% in Vals AI's run. Vendor tables can also show rivals ahead: Anthropic's own Opus 5.5 table reports GPT-6 Astra leading on Terminal-Bench-Science (64.6% versus 58.7%) and AutomationBench (41.4% versus 40.0%). Google reports 51.3% on AutomationBench for Gemini 4 Argon, measured in Google's own run. These are different runs and should not be merged into one ranking.

Effort settings move scores more than vendors do. Artificial Analysis scores Claude Opus 5.5 at 58 on its Intelligence Index at maximum effort and 51 at medium; GPT-6 Astra scores 53 at maximum and 50 at medium. Any comparison that does not state the effort level, tools and harness is incomplete.

Benchmarks saturate and leak. The 2026 AI Index reports that performance on SWE-bench Verified rose from 60% to near 100% in a year. In February 2026 OpenAI stopped treating it as a frontier measure: its audit of 138 hard problems found 59.4% had flawed tests or task descriptions, and models could reproduce solutions from a task identifier alone, a sign of training contamination. OpenAI recommended SWE-bench Pro instead.

Differences are often within noise. Anthropic reports a standard error of about 2.6 points on Terminal-Bench 4.0 for Opus 5.5. On LMArena's human-preference leaderboard, the top ten models on 30 September sat within 31 points, with Gemini 4 Argon's lead marked preliminary.

BenchmarkWhat it measuresWatch for
Terminal-Bench 4.0Agentic tasks in a command-line environmentHarness differences; vendor vs independent runs
GDPval-AAProfessional knowledge-work tasks, scored as EloGrader design; relative, not absolute, scores
Humanity's Last ExamExpert-level questions across disciplinesResults with and without tools are not comparable
AutomationBenchBusiness workflow automationDifferent vendors' runs; low absolute scores
OSWorldComputer use in a desktop operating systemPartial vs full task sets
LMArenaHuman preference in blind comparisonsStyle can win over accuracy; wide confidence intervals
SWE-bench VerifiedFixing real GitHub issuesSaturated and contaminated; no longer used by OpenAI

Independent aggregators reduce these problems but do not remove them. As of 30 September 2026, Artificial Analysis ranks Claude Opus 5.5 first, while the Vals Index and LMArena both rank Gemini 4 Argon first, a model most enterprises cannot yet buy.

Inference cost: price per task, not per token

List prices span more than two orders of magnitude.

Model (access)InputCached inputOutputNotes
GPT-6 Astra (OpenAI)10.001.0050.00Batch 50% off; higher rates above 272K input
GPT-6.1 Sol (OpenAI)2.000.1010.00Batch 50% off; above 272K, 2x input and 1.5x output
GPT-6 Luna (OpenAI)0.100.010.50Batch 50% off
Claude Fable 5.1 (Anthropic)10.000.2550.00Batch 50% off; cache writes 1.25x input
Claude Opus 5.5 (Anthropic)4.000.2020.00Batch 50% off; fast mode 8.00 / 40.00
Claude Sonnet 5.5 (Anthropic)2.000.2010.00Batch 50% off
Claude Haiku 4.5 (Anthropic)1.000.105.00Batch 50% off; 200K context
Gemini 3.1 Pro preview (Google)2.000.2012.00Up to 200K; 4.00 / 18.00 above; cache storage extra
Gemini 3.8 Flash (Google)0.750.0753.75Introductory; doubles on 1 January 2027
Gemini 3.1 Flash-Lite (Google)0.250.0251.50Text, image and video input rate
Gemini 4 Argon (Google, limited)2.000.1010.00Introductory; 4.00 / 20.00 afterwards
Muse Spark 1.3 (Meta)1.250.154.25Meta Model API
DeepSeek V4.1 Flash (DeepSeek)0.300.0061.20Peak rate; off-peak 50% lower
DeepSeek V4 Pro (DeepSeek)1.320.0443.96Peak rate; off-peak 50% lower

US dollars per million tokens, standard tier, as of 1 October 2026. Sources: OpenAI, Anthropic, Google, Meta and DeepSeek pricing pages. Cloud-route, regional and negotiated prices differ.

To show what this means in practice, we priced three hypothetical monthly workloads at list prices.

Figure 5Illustrative

Model choice moves the bill by two orders of magnitude

Monthly list-price token cost (USD) for three hypothetical workloads. Logarithmic scale.

A. Batch document extraction

1,000M input, 100M output tokens; batch discount where offered

  • GPT-6 Astra
    $7,500
  • Claude Fable 5.1
    $7,500
  • Claude Opus 5.5
    $3,000
  • Meta Muse Spark 1.3
    $1,675
  • Gemini 3.1 Pro (preview)
    $1,600
  • Claude Sonnet 5.5
    $1,500
  • GPT-6.1 Sol
    $1,500
  • Claude Haiku 4.5
    $750
  • Gemini 3.8 Flash
    $562
  • DeepSeek V4.1 Flash (peak)
    $420
  • Gemini 3.1 Flash-Lite
    $200
  • GPT-6 Luna
    $75

B. Internal knowledge assistant

500M input (70% cache hits), 50M output; real time

  • GPT-6 Astra
    $4,350
  • Claude Fable 5.1
    $4,088
  • Claude Opus 5.5
    $1,670
  • Gemini 3.1 Pro (preview)
    $970
  • Claude Sonnet 5.5
    $870
  • GPT-6.1 Sol
    $835
  • Meta Muse Spark 1.3
    $452
  • Claude Haiku 4.5
    $435
  • Gemini 3.8 Flash
    $326
  • Gemini 3.1 Flash-Lite
    $121
  • DeepSeek V4.1 Flash (peak)
    $107
  • GPT-6 Luna
    $44

C. Long-running agent

2,000M input (90% cache hits), 80M output; real time

  • GPT-6 Astra
    $7,800
  • Claude Fable 5.1
    $6,450
  • Claude Opus 5.5
    $2,760
  • Gemini 3.1 Pro (preview)
    $1,720
  • Claude Sonnet 5.5
    $1,560
  • GPT-6.1 Sol
    $1,380
  • Meta Muse Spark 1.3
    $860
  • Claude Haiku 4.5
    $780
  • Gemini 3.8 Flash
    $585
  • Gemini 3.1 Flash-Lite
    $215
  • DeepSeek V4.1 Flash (peak)
    $167
  • GPT-6 Luna
    $78
$10$100$1K$10K
INVISOM analysis using published standard-tier list prices as of 1 October 2026 (OpenAI, Anthropic, Google, Meta, DeepSeek pricing pages). Excludes cache-write charges, long-context surcharges (prompts are assumed under 200K tokens), tokenizer differences, reasoning-token variation, tools, negotiated discounts, VAT and cloud-route premiums. No batch discount applied to Meta or DeepSeek; DeepSeek off-peak rates are half the peak rates shown. Gemini 3.8 Flash uses introductory pricing that doubles on 1 January 2027.

The spread is wide: the same long-running agent workload costs about US$78 a month on GPT-6 Luna and about US$7,800 on GPT-6 Astra. But per-token prices understate how costs actually behave, for three reasons.

  1. Tokens are not equal. Each vendor tokenizes text differently; Anthropic notes that its tokenizer for Claude 4.7 and later produces about 30% more tokens for the same text than its predecessor. A cheaper per-token price can still produce a larger bill.
  2. Reasoning tokens are billed as output, and their volume depends on the effort setting and the model.
  3. Agents differ in how many steps they take. On Vals AI's Terminal-Bench 4.0 run, average cost per task was US$9.58 for GPT-6 Astra, US$13.20 for Claude Opus 5.5 and US$16.51 for Claude Sonnet 5.5. Sonnet 5.5 has half Opus 5.5's per-token price but cost more per task in that run.

Measure cost per completed task on your own workload. Also budget for change: Google has announced that Gemini 3.8 Flash prices double in January 2027, Anthropic dropped a planned price rise for Claude Sonnet 5, and OpenAI renamed its priority tier in July 2026. Lifecycles are short too: Anthropic's retirement commitment for Claude Haiku 4.5 runs only to 15 October 2026.

Self-hosting changes the cost model from tokens to capacity: GPUs (owned or reserved), serving software, engineers, monitoring and evaluation. It tends to pay off only with high, steady utilization or when residency leaves no alternative. Model the break-even with your own utilization figures; published hardware or token prices will not do it for you.

A model-selection scorecard

Score each shortlisted model on your own evidence. Hard requirements act as gates: a model that fails residency, licensing or security requirements is excluded whatever its score.

CriterionSuggested weightEvidence required
Task quality on your evaluation set30%Pass rate on real tasks, with confidence intervals, at a stated effort setting
Cost per completed task15%Measured tokens, retries and reasoning, at your route's price
Deployment route, residency and security15%Regions, retention terms, identity integration, certifications, in writing
Tool use and structured output10%Schema adherence and tool-call error rates on your tools
Latency and throughput10%Median and 95th-percentile latency measured on your route
Lifecycle and vendor stability10%Retirement commitments, version pinning, history of price changes
Licence, transparency and legal terms10%Licence, IP indemnity, data-use terms, disclosure of changes

INVISOM framework. Adjust weights by use case: a customer-facing assistant weights latency higher; a regulated workflow weights residency and auditability higher.

The scorecard only works if the evidence behind it is produced the same way for every candidate. That is the job of the evaluation pipeline.

Figure 6

Your own evaluation set is the only benchmark that transfers

A six-stage pipeline that keeps model choice evidence-based and reversible

  1. Stage 1

    Define

    Write down the task, the users, what a correct answer looks like and what failure costs. Collect real examples with known good outputs.

  2. Stage 2

    Screen

    Use public and independent leaderboards to build a shortlist of three to five models across at least two vendors. Treat them as a floor check only.

  3. Stage 3

    Test offline

    Run every candidate on your own set in the same harness. Record model version, date, prompt, reasoning effort and tools for each run.

  4. Stage 4

    Price per task

    Measure tokens, latency and cost per completed task, including retries and reasoning tokens, not price per token.

  5. Stage 5

    Stress

    Red-team for prompt injection, data leakage, refusals and language coverage, including Arabic where relevant.

  6. Stage 6

    Shadow and monitor

    Run in shadow or canary mode, then monitor quality and cost in production. Re-run the suite on every model or prompt change.

↻Repeat from stage 3 whenever a provider ships a new version, changes a price or retires a model.

INVISOM framework, informed by benchmark-methodology caveats from OpenAI (SWE-bench Verified audit, 2026), Stanford HAI AI Index 2026 and independent evaluators.

Two design choices make the pipeline durable. Keep the evaluation set and harness independent of any single vendor's tooling, so the comparison is fair. And put a thin routing layer between applications and models, so a better or cheaper model can be swapped in after it passes the suite. The use-case selection discipline that decides what to build should also decide which tasks go into the set.

The Gulf view: residency first, then language

For organizations in Saudi Arabia and the UAE, two questions often narrow the field before capability does.

Where does inference run? Options vary by provider and route. OpenAI offers data residency and inference residency in the UAE for eligible ChatGPT business customers. Claude models can be called on Amazon Bedrock from the UAE and Bahrain regions through global cross-Region inference, which keeps data at rest in the source region while inference may run elsewhere. Anthropic's own API offers US or global inference. Where personal or classified data must be processed in-country, an open-weight model on in-country infrastructure is often the only option that satisfies the requirement. Our NDMO implementation guide covers the classification rules that drive this.

How well does it handle Arabic? The region has its own models: Jais 2, a 70-billion-parameter open-weight model trained from scratch, and ALLaM and Falcon Arabic, built by continued pretraining of existing models. Global frontier models also handle Modern Standard Arabic well in many tasks. Dialects, tone and domain terminology are where differences appear, so Arabic test cases belong in the evaluation set from the start. A separate INVISOM analysis covers Arabic-first models in depth.

What to do in the next 90 days

  1. Pick two or three deployment routes, not one model. Typically a managed-cloud default, a direct API for frontier features, and an open-weight option for residency-bound or high-volume work.
  2. Build an evaluation set from real work: a few hundred tasks with known good outcomes per priority use case, including Arabic where relevant.
  3. Run a structured bake-off across at least two proprietary vendors and one open-weight model, recording model version, date, effort setting and tools for every run.
  4. Report cost per completed task and 95th-percentile latency, not price per token, to whoever approves the budget.
  5. Put a routing layer in place so applications call a capability, not a model name, and pin model versions in production.
  6. Get residency, retention and licence terms in writing for each route, model by model.
  7. Set a re-evaluation trigger: every major model release, price change or retirement notice, and at least quarterly.

Frequently asked questions

Which LLM is best for enterprises in 2026?

No single model is best. As of 1 October 2026, Artificial Analysis ranks Claude Opus 5.5 first, while the Vals Index and LMArena rank Google's Gemini 4 Argon first, and GPT-6 Astra leads some vendor-reported benchmarks. Choose by testing on your own tasks, cost per task and deployment route.

How much do enterprise LLMs cost per million tokens?

As of 1 October 2026, list prices range from US$0.10 input and US$0.50 output (GPT-6 Luna) to US$10 input and US$50 output (GPT-6 Astra, Claude Fable 5.1). Cached input, batch discounts, long-context surcharges and reasoning tokens change real costs substantially.

Are open-weight models good enough for enterprise use?

For many tasks, yes. Stanford's AI Index found the open-closed gap narrowed to 1.7% on some benchmarks. DeepSeek, Qwen, OpenAI's gpt-oss, Meta's Muse Glimmer and Mistral all publish open-weight models. Check licences, hardware needs and your own evaluation results.

Why can't we rely on published LLM benchmarks?

Scores depend on who runs the test, the effort setting and the harness, and popular benchmarks saturate or leak into training data. OpenAI stopped using SWE-bench Verified in 2026 after finding flawed tests and contamination. Use public benchmarks to shortlist, then test on your own tasks.

Sources and further reading

Work with INVISOM

Build a vendor-neutral LLM evaluation and selection programme

Start a conversation

Related Insights

Continue the Topic

View all insights

Knowledge IntelligenceComparison

Arabic-First LLMs vs Global Models

Regional models win on dialect, control and cost per Arabic document, not by default. Public leaderboards mostly test exam-style standard Arabic, so the choice belongs to an evaluation set built from your own users.

Sep 27, 2026 · 16 min read

Knowledge IntelligenceComparison

RAG, Fine-Tuning or a Custom LLM?

Retrieval owns facts. Tuning owns behaviour. Almost nobody should train their own model. And in 2026 the market itself has started making the decision for you.

Aug 16, 2026 · 10 min read

GovernanceImplementation Guide

AI Data Residency in KSA and the UAE

Hosted in the region is not the same as processed in the region. For most frontier models, a Gulf deployment today stores your data locally and sends your prompts abroad, so residency has to be decided per workload and per feature.

Sep 29, 2026 · 15 min read