Evaluation of a Foundation Model

1. What is a foundation model?

A foundation model is a model trained on broad data, usually at large scale, that can be adapted to many downstream tasks. The adaptation may happen through prompting, retrieval-augmented generation, tool use, fine-tuning, preference optimization, adapters, or by placing the model inside a larger agentic system.

The important word is foundation: the model is not designed for only one narrow prediction problem. It provides general representations and capabilities that can support many applications. A language foundation model may summarize, translate, write code, answer questions, classify documents, operate software tools, and participate in multimodal workflows. Vision, audio, scientific, robotics, and multimodal foundation models follow the same broad idea.

A model is not the entire AI product

A production AI system normally contains more than the neural network. It may also contain a system prompt, search or retrieval, tool definitions, memory, safety filters, a model router, retry logic, a code execution sandbox, document parsers, and user-interface rules. Therefore, an excellent model can perform poorly inside a weak application harness, while a smaller model can perform remarkably well inside a carefully engineered system.

2. Examples in the current model landscape

The following examples illustrate the diversity of foundation-model releases. They are not a permanent ranking. Model families, aliases, prices, and availability change frequently, so the linked provider documentation should be treated as the current source of truth.

3. Closed, open-weight, and fully open models

The phrase “open-source model” is frequently used too loosely. In conventional software, source code is the preferred form for modification. In machine learning, the final weights are important, but they are only one part of the system. A serious evaluation should distinguish at least three release patterns.

Why the distinction matters

Open weights enable local deployment, quantization, fine-tuning, controlled versioning, and deeper inspection. They do not automatically reveal why the model behaves as it does, what copyrighted or sensitive data may have been used, how the data were filtered, or whether the model can be recreated. Fully open releases provide much stronger scientific transparency, but they may still trail closed frontier systems on some capabilities or require substantial engineering effort to deploy efficiently.

A licence must be evaluated separately from technical availability. A model may be downloadable yet impose restrictions on commercial use, redistribution, user scale, specific applications, or derivative models. “Available on Hugging Face” is not the same as “unrestricted open source.”

4. For an open model, where are the code, algorithm, and weights?

There is rarely one file called “the algorithm.” The model is distributed as a collection of artifacts. The architecture specifies the mathematical structure; the weights contain the learned numerical values; the tokenizer converts text into tokens; training code produces the checkpoint; and inference code executes it efficiently.

Files commonly found in a model repository

A repeatable inspection method

1. Start at the provider’s official release page or verified Hugging Face organization.
2. Read the model card before downloading anything.
3. Check the exact licence and whether the release is base, instruct, reasoning, multimodal, or task-specific.
4. Open config.json to inspect the architecture and parameterization.
5. Inspect the “Files and versions” tab for checkpoint shards, tokenizer files, and revisions.
6. Follow links to the technical report and source repository.
7. Look for training scripts, data manifests, optimizer settings, intermediate checkpoints, logs, and evaluation harnesses.
8. Record the commit hash or model revision used in your evaluation.

Example: loading and inspecting a Hugging Face model

from transformers import AutoConfig, AutoTokenizer
from huggingface_hub import model_info

MODEL_ID = "organization/model-name"

info = model_info(MODEL_ID)
config = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=False)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)

print("Revision SHA:", info.sha)
print("Architecture:", getattr(config, "architectures", None))
print("Hidden size:", getattr(config, "hidden_size", None))
print("Layers:", getattr(config, "num_hidden_layers", None))
print("Attention heads:", getattr(config, "num_attention_heads", None))
print("Vocabulary size:", tokenizer.vocab_size)

5. Anatomy of a model release

When a provider announces a new model, the announcement usually combines several layers that should be evaluated separately.

Two services can host the same open-weight checkpoint yet deliver different latency, output quality, maximum context, concurrency, and cost. Similarly, a provider may silently update a model alias while retaining the same friendly name. For reproducibility, prefer versioned model IDs and pin exact checkpoint revisions.

6. What are parameters and weights?

A parameter is a numerical value inside the neural network that can be adjusted during training. Weights are the major class of learned parameters that control how strongly one representation influences another. Biases, normalization scales, embedding tables, routing values, and other learned tensors are also parameters.

During pretraining, the model repeatedly predicts missing or next tokens, compares its predictions with the training target, calculates an error, and updates parameters using gradient-based optimization. After many updates, the final collection of values is stored as a checkpoint. Inference uses those learned values to transform an input sequence into probabilities over possible outputs.

What does “a 1 billion parameter model” mean?

It means the model contains approximately one billion learned scalar values. It does not mean the model stores one billion facts, has one billion lines of code, or has a directly measurable amount of intelligence. Parameter count describes scale, but capability also depends on architecture, training data, number of training tokens, optimization, post-training, context design, and inference-time computation.

Dense models versus mixture-of-experts models

In a dense model, nearly all parameters participate in each token’s forward pass. In a mixture-of-experts model, a router selects only a subset of expert blocks for each token. Therefore, an MoE release may advertise both total parameters and active parameters. Total parameters mainly influence storage and aggregate capacity; active parameters more strongly influence the computation required per token. They are not interchangeable measures.

For example, a model described as “235B total / 22B active” stores roughly 235 billion parameters, while the routing mechanism activates a much smaller subset for a token. Its memory footprint can resemble a very large model, while its per-token compute can be closer to a smaller dense model—although communication, routing, cache, and parallelism overhead still matter.

7. How much memory does a model need?

The simplest lower-bound estimate is parameter count multiplied by bytes per stored parameter. This estimates weight memory only.

Actual inference memory is higher because the runtime also needs temporary activations, CUDA or accelerator kernels, allocator headroom, attention workspaces, the KV cache, and sometimes duplicated buffers. Quantization metadata and dequantization work also add overhead. A practical deployment should therefore leave headroom instead of matching the theoretical weight size exactly.

The KV cache

Autoregressive generation stores attention keys and values for previous tokens so they do not need to be recomputed at every decoding step. The KV cache grows with sequence length and batch size. Long contexts and many concurrent users can therefore exhaust GPU memory even when the weights fit comfortably.

Grouped-query attention and multi-query attention reduce KV-cache size by using fewer key/value heads than query heads. This is one reason architecture matters independently of parameter count.

Training requires far more memory than inference

Full training stores weights, gradients, optimizer states, activations, and communication buffers. A basic Adam-style optimizer can require several times the raw parameter memory before activation memory is counted. Distributed training techniques such as tensor parallelism, pipeline parallelism, data parallelism, ZeRO, sharding, checkpointing, and mixed precision are used because frontier models cannot fit on one accelerator.

8. How much training was done?

“How large is the model?” and “how much was it trained?” are different questions. Parameter count measures the size of the learned state. Training amount describes the data and compute used to produce that state. A model release should ideally provide enough information to understand both.

Training information worth recording

Closed frontier providers commonly disclose only part of this information. The absence of disclosure should not be filled with guesses. Record it as not disclosed and treat transparency as a separate evaluation dimension. Open-weight releases also vary considerably: some publish only the final checkpoint, while projects such as OLMo intentionally publish data, code, recipes, logs, and intermediate checkpoints.

9. Base, instruct, reasoning, and agentic variants

A model family often contains several variants that should not be compared as though they were identical.

Reasoning models require special care. A score obtained with maximum reasoning effort and a large token budget should not be compared directly with a fast default mode. Record the reasoning setting, output-token count, number of attempts, tool access, and total cost.

10. Evaluation criteria in priority order

The correct priority order begins with the actual use case, not with public leaderboards. The following order is a strong default for enterprise and engineering evaluations. The weights can be adjusted, but the ordering prevents teams from optimizing cost or benchmark scores before establishing usefulness and safety.

1. Task success on representative work. Build a private evaluation set from real prompts, documents, languages, edge cases, and failure modes. Measure complete task success rather than surface fluency.
2. Reliability, factuality, and grounding. Measure unsupported claims, citation correctness, consistency across repeated runs, uncertainty calibration, and the ability to say that evidence is insufficient.
3. Safety, security, privacy, and compliance. Evaluate prompt injection, data leakage, unsafe tool calls, policy compliance, access controls, retention, residency, auditability, and regulatory obligations.
4. Instruction following and controllability. Test format compliance, system-prompt adherence, structured output, tool-use rules, tone, verbosity, and resistance to conflicting instructions.
5. Latency and throughput under realistic load. Record time to first token, generation speed, end-to-end latency, p95 and p99 tails, concurrency, queueing, and cold starts.
6. Total cost per successful task. Include input and output tokens, reasoning tokens, retries, retrieval, tool calls, hosting, GPUs, engineering time, and human review—not merely list price per million tokens.
7. Context, retrieval, and tool-use quality. Test the lengths and document types you actually use. Measure information retrieval, multi-document reasoning, context dilution, and tool-selection accuracy.
8. Deployment control and ecosystem fit. Consider licence, self-hosting, regional availability, fine-tuning, observability, version pinning, framework support, quantization, and vendor lock-in.
9. Transparency and reproducibility. Assess model cards, technical reports, system cards, data disclosure, evaluation harnesses, checkpoint hashes, and change logs.
10. Public benchmark evidence. Use reputable benchmarks as supporting evidence and diagnostic clues, not as the final selection decision.

A suggested weighted score

11. Standard and reputable benchmark families

No single benchmark measures “intelligence.” A defensible evaluation uses a portfolio selected for the intended capabilities, then validates the findings on private tasks. The table below emphasizes benchmarks with public papers, code, leaderboards, or transparent methodologies.

Benchmarks should be grouped by purpose

12. Inference and serving metrics

Inference metrics describe what happens when a trained model is used to generate outputs. They depend on the checkpoint, precision, hardware, serving engine, prompt length, output length, batch size, concurrency, and network path. A single “tokens per second” number without these conditions is not meaningful.

Measure under a latency–throughput curve

Serving systems trade individual-user latency against aggregate throughput. Larger batches improve accelerator utilization and throughput but can increase queueing and inter-token delays. Report several concurrency points rather than one best-case number.

13. Building a benchmark report card

A report card should make test conditions visible and prevent a high score in one area from hiding unacceptable failures elsewhere. Separate externally reported benchmark numbers from your own measurements.

Report-card header

Illustrative report card — hypothetical values

In this hypothetical example, Model C has the highest total because speed and economics matter to the chosen workload, even though Model A has the highest raw quality. Model B remains strategically attractive where self-hosting, inspection, customization, or data control outweighs some capability loss. The report card makes the trade-off explicit rather than pretending that one model is universally best.

Examples of real release documentation

• Anthropic Claude Opus release documentation and system cards
• xAI Grok release documentation and model catalogue
• DeepSeek model cards and technical reports
• OpenAI model release documentation
• Google Gemini model documentation
• Ai2 OLMo artifacts and technical reports

14. A practical model-evaluation workflow

Step 1 — Define the decision

State the application, user population, risk level, languages, data sensitivity, expected volume, latency target, budget, deployment constraints, and what a successful answer or action looks like.

Step 2 — Build a private evaluation set

Collect representative tasks from real work. Include normal cases, difficult cases, ambiguous requests, adversarial instructions, long documents, multilingual examples, missing information, and tasks for which the correct action is to abstain or ask a question. Remove personal or confidential data unless the evaluation environment is approved for it.

Step 3 — Create an evaluation contract

Freeze the system prompt, tool definitions, retrieval configuration, sampling settings, output schema, reasoning effort, maximum tokens, retry policy, and scoring rubric. Without a contract, teams unintentionally tune each model differently and invalidate the comparison.

Step 4 — Establish baselines

Include the current production system, a simple non-AI method where applicable, one strong closed model, one economical model, and one self-hostable model. Baselines reveal whether increased complexity produces meaningful value.

Step 5 — Run capability and behavioral tests

Use private task evaluations first, followed by selected public benchmarks. Run repeated trials where outputs are stochastic. Measure task success, groundedness, instruction compliance, safety, and consistency separately.

Step 6 — Benchmark the full system

Test realistic prompt lengths, output lengths, concurrency, retrieval, tool use, network conditions, and failure recovery. Measure latency distributions and error rates rather than only averages.

Step 7 — Normalize economics

Calculate cost per accepted task. Include retries, long reasoning traces, cached input, vector search, tool calls, human review, and self-hosting infrastructure. A cheaper token price can still yield a more expensive completed task.

Step 8 — Perform security and governance review

Review data handling, retention, region, provider terms, licence, access controls, logging, prompt injection, tool permissions, supply-chain risk, model provenance, and incident procedures.

Step 9 — Pilot with shadow traffic

Run candidate models on sampled real traffic without allowing them to take irreversible actions. Compare against the production outcome and collect domain-expert judgments.

Step 10 — Select, version, and monitor

Pin model versions where possible, establish regression gates, monitor quality and latency, and schedule re-evaluation when the provider updates an alias, changes pricing, retires a model, or releases a materially different version.

15. Common benchmark and model-comparison errors

16. Where to check when new models are released

Use a layered monitoring strategy. Provider documentation establishes what was released; independent evaluations show how it compares; reproducible frameworks help verify claims; and model repositories expose artifacts for inspection.

Primary provider sources

• Anthropic: model overview, news, and system cards
• OpenAI: model documentation, API changelog, and release posts
• Google: Gemini model catalogue and Google DeepMind news
• xAI: model documentation and release news
• DeepSeek: transparency centre and API change log
• Meta Llama: official site, Meta AI blog, and model repositories
• Qwen: project blog and model repositories
• Mistral: model overview, news, and model repositories
• Ai2 OLMo: project page and source repository

Independent and reproducible comparison sources

17. Final selection checklist

• The exact model version or checkpoint revision is recorded.
• The tested object—raw model, API, chat product, or complete application—is clearly identified.
• The release type is described accurately as closed, open-weight, partially open, or fully open.
• The licence and commercial-use restrictions have been reviewed.
• Parameter count, active parameters for MoE models, precision, and quantization are recorded.
• Weight memory, KV-cache growth, concurrency, and hardware headroom have been estimated.
• Training data, token count, compute, and post-training disclosure are recorded as published or not disclosed.
• Representative private tasks are the highest-weight part of the evaluation.
• Public benchmarks are selected by capability and their versions and harnesses are recorded.
• Reasoning effort, tools, search, retries, and token budgets are aligned across candidates.
• TTFT, TPOT or ITL, end-to-end latency, throughput, tail latency, and errors are tested under realistic load.
• Economics are calculated per successful task, including retries, tools, hosting, and human review.
• Groundedness, citation quality, calibration, safety, security, privacy, and prompt injection are tested.
• The selected model passes a pilot or shadow-traffic stage before receiving consequential permissions.
• Regression tests and a re-evaluation trigger are defined for model updates, alias changes, and retirements.

Further reading and live resources

Foundation models and transparency

• Stanford CRFM — On the Opportunities and Risks of Foundation Models
• Open Source Initiative — Open Source AI Definition
• Open Source Initiative — Open Weights versus Open Source AI
• Stanford Foundation Model Transparency Index

Model repositories and inspection

• Hugging Face — Model cards
• Hugging Face — Model repositories and files
• Hugging Face Hub — Downloading and pinning repository files
• Hugging Face Accelerate — Model memory estimator
• Hugging Face Transformers — Quantization overview

Evaluation and inference

• Stanford HELM
• EleutherAI language-model evaluation harness
• vLLM serving metrics
• NVIDIA LLM inference metric definitions
• MLCommons benchmark suites