Large Language Models

Large language models are deep learning models, usually transformer-based, trained to predict tokens from large corpora. After pretraining, they are often instruction-tuned and aligned so they can follow prompts and assist users.

How can LLM hallucinations be evaluated?

LLM hallucinations are evaluated by checking generated claims against reliable evidence and counting how often the model invents, contradicts, or fails to support information. A good evaluation distinguishes factual correctness (is the claim true?) from faithfulness (does the supplied context support it?). There is no universal hallucination metric: the choice depends on whether the assistant answers factual questions, summarizes documents, or uses a retrieval system. 1. Create a reliable test set Collect representative user questions, model outputs, and, where relevant, authoritative source documents . Include straightforward questions, ambiguous prompts, missing information cases, outdated documents, and difficult edge cases. Decide in advance whether an answer is allowed to use outside knowledge or must use only the supplied context . Otherwise, a true but uncited statement might be mislabeled as false. 2. Evaluate individual factual claims Instead of scoring an entire paragraph as simply correct or incorrect: 1. Break the response into atomic, independently checkable claims . 2. Find evidence for each claim. 3. Label each claim supported, contradicted, or not verifiable under the evaluation rules. 4. Report the claim level and answer level results separately. FActScore follows this fine grained approach, measuring the fraction of atomic facts supported by reliable knowledge sources. It is especially useful for long form factual generation . 3. Evaluate faithfulness to retrieved context For a RAG or document grounded assistant, check whether the response's claims follow from the retrieved passages . Ragas Faithfulness uses the proportion of generated claims supported by retrieved context. A model can be faithful but factually wrong if the supplied source itself is outdated or incorrect. Conversely, an answer can be factually correct but unfaithful to a strict source only task. 4. Numerical example Suppose an assistant generates 5 answers containing 20 atomic factual claims . Reviewers find: 15 supported claims 3 contradicted claims 2 claims not verifiable from the required sources For this source grounded evaluation: Suppose those five unsupported claims appear across 3 distinct answers : These results measure different things . Never call the 25% claim level rate a 60% answer level rate, or treat a lack of source support as proof of factual falsehood. 5. Other evaluation methods Human review: Trained annotators verify claims and resolve ambiguous cases using a shared rubric. Double review a sample and track disagreement. LLM as a judge: A second model identifies claims or judges their evidential support. It can scale reviews, but may itself hallucinate or show systematic bias. Calibrate it using expert reviewed examples. Automated fact verification: Retrieval plus entailment or natural language inference checks can help flag unsupported statements. Verify citations and source relevance. Specialized benchmarks: HalluLens tests particular hallucination behaviors using multiple task types. Benchmarks complement, rather than replace, application specific tests. Abstention tests: Ask questions whose answers are missing from the evidence. Check whether the model appropriately says it cannot determine the answer instead of guessing. Practical evaluation workflow Preserve the model version, prompts, retrieved documents, sampling settings, rubric, and evaluator version. Sample enough examples for meaningful conclusions and report uncertainty when comparing systems. Common mistakes Treating any unsupported claim as certainly false: It may be true but ungrounded. Using only one score: Factuality, faithfulness, citation accuracy, abstention, and answer level failure can disagree. Trusting model judges blindly: Automatic graders need human validation. Ignoring retrieval quality: A bad source can make an otherwise capable generator appear unreliable. Testing only easy questions: Include unanswerable, misleading, and production like cases. In short: Evaluate hallucinations by breaking answers into checkable claims, verifying them against the right evidence, and reporting clearly defined error rates . Review the examples behind the scores to understand whether failures come from retrieval, unsupported generation, or false facts. For background, see “What is hallucination in an LLM?” and “What are the different types of LLM hallucinations?” Research references: FActScore: Fine grained Atomic Evaluation of Factual Precision, Ragas Faithfulness metric, and HalluLens: LLM Hallucination Benchmark.

What is factuality in LLM evaluation?

Factuality in LLM evaluation means how accurately a model's statements agree with verifiable facts about the world . A factual answer makes claims that reliable evidence supports; a nonfactual answer may contain false dates, invented citations, or incorrect numbers. Factuality is not the same as fluency (how natural a sentence sounds), confidence (how certain it sounds), or faithfulness (whether it follows a provided document). Factuality vs. faithfulness Suppose a company document incorrectly says the return period is 45 days , although the current official policy is 30 days . “The period is 45 days” is faithful to that outdated document but factually incorrect about the current policy. “The period is 30 days” is factually correct but not supported by that document alone. This distinction matters for RAG systems: grounding an answer in a source does not guarantee that the source is true or current. How do you measure factuality? There are two common task formats. 1. Short, verifiable questions: Provide questions with trusted reference answers, such as the capital of a country or a dated historical event. Grade model responses as correct , incorrect , or not attempted . SimpleQA uses this style of short form factuality evaluation; an appropriate abstention can be preferable to an invented answer. 2. Long form answers: Split a response into atomic factual claims and verify them individually against reliable sources. FActScore measures factual precision using the proportion of atomic claims supported by trustworthy evidence. A different method may be needed for an opinion, creative writing request, or an unanswerable question. Avoid forcing subjective claims into true/false categories. Beginner friendly numerical example Imagine evaluating 10 short factual questions : The model answers 7 correctly . It answers 2 incorrectly . It says “I don't know” once. Using clearly stated denominators: The 70% figure counts unanswered questions as not correct. The 77.8% figure measures accuracy only when the model attempts an answer . Reporting both is useful: a model might improve attempted accuracy simply by answering fewer difficult questions. For a long response with 8 checkable claims , if 6 are verified , its illustrative claim level factual precision is: An actual evaluation must specify how ambiguous, unsupported, or undecidable claims are handled. Do not count an unverified claim as definitely false without appropriate evidence. Practical factuality evaluation workflow 1. Choose representative questions: Include easy and difficult cases from the intended application, not just public trivia. 2. Establish trusted references: Use reliable, sufficiently recent sources; preserve source dates for facts that can change. 3. Run the model consistently: Keep prompt templates, retrieval tools, and generation settings controlled. 4. Score answers appropriately: Short answers may use answer level correctness; longer responses may need claim level verification. 5. Track abstentions separately: Measure coverage (questions answered) alongside correctness among attempted answers. 6. Audit borderline cases: Use human reviewers to check whether evidence truly supports the claim and update the dataset when references become outdated. Common mistakes Equating factuality with faithfulness: A perfectly quoted but outdated source can still lead to a factually wrong answer. Equating eloquence with truth: A fluent answer may contain invented details. Reporting only attempted accuracy: A system that refuses most questions can appear unusually accurate among its few attempts. Using a single automatic judge uncritically: Verify the grader against expert reviewed examples. Treating every claim without evidence as false: Some claims are undecidable given available sources. Assuming public benchmark scores generalize: Production questions may be more specialized or time sensitive. Useful factuality benchmarks SimpleQA: Short form, fact seeking questions with verifiable answers and explicit abstention handling. FActScore: Atomic claim factual precision for long form generation. FactBench: Evidence based evaluations on diverse prompts, including supported, unsupported, and undecidable content. See “How can LLM hallucinations be evaluated?” for the broader hallucination evaluation workflow. A later question, “What is LLM calibration?” , explores whether model confidence corresponds to real accuracy. In short: Factuality asks “Are the model's factual claims true?” Measure it using reliable external evidence, appropriate scoring rules, and separate reporting of errors and abstentions. References: OpenAI SimpleQA, SimpleQA paper, FActScore, and FactBench.

What is LLM calibration?

LLM calibration describes whether a model's estimated confidence matches its observed probability of being correct . A model is well calibrated if, across many questions for which it reports 80% confidence , approximately 80% of its answers are correct . Accuracy and calibration are different: A model can be accurate but overconfident, or less accurate yet reasonably calibrated. A beginner friendly example Suppose an LLM answers 100 independently reviewed factual questions and reports 90% confidence for every answer. If 90 answers are correct, its confidence is well aligned for that group . If 60 answers are correct, it is overconfident by 30 percentage points . If 98 answers are correct, it is underconfident by 8 percentage points . Calibration is a statistical property across many predictions , not a promise that an individual "90% confident" answer is right. What confidence are we measuring? In LLMs, a confidence estimate might come from: A verbalized number : "I'm 80% confident my answer is correct." A model generated uncertainty label : "likely" or "uncertain." A probability computed from model scores/logits , with a task specific method for converting them to answer correctness . An external confidence estimator trained on validation examples. Important: The probability assigned to an individual next token is not automatically the probability that the whole answer is factually correct. Likewise, fluent wording and an assertive tone are not calibrated confidence estimates . Research finds that linguistic and model internal confidence can differ. How is calibration evaluated? 1. Reliability diagram: Group answers by reported confidence and compare their average confidence to the fraction actually correct. Perfectly calibrated groups sit close to the diagonal. 2. Expected Calibration Error (ECE): Compute a weighted average of the absolute gap between accuracy and confidence in each group: For example, if a bin holds 40% of questions , with mean confidence 80% and accuracy 60% , that bin contributes: That is 8 percentage points toward overall ECE. To obtain full ECE, add contributions from all bins. ECE depends on binning choices and can conceal subgroup errors. 3. Brier score: For a binary correct/incorrect outcome, average the squared error between estimated correctness probability p and actual outcome y (1 for correct, 0 for incorrect): Lower is better for the standard binary Brier score. It evaluates the quality of probabilistic forecasts, not calibration alone. Why does calibration matter in practice? Imagine a support assistant giving answers about account policies: With high estimated confidence, it may provide a factual answer subject to normal safeguards . With uncertain answers, it can request additional evidence, search a source, abstain, or route to human review . Teams can tune these thresholds based on acceptable errors and the cost of escalation. A calibrated 70% estimate means roughly 30 out of 100 similarly scored predictions may still be wrong . It does not make any specific response safe without verification. How can calibration be improved? Collect held out answers with correctness labels that represent the production workload. Measure confidence versus actual correctness before changing the system. Apply post hoc calibration (for example, temperature scaling or isotonic regression) to a suitable confidence signal, validating on a separate held out set. Train a confidence estimator or improve uncertainty expression where appropriate. Reassess after deployment changes to the model, prompting, retrieval, or domain. Calibration can worsen under distribution shift ; a reliability curve from trivia questions may not apply to legal document extraction. Common mistakes Confusing token probability with truth: High likelihood for generated text does not prove factual correctness. Mistaking confidence wording for probability: "Definitely" or "I think" needs empirical validation. Reporting ECE alone: Check accuracy, Brier score, sample size, and calibration across relevant slices. Overfitting the calibration set: Evaluate the final calibrator on different held out questions. Treating calibration as hallucination prevention: Better uncertainty estimates can support safer decisions but do not fix missing or false facts directly. In short: Calibration asks "When the model says it is 80% likely to be right, is it actually right about 80% of the time?" Evaluate this across many labeled answers, rather than trusting the tone of any individual response. For the distinction between true answers and unsupported answers, see "What is factuality in LLM evaluation?" Research references: How Can We Know When Language Models Know?, Reducing Conversational Agents' Overconfidence Through Linguistic Calibration, Linguistic Calibration of Long Form Generations, and ConfidenceBench.

What are the main limitations of Large Language Models?

Large Language Models (LLMs) are powerful at generating and transforming language, but they do not guarantee factual accuracy, robust reasoning, up to date knowledge, or safe decisions. Their limitations depend on the model, task, context, deployment setup, and available tools. 1. Incorrect or invented information LLMs can produce plausible but false claims, including invented citations or policy details. Confident wording is not proof of correctness . See “What is hallucination in an LLM?” for the dedicated explanation. 2. Knowledge gaps and changing information A model's learned parameters do not automatically track new regulations, product prices, current events, or revised company policies. Even models using search or RAG can retrieve outdated material or misunderstand it. Example: An assistant trained on an old reimbursement policy gives yesterday's limit instead of today's. Mitigation: Retrieve the current, authoritative policy and validate the relevant date. 3. Limited and imperfect use of context A model can process only a bounded amount of input in one request. Importantly, having room for a long document does not mean using every part reliably : the Lost in the Middle study found that performance can vary with the location of relevant evidence in long inputs. Example: In a 150 page contract, the model misses a crucial exception buried among repetitive clauses. Mitigation: Targeted retrieval, structured extraction, and testing with relevant facts at different positions. 4. Unreliable multi step reasoning and arithmetic Models may make mistakes in calculations, logic chains, counterfactuals, or detailed planning, even when an individual step is simple. They can also follow an early mistaken assumption through the rest of an answer. Example: An invoice has 3 items at ₹400 each plus 18% tax. The correct amount is ₹1,416 , but a model might give a different figure. Mitigation: Use a calculator, programmatic checks, and unit tests instead of relying on free form generation for exact arithmetic. 5. Sensitivity to wording and formatting Two semantically equivalent prompts can receive different answers. Instruction following, structured output, and long sequences of constraints can also fail. Example: An API assistant returns valid JSON for one prompt but adds explanatory prose when the prompt is rephrased. Mitigation: Schema constrained outputs, validation, and tests with prompt variations. 6. Bias and uneven performance Training data and training objectives can reproduce stereotypes or produce uneven results across languages, cultures, populations, and specialized domains. Mitigation: Measure relevant subgroups and real use cases, involve domain reviewers, and monitor deployment performance. 7. Privacy, security, and misuse risks Models can reveal sensitive information included in prompts or tool outputs if applications are poorly designed. External content can also contain prompt injection , attempting to redirect the assistant away from authorized instructions. Mitigation: Data minimization, access controls, tool permissions, output validation, and human approval for consequential actions. 8. Cost, latency, and operational constraints Long prompts, high concurrency, and generation of many tokens consume compute, memory, time, and money. A model that works in a demo may not meet production latency or throughput targets. Example: A support bot needs answers in under 2 seconds but exceeds its target on long prompts. Mitigation: Measure TTFT and end to end latency, then consider shorter context, smaller models, caching, batching, or optimized serving. 9. Lack of guaranteed consistency and accountability Sampling and changes to prompts or models can change answers. Even deterministic settings do not eliminate factual or reasoning errors. For medicine, finance, legal matters, or production changes, an LLM should not be the sole authority without suitable oversight. How should you assess these limitations? Test the actual application rather than treating a leaderboard score as proof of reliability: Factuality: Verify important claims against trusted sources. Task success: Test realistic, difficult, and unanswerable requests. Robustness: Rephrase prompts and vary document length and evidence position. Safety/security: Test private data handling and untrusted inputs. Operations: Track latency, throughput, cost, and failures. Key distinction: An LLM's limitations are not all fundamental impossibilities . Retrieval, specialized tools, stronger training, better interfaces, and verification can improve many outcomes, but they do not guarantee perfection. In short: Use LLMs as adaptable language processing components; add reliable evidence, deterministic tools, validation, and oversight wherever correctness matters. Research references: NIST Generative AI Risk Management Profile and Lost in the Middle: How Language Models Use Long Contexts.

Why do LLMs sometimes fail at seemingly simple tasks?

LLMs can solve advanced tasks yet fail at apparently simple ones because success on familiar language patterns does not guarantee consistent generalization to every wording, direction of a relationship, or rule. A simple looking question may be unusual relative to the patterns the model learned. 1. The reversal curse: knowing A → B does not guarantee B → A Suppose a model was trained on this made up statement: It might correctly answer "Who wrote The Silver Orchard?" yet fail another direction of a learned relation, depending on which relation was emphasized during training. Research on the reversal curse found that some autoregressive models trained on "A is B" did not reliably infer "B is A" , even when the relationship should be reversible. This is a finding about specific experimental conditions, not proof that every modern LLM fails every reverse question . Providing the statement in the current prompt can change the result. 2. Superficial changes can change the answer These prompts should usually test the same knowledge: "Which option is correct? A. ... B. ..." "Choose the right response: B. ... A. ..." Yet models can be sensitive to answer option position , punctuation, formatting, and phrasing. In controlled studies, changing the order of multiple choice options sometimes significantly changed performance. Why? Learned positional and format preferences can interfere with applying the underlying task consistently. This problem is often called prompt brittleness . 3. The model may match a pattern instead of applying a general rule A model may complete a familiar numerical or word pattern correctly but misapply it to unfamiliar symbols, longer sequences, or a slightly changed rule. Example: A model that answers this correctly once may still fail when labels or irrelevant details are changed. One success is not enough to establish reliable rule use. 4. Multi step dependence makes small tasks fragile Even a task that looks short may require several decisions: interpret a condition, keep intermediate state, handle an exception, and format a final response. Errors in any step can change the answer. For a workflow with five required decisions , if each independently succeeds 90% of the time , a toy calculation gives: This is illustrative only : real LLM error events are not generally independent, and the assumption does not establish a real model's task success rate. 5. Plausible text is not built in verification A model may produce an answer that is fluent and stylistically appropriate without executing code, checking a mathematical relation, or verifying a factual claim. Low temperature or deterministic generation can make results more repeatable but cannot guarantee they are correct . Practical way to identify these failures Instead of testing a single prompt, build a small robustness suite around the same underlying task: 1. Paraphrase: Ask the identical question using several equivalent wordings. 2. Reverse relationships: Test both directions where the relation logically permits it. 3. Shuffle answer options: Reorder choices while keeping the correct answer unchanged. 4. Rename entities: Replace familiar names with made up ones so memorization is less useful. 5. Add irrelevant details: See whether distracting context causes errors. 6. Check exact outputs: For arithmetic or code, compare results with a calculator or executable tests. For example, if a model gets 9 of 10 original questions right but only 6 of 10 meaning preserving variants right: That gap reveals a robustness concern on this particular sample, but does not by itself identify the cause. Use more representative cases and repeat trials before drawing broader conclusions. What can improve reliability? Use clear, unambiguous prompts and consistent structured formats. Include varied paraphrases and reversed relations in evaluation or suitable training data. Use deterministic tools for exact arithmetic, constraints, and programmatic tasks. Break complex dependencies into testable steps where that improves verification . Track the worst performing variations , not only mean accuracy on a favored prompt. In short: "Simple for a person" does not mean "reliably learned by an LLM." Robustness testing reveals where a model has learned a surface pattern rather than a rule that transfers consistently. See "What are the main limitations of Large Language Models?" for the wider overview of operational and reliability limitations. Research references: The Reversal Curse (ICLR 2024), Sensitivity to option order (NAACL 2024), and When Punctuation Matters (EMNLP 2025).

What are RMSNorm and MLP in Transformer architectures?

RMSNorm (Root Mean Square Normalization) and the MLP (Multilayer Perceptron) are two different components inside many modern Transformer blocks. RMSNorm stabilizes the scale of hidden activations; the MLP transforms each token's representation through learned nonlinear layers. Where do they appear? A typical pre normalized , decoder only Transformer block can be illustrated as: This is a simplified Llama style pattern. Specific architectures differ; some use standard LayerNorm, different MLP activations, or different normalization placement. 1. What is RMSNorm? RMSNorm divides a hidden vector by its root mean square (RMS) , then scales it using learned weights. Unlike standard LayerNorm, it does not subtract the mean . For a vector x with d elements: Here, epsilon is a small positive constant to avoid division by zero. The learned per feature weights let the model adjust the normalized activations. Numerical example: Let x = [3, 4], epsilon ≈ 0, and both learned weights equal 1: The output is not forced to have zero mean . That is the key distinction from LayerNorm. Why use it? RMSNorm performs a simpler calculation and can reduce normalization overhead, while providing useful activation rescaling. Its efficiency and effect on convergence depend on the implementation and workload. 2. What is the MLP in a Transformer? The MLP, also called the feed forward network (FFN) , applies learned transformations independently at each token position . Self attention mixes information between token positions ; the MLP performs nonlinear feature transformation within each token's hidden representation . A common basic form is: The hidden vector is usually expanded to a larger intermediate dimension before being projected back. Simple shape example: Each token undergoes the same learned MLP transformation, but with its own input vector . 3. What is SwiGLU, and why does it matter? Many Llama style LLMs use a gated MLP rather than the simple two linear layer example. One common formulation is: One projection produces features passed through SiLU . Another projection produces features to be multiplied elementwise with the activated gate. A final projection maps the intermediate representation back to the model hidden size. The gate helps modulate which transformed features contribute to the output. The exact parameter dimensions and bias choices depend on the architecture. 4. How do RMSNorm and MLP work together? For a pre norm decoder block, a simplified expression is: RMSNorm: Controls the input scale before each sublayer. Attention: Combines information across the token sequence. MLP: Applies token wise learned nonlinear transformations. Residual connections: Add sublayer outputs back to their inputs, helping signal and gradient flow through a deep network. Important: RMSNorm does not replace the MLP, and the MLP is not the attention mechanism. They perform different, complementary jobs. Practical PyTorch example This demonstrates RMSNorm followed by a simple MLP ; it is not a full Transformer block and does not implement the three projection SwiGLU variant. In short: RMSNorm normalizes feature magnitudes without mean subtraction; MLPs transform each token's features. Together with attention and residual connections, they form core parts of many decoder only LLM blocks. References: Root Mean Square Layer Normalization, GLU Variants Improve Transformer, Llama in Hugging Face Transformers.

How do you load a Large Language Model?

Loading a Large Language Model (LLM) means reading its architecture configuration, trained weights, and tokenizer from a saved checkpoint or model repository, placing the model on available compute hardware, and preparing it for inference. Loading is different from training: a pretrained model already contains learned parameters. Loading restores them; it does not retrain the model. What do you need? Model weights: The learned numerical parameters, often stored in safetensors checkpoint files. Configuration: Model details such as hidden size, layer count, and vocabulary size, usually in config.json. Tokenizer: Converts text into token IDs and token IDs back into text. Use the tokenizer intended for the model. Runtime and hardware: Compatible software, adequate RAM or GPU VRAM, and permission to access the model files. Typical loading flow Beginner friendly Python example Install compatible versions of PyTorch , Transformers , and, where needed, Accelerate : Then load a small, public causal language model: This example runs on CPU by default and is suitable for illustrating the loading process. The output is a text continuation, not a verified factual answer . Some chat tuned models require their own chat template rather than a raw text prefix. Loading larger models on a GPU For supported Hugging Face Transformers versions, you can request automatic device placement and checkpoint precision: Replace the placeholder with a real compatible model ID that your account can access. device map="auto" may distribute weights across available devices, including CPU offload where supported. It is not a guarantee of fast execution or sufficient memory. Version note: Current Transformers v5 documentation uses dtype="auto"; older v4 examples often use torch dtype="auto". Match the argument name to your installed version. How much memory is required? A useful weights only estimate is: For a hypothetical 7 billion parameter dense model: FP32 (4 bytes): about 28 GB of weight storage. FP16/BF16 (2 bytes): about 14 GB of weight storage. 4 bit packed weights (0.5 bytes): ideally about 3.5 GB before scales, metadata, and implementation overhead. This is not total runtime memory . Activations, the KV cache, temporary buffers, allocator overhead, and loading time peaks add to the requirement. Quantization formats and hardware support also affect what is practical. Common loading problems Out of memory: Model weights plus runtime buffers exceed available RAM or VRAM. Consider a smaller or quantized model, offloading, or distributing the model. Access denied: Some model repositories require acceptance of license terms and an authenticated account. Tokenizer mismatch: A tokenizer from a different model may produce incompatible token IDs or poor outputs. Unsupported architecture or versions: Use a compatible model class and library release. Unsafe checkpoint code: Treat downloaded models as untrusted software. Prefer safetensors, review repository code, and avoid enabling trust remote code=True unless the source has been vetted. Missing chat formatting: Instruction tuned chat models may expect role markers and a model specific chat template. Loading vs. inference serving from pretrained is a straightforward way to load a model for experiments and applications. Production services may use optimized inference engines with batching, KV cache management, and concurrency. Those are serving optimizations , not the basic act of loading model weights. In short: Choose a compatible checkpoint, load its tokenizer and architecture/weights, place it on supported hardware, and verify it can generate a short response. Check memory, model permissions, tokenizer compatibility, and checkpoint trust before deployment. References: Hugging Face Transformers: Quickstart, Loading models, Model loading parameters.

What is the production risk of just using a standard, pre-trained Llama-3 tokenizer to train a new model for the legal and medical domain ?

Using the original Llama 3 tokenizer to train a new legal/medical LLM is not inherently wrong—but the production risk is that specialized text may be represented with more tokens than necessary. That can increase training and inference cost, consume the context window faster, and potentially make domain specific patterns harder to learn. The right decision is measure first, then compare tokenizers on realistic data rather than automatically rebuilding the vocabulary. Why can a general purpose tokenizer be inefficient? Llama 3 uses a roughly 128K token vocabulary designed to encode broad text efficiently. It is a TikToken based, byte level BPE tokenizer ; unusual words can generally be represented by smaller pieces rather than becoming unrecognized tokens. But legal and medical documents contain unusual forms such as: Medical terminology: rare drug names, genomic variants, disease codes, measurements, and abbreviations. Legal terminology: jurisdiction specific citations, clause numbering, case references, and specialized phrases. Some of these strings may be broken into many subword tokens. The risk is fragmentation, not necessarily an unknown token error. A large vocabulary does not guarantee efficient segmentation for every domain. A simple numerical example Suppose we tokenize the same 10,000 word collection of legal and medical documents using two candidates: Original Llama 3 tokenizer: 14,000 tokens. Tokenizer trained on a representative domain corpus: 10,000 tokens. These are illustrative numbers , not published measurements of Llama 3. The exact amount of meaningful text depends on the data. Extra tokens increase sequence length and often memory/latency, but do not automatically imply a 40% increase in total GPU time : batch shape, padding, caching, and attention implementation matter. What could go wrong in production? 1. Higher serving cost: More tokens for the same case history, contract, or clinical note increase input processing, context usage, and potentially billing. 2. Context truncation: More tokenized fragments may push a crucial clause, medication dosage, or diagnosis outside a fixed context budget. 3. Longer training: The model may need to process more token positions for the same volume of raw documents, affecting compute planning. 4. Harder domain representation: Rare technical strings split across many pieces can demand more compositional learning. But fewer tokens do not automatically guarantee higher accuracy . 5. Compatibility risks if tokenizer changes later: Token IDs must match the learned input embedding and output projection rows. A mismatched tokenizer can render a checkpoint's word representations invalid. What is different when training a new model from scratch? Scenario A — Training a new model from scratch You can reuse the Llama 3 tokenizer while randomly initializing the new model's embeddings. You can also train a new domain oriented tokenizer and size the new embedding/output matrices to its vocabulary. Reusing the tokenizer does not transfer Llama 3's pretrained language knowledge . It transfers token segmentation and token ID conventions only. Evaluate the tokenizer together with the new model , not just by counting tokens. Scenario B — Continuing training or fine tuning pretrained Llama 3 weights Keep the pretrained tokenizer initially; its token IDs are already tied to learned embedding/output parameters. Replacing it wholesale breaks compatibility with those pretrained rows and can discard much of the existing representation benefit. Carefully adding tokens is possible, but the new embeddings and possibly output rows must be resized, initialized, and trained; results require validation. These scenarios should not be conflated. How should an engineering team make the decision? 1. Sample real workloads: Obtain permitted, de identified legal and clinical text reflecting production languages, document types, codes, and punctuation. 2. Measure compression: Compare tokens per document, tokens per character/word, the distribution of unusually fragmented terms, and how often a text exceeds the target context length. 3. Create a candidate: Train a domain tokenizer on a representative corpus, or retain Llama 3 as the baseline. Keep special token conventions and downstream infrastructure in scope. 4. Train fair pilot models: Use comparable training budgets and controlled setups to test whether the candidate actually improves validation loss and domain task results. 5. Evaluate safety critical details: Test exact extraction of dosages, dates, identifiers, legal citations, and negation; check end to end throughput, memory, and latency as well as accuracy. 6. Freeze and version the winner: Save tokenizer files with the model checkpoint, maintain strict tokenizer/model ID compatibility, and confirm production data processing consistency. Decision rule: If Llama 3 tokenizes the domain efficiently and pilot model quality is good, reuse it . If domain specific fragmentation is substantial and a controlled pilot demonstrates meaningful end to end gains, a custom tokenizer may justify the added training and deployment complexity. Research evidence Meta describes Llama 3's 128K vocabulary and improved general token efficiency. Work on domain specific clinical/biomedical models shows that vocabulary design can materially affect task performance , but the best choice depends on the corpus and workload. Hugging Face documents the need to resize token embeddings if new vocabulary items are added to an existing model. In short: The main risk is inefficient encoding and the resulting cost/context trade off , not an inability to represent medical or legal text. Measure fragmentation and domain accuracy before changing a tokenizer; also distinguish new training from modifying pretrained model weights . References: Meta: Introducing Llama 3, Meta Llama 3 model card, Clinical and biomedical vocabulary design study, Hugging Face: Customizing tokenizers, and Hugging Face: Tokenizer/vocabulary resizing.

We have a fixed compute budget - 32 H100s for two weeks. To get the best possible model, should we train a larger model on less data, or a smaller model on more data? How do you begin to answer this?

Start with compute optimal scaling, then validate it with small pilot runs. Under a fixed budget of 32 NVIDIA H100 GPUs for two weeks , choosing between a larger model trained on fewer tokens and a smaller model trained on more tokens is a joint optimization problem—not a rule that "bigger always wins." For dense decoder only Transformers , a useful first approximation for pretraining arithmetic is: At fixed C , doubling N approximately halves the number of training tokens D you can afford. This 6ND approximation ignores some attention, embedding, optimizer, and distributed system costs; it is a planning estimate , not a complete wall clock model. Step 1 — Turn the GPU rental into a usable training budget The allocated time is: Critical distinction: The H100's advertised peak Tensor Core FLOPs are not your model's achieved training FLOPs. Real performance depends on H100 variant, precision, model shape, sequence length, memory, parallelism, communication, input pipeline, failures, and checkpoint overhead. Illustrative assumption only: Suppose a representative pilot sustains 200 trillion useful model FLOPs/s per GPU when averaged over the chosen training setup. This is not a claimed benchmark for all H100s. This is a conditional 14 day ceiling . In practice, subtract time for environment setup, experiments, evaluation, validation, maintenance, and outages. Recalculate after measuring actual tokens/sec and model FLOPs utilization (MFU) . Step 2 — Compare plausible model size / token count choices Under the same hypothetical 7.74 × 10²¹ FLOPs and the approximation D ≈ C / (6N) : 4B parameter model: about 323B training tokens (~81 tokens per parameter). 8B parameter model: about 161B training tokens (~20 tokens per parameter). 16B parameter model: about 81B training tokens (~5 tokens per parameter). These figures are compute allocations , not measurements of final model quality or guarantees that each configuration fits, trains efficiently, or has enough unique high quality text. Step 3 — Use Chinchilla as the first baseline In Training Compute Optimal Large Language Models (Hoffmann et al., 2022), the researchers found that model size and training tokens should grow roughly together along the compute optimal frontier for their studied settings. A familiar approximate rule of thumb from that work is ~20 training tokens per parameter . Under our assumed budget, that heuristic points toward an ~8B parameter model trained on ~160B tokens . Do not treat 20:1 as a universal formula. It is an empirical starting point derived from specific model families, corpora, and objectives. Domain, architecture, data quality, training regime, and evaluation target can move the optimum. Step 4 — Decide what "best model" actually means Best validation loss at training completion is not necessarily the best choice for product deployment: One time research / maximum final loss efficiency: Explore models around the compute optimal frontier, measuring quality on the relevant held out text. High volume inference: A smaller model trained for more tokens may be cheaper to deploy and serve. Meta's Llama 3 study found that its 8B model kept improving when trained much longer than the older Chinchilla optimal token estimate. Specialized tasks: Evaluate the actual target capabilities (e.g., instruction following, code, multilingual performance), rather than selecting solely on overall perplexity. Limited high quality text: Extra epochs are not the same as new, independent tokens; repetition and data quality can alter the optimum. Step 5 — Run short, controlled pilot experiments Before committing two weeks to one expensive run: 1. Benchmark the hardware and training stack. Profile tokens/s, useful FLOPs/s, memory, collective communication, data loading, and checkpoint time for representative model sizes. 2. Create a small scaling grid , such as multiple parameter sizes and training token budgets, using the same architecture family, tokenizer, dataset mix, validation protocol, and appropriate tuned training settings. 3. Fit or inspect loss vs compute curves. Compare candidates after equal compute , not equal optimization steps, and extrapolate cautiously. 4. Check important downstream tasks using uncontaminated, held out benchmarks and deployment specific requirements. 5. Plan the remaining runway. Keep a buffer for checkpointing, evaluation, and recovery; avoid budgeting 100% of the 14 days as uninterrupted training. 6. Select the final N and D pair , then instrument training to catch under utilization or instability early. Tiny experiments cannot prove the optimum for all scales , but they are vastly more informative than assuming advertised H100 peak performance or blindly applying a generic scaling ratio. Why not simply train the largest model possible? A huge model may receive too few tokens to learn effectively under the budget. It can also have lower utilization, require more model/optimizer state sharding, create expensive inter GPU communication, and cost more per generated token in production. Conversely, an excessively small model may become capacity limited even if trained on many tokens. The goal is to balance both failure modes. Interview ready recommendation "I'd first determine whether success means final validation loss or downstream/deployment quality. I'd profile the real H100 training setup and convert the time budget into achieved useful training FLOPs. Then I'd use the approximate 6ND constraint and Chinchilla scaling laws to choose candidate model sizes and token counts. As a rough illustration, with 200 TFLOP/s per GPU, the full 14 days correspond to about 7.74 × 10²¹ useful FLOPs, putting an ~8B/160B token configuration near a classical 20:1 heuristic. I'd verify this with pilot scaling runs, data quality checks and downstream evaluation, reserve operational headroom, and consider smaller, longer trained models when inference cost matters. I'd never promise one size is optimal without measurements." In short: Profile → estimate usable compute → build candidate N/D pairs → fit scaling curves → test real tasks → choose for total lifecycle cost. Under a fixed budget, the size vs data trade off is an empirical engineering decision . Research: Hoffmann et al., Training Compute Optimal Large Language Models ; NeurIPS 2022 paper abstract; Meta's Llama 3 scaling findings; NVIDIA H100 specifications.

Your team is struggling with training instability and exploding gradients in a new 100B+ model. The original ’Attention Is All You Need’ paper used post-norm with learning rate warm-up. Why is that a bad idea for deep models, and what’s the one simple architectural change that solves this?

For a very deep Transformer with unstable gradients, one of the first architectural changes to test is switching from Post LayerNorm (Post LN) to Pre LayerNorm (Pre LN). In Pre LN, normalization occurs before the attention or feed forward sublayer, leaving an uninterrupted identity residual path through the block. Important qualification: Post LN is not inherently "bad," and Pre LN does not guarantee that a 100B+ model will train stably. The right diagnosis still depends on learning rate, initialization, optimizer, numerical precision, residual scaling, and distributed training. 1. What is Post LN versus Pre LN? A Transformer repeatedly applies sublayers such as attention and an MLP. Let x be the current hidden state and F represent one such sublayer. Post LN (original 2017 Transformer): Pre LN: The original Attention Is All You Need architecture applied normalization after each residual addition. In Pre LN, the normalization is moved inside the residual branch, before attention and the MLP. Modern LLMs often use a Pre Norm layout with RMSNorm instead of classic LayerNorm, but RMSNorm and normalization placement are separate design choices. 2. Why can Post LN be difficult to train? Gradients must travel backward through many blocks. In Post LN, every residual block's output passes through a normalization operation. Backpropagation repeatedly multiplies by the Jacobian of those normalizations and sublayers. This can affect gradient magnitude and conditioning. Xiong et al., in On Layer Normalization in the Transformer Architecture (ICML 2020) , showed under their theoretical assumptions that Post LN can have large expected gradients near the output layers at initialization . A large learning rate can then yield overly large updates and unstable optimization; warm up can help. With Pre LN , the unmodified residual addition gives gradients a direct identity component. In Jacobian notation: Here I is the identity matrix, J F the sublayer Jacobian, and J LN the normalization Jacobian, evaluated at their respective inputs. The Pre LN expression contains an explicit identity path that is not multiplied by the normalization Jacobian at that block's output. This tends to improve gradient flow. Caution: The identity term is not a proof that every gradient remains bounded: the learned branch can still create problematic updates. 3. Beginner friendly numerical intuition Suppose you have 20 stacked blocks and, in a deliberately simplified scalar analogy , a gradient component retains 0.7× its magnitude each time it passes through an unavoidable transformation. A direct identity residual path can preserve a route for gradient information instead of forcing it through all those transformations. This is only an illustration , not an actual gradient measurement for LayerNorm. Real Transformer Jacobians are matrices and can amplify, attenuate, or rotate gradient components. In particular, the documented Post LN instability is often about large gradients near the final layers , not a claim that every Post LN gradient vanishes. 4. What would you actually change? For a new 100B+ parameter model, test a Pre Norm residual block : Compare it with the Post Norm pattern: In PyTorch's reference implementation, torch.nn.TransformerEncoderLayer(..., norm first=True) selects Pre LN; norm first=False selects Post LN. Note: Changing placement alters the computation. You cannot generally convert a fully trained Post LN checkpoint into an equivalent Pre LN checkpoint simply by moving the layers; retraining or substantial adaptation may be necessary. 5. Why doesn't learning rate warm up solve everything? Warm up starts with smaller updates while optimization is most sensitive, but it does not remove the structural placement of normalization . Pre LN can reduce this particular source of sensitivity. Xiong et al. demonstrated settings where Pre LN trained effectively without warm up. However, retaining warm up can still be appropriate for very large models because there are other sources of early instability. Do not remove warm up automatically when moving to Pre LN. 6. What else must you check before blaming Post LN? In an actual exploding gradient incident, review: 1. Gradient norms per layer and their changes before the first spike, including loss and whether gradients become NaN/Inf. 2. Learning rate schedule , optimizer configuration, weight decay, and warm up; check whether the current peak LR is suitable for the new depth and scale. 3. Initialization and residual branch scaling , especially across many blocks. 4. Precision and numerics: mixed precision, loss scaling where applicable, LayerNorm/RMSNorm epsilon, optimizer state precision, and overflow. 5. Gradient clipping: it can limit update spikes but is not a substitute for stable architecture and learning rate. 6. Distributed training correctness: gradient accumulation, reductions, sharding, checkpoint restoration, and global batch size changes. Run a controlled smaller scale A/B experiment that holds dataset, optimizer, budget, and architecture family as steady as possible while changing norm placement . Compare loss curves, gradient distributions, number of failed steps, and downstream quality. 7. Is Pre LN always the best final design? No. Pre LN's easier optimization can come with different representational and final quality trade offs depending on depth and setup. Some designs use scaled residuals or hybrid approaches. DeepNorm / DeepNet modifies residual connections and initialization to stabilize extremely deep Transformers while retaining desirable Post LN characteristics. This means Pre LN is the simplest strong baseline to try , not the only method, and not a guarantee of superior final quality. Interview ready answer "The original Transformer uses Post LN, so the residual path goes through LayerNorm after every sublayer. In very deep models this changes the gradient path and can create large output layer gradients at initialization, making optimization sensitive to learning rate and warm up. I'd first test Pre LN—normalize inputs to attention and the MLP, then add each result to the untouched residual stream. This exposes an identity gradient path and often improves stability. But for a 100B+ run I'd also inspect per layer gradient norms, learning rate schedule, initialization, precision and optimizer behavior; Pre LN alone is not a guarantee. DeepNorm style alternatives can also work." In short: Post LN = normalize after the residual sum; Pre LN = normalize before the sublayer. Moving normalization inside the residual branch usually makes very deep Transformers easier to optimize, but stability still requires careful training system validation. References: Vaswani et al., Attention Is All You Need; Xiong et al., On Layer Normalization in the Transformer Architecture; Wang et al., DeepNet: Scaling Transformers to 1,000 Layers; PyTorch TransformerEncoderLayer: norm first; Nguyen & Salazar, Transformers without Tears.