Research

Toolformer: Language Models Can Teach Themselves to Use Tools

Timo Schick12 min readResearch
FOUNDATIONALAI AgentsAdvanced

Key takeaway

Foundational work on tool calling.

Updated on Oct 3, 2026

01

Abstract

  • Toolformer is a 6.7B-parameter GPT-J-based language model trained to decide which external API to call, when to call it, what arguments to provide, and how to incorporate the returned result.
  • The method requires only a handful of demonstrations per API rather than large human-annotated tool-use datasets.
  • Toolformer uses self-supervised filtering: candidate API calls are retained when their results reduce the model's loss on future tokens.
  • The paper evaluates a question-answering system, Wikipedia search, calculator, calendar, and machine translation system.

02

Introduction

  • The paper starts from a tension in LMs: scaling improves broad language abilities but does not reliably solve precise arithmetic, factual lookup, current information, low-resource language understanding, or temporal awareness.
  • Tool use can address these weaknesses, but prior approaches either require substantial human supervision or are narrowly tied to a specific task.
  • Toolformer aims to make tool use general: the model itself decides whether a tool is useful instead of being told in advance which tool to invoke.
  • The authors also want tool augmentation without sacrificing the base model's general language-modeling capabilities.

03

Problem

  • The core problem is learning useful API calls without manually annotating where every tool should be used.
  • A useful tool call must provide information that actually improves prediction, rather than merely producing an externally valid answer.
  • The model therefore needs to learn three decisions jointly: whether to call a tool, which tool to call, and what input to send.
  • The training process must preserve the original language-modeling distribution closely enough that tool learning does not damage general language ability.

04

Methodology

  • API calls are represented as tuples c=(a_c,i_c), with the API name and its input, and are linearized using special API delimiters plus an arrow to the result.
  • The pipeline has three major stages: sample candidate calls, execute them, and filter them using their effect on future-token prediction.
  • Candidate call positions are sampled where the model assigns probability above a threshold to starting an API call; at most k positions are retained.
  • At each candidate position, up to m possible API calls are sampled from the model using an API-specific prompt.
  • The remaining calls are interleaved with the original text to create the augmented training sequence.
Figure 1

05

Architecture

  • Toolformer treats an API call as part of the language sequence rather than adding a separate neural controller architecture.
  • During inference, decoding proceeds normally until the model emits the API-result marker; decoding is then paused while the external API executes.
  • The API response and closing API token are inserted into the sequence before generation resumes.
  • The model therefore learns both tool invocation syntax and the contextual conditions under which tool results improve continuation probabilities.
  • In practice the paper uses token sequences such as “[”, “]”, and “->” instead of expanding the model vocabulary with new special tokens.

06

Dataset

  • The Question Answering tool is based on Atlas, a retrieval-augmented LM fine-tuned on Natural Questions.
  • The Calculator supports the four basic arithmetic operations and rounds results to two decimal places.
  • Wikipedia Search uses a BM25 retriever over the KILT Wikipedia dump and returns short text snippets.
  • Machine Translation uses the 600M-parameter NLLB model covering 200 languages, with fastText automatically detecting the source language and English as the target.
  • Calendar returns the current date without requiring an input argument.

07

Training

  • The base language model is GPT-J with 6.7B parameters.
  • The source corpus C is a subset of CCNet; the same underlying text is retained after API calls are inserted.
  • Candidate API calls are filtered using a weighted future-token cross-entropy loss rather than human labels.
  • The final model is fine-tuned with batch size 128, learning rate 1e-5, and linear warmup over the first 10% of training.
  • API-specific heuristics reduce the annotation cost; for example, calculator candidates are considered only for texts containing at least three numbers.

08

Experiments

  • The filtering loss compares prediction after receiving both API input and output against the better of receiving nothing or only the API input.
  • A call is retained only when the API result reduces future-token loss by at least a tool-specific filtering threshold.
  • The authors use exponentially decreasing weights so information close to the API call receives more emphasis.
  • At training time, calls for different tools are merged into the original text and the model is trained with the standard language-modeling objective.
  • The approach intentionally does not require modifying the vocabulary because API syntax is represented with existing token sequences.

09

Baselines

  • Downstream evaluation uses zero-shot prompting: models receive natural-language task instructions but no task-specific examples showing how a tool should be used.
  • LAMA evaluation covers SQuAD, Google-RE, and T-REx, with the correct word required to occur within the first five generated words.
  • Mathematical evaluation uses ASDiv, SVAMP, and MAWPS.
  • Question answering evaluation uses Web Questions, Natural Questions, and TriviaQA, while temporal evaluation uses TEMPLAMA and DATESET.
  • MLQA tests Arabic, German, Spanish, Hindi, Vietnamese, and Chinese questions against English contexts.

10

Results

  • On LAMA, Toolformer reaches 33.8 on SQuAD, 11.5 on Google-RE, and 53.5 on T-REx, compared with GPT-J scores of 17.8, 4.9, and 31.9.
  • On the math benchmarks, Toolformer reaches 40.4 on ASDiv, 29.4 on SVAMP, and 44.0 on MAWPS.
  • For Web Questions, Natural Questions, and TriviaQA, Toolformer reaches 26.3, 17.7, and 48.8 respectively.
  • On TEMPLAMA and DATESET, Toolformer reaches 16.3 and 27.3 versus 13.7 and 3.9 for GPT-J.
  • Toolformer beats much larger OPT-66B and often GPT-3-175B, although GPT-3 remains stronger on several QA tasks.

11

Conclusion

  • Error cases reveal that the Wikipedia search engine can return poor matches and Toolformer cannot reformulate a query or browse multiple results.
  • The model is restricted to at most one API call per input during the main inference setup, which prevents chained tool use.
  • TEMPLAMA improvement is not primarily caused by Calendar: Calendar is used in only 0.2% of examples, while QA and Wikipedia tools dominate.
  • DATESET improvement is attributable to Calendar, which is used in 54.8% of examples.
  • In MLQA, machine translation is used between 63.8% and 94.9% of examples depending on language, except Hindi at only 7.3%.
  • Tool-use ability emerges around 775M parameters in the GPT-2 scaling experiments; smaller models show little benefit from tools.

12

References

  • Toolformer shows that self-supervised API-call filtering can turn a general language model into a selective tool user without task-specific tool-use annotations.
  • The approach improves zero-shot performance across factual, mathematical, multilingual, and temporal tasks while retaining general language-modeling ability when API calls are disabled.
  • The paper's main limitations are independent API-call generation, no interactive browsing/query reformulation, sensitivity to exact wording, sample inefficiency for some tools, and no explicit API-computation-cost objective.
  • The authors suggest iterative application of the bootstrapping procedure as a possible route to improve sample efficiency.

Continue reading

Related research

Browse all research