Toolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick12 min readResearch
FOUNDATIONALAI AgentsAdvanced
Key takeaway
Foundational work on tool calling.
Updated on Oct 3, 2026
01
Abstract
- Toolformer is a 6.7B-parameter GPT-J-based language model trained to decide which external API to call, when to call it, what arguments to provide, and how to incorporate the returned result.
- The method requires only a handful of demonstrations per API rather than large human-annotated tool-use datasets.
- Toolformer uses self-supervised filtering: candidate API calls are retained when their results reduce the model's loss on future tokens.
- The paper evaluates a question-answering system, Wikipedia search, calculator, calendar, and machine translation system.
02
Introduction
- The paper starts from a tension in LMs: scaling improves broad language abilities but does not reliably solve precise arithmetic, factual lookup, current information, low-resource language understanding, or temporal awareness.
- Tool use can address these weaknesses, but prior approaches either require substantial human supervision or are narrowly tied to a specific task.
- Toolformer aims to make tool use general: the model itself decides whether a tool is useful instead of being told in advance which tool to invoke.
- The authors also want tool augmentation without sacrificing the base model's general language-modeling capabilities.
03
Problem
- The core problem is learning useful API calls without manually annotating where every tool should be used.
- A useful tool call must provide information that actually improves prediction, rather than merely producing an externally valid answer.
- The model therefore needs to learn three decisions jointly: whether to call a tool, which tool to call, and what input to send.
- The training process must preserve the original language-modeling distribution closely enough that tool learning does not damage general language ability.
04
Methodology
- API calls are represented as tuples c=(a_c,i_c), with the API name and its input, and are linearized using special API delimiters plus an arrow to the result.
- The pipeline has three major stages: sample candidate calls, execute them, and filter them using their effect on future-token prediction.
- Candidate call positions are sampled where the model assigns probability above a threshold to starting an API call; at most k positions are retained.
- At each candidate position, up to m possible API calls are sampled from the model using an API-specific prompt.
- The remaining calls are interleaved with the original text to create the augmented training sequence.
Figure 1
05
Architecture
- Toolformer treats an API call as part of the language sequence rather than adding a separate neural controller architecture.
- During inference, decoding proceeds normally until the model emits the API-result marker; decoding is then paused while the external API executes.
- The API response and closing API token are inserted into the sequence before generation resumes.
- The model therefore learns both tool invocation syntax and the contextual conditions under which tool results improve continuation probabilities.
- In practice the paper uses token sequences such as “[”, “]”, and “->” instead of expanding the model vocabulary with new special tokens.
06
Dataset
- The Question Answering tool is based on Atlas, a retrieval-augmented LM fine-tuned on Natural Questions.
- The Calculator supports the four basic arithmetic operations and rounds results to two decimal places.
- Wikipedia Search uses a BM25 retriever over the KILT Wikipedia dump and returns short text snippets.
- Machine Translation uses the 600M-parameter NLLB model covering 200 languages, with fastText automatically detecting the source language and English as the target.
- Calendar returns the current date without requiring an input argument.
07
Training
- The base language model is GPT-J with 6.7B parameters.
- The source corpus C is a subset of CCNet; the same underlying text is retained after API calls are inserted.
- Candidate API calls are filtered using a weighted future-token cross-entropy loss rather than human labels.
- The final model is fine-tuned with batch size 128, learning rate 1e-5, and linear warmup over the first 10% of training.
- API-specific heuristics reduce the annotation cost; for example, calculator candidates are considered only for texts containing at least three numbers.
08
Experiments
- The filtering loss compares prediction after receiving both API input and output against the better of receiving nothing or only the API input.
- A call is retained only when the API result reduces future-token loss by at least a tool-specific filtering threshold.
- The authors use exponentially decreasing weights so information close to the API call receives more emphasis.
- At training time, calls for different tools are merged into the original text and the model is trained with the standard language-modeling objective.
- The approach intentionally does not require modifying the vocabulary because API syntax is represented with existing token sequences.
09
Baselines
- Downstream evaluation uses zero-shot prompting: models receive natural-language task instructions but no task-specific examples showing how a tool should be used.
- LAMA evaluation covers SQuAD, Google-RE, and T-REx, with the correct word required to occur within the first five generated words.
- Mathematical evaluation uses ASDiv, SVAMP, and MAWPS.
- Question answering evaluation uses Web Questions, Natural Questions, and TriviaQA, while temporal evaluation uses TEMPLAMA and DATESET.
- MLQA tests Arabic, German, Spanish, Hindi, Vietnamese, and Chinese questions against English contexts.
10
Results
- On LAMA, Toolformer reaches 33.8 on SQuAD, 11.5 on Google-RE, and 53.5 on T-REx, compared with GPT-J scores of 17.8, 4.9, and 31.9.
- On the math benchmarks, Toolformer reaches 40.4 on ASDiv, 29.4 on SVAMP, and 44.0 on MAWPS.
- For Web Questions, Natural Questions, and TriviaQA, Toolformer reaches 26.3, 17.7, and 48.8 respectively.
- On TEMPLAMA and DATESET, Toolformer reaches 16.3 and 27.3 versus 13.7 and 3.9 for GPT-J.
- Toolformer beats much larger OPT-66B and often GPT-3-175B, although GPT-3 remains stronger on several QA tasks.
11
Conclusion
- Error cases reveal that the Wikipedia search engine can return poor matches and Toolformer cannot reformulate a query or browse multiple results.
- The model is restricted to at most one API call per input during the main inference setup, which prevents chained tool use.
- TEMPLAMA improvement is not primarily caused by Calendar: Calendar is used in only 0.2% of examples, while QA and Wikipedia tools dominate.
- DATESET improvement is attributable to Calendar, which is used in 54.8% of examples.
- In MLQA, machine translation is used between 63.8% and 94.9% of examples depending on language, except Hindi at only 7.3%.
- Tool-use ability emerges around 775M parameters in the GPT-2 scaling experiments; smaller models show little benefit from tools.
12
References
- Toolformer shows that self-supervised API-call filtering can turn a general language model into a selective tool user without task-specific tool-use annotations.
- The approach improves zero-shot performance across factual, mathematical, multilingual, and temporal tasks while retaining general language-modeling ability when API calls are disabled.
- The paper's main limitations are independent API-call generation, no interactive browsing/query reformulation, sensitivity to exact wording, sample inefficiency for some tools, and no explicit API-computation-cost objective.
- The authors suggest iterative application of the bootstrapping procedure as a possible route to improve sample efficiency.
Source trail
Primary papers, implementation pages, and external source records used to ground this explainer.
Continue reading
Related research
AI Agents
GitSwarm
GitSwarm turns inference-time work into a persistent, branchable body of reusable contributions that later agents can build on.
AI AgentsLongHarness
LongHarness helps agents reason over 100M-token document collections by keeping evidence and conclusions as explicit, revisable states that guide one another.
AI AgentsGenerative Agents
An agent architecture combining memory, retrieval, reflection, and planning to simulate believable long-horizon behavior.