Research

HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Yongliang Shen12 min readResearch
FOUNDATIONALAI AgentsAdvanced

Key takeaway

Foundational work on llm as orchestrator.

Updated on Oct 3, 2026

01

Abstract

  • HuggingGPT uses ChatGPT as a controller to connect with hundreds of specialized models on Hugging Face.
  • The controller receives a user request, plans which models are required, executes them, and summarizes their outputs.
  • The architecture separates task planning from specialized model execution.
  • HuggingGPT supports multimodal and multi-task requests by composing models trained for individual capabilities.
  • The paper positions the LLM as a central orchestrator rather than requiring one model to perform every task itself.

02

Introduction

  • Large language models have broad reasoning and language abilities but are not necessarily optimal for every specialized task.
  • Hugging Face provides a large ecosystem of models for vision, speech, language, and multimodal tasks.
  • A controller is needed to translate a natural-language request into a sequence of specialized model calls.
  • The system must understand task dependencies because later tasks may consume outputs from earlier tasks.
  • HuggingGPT therefore treats an LLM as a planning and coordination layer over a model ecosystem.

03

Problem

  • The problem is how to solve complex AI requests that span multiple task types and specialized models.
  • A single general-purpose LLM may lack the capability or efficiency required for image generation, object detection, speech recognition, or other specialist tasks.
  • The controller must select appropriate models from a large model repository.
  • It must construct an executable task plan and pass intermediate outputs between tasks.
  • It must also summarize heterogeneous outputs into a coherent final response.

04

Methodology

  • HuggingGPT has four major stages: task planning, model selection, task execution, and response generation.
  • Task planning decomposes the user request into individual tasks and identifies dependencies among them.
  • Model selection maps each task to an appropriate Hugging Face model using model descriptions and task metadata.
  • Task execution runs the selected models and records their outputs.
  • Response generation uses ChatGPT to synthesize the completed task results into the final answer.
  • The controller can execute tasks sequentially or in parallel when dependencies permit.
Figure 1

05

Architecture

  • The system represents the plan as a sequence or graph of tasks with explicit dependencies.
  • Each task contains a task type, input, and selected model.
  • Outputs from one model can become inputs to downstream tasks.
  • The LLM controller uses tool descriptions to reason about which specialist is appropriate.
  • Execution results are returned to the controller so it can continue planning or produce the final response.
  • This design enables multi-model pipelines such as detection → classification → captioning or speech → text → question answering.

06

Dataset

  • The paper evaluates HuggingGPT on a broad collection of AI tasks spanning language, vision, speech, and multimodal capabilities.
  • The experiments use publicly available models from the Hugging Face model ecosystem rather than training a new specialist for each request.
  • Task planning is evaluated on complex instructions requiring multiple models.
  • The system is tested on tasks involving object detection, image classification, image captioning, visual question answering, speech recognition, and text generation.
  • Intermediate model outputs are used as inputs to later tasks, allowing compositional pipelines.

07

Training

  • The controller is based on ChatGPT/GPT-3.5-turbo-class language-model capabilities available at the time of the paper.
  • Specialized Hugging Face models are selected at inference time from their model cards and task descriptions.
  • The system does not train a unified multimodal model from scratch.
  • Prompt templates guide task decomposition, model selection, execution, and final response generation.
  • The controller can reason over prior task outputs when constructing subsequent model inputs.

08

Experiments

  • The paper evaluates both single-task and multi-task scenarios to test whether the controller can correctly orchestrate specialists.
  • Complex requests are decomposed into subtasks whose dependencies determine execution order.
  • The authors evaluate whether selected models can successfully perform their assigned tasks.
  • They also evaluate the quality of the final generated response after multiple specialist outputs are combined.
  • Failure analysis examines planning mistakes, incorrect model selection, and execution errors.

09

Baselines

  • Direct ChatGPT responses provide a baseline for requests that can be solved without specialist models.
  • Specialist Hugging Face models provide the execution baseline for individual tasks.
  • The system is also compared with task-specific pipelines where applicable.
  • The important comparison is whether LLM-driven orchestration can combine specialist models more flexibly than manually designed pipelines.
  • Model-selection quality is evaluated separately from the final answer quality.

10

Results

  • HuggingGPT demonstrates that an LLM can coordinate specialist models for tasks beyond its own native generation capabilities.
  • The architecture enables multi-modal pipelines without requiring the controller to directly perform every modality-specific computation.
  • Model descriptions function as an executable capability registry that the controller can reason over.
  • The separation between planning and execution makes the system extensible: adding a new model can expand the available capability set.
  • Intermediate outputs allow the system to construct multi-step solutions instead of treating each user request as a single prediction.

11

Conclusion

  • HuggingGPT established an influential architecture in which an LLM acts as a general-purpose planner and orchestrator over a model ecosystem.
  • Its four-stage pattern—planning, model selection, execution, and response generation—is a foundation for later agent and tool-use systems.
  • The paper demonstrates that heterogeneous specialist models can be composed dynamically through natural-language interfaces.
  • The architecture naturally extends to multimodal agents because each modality can be delegated to an appropriate expert.
  • Its main engineering lesson is that orchestration, interface compatibility, and reliable tool selection are as important as the underlying specialist models.

12

References

Primary source: Shen et al., “HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face,” NeurIPS 2023, arXiv:2303.17580.

  • The paper introduces an LLM-centered controller for orchestrating Hugging Face models.
  • Its pipeline explicitly separates task planning, model selection, task execution, and response generation.
  • The system is designed for multi-task and multimodal requests.
  • The work is an orchestration framework rather than a new foundation-model pretraining method.
  • Key limitations include planning errors, model-selection errors, execution incompatibilities, latency, and dependence on external model quality.

Continue reading

Related research

Browse all research