Research

AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation

Qingyun Wu12 min readResearch
FOUNDATIONALAI AgentsAdvanced

Key takeaway

Foundational work on multi-agent collaboration.

Updated on Oct 3, 2026

01

Abstract

  • AutoGen is an open-source framework for building LLM applications by composing multiple agents that converse to accomplish tasks.
  • Agents are customizable and conversable, with combinations of LLMs, human inputs, and tools.
  • Conversation behavior can be programmed using natural language and computer code.
  • The paper targets mathematics, coding, question answering, operations research, online decision-making, and entertainment.
  • AutoGen is presented as generic infrastructure rather than a single task-specific agent.

02

Introduction

  • Increasing task complexity motivates cooperation among multiple LLM agents.
  • Multi-agent collaboration can encourage divergent thinking, improve factuality and reasoning, and provide guardrails.
  • Existing approaches were fragmented across single-agent systems and task-specific multi-agent systems.
  • AutoGen unifies LLM agents, human participation, and tool execution under conversational interaction.
  • The core abstraction is a conversable agent whose behavior is driven by messages.
  • The framework supports different agent counts, interaction patterns, and autonomy levels.
  • The paper introduces conversation programming as the programming model for multi-agent applications.

03

Problem

  • The core problem is how to develop multi-agent LLM applications across domains and complexity levels without building a new orchestration system for each task.
  • Applications may require one-shot or multi-round interaction.
  • Some applications require humans continuously in the loop, while others can run autonomously.
  • Agents may need LLM reasoning, code execution, external tools, or human expertise.
  • Static workflows are insufficient when the next action depends on conversation state.
  • The paper therefore treats computation and control flow as conversation-centric.

04

Methodology

  • AutoGen introduces conversation programming to specify both computation and control flow.
  • Computation describes the actions agents take to produce responses during a conversation.
  • Control flow describes the order and conditions under which those computations execute.
  • Agent computation is conversation-centric: an agent acts based on conversations it participates in.
  • Message passing creates subsequent computation and interaction.
  • Auto-reply functions provide a unified interface for automated conversation.
  • Natural language can encode behavior through system messages and instructions.
  • Computer code can encode termination conditions, reply functions, human-intervention rules, and other control logic.
  • The two mechanisms can be combined in the same application.
  • Function calling can connect natural-language decisions with executable tools.
  • Group conversations can dynamically select the next speaker based on the conversation.
Figure 1

05

Architecture

  • A typical two-agent setup pairs an AssistantAgent with a UserProxyAgent.
  • The assistant can generate reasoning or executable Python code.
  • The user proxy can execute generated code and return execution results to the assistant.
  • Human feedback can be inserted when the user proxy is configured to request it.
  • AutoGen supports multi-agent conversations rather than requiring a fixed two-agent topology.
  • GroupChatManager can coordinate dynamic group conversations.
  • Agents can be specialized by role, capability, or tool access.
  • The same agent abstraction can support coding, retrieval, decision-making, mathematics, and interactive tasks.
  • Conversation acts as the mechanism through which state and control are propagated.

06

Dataset

  • The paper evaluates AutoGen across multiple application domains rather than a single benchmark.
  • Mathematical problem solving uses the MATH dataset.
  • The reported MATH evaluation includes the full dataset and a 120-question level-5 subset.
  • Retrieval-augmented generation experiments include Natural Questions.
  • Embodied decision making is evaluated in ALFWorld.
  • Multi-agent coding is evaluated with the OptiGuide application.
  • Online decision-making is represented by browser interaction experiments including MiniWoBChat.
  • The paper also demonstrates dynamic group chat, conversational chess, and other application scenarios.
  • These evaluations demonstrate framework generality rather than training a new foundation model.

07

Training

  • AutoGen is primarily an inference-time framework; the paper does not require retraining the underlying LLM to create its multi-agent workflows.
  • The framework composes existing LLM capabilities through prompting, conversation, tools, and human feedback.
  • System messages can define agent roles and behavioral constraints.
  • Generated code can be executed and the resulting output fed back into the conversation.
  • Human feedback can be incorporated without changing model parameters.
  • The framework improves task execution through interaction and orchestration rather than parameter updates.
  • Reported experiments use capable pretrained LLMs, including GPT-4 and GPT-3.5-era models depending on the application.

08

Experiments

  • The paper studies six representative application categories: mathematics, retrieval-augmented generation, ALFWorld decision making, multi-agent coding, dynamic group chat, and conversational chess.
  • Mathematical problem solving tests whether conversation and code execution improve difficult reasoning.
  • RAG tests whether an assistant can request additional context when initial retrieval is insufficient.
  • ALFWorld tests whether multiple agents can improve grounding in an embodied text environment.
  • OptiGuide tests multi-agent collaboration for code generation and optimization guidance.
  • Group-chat experiments study dynamic speaker selection.
  • Chess demonstrates conversational coordination in a structured game.
  • The experiments compare different agent configurations and interaction patterns rather than treating multi-agent collaboration as one fixed algorithm.

09

Baselines

  • Mathematical comparisons include GPT-4, ChatGPT with Code Interpreter, and ChatGPT with a plugin in the reported setups.
  • Full MATH comparison reports AutoGen at 69.48% versus ChatGPT+Code Interpreter at 48.33%.
  • On the level-5 120-question subset, AutoGen is compared against ChatGPT+Code Interpreter and ChatGPT+Plugin.
  • RAG comparisons include a non-interactive retrieval setup against the interactive AutoGen pattern.
  • ALFWorld compares two-agent and three-agent configurations.
  • OptiGuide compares multi-agent coding against a single-agent setup.
  • MiniWoBChat results are compared with the specialized RCI baseline.
  • Group-chat experiments compare role-play-based speaker selection with task-based selection.

10

Results

  • AutoGen achieved 69.48% success on the reported MATH evaluation.
  • ChatGPT+Code Interpreter achieved 48.33% in the corresponding comparison.
  • On the level-5 120-question subset, AutoGen achieved 52.5%.
  • The level-5 result exceeded the reported ChatGPT+Code Interpreter result of 48.33% and ChatGPT+Plugin result of 30.0%.
  • Interactive RAG improved Natural Questions F1 from 22.79% to 25.88% in the reported experiment.
  • Adding a grounding agent increased ALFWorld average success from 54% to 69%.
  • ALFWorld best-of-3 success increased from 63% to 77% with the additional grounding agent.
  • Multi-agent OptiGuide improved F1 by 8% with GPT-4 and 35% with GPT-3.5 compared with single-agent configurations.
  • The OptiGuide workflow reduced core workflow code from 430 lines to 100 lines.
  • MiniWoBChat reached 52.8%, reported as 3.6 percentage points below the specialized RCI result.

11

Conclusion

  • AutoGen argues that multi-agent conversation can serve as reusable infrastructure for complex LLM applications.
  • Its key abstraction is a customizable conversable agent.
  • Conversation programming unifies computation and control flow around message passing.
  • Natural language, executable code, tools, and humans can participate in the same workflow.
  • Experiments demonstrate gains in mathematics, RAG, ALFWorld, and multi-agent coding.
  • The framework also supports dynamic group conversation and interactive application patterns.
  • The work establishes conversation-centric orchestration as a practical design pattern for LLM applications.

12

References

  • Primary paper: Wu et al., “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation,” arXiv:2308.08155.
  • The work was published as a COLM 2024 conference paper.
  • Authors include Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang.
  • The framework was released as open-source software.
  • The paper evaluates mathematics, RAG, ALFWorld, coding, group chat, and online decision-making scenarios.
  • Primary source: https://arxiv.org/abs/2308.08155

Continue reading

Related research

Browse all research