Function Calling and Structured Outputs Explained: How LLMs Call Your Code

Function calling is the model-provider feature that lets an LLM return a JSON object matching a schema you define, which your code then parses and executes. Structured outputs extend the same guarantee to arbitrary responses: with strict mode on, the model is constrained during decoding so the result is valid against your JSON Schema on every call.

What Function Calling Actually Does

Function calling does not call anything. The model never touches your functions, your database, or the network. It produces a structured message — a tool name plus a JSON object of arguments — and hands control back to your runtime. Your code decides whether to execute it, what to do with the result, and what to send back.

The full cycle has three participants and four steps:

1. Declare: You attach tool definitions to the request — a name, a description, and a JSON Schema describing the arguments.
2. Decide: The model chooses whether to respond with plain text or with one or more tool calls, based on the descriptions it was given.
3. Execute: Your code validates the arguments, runs the function, and appends a tool message containing the result to the conversation.
4. Continue: The model sees the result and either answers the user or requests another tool — the loop repeats until it stops asking.

Because the model only proposes and your code only executes, the security boundary stays on your side of the wire: an argument that fails validation never reaches the function, and a function the model did not list in its schema can never be invoked.

The Agent Loop in Code

Everything an agent does — search, call an API, read a file, ask a follow-up — is this loop repeated. The practical details that separate a demo from production are the step budget, the parallel tool calls, and the message history you append in exactly the shape the provider expects.

import OpenAI from "openai";

const openai = new OpenAI();

const tools = [
  {
    type: "function",
    function: {
      name: "get_weather",
      description: "Get the current weather for a city.",
      parameters: {
        type: "object",
        properties: {
          city: { type: "string", description: "City name, e.g. Lisbon" },
          unit: { type: "string", enum: ["celsius", "fahrenheit"] },
        },
        required: ["city", "unit"],
        additionalProperties: false,
      },
    },
  },
  {
    type: "function",
    function: {
      name: "search_orders",
      description: "Search orders by customer email or order ID.",
      parameters: {
        type: "object",
        properties: {
          query: { type: "string" },
        },
        required: ["query"],
        additionalProperties: false,
      },
    },
  },
] as const;

type Message = OpenAI.Chat.Completions.ChatCompletionMessageParam;

async function runAgent(userInput: string): Promise<string> {
  const messages: Message[] = [{ role: "user", content: userInput }];

  for (let step = 0; step < 8; step++) {
    const response = await openai.chat.completions.create({
      model: "gpt-4.1",
      messages,
      tools,
      parallel_tool_calls: true,
    });

    const message = response.choices[0].message;
    messages.push(message);

    if (!message.tool_calls || message.tool_calls.length === 0) {
      return message.content ?? "";
    }

    for (const call of message.tool_calls) {
      const args = JSON.parse(call.function.arguments);
      const result = await dispatch(call.function.name, args);
      messages.push({
        role: "tool",
        tool_call_id: call.id,
        content: JSON.stringify(result),
      });
    }
  }

  throw new Error("Agent did not finish within the step budget");
}

async function dispatch(name: string, args: Record<string, unknown>) {
  switch (name) {
    case "get_weather":
      return weatherApi.current(args.city as string, args.unit as string);
    case "search_orders":
      return ordersDb.search(args.query as string);
    default:
      throw new Error("Unknown tool: " + name);
  }
}

Three details in that loop are load-bearing. The step budget stops runaway agents that never choose to finish. parallel_tool_calls lets the model emit several independent calls in one turn — OpenAI, Anthropic, and Google all support this — which collapses round trips when the calls do not depend on each other. And the tool result must be echoed back with the matching tool_call_id, or the provider rejects the next request.

Schemas Are the Contract

The tool description and JSON Schema are the only things the model sees when it decides whether and how to call a tool. This makes schema design the highest-leverage part of the feature. A vague description produces wrong calls; a loose schema produces arguments your code cannot trust.

• Enumerate, do not free-text: Any field with a known set of values — status, currency, priority — should be an enum. It removes an entire class of near-miss values.
• Require everything: List every field in required and set additionalProperties: false. Optional fields become nullable-and-required under strict mode rather than absent.
• Describe arguments, not intents: The tool description explains what the tool does; argument descriptions explain units, formats, and examples. Mixing the two degrades selection accuracy.
• Keep tools orthogonal: Overlapping tools — search_orders versus get_order with ambiguous descriptions — make the model guess. Rename until the boundary is obvious.

Once the schema is decided, four ways to get structured data out of a model are worth telling apart, because they differ in what they guarantee rather than what they promise.

Verdict: if any code downstream does JSON.parse and then branches on the result, use strict structured outputs. Prompted JSON is fine only when a human reads the output. Function calling in default mode is the right default for agents, where the argument is validated by your dispatcher anyway.

How Strict Mode Works

Strict mode — OpenAI's structured outputs, Anthropic's strict tool schemas, Google's response_schema — is implemented with constrained decoding. During sampling, tokens that would make the output invalid against the schema are masked before the argmax step, so the model can only emit a valid object. Nothing is validated after the fact; the invalid string is never generated.

The result is measurable. When OpenAI introduced structured outputs in August 2024, its strict mode on gpt-4o-2024-08-06 reached 100% validity on the complex-schema evaluation set, against under 40% for gpt-4-0613 with prompting alone. That gap is the entire argument for the feature: not better answers, but answers your parser can survive.

Constraints have a cost. Strict schemas restrict what you can express — no recursive definitions in early versions, every property required, no additional properties — and constrained decoding can add prefill latency on long schemas. For a twenty-field object called a few hundred times a day, that cost is irrelevant. For a two-field object called in a hot loop, measure before you adopt.

from enum import Enum

from openai import OpenAI
from pydantic import BaseModel, Field


class Priority(str, Enum):
    low = "low"
    medium = "medium"
    high = "high"


class Ticket(BaseModel):
    title: str = Field(description="One-line summary of the request")
    priority: Priority
    affected_services: list[str]
    customer_impact: int = Field(ge=0, le=5)


client = OpenAI()

response = client.chat.completions.parse(
    model="gpt-4.1-mini",
    messages=[
        {"role": "system", "content": "Classify the support ticket."},
        {"role": "user", "content": ticket_text},
    ],
    response_format=Ticket,
)

message = response.choices[0].message
if message.parsed is None:
    raise ValueError("Model returned no parseable ticket: " + str(message.refusal))

ticket = message.parsed
print(ticket.priority, ticket.affected_services)

The Pydantic model is the schema: the SDK compiles it to JSON Schema, sends it as the response format, and parses the response back into a typed object. Note the explicit null check — strict mode guarantees syntactic validity against the schema, not that the model had enough information to fill it honestly. Refusals and confident nonsense are still possible; they simply arrive as well-formed JSON.

Provider Differences

The three major providers expose the same underlying idea with different surface syntax. If you are writing a provider-neutral layer, these are the deltas you have to abstract:

The one genuine asymmetry is the last row. Anthropic historically had no equivalent of response_format, so teams that needed a typed final answer declared a "finish" tool and forced it. Treat that as a provider detail, not an architecture: your application code should consume one typed result either way.

Failure Modes You Must Design For

A valid schema is a statement about syntax, not about correctness. Production systems fail in a consistent handful of ways, and each has a specific countermeasure:

• Invalid arguments in default mode: Without strict mode, JSON.parse on function.arguments can throw. Mitigate by parsing defensively and re-prompting with the schema error, or by turning on strict mode.
• Plausible but wrong values: The model fills customer_id with an ID that does not exist. Mitigate by validating references against your own store before executing, and by treating unknown IDs as a tool error the model can retry.
• Non-idempotent execution: A retried tool call that charges a card or deletes a record causes real damage. Mitigate by passing an idempotency key derived from the call arguments and caching results per conversation.
• Prompt injection steering tool use: Content you fetched — a web page, a ticket comment — can tell the model to call delete_account. Mitigate by never exposing destructive tools to untrusted-context turns, and by requiring human approval for write operations.
• Unbounded loops: The model keeps calling tools without answering. Mitigate with a hard step budget, a per-run token budget, and a timeout that returns partial state instead of hanging.

Log every tool name, the raw argument string, the parsed arguments, and the result alongside the model version. When a workflow regresses after a model update, that log is the only thing that tells you whether selection changed or execution did.

Choosing Between Function Calling and Structured Outputs

These are not competing features; they answer different questions. Function calling answers "what should the application do next" and is inherently about control flow. Structured outputs answer "what shape is this data" and are about serialization.

Use function calling when the model is selecting among actions: an agent, a router, a workflow engine, anything where the next step depends on reasoning over context. Use structured outputs when the model is producing data your code will ingest: an extraction pipeline, a classification step, a row you are about to write to a database. Use both together when an agent must return a typed summary at the end of its run — tool selection in default mode, a forced final tool with strict: true for the handoff.

Skip both when a human is the consumer. If someone is reading the output, asking for JSON in the prompt is enough, and the constraints you add for machines only make the prose worse.