AI agents don't just call a model and return its response. Instead, they orchestrate entire workflows that involve selecting tools for LLMs to use, evaluating guardrails, routing between LLM calls, and managing context across conversation turns. This means that a single user request can spawn dozens of internal steps, each one a potential point of failure, latency, or unexpected cost.

AI agent observability is the practice of capturing these execution flows so you can understand what the agent did, why it chose that path, and how it performed. In practice, this means tracing an agent run's execution path, capturing inputs and outputs at each step, and tracking cost, token usage, latency, and error rates.

AI agent tracing builds the foundation for AI agent observability. Without tracing, debugging an AI agent becomes a game of trying to guess which event triggered what decision in the agent workflow, and why.

A request can return successfully while the agent gives an incorrect answer, takes too long on a particular step, or triggers duplicate LLM or tool calls. Agent observability gives engineers the evidence to investigate these problems and compare behavior across runs.

Let's look at some of these problems in more detail.

A single LLM call is already non-deterministic: the same prompt can produce different outputs across runs. Agents amplify this because they chain multiple LLM calls together, where the output of one step determines the input of the next.

This means variance doesn't just exist at one point. Rather, it compounds across the workflow. A slightly different model response in step one can cause the agent to select a completely different tool in step two, which changes the context passed to step three, and so on. Reproducing a specific agent run becomes nearly impossible without capturing the full execution path.

Traditional software can also return incorrect results without throwing an error. With agents, these failures can be harder to catch because each run can take a different path: the workflow completes without any error, but the agent selected the wrong tool, retrieved stale context, or returned a response that is confidently incorrect.

Without observability, these silent failures only surface when a user complains or when downstream systems consume bad output. Tracing each decision in the agentic loop, such as which tool was selected, what context was available, what the model actually returned, is the only reliable way to evaluate whether an agent's "successful" run was actually logically correct.

Agents decide at runtime how many tools to call and how many LLM roundtrips to make. A simple user query might resolve in one LLM call, while a similar query might trigger several tool calls and model invocations depending on the agent's reasoning.

This makes cost and latency inherently unpredictable. Observability lets engineers track token usage, tool call frequency, and per-step latency across runs to identify expensive patterns early. For example, if two models with different pricing tiers perform comparably for a given workload, engineers can switch to the cheaper option — but only if they have the data to make that comparison.

OpenTelemetry (OTel) is a CNCF graduated project for instrumenting applications to emit telemetry data (primarily traces, metrics, and logs) in a vendor-neutral format. What makes it uniquely suited for agentic applications is that your visibility extends beyond the AI agent to the entire request: what endpoint did the user call, what DB queries ran, were any background jobs enqueued, did the application make any downstream API calls?

Prerequisites

Python 3.12 or newer. Download the latest version.

A SigNoz Cloud account for visualizing telemetry data.

An OpenAI API key to run the agent.

An API client like Bruno or Postman to test endpoints.

Setting up SigNoz Cloud

SigNoz Cloud is an OpenTelemetry-native observability platform that unifies traces, metrics, and logs in a single interface.

Clone and navigate to the demo application:

Copygit clone --filter=blob:none --sparse https://github.com/SigNoz/examples.git git -C examples sparse-checkout set python/otel-ai-agent-observability-demo cd examples/python/otel-ai-agent-observability-demo

Create and activate a Python virtual environment:

Copypython3.12 -m venv .venv source .venv/bin/activate

Install the application and OpenTelemetry Python dependencies:

Copypip install -r requirements.txt

The requirements.txt file pins the following OTel packages:

Next, run opentelemetry-bootstrap to detect widely used Python libraries (like the FastAPI web framework and HTTPX client library) and install their corresponding instrumentation packages:

Copyopentelemetry-bootstrap --action=install

Dangling OpenAI instrumentation package

opentelemetry-bootstrap currently installs a deprecated opentelemetry-instrumentation-openai-v2 package by default.

However, because our requirements.txt explicitly installs the modern opentelemetry-instrumentation-genai-* libraries, OpenTelemetry resolves to the newer packages at runtime.

You can safely leave the v2 package alone or uninstall it.

Start the application using opentelemetry-instrument to automatically instrument our application, including the AI agent workflow:

CopyOPENAI_API_KEY="<your-openai-api-key>"
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.<your-region>.signoz.cloud:443"
OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>"
OTEL_SERVICE_NAME="otel-ai-agent-observability-demo"
OTEL_RESOURCE_ATTRIBUTES="service.version=1.0.0,deployment.environment=dev"
OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only
opentelemetry-instrument fastapi run --port 8085 --workers 1

Here is what these OpenTelemetry environment variables configure:

Correlating logs to traces

Application logs are automatically correlated with traces by the opentelemetry-bootstrap-installed logging instrumentation. This eliminates the need to set the OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED environment variable, which has been deprecated in the more recent OpenTelemetry releases.

The FastAPI server starts on port 8085. Calling / returns a quick health check response:

Copy(.venv) ❯ curl "http://127.0.0.1:8085/" {"message":"OpenTelemetry NBA agent demo is running"}

The demo is now running. Before exploring the agent code, let’s look at the OpenTelemetry conventions and instrumentation packages used to capture AI agent workflows.

While early instrumentation discussions focused solely on "LLM tracing", OpenTelemetry standardizes AI telemetry under the broader umbrella of Generative AI (GenAI), using semantic conventions prefixed with gen_ai.*.

Because GenAI patterns and agentic harnesses evolve far more rapidly than traditional HTTP or database libraries, OpenTelemetry created a dedicated repository, open-telemetry/opentelemetry-python-genai, to enable maintainers to independently develop and publish these libraries.

This migration introduced a cleaner, split-scope architecture:

The earlier Agents instrumentation combined workflow and model-call tracing in a single package.

Additionally, because opentelemetry-bootstrap has not yet added these dedicated GenAI packages to its automated discovery list, you must install and track them separately through your requirements.txt or pyproject.toml files.

Varying maturity across GenAI instrumentation packages

Coverage varies across GenAI instrumentation packages, especially for agent features such as handoffs and guardrails. It's recommended that you pin your instrumentation packages explicitly and check their changelogs when upgrading dependencies.

Now that we've seen how OpenTelemetry separates agent orchestration from model calls, let's look at the code behind the NBA agent's workflow. Skip to the next section if you want to jump directly to generating and inspecting traces.

FastAPI serves our HTTP endpoints, the OpenAI Agents SDK manages the agent loop, and the OpenAI Python API library handles the underlying model calls. This separation is why we instrument both the Agents SDK and the API library: one captures the workflow, while the other captures each model call.

Our web application exposes an NBA Reporter agent that answers questions on NBA topics (e.g., general news or finals), performs web research, calculates team win percentages, and retains conversation history across turns via server-managed session persistence.

The agent_service.py file defines the agent configuration and the execution function used by our FastAPI endpoint.

We initialize the agent with a system prompt, tools, input guardrails, and model selection. The application uses gpt-6-luna (OPENAI_MODEL = "gpt-6-luna" in app/config.py), which delivers consistent multi-turn reasoning and tool formatting:

CopyNBA_AGENT = Agent( name="NBA_Reporter", instructions=NBA_INTERACTIVE_PROMPT, tools=[WebSearchTool(), calculate_win_percentage], input_guardrails=[nba_content_guardrail], model=OPENAI_MODEL, )

The calculate_win_percentage function tool gives the agent a deterministic way to calculate and display team win percentages without the risk of arithmetic hallucinations:

Copy@function_tool def calculate_win_percentage(wins: int, losses: int) -> str: """Calculates the winning percentage for an NBA team given their wins and losses."""

total_games = wins + losses if total_games == 0: return ".000"

return f"{wins / total_games:.3f}".lstrip("0")

To prevent misuse or off-topic queries, we define an input guardrail. When a user prompt does not match relevant basketball keywords, the function returns a GuardrailFunctionOutput with tripwire_triggered=True:

app/guardrails.py

@input_guardrail() def nba_content_guardrail( context: RunContextWrapper[None], agent: Agent, input_data: str | list, ) -> GuardrailFunctionOutput: """Ensures the user query is relevant to basketball/NBA."""

keywords = [ "nba", "basketball", "player", ... # truncated for brevity ]

extract the last user message from chat history

if isinstance(input_data, list): latest_user_message = next( ( item.get("content", "") for item in reversed(input_data) if isinstance(item, dict) and item.get("role") == "user" ), "", ) input_query = latest_user_message else: input_query = input_data

input_query = input_query.lower() is_relevant = any(keyword in input_query for keyword in keywords)

if len(input_query) < 5 or is_relevant: return GuardrailFunctionOutput(tripwire_triggered=False, output_info=None)

return GuardrailFunctionOutput( tripwire_triggered=True, output_info={ "reason": "The request is off-topic. Please ask questions relevant to NBA or basketball." }, )

Setting tripwire_triggered=True instructs the Agents SDK to abort execution and raise an InputGuardrailTripwireTriggered exception. We intercept this in a FastAPI exception handler to record the exception event directly on the active OpenTelemetry span before returning an HTTP 400 response:

Copy@app.exception_handler(InputGuardrailTripwireTriggered) async def handle_guardrail_block( request: Request, exc: InputGuardrailTripwireTriggered, ) -> JSONResponse: guardrail_msg = exc.guardrail_result.output.output_info span = trace.get_current_span()

span.record_exception(exc) return JSONResponse(status_code=400, content={"detail": guardrail_msg})

Managing session context across turns

The run_agent_turn function completes our implementation. The agent uses the OpenAIConversationsSession object to persist and gather conversation context across turns.

If no session_id was passed, the session object triggers the creation of a new conversation session. In a follow-up request, the user can attach this ID to ensure the agent has context of the conversation thus far.

app/agent_service.py

def run_agent_turn( topic: str, user_messag