How It Works

The Problem

Production AI agents are distributed systems: they depend on LLM APIs, tools, context windows, and multi-step orchestration. Each of these can fail. Today's tools don't answer the questions that matter:

• What happens when the agent's tools fail? — A search API returns 503. A database times out. Does the agent degrade gracefully, hallucinate, or fabricate data?
• Does the agent always follow its rules? — Must it always cite sources? Never return PII? Are those guarantees maintained when the environment is degraded?
• Did we fix the production incident? — After a failure in prod, how do we prove the fix and prevent regression?

Observability tools tell you after something broke. Eval libraries focus on output quality, not resilience. No tool systematically breaks the agent's environment to test whether it survives. Flakestorm fills that gap.

The Solution: Four Pillars

Like Chaos Monkey for infrastructure, Flakestorm injects failures into tools, APIs, and LLMs — and tests how the agent handles hostile or malformed inputs — then verifies that the agent obeys its behavioral contract and recovers gracefully.

Who Flakestorm Is For

• Teams shipping AI agents to production — Catch failures before users do
• Engineers running agents behind APIs — Test against real-world abuse patterns
• Teams already paying for LLM APIs — Reduce regressions and production incidents
• CI/CD pipelines — Automated reliability gates before deployment

Flakestorm is built for production-grade agents handling real traffic. While it works for exploration and hobby projects, it's designed to catch the failures that matter when agents are deployed at scale.

Scores at a Glance

Why This Matters

How Flakestorm Works

You define golden prompts, invariants (or a full contract), and optionally chaos, replay, and adversarial input settings. Flakestorm runs the chosen mode(s), checks responses against your rules, and produces a robustness or resilience score plus an HTML report.

• Chaos only — Golden prompts → agent with fault-injected tools/LLM → invariants.
• Contract — Golden prompts → agent under each chaos scenario → verify named invariants across a matrix.
• Replay — Recorded production input + tool responses → agent → contract.
• Adversarial inputs — Golden prompts → adversarial variations (24 types) → agent (with or without chaos) → invariants.

Define Once, Run Any Mode

Define golden prompts, invariants (or a full contract), and optionally chaos, replay, and adversarial input settings in a simple YAML file. Flakestorm handles fault injection, adversarial generation, and assertion checking.

• Adversarial inputs: 24 mutation types; use with chaos to test both bad inputs and bad environments
• Environment chaos: timeouts, errors, rate limits, malformed tool/LLM responses
• Behavioral contracts: named invariants × chaos matrix; severity-weighted resilience score
• Invariants & assertions: latency, JSON validity, semantic similarity, safety (PII, refusal)
• Reports: interactive HTML and JSON; contract matrix and replay reports

version: "1.0"
agent:
  endpoint: "http://localhost:8000/invoke"
  type: "http"
golden_prompts: ["Book a flight to Paris", "What's my balance?"]
invariants: [{ type: "latency", max_ms: 2000 }]

Try Flakestorm in ~60 Seconds

pip install flakestorm → flakestorm init → edit flakestorm.yaml (agent.endpoint) → flakestorm run. With v2 config you can also run --chaos, contract run, replay run, or flakestorm ci.

You get a robustness score (adversarial runs) or resilience score (chaos/contract/replay), plus a report. For local adversarial generation you'll need Ollama — see the docs.

How Flakestorm Is Different

Most tools test output quality. Flakestorm tests survivability: does the agent handle broken tools, rate limits, degraded environments, and adversarial inputs while still obeying its rules?

• Environment chaos (fault injection into tools and LLMs)
• Behavioral contracts verified across a chaos matrix
• Replay of production failures as deterministic tests
• Adversarial inputs (24 mutation types) for both bad inputs and bad environments

Built for production AI agents, where failures are gradual, inconsistent, and hard to reproduce — and where "did we fix the incident?" matters.

Flakestorm vs Manual Testing vs Other Tools

Use Cases

• Catch failures before users do — pre-deployment chaos, contract, replay, and adversarial testing
• CI/CD reliability gates — overall weighted score for blocking PRs
• Verify behavioral contracts when tools and LLMs fail
• Replay production incidents as deterministic tests
• Detect prompt injection, context attacks, and instruction leakage under stress

Open Source vs Cloud

We do not cripple the OSS version. Cloud exists to remove operational pain, not to lock features.