Know what changes before you ship.

Test agent changes against real production scenarios to understand their impact on behavior, latency, and cost before you merge.

Connect to Github Book a demo

Raindrop Simulations reviewing the pull request Batch calendar edits to reduce API calls

Catch regressions before production

Simulations run on every PR to surface potential regressions directly in GitHub.

Return only the latest order from lookup_order #482

Openmaya wants to merge 3 commits into main from orders/latest-only

raindrop-ai bot commented 4 minutes ago

Simulations

`support-agent` at `a1f3c9e` · Worth reviewing

Change: lookup_order now returns only the customer’s most recent order instead of every open order.

Observed: Refunds and order status still work. In 2 address-change conversations, the agent updated the newest order while the customer was asking about an older one.

Simulation faithfulness: High · 7 of the 10 conversations we replayed touched this change · 2 got worse · See them side by side

Raindrop Simulations / support-agent 2 of 7 affected conversations got worseDetails

CI / test Successful in 1m 12sDetails

“Simulations shows me whether a prompt change actually fixes the problem I’m trying to solve. I can test it against realistic conversations, see what worked and what didn’t, and keep iterating before I ship.”

Chris CataneseFounding Engineer, Caddy

Understand the full impact of a change

Raindrop replays production scenarios to reveal changes in agent behavior across workflows.

Conversation 3 of 10 · Address change Got worse

User

Can you ship my blue jacket order to my new address? 41 Elm St, Portland.

Productionmain · 7c02e41

lookup_ordercustomer: cus_8812 → 3 open orders
ReasoningThe blue jacket is ord_9931.
update_addressorder: ord_9931 · 41 Elm St
ReplyDone. Your blue jacket (ord_9931) now ships to 41 Elm St.

This PR#482 · a1f3c9e

lookup_ordercustomer: cus_8812 → 1 order
ReasoningThe only open order is ord_9977.
update_addressorder: ord_9977 · 41 Elm St
ReplyDone. Your order (ord_9977) now ships to 41 Elm St.

Regression. The agent changed the address on the customer’s headphones order, not the jacket they asked about.

Tool calls 2 → 2Latency 4.1s → 3.8sCost $0.018 → $0.016Signals wrong_order +1

Test in a realistic environment

Scenarios run in simulated environments that recreate your agent’s tools and services using production traces.

Shopifysimulated

orders

ord_9931Blue jacket

ord_9952Running socks

ord_9977Headphones

Stripesimulated

charges

ch_3P1x$148.00

ch_3P2k$12.00

ch_3P4m$219.00

Zendesksimulated

tickets

#40418Address change

#40402Late delivery

Gmailsimulated

messages

Re: your ordermaya@…

Shipping updatemaya@…

Supabasesimulated

customers

cus_8812Portland, OR

cus_8813Austin, TX

support-agenta1f3c9e

Your code from the PR, running in a Raindrop sandbox

lookup_order → Shopifyupdate_address → Shopifysend_email → Gmail

Built from 1,284 production traces5 services · 11 toolsProduction untouched

The harness playground

Test changes in the harness playground without opening a PR. It runs your full agent harness, with its tools, skills, and workflows, and compares results side by side with production.

Raindrop harness playground comparing a production run with a simulated run

“Raindrop helps us go from finding an issue in production to verifying the fix before it ships. Our coding agents generate PRs, and Simulations lets us replay the original user issue in a full world to see whether the change actually improves the agent’s behavior. It’s especially valuable now that agents are writing most of our PRs.”

Mel GeorgeLEAP CRM, LEAP Legal

Connect to Github Book a demo

FAQs

What do I need to get started?

A GitHub repository with a TypeScript, JavaScript, or Python agent, and that agent sending events to Raindrop. Install the Raindrop GitHub app, pick the agent, and Raindrop builds its simulated world from your code and traces. After that, every pull request that changes the agent gets a Simulations comment. Simulations is in public beta.

How does Raindrop create simulation scenarios?

From your production traffic. When a pull request changes your agent, Raindrop picks recent production conversations that exercise the changed code and replays each one against the new version. The PR comment tells you how many of the replayed conversations touched the change, and how many got better or worse.

How are simulated environments created?

Raindrop reads your agent’s code to find every tool and service it calls, then uses the tool calls in your production traces to build a stateful copy of each one, filled with data shaped like yours. Your agent reads and writes to that copy the same way it would in production. When a pull request adds a tool that no trace has seen yet, Raindrop generates realistic responses for it.

How is Simulations different from traditional evals?

Evals score outputs against a dataset you write and keep up to date. Simulations runs your real agent code from the pull request through real production scenarios, then compares each run with what happened in production. You see every workflow a change affects, including the ones you would not have thought to test. You can still add your own evals and Signals to grade each run.

Does Simulations work with my agent framework?

Yes. Raindrop runs your agent’s own code, so it works with the framework you already use, from the Vercel AI SDK and OpenAI Agents SDK to Mastra, LangChain, or a custom loop. Today Simulations runs TypeScript, JavaScript, and Python agents.

Can simulations affect my production systems?

No. During a simulation, every tool call your agent makes goes to the simulated copy of that service, never to the real one. Raindrop needs read access to your repository and the traces you already send. Messages, refunds, and database writes stay inside the simulation.

How much do Simulations cost?

Simulations is in public beta, and pricing will be set before general availability. Book a demo to talk through pricing for your team.