Skip to main content
Change Reports are a type of brief, alongside the Daily Digest and custom briefs. After connecting your GitHub, every change you make to your agent (harness, tools, prompt, cache settings, and more) will be automatically monitored. Your project’s Slack channel (or, optionally, the PR author directly) will receive updates on how the change is affecting users or the behavior of your agent.

​
Why

Raindrop covers the full lifecycle of agent quality. Simulations predict what a change will do. To use a sports analogy (every developer’s favorite), it’s a little bit like a scrimmage before the big game day. Ultimately, though, production is the source of truth for how your agent is truly performing. And there are many types of changes that can only be faithfully monitored in production. When things go wrong in production, Raindrop Issues tell you via email or Slack. But imagine you add a send_email tool and the agent never uses it. Or you make a PR to improve caching performance and it doesn’t reduce your bill. Maybe nothing “went wrong”, but you’d certainly want to know. And, especially in the first hours, the number of affected users could be very small. A Change Report catches these because it starts by analyzing the change itself. Flagging things that could be related to the change you just made, and keeping you updated in the hours after it merges.

​
How it works

Raindrop reads your agent harness automatically, meaning the prompts, models and tools that make up each agent in your repository. When a pull request merges to your default branch, Raindrop works out which agents it changed and how (tools added or removed, model or settings changed, instructions substantially changed), then associates that change with the agent’s production traffic. From there it watches the traffic that followed the merge and reports back:
  • An early check soon after the merge. If anything in the first stretch of traffic plausibly relates to the change (the new tool, the changed rule, the new model), you hear about it right away. If nothing does, no message is sent.
  • The full report after the change has been live for a while. A plain-language verdict, how many events and users the agent handled, whether each changed tool was actually called and by how many users, negative signals on events that used it, related stumbles, and highlights and lowlights with quotes from real conversations.

​
Set it up

  1. Connect GitHub from Briefs → Change Reports, or ask an admin to. Raindrop finds the agents in your repository and works out which production traffic belongs to each one.
  2. Turn on change reports per agent at Settings → Notifications → Change reports, and set a max per day if you want to cap them.
  3. Choose where reports go: the project’s Slack channel, or the PR author’s DMs. DMs fall back to the channel when the author has no matching Slack account.
Log the commit SHA with your events (raindrop.app.commit_sha) so Raindrop can tell exactly which traffic ran on the new code. Our most recent SDKs discover it automatically in most setups; see application Git provenance for when to set it yourself. Without it the report still runs, but usage is scoped by time instead of by commit, which is less accurate.

​
Read them

Every report lives in Briefs alongside the Daily Digest and your custom briefs. Filter to reports for changes you merged, and open any report to see the early check and the full report with the conversations behind each claim.
  • Briefs for the other kinds of briefs
  • Issues for failure modes Raindrop detects across your conversations
  • Triage Agent to ask follow-up questions about a report