Policy replay & what-if analysis
Know how policy changes behave before you deploy
Replay historical decision traces against a new policy configuration and see exactly which sessions would have had a different outcome — without risking production traffic.
Trace replay
Run any batch of historical traces through a new policy set and compare original vs. replayed decisions.
Decision diff
See side-by-side: what the agent decided before vs. what it would decide with your new policy.
Decision audit log
Every policy evaluation is logged — which policies matched, why, and what the winning action was.
Replay summary
Aggregate results: how many traces changed, which new decisions dominate, confidence of the shift.
Targeted replay
Filter to a specific agent, time window, or session subset before replaying for focused analysis.
Safe iteration
Test policy changes on historical data without exposing users to the new behavior until you're ready.
At a glance
- Replay decision traces against any new policy set
- Side-by-side original vs. replayed decision diff
- Full decision audit log with matched policy details
- Aggregate changed / unchanged summary per replay
- No production traffic affected during evaluation
Common questions
What is the difference between experiments and policy replay?
Experiments run new live sessions with a modified configuration — they use real production traffic and measure actual outcomes. Policy replay is a what-if analysis: it takes historical sessions that already ran and re-evaluates them against a new policy to predict how they would have been handled differently. Replay involves no live traffic.
Does replaying traces affect my production agents?
No. Replay is a fully read-only operation on your stored event data. Your live agents continue running normally and are completely unaffected. The replay executes on the Niitaka backend against historical snapshots — nothing is written back to your agent or its environment.
More general questions? See the full FAQ →
Ready to get started?
Connect your first agent in under 5 minutes. Free to start, no credit card required.
Next: Versioning