Setting up variants

What a variant contains

Each variant stores an optional config object with fields the SDK reads flat off the runtime config response — there's no llm wrapper key:

Control modes

Every experiment has one control_mode, set once for all its variants — it decides which of the fields above Niitaka actually serves, and which stay entirely in your own code. The dashboard's variant setup form shows only the fields relevant to the selected mode.

In traffic_split mode, get_runtime_config() is still worth calling — it's what resolves which variant this session was assigned to and records that assignment for the comparison later — you just don't read a model or prompt out of its response:

with niitaka.start_session(
    agent_id="report-summariser",
    experiment_id="exp_a1b2c3d4e5f6",
    variant="gpt-4o",
) as session:
    # Traffic split mode: Niitaka only assigns and compares. Model and
    # prompt selection stay entirely in your own code.
    response = openai_client.chat.completions.create(
        model="gpt-4o",   # chosen by you, not fetched from Niitaka
        messages=[{"role": "user", "content": "..."}],
    )

Creating variants in the dashboard

1. 1

   Open the experiment

   Go to Experiments and open a Draft experiment. If you haven't created one yet, click New experiment and link it to your agent.

2. 2

   Add a variant

   Click Add variant. Give it a short, descriptive name that matches what you'll pass to start_session(variant=...) — for example gpt-4o or gemini-flash. Names are case-sensitive.

3. 3

   Pick a control mode, then configure the variant

   Choose full_config, guardrails_only, or traffic_split for the experiment (see Control modes below) — this decides which fields show up. Fields left blank inherit your agent's default policy configuration.

4. 4

   Repeat for all arms

   Add one variant per arm. A two-arm experiment (one baseline + one treatment) is the simplest and reaches significance fastest. You can add up to 8 variants total.

5. 5

   Set the baseline

   One variant must be marked as the baseline (control arm). All statistical comparisons are made relative to it. Typically this is your current production config.

Traffic weights

Each variant has a traffic_weight. Sessions are distributed proportionally — a weight of 1.0 on each of three variants gives a 33/33/33 split. You can skew the split if you want to limit exposure to an untested variant:

• Weights are relative, not percentages. 0.5 / 0.5 and 1.0 / 1.0 produce the same 50/50 split.
• For multi-variant experiments, equal weights maximise statistical power for a fixed total session count.
• You can update weights while an experiment is Running — note this may cause a Sample Ratio Mismatch (SRM) if the shift is large.

Assigning sessions to variants

Pass experiment_id and variant to start_session(). Both are required for the session to be counted in the experiment.

with niitaka.start_session(
    goal="Summarise quarterly report",
    agent_id="report-summariser",
    experiment_id="exp_a1b2c3d4e5f6",   # copy from Experiments dashboard
    variant="gpt-4o",                    # must match the variant name you created exactly
) as session:
    response = openai_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "..."}],
    )

Using Runtime Config with variants

If you filled in the variant's LLM and guardrail fields, the SDK can automatically fetch that config at session start. This lets you write one agent that adapts to whichever variant it's assigned to, without if/else branches.

with niitaka.start_session(
    goal="Summarise quarterly report",
    agent_id="report-summariser",
    experiment_id="exp_a1b2c3d4e5f6",
    variant="gpt-4o",
) as session:
    # SDK auto-fetches this variant's config at session start.
    # Calling get_runtime_config() also applies any guardrails you set on the
    # variant automatically — nothing further to do for cost/step limits.
    config = niitaka.get_runtime_config(agent_id="report-summariser")

    model         = config["model"]          # e.g. "gpt-4o" — top-level, no "llm" wrapper
    system_prompt = config["system_prompt"]

    response = openai_client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user",   "content": "..."},
        ],
    )

Naming conventions

• Use the model name for model comparison experiments: gpt-4o, gemini-2.5-flash.
• Use descriptive slugs for prompt or guardrail experiments: v1-concise-prompt, high-cost-limit.
• Avoid spaces and special characters — lowercase letters, numbers, and hyphens only.