Test three models without changing production.

1. 1. Choose a workload. Open Model recommendations and review its top three policy-approved candidates.
2. 2. Choose your test. Compare cost, speed and reliability, or include hosted quality testing from the Compare Mode page.
3. 3. Turn on Compare Mode. Add one option to the existing Python decorator or Node.js workload wrapper.
4. 4. Run staging traffic. Your normal model still returns the application response. Candidate calls run separately.
5. 5. Review the result. Compare the available results in Model recommendations. The crown appears only when hosted quality testing produces a passing winner.

@spendlensai.observe(
    workload="customer-support-reply",
    compare_mode=True,
)
def generate_reply(message: str):
    return client.chat.completions.create(...)

compare_mode=True is the only code change after normal SpendLens instrumentation.

const generateReply = observe(
  { workload: "customer-support-reply", compareMode: true },
  async (message) => openai.chat.completions.create(...)
);

compareMode: true uses the Compare Mode setup for the same workload name.

Set only the keys required by the candidates enabled for this campaign. Keep them in your staging environment or secret manager. These keys are read locally by the SDK and are never sent to SpendLens.

# Set only the keys needed by your selected candidates.
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
DEEPSEEK_API_KEY=sk-...
MISTRAL_API_KEY=...
GOOGLE_API_KEY=...       # Gemini (GEMINI_API_KEY is also accepted)
DASHSCOPE_API_KEY=...    # Qwen / Alibaba Cloud Model Studio

GOOGLE_API_KEY is preferred for Gemini, with GEMINI_API_KEY supported as an alias. Qwen uses DASHSCOPE_API_KEY. These provider keys are separate from SPENDLENS_API_KEY, which authenticates telemetry.

No SDK quality variables are required. Optional quality evaluation runs on SpendLens infrastructure using the dashboard setting, the redacted template and safe generated data.

• The SDK runs candidate tests separately in your staging process. Your original provider call and response are never replaced.
• The original provider call stays on the normal application path. Candidate failures cannot replace its response.
• Hosted quality testing is optional, runs on SpendLens infrastructure and is controlled from the dashboard, not an SDK environment variable.
• SpendLens uses only the redacted prompt template and safe generated examples for hosted quality testing.
• Dynamic prompt values, production customer inputs and production responses are not used for hosted quality tests.
• The dashboard request limit and cost limit stop further comparison calls automatically.

compare_mode=True or compareMode: true remains the only SDK switch. The Compare Mode dashboard decides whether hosted quality testing is included.

Hosted quality testing currently supports OpenAI, Anthropic, Gemini, DeepSeek, Mistral and Qwen. Other providers are coming soon.

Pro includes $40 in Compare credits each month. Growth includes $100. Credits reset monthly and the dashboard shows the remaining balance.

Template-based AI scores are directional. Validate the winning model with your own staging data before changing production.

Compare Mode currently supports non-streaming chat message requests. OpenAI Responses input and arbitrary direct REST calls are not compared yet. Normal SpendLens telemetry support is unchanged.

If Compare Mode is marked in the SDK without an active campaign, normal telemetry continues. No additional model calls run inside your application.