The gap traditional testing leaves open

Unit tests assert that add(2, 3) === 5. That determinism breaks the moment your code calls a language model. The same prompt on different days, with different model versions, or after a seemingly harmless wording tweak can produce outputs that score completely differently on quality. Standard test suites cannot catch that drift.

Braintrust is an AI evaluation platform built specifically for this problem. Instead of one-shot assertions, you run experiments — your AI function against a dataset of inputs, scored automatically, with every result stored and compared over time. By the end of this tutorial you will have:

A local experiment runner scoring LLM outputs against ground truth
An LLM-as-a-judge scorer for open-ended quality
Prompt management pulled from the Braintrust dashboard at runtime
Production trace logging with zero extra instrumentation on your OpenAI client
A CI/CD gate that fails when quality drops below a threshold

Prerequisites

Node.js 20 or newer
TypeScript 5+
An OpenAI API key (any compatible provider works)
A Braintrust account — the free tier covers this tutorial
Familiarity with async TypeScript and basic LLM concepts

Step 1: Project setup

Create a fresh TypeScript project or add Braintrust to an existing one.

npm install braintrust autoevals openai zod

Add your keys to .env.local:

BRAINTRUST_API_KEY=your_braintrust_key
OPENAI_API_KEY=your_openai_key
BRAINTRUST_PROJECT_NAME=my-ai-app

Create a tsconfig.json that targets ES2022 or newer so top-level await works in eval scripts:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "outDir": "dist"
  }
}

Step 2: Your first experiment

An experiment in Braintrust has three parts: a dataset of inputs (and optional expected outputs), a task function that calls your AI, and one or more scorers that rate each output.

Create lib/eval/summarize.eval.ts:

import { Eval } from "braintrust";
import OpenAI from "openai";
 
const openai = new OpenAI();
 
// Task: the AI function under evaluation
async function summarize(input: { text: string }): Promise<string> {
  const response = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content: "Summarize the following text in two to three concise sentences.",
      },
      { role: "user", content: input.text },
    ],
  });
  return response.choices[0].message.content ?? "";
}
 
// Scorer: rewards concise summaries (under 60 words)
function concisenessScorer(args: { output: string }) {
  const wordCount = args.output.split(/\s+/).length;
  const score = wordCount <= 40 ? 1 : wordCount <= 70 ? 0.7 : 0.3;
  return { name: "conciseness", score };
}
 
// Dataset: a small ground-truth set
const dataset = [
  {
    input: {
      text: "Next.js 16 introduced major improvements to the App Router, including faster Server Actions, redesigned caching with explicit developer control, improved streaming for large data, and a new Partial Prerendering mode that blends static and dynamic rendering on the same page.",
    },
    expected:
      "Next.js 16 improves the App Router with faster Server Actions, redesigned caching, better streaming, and a new Partial Prerendering mode.",
  },
  {
    input: {
      text: "TypeScript 6 brought significant performance improvements to the type checker, cutting check times by up to 40% on large codebases. It also introduced isolated declarations for faster parallel builds, and stricter inference for mapped types that catches more errors at compile time.",
    },
    expected:
      "TypeScript 6 speeds up the type checker by up to 40%, adds isolated declarations for parallel builds, and improves mapped-type inference.",
  },
];
 
Eval("text-summarization", {
  data: dataset,
  task: summarize,
  scores: [concisenessScorer],
});

Run the experiment:

npx braintrust eval lib/eval/summarize.eval.ts

Braintrust prints a link to your dashboard where you can inspect each input, the output your function produced, the expected value, and every score. The first run becomes your baseline.

Step 3: LLM-as-a-judge scoring

String matching is brittle for open-ended outputs. A better approach is using a second LLM call to judge quality. The autoevals package ships pre-built scorers for common tasks.

import { Factuality, Similarity } from "autoevals";
 
Eval("text-summarization-v2", {
  data: dataset,
  task: summarize,
  scores: [
    // Checks factual accuracy against the expected output
    Factuality,
    // Measures semantic similarity on a 0-1 scale
    Similarity,
    // Your custom conciseness check
    concisenessScorer,
  ],
});

Factuality uses a chain-of-thought LLM judge to rate whether your output introduces hallucinations relative to the expected answer. It catches quality regressions that simple string comparison misses entirely.

Available scorers in autoevals:

Factuality — hallucination detection
Similarity — semantic closeness
AnswerCorrectness — for question-answering tasks
AnswerRelevancy — checks if the answer addresses the question
ContextRecall — for RAG pipelines
Humor, Toxicity, Moderation — content quality

Step 4: Custom LLM judge template

For domain-specific scoring, define your own judge prompt:

import { LLMClassifierFromTemplate } from "autoevals";
 
const faithfulnessJudge = LLMClassifierFromTemplate({
  name: "faithfulness",
  promptTemplate: `You are evaluating whether an AI-generated summary is faithful to the source text.
 
Source text:
{{input.text}}
 
Summary:
{{output}}
 
Rate faithfulness:
- "A" — fully accurate, no hallucinations
- "B" — mostly accurate, minor omissions
- "C" — inaccurate or misleading
 
Answer with a single letter.`,
  choiceScores: { A: 1, B: 0.6, C: 0 },
  useCoT: false,
});

The {{input.text}} and {{output}} placeholders are filled by Braintrust at runtime. useCoT: true asks the judge to reason before answering, which improves accuracy on complex tasks at the cost of extra tokens.

Step 5: Wrapping your OpenAI client for production tracing

In production you want every LLM call logged automatically — model name, prompt, response, latency, and token cost — without manually instrumenting each call.

import { wrapOpenAI, initLogger } from "braintrust";
import OpenAI from "openai";
 
// Initialize once at module or app startup
initLogger({
  projectName: process.env.BRAINTRUST_PROJECT_NAME!,
  apiKey: process.env.BRAINTRUST_API_KEY,
  asyncFlush: true, // non-blocking for serverless environments
});
 
// The wrapped client is a drop-in replacement for the standard client
export const openai = wrapOpenAI(new OpenAI());

Every call you make through openai now appears in your Braintrust project as a logged span — no other changes needed. You can filter by model, date range, prompt content, latency percentile, or token cost in the dashboard.

Step 6: Adding manual spans for complex workflows

When your AI pipeline does more than one LLM call — retrieval, reranking, multi-step reasoning — use traced to group them:

import { traced } from "braintrust";
 
export async function answerWithContext(question: string, docs: string[]) {
  return traced(
    async (span) => {
      // Log the inputs to this span
      span.log({ input: { question, docCount: docs.length } });
 
      // Retrieval step (logged automatically if you wrap the client)
      const context = docs.join("\n\n");
 
      const answer = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [
          {
            role: "system",
            content: `Answer the question using only the provided context.\n\nContext:\n${context}`,
          },
          { role: "user", content: question },
        ],
      });
 
      const output = answer.choices[0].message.content ?? "";
 
      // Log the final output and any metadata
      span.log({
        output,
        metadata: { tokensUsed: answer.usage?.total_tokens },
      });
 
      return output;
    },
    { name: "rag-answer" }
  );
}

The resulting trace in Braintrust shows the full span tree: retrieval timing, LLM call details, and overall response time in a single view.

Step 7: Next.js App Router integration

Wire the traced client into your API route:

// app/api/chat/route.ts
import { NextResponse } from "next/server";
import { traced, initLogger } from "braintrust";
import { openai } from "@/lib/braintrust"; // your wrapped client
 
// Initialize the logger at module level (runs once per cold start)
initLogger({
  projectName: process.env.BRAINTRUST_PROJECT_NAME!,
  apiKey: process.env.BRAINTRUST_API_KEY,
  asyncFlush: true,
});
 
export async function POST(req: Request) {
  const { message, userId } = (await req.json()) as {
    message: string;
    userId: string;
  };
 
  const response = await traced(
    async (span) => {
      span.log({ input: { message }, metadata: { userId } });
 
      const completion = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: message }],
      });
 
      const output = completion.choices[0].message.content ?? "";
      span.log({ output });
 
      return output;
    },
    { name: "chat" }
  );
 
  return NextResponse.json({ response });
}

Because asyncFlush: true is set, the trace is sent to Braintrust in the background after the response is returned — your users see no added latency.

Step 8: CI/CD quality gate

Automate evals on every pull request that touches AI code. Create .github/workflows/eval.yml:

name: AI Evaluation Gate
 
on:
  pull_request:
    paths:
      - "lib/ai/**"
      - "lib/eval/**"
      - "prompts/**"
      - "app/api/**"
 
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci
      - name: Run Braintrust evals
        env:
          BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          npx braintrust eval lib/eval/*.eval.ts --threshold 0.75

The --threshold 0.75 flag exits with a non-zero code if the average score across all experiments drops below 75%, blocking the merge. The threshold is per-project — raise it as your evals mature.

For GitLab CI, the equivalent .gitlab-ci.yml stage:

eval:
  stage: test
  image: node:20
  only:
    changes:
      - lib/ai/**
      - lib/eval/**
  script:
    - npm ci
    - npx braintrust eval lib/eval/*.eval.ts --threshold 0.75
  variables:
    BRAINTRUST_API_KEY: $BRAINTRUST_API_KEY
    OPENAI_API_KEY: $OPENAI_API_KEY

Step 9: Comparing experiments (A/B prompt testing)

Run the same dataset through two prompt variants and compare scores side by side:

import { Eval } from "braintrust";
import { Factuality } from "autoevals";
 
const DATASET = [
  /* your test cases */
];
 
async function summarizeV1(input: { text: string }) {
  const res = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: "Summarize briefly." },
      { role: "user", content: input.text },
    ],
  });
  return res.choices[0].message.content ?? "";
}
 
async function summarizeV2(input: { text: string }) {
  const res = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      {
        role: "system",
        content:
          "You are a technical writer. Summarize the text below in two sentences, preserving all key facts.",
      },
      { role: "user", content: input.text },
    ],
  });
  return res.choices[0].message.content ?? "";
}
 
// Run both experiments
Eval("summarization-v1", { data: DATASET, task: summarizeV1, scores: [Factuality] });
Eval("summarization-v2", { data: DATASET, task: summarizeV2, scores: [Factuality] });

Open the Braintrust Compare view to see which variant wins on each scorer, for each individual example. This makes prompt engineering an evidence-based process rather than guesswork.

Step 10: Tracking regressions over time

Braintrust automatically compares each new experiment run against the previous one. Scores that drop appear in red; improvements in green. To mark an experiment as the official baseline:

npx braintrust eval lib/eval/summarize.eval.ts --set-baseline

Future runs reference this baseline in the dashboard. If a model upgrade accidentally regresses your summarization quality, you see it immediately — not three sprints later when a user reports it.

Troubleshooting

Scores not appearing in the dashboard: Confirm your scorer returns { name: string; score: number }. Both fields are required; returning only the number silently drops the score.

Traces missing in production: On Vercel or Cloudflare Workers, the process may terminate before the async flush completes. Add await logger.flush() at the end of your handler for guaranteed delivery.

Eval runs consuming too many tokens: Use gpt-4o-mini or claude-haiku-4-5 as your judge model for simple classification tasks. Reserve frontier models for tasks that genuinely require deeper reasoning.

Rate limits during parallel eval runs: Braintrust runs dataset cases concurrently by default. Wrap your task with a token-bucket rate limiter, or set the --concurrency flag:

npx braintrust eval lib/eval/*.eval.ts --concurrency 3

Next steps

Explore the Braintrust MCP server to use Claude or GPT-4o to help design eval datasets from natural language descriptions
Build an online eval loop that samples 1–5% of production traces and re-scores them nightly, alerting on drift
Use Braintrust Datasets API to curate golden examples from production logs and continuously expand your test suite
Combine with Langfuse for full-lifecycle observability: Braintrust handles evals, Langfuse handles runtime traces and prompt versioning

Conclusion

You have built a complete AI evaluation pipeline: experiments that score LLM outputs automatically, an LLM judge for open-ended quality, production tracing with zero extra code on your OpenAI client, and a CI/CD gate that blocks deploys when quality regresses. Prompt changes are now treated with the same rigor as code changes — and your users benefit from a measurable, continuously improving AI experience.