writing/tutorial/2026/05
TutorialMay 20, 2026·32 min read

Build a Distributed Task Queue with Hatchet, Postgres and TypeScript

Learn how to build production-grade background jobs and durable workflows with Hatchet — a Postgres-backed task queue for TypeScript. This tutorial covers tasks, multi-step DAGs, retries, rate limiting, concurrency controls, cron schedules and event-driven triggers.

Every serious web app eventually outgrows synchronous request/response code. Sending emails, processing uploaded images, calling slow AI APIs, syncing data across services, running scheduled reports — none of these belong inside an HTTP handler. They need to happen reliably in the background, survive deploys, retry on failure, and scale horizontally across workers.

The classic stack for this in TypeScript has been BullMQ on Redis plus a custom dashboard, or fully managed services like Inngest and Trigger.dev. Hatchet takes a different angle: it is an open-source, self-hostable task queue built directly on Postgres. No Redis. No Kafka. The same database you already trust for your application data also coordinates your background work, with durable state, exactly-once steps, rate limits, concurrency controls and cron schedules built in.

In this tutorial, you will build a media processing and notification pipeline for a Next.js app: when a user uploads an image, Hatchet orchestrates a multi-step workflow that validates the file, generates thumbnails, runs an AI caption, stores results, and sends a notification. Along the way you will learn how Hatchet models tasks, workflows, retries, rate limits, cron schedules and event triggers — and how to deploy a worker fleet in production.

Prerequisites

Before starting, make sure you have:

  • Node.js 20+ installed
  • Docker Desktop running locally (for the Hatchet engine + Postgres)
  • Basic knowledge of TypeScript and async/await
  • Familiarity with Next.js App Router (route handlers, server actions)
  • A code editor such as VS Code

You do not need a paid Hatchet Cloud account for this tutorial — everything runs locally on Docker.

What You Will Build

A media processing pipeline composed of:

  • validateUpload task — checks file size, mime type, and dimensions
  • generateThumbnails task — produces small, medium and large variants
  • captionImage task — calls an AI captioning API with retries
  • persistMetadata task — writes results to your Postgres app schema
  • notifyUser task — sends a push or email notification
  • media-pipeline workflow — a DAG that chains the tasks together with parallel branches
  • A cron-driven cleanup workflow that runs nightly
  • A Next.js route handler that triggers the pipeline on upload

By the end you will understand how Hatchet differs from queue-only systems, when to choose tasks versus full workflows, and how to operate workers in production.

Why Hatchet Instead of BullMQ or Inngest

A short orientation before we code:

  • BullMQ is a Redis-based queue. Excellent for FIFO/priority queues, but you build durable workflows, retries beyond simple counts, dashboards and observability on top of it yourself.
  • Inngest / Trigger.dev are SaaS-first durable execution platforms. They are great DX but introduce vendor lock-in and a billing surface that grows with usage.
  • Hatchet is open source, self-hostable, and uses Postgres as the source of truth. You get durable state, DAG workflows, fan-out, rate limits and concurrency keys with one extra database — often the one you already run.

If your stack already has Postgres and you want to avoid adding Redis and a SaaS dependency at the same time, Hatchet is a strong default.

Step 1: Run Hatchet Locally with Docker

Create a new project directory and start the local Hatchet stack. Hatchet ships a single docker-compose.yml that boots Postgres, RabbitMQ (used internally as a low-latency notification channel) and the Hatchet engine and dashboard.

mkdir hatchet-media-pipeline && cd hatchet-media-pipeline
curl -L https://hatchet.run/install/docker-compose.yml -o docker-compose.yml
docker compose up -d

When the stack finishes booting, the Hatchet dashboard is available at http://localhost:8080. Log in with the default credentials printed in the terminal, create a tenant, and generate an API token from the Settings → API Tokens page. Save it as an environment variable:

echo "HATCHET_CLIENT_TOKEN=your_token_here" > .env
echo "HATCHET_CLIENT_TLS_STRATEGY=none" >> .env

The tls_strategy=none flag is important for local dev — Hatchet uses gRPC over TLS in production, but the local stack runs in cleartext.

Step 2: Install the TypeScript SDK

Initialize a TypeScript project and add the official SDK:

npm init -y
npm install @hatchet-dev/typescript-sdk
npm install -D typescript tsx @types/node dotenv
npx tsc --init

Update tsconfig.json so it works smoothly with modern Node and ESM modules:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "Bundler",
    "esModuleInterop": true,
    "strict": true,
    "skipLibCheck": true,
    "outDir": "dist",
    "rootDir": "src"
  },
  "include": ["src/**/*"]
}

Create the entrypoint for shared Hatchet config:

// src/hatchet.ts
import "dotenv/config";
import { Hatchet } from "@hatchet-dev/typescript-sdk";
 
export const hatchet = Hatchet.init();

Hatchet.init() reads your API token and connection settings from environment variables. No further configuration is needed for the local stack.

Step 3: Define Your First Task

Tasks are the smallest unit of work in Hatchet. Each task has typed inputs and outputs, automatic retries, and durable state. Create the upload validation task:

// src/tasks/validate-upload.ts
import { hatchet } from "../hatchet";
 
type Input = {
  fileUrl: string;
  mimeType: string;
  sizeBytes: number;
};
 
type Output = {
  ok: boolean;
  width: number;
  height: number;
};
 
export const validateUpload = hatchet.task({
  name: "validateUpload",
  retries: 2,
  fn: async (input: Input): Promise<Output> => {
    if (input.sizeBytes > 25_000_000) {
      throw new Error("file too large: must be under 25 MB");
    }
    if (!["image/png", "image/jpeg", "image/webp"].includes(input.mimeType)) {
      throw new Error(`unsupported mime type: ${input.mimeType}`);
    }
    const dims = await probeImage(input.fileUrl);
    return { ok: true, width: dims.width, height: dims.height };
  },
});
 
async function probeImage(url: string) {
  return { width: 1920, height: 1080 };
}

Several things to notice:

  • Typed input and output. The SDK is fully type-safe end to end, so when other tasks consume this output they get autocompletion.
  • retries: 2. Hatchet will retry up to two times on thrown errors before marking the task failed.
  • Pure async function. No queue plumbing, no done() callbacks. Throwing an error fails the task; returning a value succeeds it.

Step 4: Run a Worker

A task definition alone does nothing until a worker picks it up. Workers are long-running processes that subscribe to one or more tasks and execute them as they are scheduled.

// src/worker.ts
import { hatchet } from "./hatchet";
import { validateUpload } from "./tasks/validate-upload";
 
async function main() {
  const worker = await hatchet.worker("media-worker", {
    workflows: [validateUpload],
    slots: 10,
  });
  await worker.start();
}
 
main().catch((err) => {
  console.error(err);
  process.exit(1);
});

The slots: 10 value tells the worker it can run up to ten tasks concurrently. In production you tune this to match the CPU and memory profile of the work the worker performs.

Start it in a separate terminal:

npx tsx src/worker.ts

The worker will register itself with the Hatchet engine and become visible in the dashboard.

Step 5: Trigger the Task

In a third terminal, run a small script that fires the task and waits for its result:

// src/trigger.ts
import { validateUpload } from "./tasks/validate-upload";
 
const result = await validateUpload.run({
  fileUrl: "https://example.com/avatar.png",
  mimeType: "image/png",
  sizeBytes: 1_240_000,
});
 
console.log("validation result:", result);
npx tsx src/trigger.ts

You should see the validated dimensions printed to the console. Open the Hatchet dashboard and you will find the run, its duration, attempts, logs and the typed input/output payload — all stored durably in Postgres.

Step 6: Compose Tasks into a Workflow

A workflow chains tasks together as a DAG, with each step receiving the outputs of its parents. Here is the full media pipeline:

// src/workflows/media-pipeline.ts
import { hatchet } from "../hatchet";
 
type PipelineInput = {
  userId: string;
  fileUrl: string;
  mimeType: string;
  sizeBytes: number;
};
 
export const mediaPipeline = hatchet.workflow<PipelineInput>({
  name: "mediaPipeline",
  on: { event: "media:uploaded" },
});
 
const validate = mediaPipeline.task({
  name: "validate",
  retries: 2,
  fn: async (input, ctx) => {
    return ctx.runTask("validateUpload", {
      fileUrl: input.fileUrl,
      mimeType: input.mimeType,
      sizeBytes: input.sizeBytes,
    });
  },
});
 
const thumbs = mediaPipeline.task({
  name: "generateThumbnails",
  parents: [validate],
  retries: 3,
  fn: async (input, ctx) => {
    const dims = ctx.parents.validate;
    return generateThumbnails(input.fileUrl, dims);
  },
});
 
const caption = mediaPipeline.task({
  name: "captionImage",
  parents: [validate],
  retries: 5,
  rateLimits: [{ key: "ai-caption", units: 1, dynamic: true }],
  fn: async (input) => {
    return aiCaption(input.fileUrl);
  },
});
 
const persist = mediaPipeline.task({
  name: "persistMetadata",
  parents: [thumbs, caption],
  fn: async (input, ctx) => {
    return saveMetadata({
      userId: input.userId,
      thumbnails: ctx.parents.generateThumbnails,
      caption: ctx.parents.captionImage,
    });
  },
});
 
mediaPipeline.task({
  name: "notifyUser",
  parents: [persist],
  fn: async (input) => sendPushNotification(input.userId, "Your upload is ready"),
});

What this gives you:

  • Parallel branches. generateThumbnails and captionImage both depend on validate and run concurrently when validation succeeds.
  • Implicit fan-in. persistMetadata only starts after both parents complete, with their typed outputs available on ctx.parents.
  • Per-task retry policies. The flaky AI caption call retries up to five times; thumbnail generation up to three.
  • Rate limiting. The AI call participates in a shared rate-limit bucket named ai-caption, so the whole worker fleet honors a single throughput ceiling.

Add the workflow to the worker registration in src/worker.ts:

const worker = await hatchet.worker("media-worker", {
  workflows: [validateUpload, mediaPipeline],
  slots: 10,
});

Step 7: Trigger Workflows from Next.js

Wire the pipeline into a Next.js App Router route handler. Place the file under app/api/uploads/route.ts:

import { NextResponse } from "next/server";
import { mediaPipeline } from "@/src/workflows/media-pipeline";
 
export async function POST(req: Request) {
  const body = await req.json();
 
  const handle = await mediaPipeline.run({
    userId: body.userId,
    fileUrl: body.fileUrl,
    mimeType: body.mimeType,
    sizeBytes: body.sizeBytes,
  });
 
  return NextResponse.json({ workflowRunId: handle.workflowRunId });
}

mediaPipeline.run() enqueues a workflow run, durably persists the input in Postgres, and returns immediately with a run ID. Your HTTP handler stays fast, and the rest of the work continues in the background — even if Next.js redeploys mid-flight.

For workflows triggered by external systems (Stripe webhooks, S3 events, Resend events) emit a Hatchet event instead:

await hatchet.events.push("media:uploaded", {
  userId: body.userId,
  fileUrl: body.fileUrl,
  mimeType: body.mimeType,
  sizeBytes: body.sizeBytes,
});

Because mediaPipeline is declared with on: { event: "media:uploaded" }, every push of that event fans out into a workflow run automatically.

Step 8: Add Concurrency Controls

Suppose you want at most three concurrent runs per user, so a single account uploading a hundred files cannot starve the queue for everyone else. Hatchet expresses this declaratively with a concurrency key:

export const mediaPipeline = hatchet.workflow<PipelineInput>({
  name: "mediaPipeline",
  on: { event: "media:uploaded" },
  concurrency: {
    expression: "input.userId",
    maxRuns: 3,
    limitStrategy: "GROUP_ROUND_ROBIN",
  },
});

The engine groups runs by input.userId, caps each group at three running workflows, and dispatches additional runs in round-robin order. No custom locks. No Redis. The state lives in Postgres.

Step 9: Schedule a Cleanup Cron

Background jobs are not just request-driven. Add a nightly cleanup that removes orphan thumbnails older than seven days:

// src/workflows/cleanup.ts
import { hatchet } from "../hatchet";
 
export const cleanupOrphans = hatchet.workflow({
  name: "cleanupOrphans",
  on: { cron: "0 3 * * *" },
});
 
cleanupOrphans.task({
  name: "deleteOrphans",
  fn: async () => {
    const removed = await deleteOrphanThumbnails({ olderThanDays: 7 });
    return { removed };
  },
});

Register the workflow on the worker and Hatchet handles cron scheduling for you, including coordination across multiple worker replicas — exactly one cron firing per interval, no duplicates.

Step 10: Test Tasks in Isolation

Hatchet tasks are plain async functions, which makes unit testing trivial. Pull the business logic into a pure module and assert it directly:

// src/lib/validate.ts
export function validateMediaInput(input: {
  mimeType: string;
  sizeBytes: number;
}) {
  if (input.sizeBytes > 25_000_000) throw new Error("file too large");
  const allowed = ["image/png", "image/jpeg", "image/webp"];
  if (!allowed.includes(input.mimeType)) throw new Error("bad mime");
  return true;
}
// src/lib/validate.test.ts
import { describe, it, expect } from "vitest";
import { validateMediaInput } from "./validate";
 
describe("validateMediaInput", () => {
  it("accepts a valid PNG under 25 MB", () => {
    expect(validateMediaInput({ mimeType: "image/png", sizeBytes: 1_000 })).toBe(true);
  });
  it("rejects oversized files", () => {
    expect(() => validateMediaInput({ mimeType: "image/png", sizeBytes: 26_000_000 })).toThrow();
  });
});

The task wrapper just calls validateMediaInput(). Your tests never have to mock Hatchet itself.

Step 11: Observe Runs in Production

The dashboard at http://localhost:8080 is the same UI you get in production. For every workflow run you see:

  • A timeline of each task, including queue time and execution time
  • The full typed input and output payload (encrypted at rest)
  • Structured logs emitted by ctx.log()
  • Retry attempts and their error messages
  • Rate-limit and concurrency-key state

For programmatic monitoring, expose Hatchet metrics to Prometheus and feed them into a Grafana dashboard alongside your existing app metrics. Latency percentiles per task are particularly useful when chasing slow steps in a DAG.

Step 12: Deploy a Worker Fleet

A production deployment usually consists of three pieces:

  1. The Hatchet engine — run via Helm chart, Fly Machines, Railway, or your existing Kubernetes cluster, pointed at a managed Postgres (Neon, Supabase, RDS).
  2. A pool of worker processes — your TypeScript code, deployed as a long-running container (Fly Machines, Render, Railway, ECS) with auto-scaling based on queue depth.
  3. Your application — Next.js on Vercel, a Bun server on Coolify, a Laravel monolith, or whatever else. It only needs the SDK to enqueue runs and push events.

Dockerize the worker with a minimal Node image:

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY dist ./dist
ENV NODE_ENV=production
CMD ["node", "dist/worker.js"]

Build once, deploy as many replicas as your throughput demands. Hatchet handles the work distribution; you focus on the business logic.

Troubleshooting

A few issues you may hit:

  • UNAVAILABLE: connection refused when starting a worker locally usually means the Docker stack is not fully booted. Wait until docker compose logs hatchet-engine shows gRPC server listening.
  • Tasks stuck in PENDING typically indicate no worker is registered for that workflow. Confirm the workflow is included in the workflows: array passed to hatchet.worker().
  • Retries never trigger if the function returns gracefully despite a logical failure. Throw an error to signal failure — that is how Hatchet decides to retry.
  • Rate-limit keys not applied across the fleet — make sure every worker registers the same rate-limit definition. Mismatched definitions create independent buckets.

Next Steps

You now have a durable, type-safe, Postgres-backed task queue powering a real media pipeline. To extend the project further:

Conclusion

Hatchet collapses the queue, the worker framework, the scheduler and the durable-state store into one Postgres-backed system. You write plain TypeScript functions, compose them into DAGs, declare retries and rate limits as data, and let the engine handle the rest. The mental model is small, the operational footprint is one extra database, and the source code is open — which makes it an attractive default for teams who already trust Postgres and want background work to feel like the rest of their codebase.

In this tutorial you set up Hatchet on Docker, defined typed tasks, composed them into a workflow with parallel branches, applied retries and rate limits, scheduled a cleanup cron, and triggered the whole pipeline from a Next.js route. The same patterns scale up to handle thousands of jobs per second across a fleet of workers, with the same dashboard and the same Postgres as your single source of truth.