AI Prompt Playground (OpenAI / Anthropic Console)

0Problem RestatementProblem

Design a web-based prompt playground, like the OpenAI Playground or the Anthropic Console. A signed-in developer writes a prompt (system message plus user messages), picks a model and settings (temperature, max tokens), clicks Run, and watches the answer stream in real time. They can save prompts, keep versions, compare runs side by side, and come back later to see history.

Anthropic also asked about very large prompts (hundreds of KB to MB): where should that content live?

1RequirementsRequirements

1.1 Functional

• An editor for prompts with variables (e.g., {{customer_name}}).
• Run against a chosen model with settings, and stream the output.
• Save prompts, create versions and view run history.
• Compare outputs (e.g., two models side by side).
• Stop a running generation.

1.2 Non-Functional

• Responsive: time to first token under a second or two, and a smooth UI while streaming.
• Durable: saved prompts and runs are never lost.
• Handles large prompts without slowing everything else.
• Cost and abuse control: rate limits and spend limits per user or organization.

1.3 Scale Estimates

• 1M monthly developers, 100K daily active, ~20 runs each → 2M runs/day ≈ 25/sec, peak 100/sec.
• A typical prompt is ~5 KB, but some are 1 MB+. Outputs are ~2 KB.
• Storage: 2M runs × ~10 KB ≈ 20 GB/day (much more if large prompts are copied into every run, which we'll avoid).

1.4 API Design

2High-Level ArchitectureArchitecture

2.1 Overview

• Frontend (React): the editor, a streaming output panel and a compare view. Autosaves drafts locally.
• Playground API: auth, rate limits and spend checks, saving prompts and versions, and starting runs.
• Run Service: builds the final request (fills in variables), calls the model gateway, streams tokens back over SSE, and records usage.
• Model Gateway: the same inference service the public API uses (routing, batching, quotas).
• Metadata DB (Postgres): prompts, versions and runs.
• Object storage: large prompt bodies and outputs, stored by content hash.

2.2 Architecture Diagram

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
    B["Browser - editor + stream view"] -->|"save"| API["Playground API"]
    B -->|"run - SSE"| RS["Run Service"]
    API --> DB[("Postgres - prompts, versions, runs")]
    API --> OS[("Object storage - large bodies by hash")]
    RS --> DB
    RS --> OS
    RS --> MG["Model Gateway"]
    MG --> GPU["Model servers"]
    RS -->|"usage"| BILL["Usage and spend limits"]

3Data ModelData model

CREATE TABLE prompts (prompt_id UUID PRIMARY KEY, org_id UUID, owner_id UUID, name TEXT, latest_version INT);
CREATE TABLE prompt_versions (
  version_id UUID PRIMARY KEY, prompt_id UUID, version INT,
  model TEXT, params JSONB,
  body_inline JSONB,         -- small prompts stored directly
  body_ref TEXT,             -- 'sha256:ab12...' when the body is large (object storage)
  body_bytes INT, created_at TIMESTAMP
);
CREATE TABLE runs (
  run_id UUID PRIMARY KEY, version_id UUID, user_id UUID, model TEXT, params JSONB,
  variables JSONB, output_inline TEXT, output_ref TEXT,
  status TEXT, input_tokens INT, output_tokens INT, latency_ms INT, created_at TIMESTAMP
);

Versions are immutable. Editing creates a new version, so every run points to exactly what was sent.

4Key FlowsFlows

4.1 Run with streaming

1. The browser posts the run request and keeps the SSE connection open.
2. The Run Service checks the rate limit and spend limit, loads the version (from inline or object storage), fills in variables, and calls the model gateway with streaming on.
3. Each token chunk is forwarded to the browser right away. The service also buffers the output.
4. On finish (or stop), it saves the run with the output and token usage, and adds the cost to the org's spend.

4.2 Stop

The browser calls stop (or closes the connection). The Run Service cancels the upstream request so GPU time isn't wasted, and saves the partial output with status stopped.

5Deep Dive A — When the prompt is a megabyteDeep dive

Developers paste entire documents into the system message. A prompt that was two kilobytes on Monday is a 900 KB contract by Friday, and every saved version stores another copy.

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  V["Save version - 900 KB body"] --> PG[("versions table")]
  PG --> TOAST["Large values stored out of line"]
  PG --> LIST["List a project's 200 versions"]
  LIST --> HEAVY["Reads drag 180 MB through the DB for a sidebar"]
  V --> DUP["Same document saved 40 times - 36 MB of identical bytes"]

%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
  B["Body"] --> T{"Under 64 KB?"}
  T -->|"yes"| PG[("Inline in Postgres - fast and simple")]
  T -->|"no"| H["Browser hashes it - SHA-256"]
  H --> EX{"Object already exists?"}
  EX -->|"yes"| REF["Just save the reference - zero bytes uploaded"]
  EX -->|"no"| PUT["Pre-signed URL - browser uploads directly to object storage"]
  PUT --> REF
  REF --> ROW["Version row holds the hash"]

6Deep Dive B — Compare, reliability and costDeep dive

• Compare mode: fire the same prompt at two models in parallel, with two streams in one SSE connection (tagged by run ID) or two connections.
• Streaming reliability: if the connection drops, the run keeps going on the server (for a short time) and the output is saved. The UI can reload the finished run by ID.
• Cost control: per-user and per-org rate limits, a monthly spend cap with alerts at 80% and 100%, and max_tokens limits.
• History at scale: runs are append-only. Partition the runs table by month and archive old outputs to object storage.

7Trade-offs & AlternativesTrade-offs

8Common Follow-up QuestionsFollow-ups

• "Sharing prompts?" Add permissions at the prompt level (private, org, link), and see the prompt-sharing design.
• "Evaluations?" Let users attach a dataset of inputs, run a version against all of them as a batch job, and score the outputs.
• "Multi-tenant enterprise?" Scope all data by org_id, add SSO, and optionally turn off saving of run contents for sensitive orgs.

9Wrap-UpWrap-up

The browser editor saves immutable prompt versions to Postgres, keeping large bodies in object storage by content hash. The Run Service fills in variables, calls the shared model gateway and streams tokens back over SSE, while saving the output and usage at the end. Rate limits and spend caps control cost, and stopping a run cancels the upstream generation.