Problem RestatementProblem
Adobe asked an ML engineer to design a color suggestion system, like Adobe Color. Given an input, suggest harmonious colors or palettes:
- from an image ("give me a palette from this photo"),
- from a partial palette ("I picked navy and orange; suggest 3 more"),
- or for a document ("suggest a background and text color that fit my design").
Suggestions should look good, be varied, respect accessibility (readable contrast), and improve from user feedback.
Approach Overview
Combine two parts:
- Rule-based color theory (fast, explainable, works with no data): complementary, analogous, triadic, split-complementary and monochrome schemes, computed on the color wheel (HSL/HSV or the perceptual LAB/LCH color spaces).
- Learned ranking (quality and taste): a model trained on millions of palettes people created, liked or used, which scores candidate palettes.
Extracting Colors from an Image
- Resize the image (e.g., to 200×200) and convert pixels to LAB color space, where distances match human perception.
- Run k-means clustering (k = 5–8) to find dominant colors. Weight by cluster size, and optionally by saliency (colors of the main subject matter more than a big blurry background).
- Remove near-duplicates (colors too close in LAB), and keep a balanced set (dominant, accent, neutral).
Deep Dive — From colours to a palette worth showingDeep dive
Extracting colours is the easy half. Turning them into five suggestions a designer will actually use is where the system earns its keep.
Return the most common colours
Take the top five colours by pixel count and show them as the palette.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
IMG["Photo of a beach"] --> CNT["Most frequent pixels"]
CNT --> OUT["Five near-identical sky blues"]
OUT --> P1["No contrast - unusable as a palette"]
OUT --> P2["The red umbrella, the actual accent, never appears"]Frequency measures area, not importance. Large flat regions dominate, so the result is five shades of the same thing, and the small high-saturation element that makes the image interesting is exactly the one that gets dropped.
Generate from harmony rules
Take the input colours and construct palettes with classical rules — complementary, analogous, triadic, split-complementary — around them.
This produces palettes with real structure, and it is a correct foundation. But a rule generates many candidates and has no opinion about which is good: a technically perfect triad can be muddy, low-contrast, or simply unlike anything people choose. The rules say what is valid, not what is liked.
Generate candidates, rank them on behaviour, then re-rank for constraints
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
IN["Image / partial palette / document"] --> EX["Extract colours - k-means in LAB"]
EX --> GEN["Candidates: harmony rules + nearest neighbours from a palette library"]
GEN --> FEAT["Features: harmony type, contrast, lightness spread, saturation balance, similarity to input, popularity"]
FEAT --> RANK["Learned ranker - trained on shown vs saved or applied"]
RANK --> RR["Re-rank: diversity + WCAG contrast at least 4.5:1"]
RR --> OUT["Top palettes"]Three stages, each doing a job the others cannot:
- Candidate generation keeps recall high by combining rules with nearest neighbours from a library of popular palettes, searched by embedding. Real palettes people made are a source of taste that no rule encodes.
- A learned ranker trained on logs — shown versus saved or applied — is what turns "valid" into "likely to be used". This is the only stage that knows anything about preference.
- Re-ranking applies the constraints a scorer will not. Diversity, because ten near-identical palettes ranked 1–10 is a worse result than five varied ones; and accessibility, enforcing WCAG contrast of at least 4.5:1 for any pair proposed as text on background. A palette that fails contrast is not a low-ranked suggestion, it is an invalid one.
Work in a perceptual colour space (LAB) throughout, for clustering and for distance. RGB distance does not match how different two colours look, so every downstream feature built on it is measuring the wrong thing.
Serving
- API:
POST /v1/colors/suggest { image_url | colors[] | document_features, n: 10 }→ palettes with hex codes and harmony labels. - Latency target: ~200 ms. Image extraction is the costly part, so cache by image hash. The ranker is small, and candidate generation is cheap math.
- Run in the design app's backend. A tiny model could run on-device for instant results.
Evaluation
- Offline: ranking metrics (NDCG, recall@10) on held-out feedback, and designer ratings on a sample.
- Online A/B: palette apply rate, save rate, time to finish a design.
- Guardrails: contrast-compliance rate, and diversity of shown palettes.
- Cold start: a new user gets popular plus rule-based palettes, and personalization grows with their saves.
Wrap-UpWrap-up
Extract dominant colors in perceptual LAB space with saliency-weighted k-means, generate candidate palettes from color-harmony rules and similar popular palettes, and rank them with a model trained on save and apply feedback plus user preferences. Re-rank for diversity and accessible contrast, cache expensive image work by hash, and evaluate with ranking metrics, designer ratings and A/B tests, retraining from feedback over time.