Problem RestatementProblem
Adobe asked a machine-learning engineer to design an end-to-end pipeline: given a document that has images and text (like a flyer, social post or slide), predict the best aspect ratio for it (e.g., 1:1, 4:5, 16:9, 9:16), perhaps to suggest a layout or crop for different platforms. The interview checks whether you can go from a vague product goal to data, features, model, evaluation and serving.
Clarify the ProblemProblem
- Output: a fixed set of common ratios (classification: 1:1, 4:5, 16:9, 9:16, A4), or any number (regression on width/height). A fixed set usually matches product needs better, so classification it is, with probabilities, so we can suggest the top 2.
- Input: the document's images (the main image, all images), text (length, headlines), and layout elements (counts, positions).
- Success: users accept the suggestion (product metric), and offline accuracy / top-2 accuracy (ML metric).
Data
- Labels: past documents where users picked a final aspect ratio (implicit labels from product logs), plus a smaller set labeled by designers for quality.
- Cleaning: remove templates the user never edited (they just kept the default), and balance classes (1:1 might dominate).
- Splits: split by user or template, not randomly, so near-duplicate documents don't leak between train and test.
Features
- Image features: embeddings from a pretrained vision model (e.g., CLIP or a ViT). Also the main image's own aspect ratio, faces and salient objects (where the important content is).
- Text features: character and word counts, number of text blocks, headline length, and text embeddings (short vs long copy matters: long text prefers taller formats).
- Layout features: number of elements, their bounding-box spread, and how much margin exists.
- Context: the target platform, if known (Instagram story → 9:16), and the user's past choices.
Deep Dive — Choosing the modelDeep dive
The task is to pick one of a handful of aspect ratios for a document made of images and text. The modelling choice is mostly about how much signal you are willing to leave on the table.
Rules from the document's own dimensions
Measure the canvas, round to the nearest standard ratio, return it.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
D["Document 1100 x 1400"] --> R["Nearest ratio: 4:5"]
R --> ECHO["Predicts what the document already is"]
ECHO --> USE["Useless for the actual question: which ratio suits this content"]
D2["A wide photo with two lines of text, laid out square"] --> ECHOIt answers a different question. The user is asking what ratio their content should be, often because the current one is wrong — so a rule that reads the current dimensions is guaranteed to agree with the mistake.
Gradient-boosted trees on hand-crafted features
Extract features — number and size of images, dominant image aspect ratios, text length, number of text blocks, whitespace, the layout's bounding boxes — and train XGBoost to classify into the ratio set.
Fast, strong, explainable, trains on modest data, and it is the right baseline to state first. Its ceiling is the feature list: a flyer and a social post can have identical counts and boxes while their content implies different crops, and no hand-crafted feature captures what the image is of or what the text says.
Multimodal encoders, with calibrated probabilities
Encode the image and the text with pretrained encoders (frozen to start, fine-tuned if the data supports it), concatenate with the layout features, and train a small MLP head with cross-entropy.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
IMG["Images"] --> IE["Image encoder - pretrained"]
TXT["Text"] --> TE["Text encoder - pretrained"]
LAY["Layout features - counts, boxes, whitespace"] --> CAT["Concatenate"]
IE --> CAT
TE --> CAT
CAT --> MLP["Small MLP head"]
MLP --> SM["Class probabilities over the ratio set"]
SM --> CAL["Temperature scaling"]
CAL --> UI{"Confident?"}
UI -->|"yes"| ONE["Suggest one ratio"]
UI -->|"no"| TWO["Offer the top two - let the user choose"]The encoders bring the semantics the tree model cannot see, and keeping the layout features alongside them matters — the pixels do not tell you there are seven text blocks.
Calibration is the part that is easy to skip and shouldn't be. Raw softmax outputs are overconfident, and this product needs the probability to mean something: the UI shows one suggestion when the model is sure and offers two when it is not. Temperature scaling on a validation set costs one parameter and makes that behaviour honest.Keep the tree model in production as a baseline to compare against. If the multimodal model is not beating it on the offline metric and in the A/B test, the extra serving cost is not buying anything.
Pipeline
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
LOGS["Product logs - final ratios"] --> DS["Dataset builder - clean, balance, split"]
LABEL["Designer labels"] --> DS
DS --> FE["Feature extraction - image + text + layout"]
FE --> FS[("Feature store")]
FS --> TR["Training + eval"]
TR --> REG[("Model registry")]
REG --> SRV["Model server"]
APP["Editor app"] -->|"document"| SRV
SRV -->|"top-2 ratios + confidence"| APP
APP -->|"accepted / changed"| LOGSEvaluation
- Offline: accuracy, top-2 accuracy, per-class precision/recall, and a confusion matrix (mixing up 4:5 and 1:1 is less bad than 16:9 vs 9:16). Check slices too: document types and languages.
- Online: an A/B test where treatment shows suggestions. Measure suggestion acceptance, time to finish a design, and export rates.
Serving
- Latency target: under ~200 ms, since the suggestion appears while editing.
- Compute image embeddings once when an image is added (cache by image hash), so prediction only runs the small head model.
- Batch requests on GPU servers, or run a distilled small model on-device for speed and privacy.
- Log inputs, predictions and user choices for monitoring and retraining.
Monitoring and Retraining
- Watch the distribution of predictions and inputs (e.g., a new platform trend like more vertical video) for drift.
- Retrain monthly (or when drift is detected) with new accepted and changed choices, and promote only if offline metrics and a small online test improve.
Wrap-UpWrap-up
Frame it as classification over common ratios with calibrated probabilities. Build labels from users' final choices (cleaned and split by user) plus designer labels, and use image embeddings, text statistics and embeddings, layout and context features. Start with gradient-boosted trees, then a multimodal network. Evaluate with top-2 accuracy, confusion analysis and an online A/B test, serve with cached image embeddings for low latency, and monitor drift for regular retraining.