Problem RestatementProblem
Salesforce asked: design a platform that connects a product to many third-party vendors: email providers (Gmail, Outlook), team messaging (Slack, Teams), CRMs, and so on. A customer authorizes their vendor account (OAuth). The product can then invoke vendor actions ("send this message to channel X", "create a contact") and receive vendor events ("a new email arrived") via webhooks. The goal: add new vendors quickly, and make integrations secure and reliable.
ArchitectureArchitecture
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
P["Product services"] -->|"action request"| IAPI["Integration API"]
IAPI --> Q[("Action queue - per vendor")]
Q --> EX["Executors - rate limited per vendor/tenant"]
EX --> CONN["Connector plugins - Slack, Gmail, CRM..."]
CONN --> V["Vendor APIs"]
V -->|"webhooks"| WH["Webhook receiver - verify signatures"]
WH --> EQ[("Event queue")]
EQ --> P
AUTH["OAuth service"] --> VAULT[("Token vault - encrypted")]
CONN --> VAULTKey Components
- Connector interface (plugin model): each vendor implements the same small interface:
authorize_url(), exchange_code(), refresh_token()
actions: { "send_message": handler, "create_contact": handler, ... }
parse_webhook(request) -> normalized events
rate_limits(), error_mapping()
Adding a vendor = a new plugin plus config, and the core platform doesn't change.
- OAuth connection: the customer clicks "Connect Slack" → vendor consent screen → callback with a code → exchange it for tokens → store them encrypted in a vault, scoped to (tenant, connection). Request minimum scopes.
- Token refresh: refresh before expiry (background) or on a 401, with a lock so only one refresh happens at a time. Revoked tokens → mark the connection "needs reauthorization" and notify the admin.
Deep Dive — Calling vendor APIs you do not controlDeep dive
"Send this Slack message" and "create this CRM record" look like function calls. They are network calls to systems with their own rate limits, outages and retry semantics.
Call the vendor inside the request
The product calls the platform, which calls Slack, and returns when Slack answers.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
APP["Product request"] --> PLAT["Integration platform"]
PLAT --> V["Vendor API"]
V -->|"429 rate limited"| ERR["Error returned to the user"]
V -->|"slow"| HOLD["Request thread held for seconds"]
V -->|"timeout after the vendor acted"| DUP["User retries - message sent twice"]The vendor's availability becomes the product's, its rate limit becomes the product's throughput limit, and an ambiguous timeout turns into a duplicate because the user retries an action that may already have happened.
Make actions async jobs with retries
Submit {connection_id, action, params}, return a job id, and have workers call the vendor with exponential backoff on failure.
The request is fast and transient failures recover on their own. Two things are still wrong. Retrying a 400 or a 403 will never succeed and burns quota forever. And more workers mean more concurrent calls, so the platform rate-limits itself by accident — the vendor's limit is shared across every tenant using that vendor.
Budget the calls, classify the errors, and key the actions
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SUB["Submit {connection_id, action, params, idempotency_key}"] --> Q[("Job queue")]
Q --> TB{"Token bucket - per vendor AND per tenant"}
TB --> CB{"Circuit breaker for this vendor"}
CB -->|"open"| WAIT["Hold in the queue - do not call"]
CB -->|"closed"| CALL["Vendor API"]
CALL -->|"429 / 5xx"| BACK["Backoff, honour Retry-After"]
BACK --> Q
CALL -->|"400 / 403"| FAIL["Permanent - fail fast, surface the reason"]
CALL -->|"success"| REC["Record the action key as completed"]
REC --> DEDUP["A replayed job with the same key is a no-op"]- Two token buckets, not one. Per vendor, so the platform stays inside Slack's published limits however many workers run; per tenant, so one customer's bulk job cannot consume the shared vendor budget everyone else needs.
- Honour
Retry-After. The vendor is telling you exactly when to come back; ignoring it in favour of your own backoff is how an integration gets throttled harder or banned. - Classify errors before retrying. 429 and 5xx are worth retrying; 400 and 403 are a bad request or a revoked token and must fail immediately with a message the customer can act on — usually "reconnect your account".
- Idempotency both ways. Pass keys to vendors that support them; for the rest, record completed action keys locally so a replayed job after a crash does not send a second message.
A circuit breaker per vendor completes it: when Slack is down, jobs wait in the queue rather than each burning a timeout, and the rest of the platform keeps working — which is the difference between one degraded integration and a degraded product.
Webhooks (vendor → us)
- A public endpoint per vendor: verify signatures (HMAC with the vendor's secret) and timestamps (to stop replays), and respond fast (200 within the vendor's timeout).
- Put events on a queue, deduplicate by vendor event ID, normalize them into our event format, and route to the right tenant and connection.
- Some vendors need subscription renewal (e.g., Microsoft Graph subscriptions expire), so a scheduler renews them.
Operations
- Per-vendor dashboards: success and error rates, latency, rate-limit hits, token refresh failures.
- Fairness: one tenant's bulk sync can't eat all of a vendor's shared app quota (per-tenant limits within the vendor limit).
- Vendor API versioning: connectors pin versions, and contract tests catch breaking changes.
Wrap-UpWrap-up
Give every vendor a connector plugin behind one interface (auth, actions, webhook parsing, limits), connect customer accounts via OAuth with minimum scopes, and keep tokens encrypted in a vault with safe single-flight refresh. Run actions as queued, idempotent jobs with per-vendor and per-tenant rate limits, retries and circuit breakers, and receive vendor events through signature-verified, deduplicated webhooks, with vendor health visible in per-vendor dashboards.