Problem RestatementProblem
NVIDIA asked: you need to deploy your services on Kubernetes in a new cloud account for the first time. Explain everything from zero: account bootstrapping, networking, the cluster, security, deployments, observability, and how you'd keep it safe and repeatable.
Kubernetes (K8s) is a system that runs containers across a group of machines, restarts them if they fail, and scales them.
Step 1: Account and Foundations
- Account structure: separate accounts or projects for
dev,stagingandprod, so mistakes in dev can't touch prod. Use an organization with central billing and guardrails (policies that block public buckets, restrict regions). - Identity: SSO for humans, no long-lived personal keys, and least-privilege roles.
- Infrastructure as Code (Terraform or Pulumi) from day one: everything below is code, reviewed and versioned, so environments are reproducible.
Step 2: Networking
- A VPC per environment, with private subnets for nodes (no public IPs) across 3 availability zones, and public subnets only for load balancers and NAT gateways.
- Plan IP ranges carefully (pods can use many IPs), and avoid overlap with offices or other VPCs you may connect later.
- Egress via NAT, and private endpoints for cloud services (storage, registry) so traffic doesn't cross the internet.
Step 3: The Cluster
- Use the cloud's managed Kubernetes (EKS/GKE/AKS), where the control plane is run for you.
- Node pools: a general pool (on-demand) for system and critical services, a spot/preemptible pool for batch work, and a GPU pool (with the NVIDIA device plugin and drivers) using taints, so only GPU workloads land there.
- Cluster autoscaler (or Karpenter) to add and remove nodes, and the Horizontal Pod Autoscaler for pods.
- Essential add-ons: ingress controller, cert-manager (TLS), external-dns, a metrics server, and a CSI storage driver.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
DEV["Git push"] --> CI["CI - build, test, scan image"]
CI --> REG[("Container registry")]
CI --> CD["CD - GitOps (Argo CD)"]
CD --> K8S["Kubernetes cluster - private subnets, 3 AZs"]
LB["Cloud load balancer + ingress"] --> K8S
K8S --> OBS["Observability - metrics, logs, traces"]
SEC["Secrets manager"] --> K8S
IAC["Terraform - VPC, cluster, IAM"] --> K8SDeep Dive — How a pod gets permission to touch cloud resourcesDeep dive
A service needs to read from a bucket and a queue. How it obtains those credentials is the security decision that matters most in a new cluster.
Long-lived keys in the environment
Create an access key, put it in the deployment manifest or a config map.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
K["Long-lived access key"] --> ENV["Env var in the manifest"]
ENV --> GIT["Manifest is in Git - the key is in history forever"]
ENV --> ANY["Any process in the pod can read it"]
ENV --> NOROT["Rotation means redeploying every service that uses it"]
ENV --> BLAST["One leaked key works from anywhere on the internet"]The credential is long-lived, copied into places you do not control, and usable from outside the cluster. Rotation is a coordinated redeploy, so in practice it does not happen.
A secrets manager synced into Kubernetes
Store credentials in a secrets manager and sync them into Kubernetes secrets, so nothing sensitive is in Git.
A genuine improvement — secrets leave source control and rotation has a central place. But what lands in the pod is still a long-lived cloud credential, readable by anything in the container and valid anywhere. The storage improved; the credential did not.
Workload identity, so there is no credential to steal
Map a Kubernetes service account to a cloud IAM role. The pod exchanges its projected service-account token for short-lived cloud credentials, scoped to that workload.
%%{init: {"look":"handDrawn","handDrawnSeed":7,"theme":"base","fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","themeVariables":{"fontFamily":"Virgil, \"Segoe Print\", \"Comic Sans MS\", cursive","fontSize":"16px","primaryColor":"#fff4e6","primaryBorderColor":"#1e1e1e","primaryTextColor":"#1e1e1e","secondaryColor":"#e7f5ff","tertiaryColor":"#ebfbee","lineColor":"#1e1e1e","textColor":"#1e1e1e","mainBkg":"#fff4e6","nodeBorder":"#1e1e1e","clusterBkg":"#f8f9fa","edgeLabelBackground":"#ffffff","classText":"#1e1e1e"}}}%%
flowchart LR
SA["Kubernetes service account"] --> TOK["Projected, short-lived token"]
TOK --> STS["Cloud identity provider"]
STS --> ROLE["IAM role scoped to this workload"]
ROLE --> CRED["Credentials valid for minutes, auto-refreshed"]
CRED --> POD["Pod - nothing to rotate, nothing to leak"]
NP["Network policy - default deny between namespaces"] --> POD
IMG["Signed images from our registry, scanned in CI"] --> POD
NR["Non-root, read-only root filesystem"] --> POD- Nothing durable to steal. Credentials expire in minutes and are tied to the workload's identity, so an exfiltrated token is useless almost immediately and only ever had that service's permissions.
- Never node-wide credentials. Attaching a role to the node gives every pod on it the same access — the mistake that turns one compromised container into cluster-wide cloud access.
- RBAC per namespace so teams deploy only to their own, and default-deny network policies so a compromised pod cannot reach services it has no business calling.
- Only signed images from your registry, scanned in CI, running non-root with a read-only root filesystem — the controls that limit what a compromise can do once it happens.
The principle to state: identity, not secrets. Every mechanism above replaces something you have to store and rotate with something that is derived, scoped and short-lived.
Step 5: Deploying Services
- Containerize each service, write Helm charts or Kustomize manifests: a Deployment with resource requests and limits, readiness and liveness probes, a Service, and an Ingress.
- CI/CD: CI builds and tests the image and pushes it to the registry. CD (e.g., Argo CD, GitOps) applies the manifests from Git, so the cluster always matches the repo.
- Safe rollouts: rolling updates with health checks, and canary releases (Argo Rollouts) that watch error rates and roll back automatically.
- PodDisruptionBudgets and multiple replicas across zones for availability.
Step 6: Observability and Operations
- Metrics (Prometheus + Grafana), logs (Fluent Bit → a log store), traces (OpenTelemetry), and alerts on SLOs.
- Backups of cluster state and persistent volumes (Velero), and a disaster recovery plan (recreate from Terraform + GitOps in another region).
- Cost controls: resource requests right-sized, spot pools for batch, autoscaling, budgets and alerts, and namespace cost reports.
Wrap-UpWrap-up
Start with separate accounts per environment, SSO and Infrastructure as Code, and a multi-AZ VPC with private subnets. Create a managed cluster with general, spot and GPU node pools plus autoscaling and core add-ons. Lock it down with workload identity, a secrets manager, RBAC, network policies and signed images. Ship services through CI plus GitOps CD with probes, canaries and disruption budgets, and finish with metrics, logs, traces, backups and cost controls.