Upgrade Shipping Kubernetes Cluster to v1.31
Perform a rolling minor-version upgrade of the shipping-prod EKS cluster from 1.29 to 1.31 ahead of AWS's extended-support deadline. Implementation window 2026-07-11 02:00–05:00 UTC. Expected customer-visible impact: none if the rolling strategy holds.
Change request details
| Submitted | 2026-07-03 10:05 PT by Jordan Lee |
| Implementation window | Saturday 2026-07-11 · 02:00–05:00 UTC (Fri 19:00–22:00 PT) |
| Expected downtime | None expected — rolling node replacement keeps ≥ 2/3 capacity per group at all times |
| Risk rating | Medium — production control-plane change, but staged upgrade path and clean staging soak |
| Service impact | No customer-facing impact expected. Internal risk: any pod without a PodDisruptionBudget could be evicted faster than expected during node draining, causing a brief internal 5xx blip absorbed by client retries. |
| Affected stakeholders | Shipping Services, Carrier Operations, Vendor Operations (indirect — shared ingress) |
| Summary | Upgrade the shipping-prod EKS control plane and node groups from Kubernetes 1.29 to 1.31, one minor version at a time (1.29 → 1.30 → 1.31), replacing nodes via managed node-group rolling update. Addresses AWS extended-support deadline for 1.29 (2026-08-01) and picks up the updated in-tree CSI driver required by label-service v4. Staging soak ran 2026-06-20 → 2026-07-02 with zero regressions. |
Backout plan
If verification (steps 07–08) fails or any P1 alert fires within the window:
- Halt the node-group rollout immediately. Notify #change-control and the CAB chair.
- Scale the old (pre-upgrade) node group back to full capacity; cordon and drain new nodes back onto it.
- The EKS control plane cannot be downgraded — if the control-plane upgrade itself is the problem, open a Sev-1 AWS support case rather than attempting a manual downgrade.
- File a post-incident review within 24 hours; do not retry the upgrade until the PIR is closed.
Approvals
| Role | Approver | Decision | Timestamp |
| Change Manager | Naomi Park | Approved | 2026-07-03 15:40 PT |
| CAB Chair | Marcus Tan — VP Engineering | Approved | 2026-07-04 09:02 PT |
| Business Owner | Priya Mehta — Tech Lead, Shipping | Approved | 2026-07-04 09:20 PT |
| Security Reviewer | Hana Becker — InfoSec | Approved | 2026-07-04 10:11 PT |
Communications plan
- T−5 days — post in #all-eng with the window and expected (nil) customer impact.
- T−24 hours — reminder in #shipping-sre and #carrier-ops; confirm on-call coverage.
- T0 — implementation Slack huddle in #change-control; live updates every 15 min.
- T+0 (post-cutover) — post "upgrade complete" summary in #change-control and #all-eng.
Implementation steps
Nine steps grouped into four phases (Pre-flight · Control plane · Node groups · Verify). Total estimated runtime ≈ 2 hours. Roles: SRE on rotation, Platform Lead (Jordan Lee), QA (Anna Voss).
Run kubectl-convert and pluto detect-helm against every Helm release in shipping-prod. Any resource using an API removed in 1.30/1.31 must be updated before the upgrade starts. Capture the report and attach it to this change ticket. Estimated runtime: 10 minutes.
Run kubectl get deploy -n shipping -o name | xargs -I{} kubectl get pdb -n shipping -l app={} style audit script (sre/scripts/pdb-audit.sh). Any deployment missing a PDB gets one added with minAvailable: 66% before proceeding — otherwise node draining can evict more replicas than intended. Estimated runtime: 15 minutes.
EKS manages etcd automatically, but trigger a manual on-demand backup checkpoint via aws eks-anywhere tooling is not applicable here — instead export the current Helm release manifests and node-group launch templates to s3://acme-eks-snapshots/shipping-prod/2026-07-11/ as a config-level rollback reference. Estimated runtime: 5 minutes.
Run eksctl upgrade cluster --name shipping-prod --version 1.30 --approve. AWS manages the control-plane upgrade with no node impact. Wait for the cluster status to show ACTIVE and the reported version to read 1.30 before continuing — this typically takes 10–15 minutes.
Repeat step 04 targeting --version 1.31. AWS does not support skipping minor versions, so this second hop is required even though nodes are still on 1.29. Estimated runtime: 10–15 minutes.
Update the launch template for shipping-ng-a to the 1.31 EKS-optimized AMI, then run eksctl upgrade nodegroup --cluster shipping-prod --name shipping-ng-a. This cordons and drains old nodes while spinning up new ones, respecting the PDBs from step 02. Confirm all pods rescheduled and Ready before starting the next node group. Repeat for shipping-ng-b and shipping-ng-c. Estimated runtime: 45 minutes total.
Trigger the post-deploy job in CI (qa/smoke-shipping-prod). Suite runs ~60 synthetic transactions against the three critical endpoints. All assertions must pass; latency report attaches to the change ticket. Estimated runtime: 15 minutes.
Check that all cluster add-ons upgraded alongside the control plane — eksctl get addon --cluster shipping-prod should show coredns, kube-proxy, vpc-cni, and aws-ebs-csi-driver all at their 1.31-compatible versions with ACTIVE status.
Once all three node groups confirm healthy for 30+ minutes, remove the superseded 1.29 launch template versions and archive the full rollout log to s3://acme-eks-snapshots/shipping-prod/2026-07-11/rollout.log. Estimated runtime: 5 minutes.
Exception Handling
| If… | Then… |
|---|---|
| A node group fails to rejoin after the rolling restart | Trigger the backout plan immediately; do not attempt a second restart without explicit SRE sign-off. |
| Add-on compatibility error detected post-upgrade | Freeze further node restarts and engage the Platform Lead for a targeted add-on patch before resuming. |
| Monitoring alerts fire during the maintenance window | Pause the rolling restart; evaluate alert context and customer impact before deciding to continue or abort. |
Notes and Tips
/infra/eks/rollback/ from the moment the change ticket is approved — do not wait until the maintenance window is open. If a node group fails to rejoin within 8 minutes of the rolling restart, invoke the backout plan immediately rather than waiting for a second attempt.- Schedule the window to begin no later than 23:00 UTC so the full rolling restart completes before the 06:00 UTC business-hours buffer — partial upgrades left overnight are high-risk.
- Monitor
kube-systemnamespace events continuously during the rolling restart; unexpected pod restarts are an early indicator of problems before metrics spike. - Post a brief status update to #all-eng at the start, midpoint, and close of the window — it reduces inbound interruptions significantly.
Post-implementation sign-off
| Verifier | Confirmation | Timestamp |
| QA — Anna Voss | Smoke suite green; p95 = 201 ms across /label, /rate, webhooks. | 2026-07-11 03:48 UTC |
| SRE on-call — Jordan Lee | No P1/P2 alerts during or 30 min after window; all node groups on 1.31.0. | 2026-07-11 04:20 UTC |
| Platform Lead — Jordan Lee | Change closed in ServiceNow; add-ons verified healthy; #all-eng notified. | 2026-07-11 04:35 UTC |