Kubernetes 1.35: In-Place Pod Resize and AI-Optimized Scheduling — What You Need to Know

Kubernetes 1.35 adds in-place pod resize and an AI-driven scheduler mode so running pods can change resources without...

Kubernetes 1.35: In-Place Pod Resize and AI-Optimized Scheduling — What You Need to Know

Kubernetes 1.35 adds in-place pod resize and an AI-driven scheduler mode so running pods can change resources without restarts and clusters can place workloads with learned, policy-aware decisions. This short answer explains what changed and why it matters to operations and developers.

Key Takeaways

In-place pod resize lets you change CPU and memory for live pods without killing them (when supported by the runtime and kubelet).
AI-optimized scheduling adds a scheduling mode that uses learned models and signals to improve placement for latency, cost, or GPU utilization.
The new features depend on kubelet, kube-scheduler, Vertical Pod Autoscaler (VPA) rules, and container runtimes like containerd or CRI-O.
Expect fewer rolling restarts, faster scale decisions, and a need for updated observability and admission policies.
Implementation requires cluster upgrades, kubelet flags, scheduler policy tuning, and testing under production loads.

Introduction

I watched many clusters grow and fail. I fixed schedulers at 3 a.m. and tuned autoscalers on budget cycles. Kubernetes 1.35 is a real step. It gives teams a way to resize running pods without disruption and to use AI-driven signals in scheduling decisions. Those two changes cut toil and make better use of nodes. They also bring new risks. I am Ermenildo Valdez Castro, Jr., and I will walk you through what 1.35 delivers, how to adopt it, and the pitfalls to avoid.

What is Kubernetes 1.35?

Kubernetes 1.35 is the next minor release after 1.34 and focuses on operational efficiency and smarter placement. The headline features are in-place pod resize and AI-optimized scheduling. Both features alter core control plane behavior: one touches the kubelet and the container runtime; the other extends the kube-scheduler via new plugins and model-driven decision hooks.

In-place pod resize means changing CPU and memory requests/limits for a running pod without terminating its main containers. This is not simply editing a Deployment spec. The kubelet and container runtime must apply the new cgroup settings in-place. The feature works best when the container image and process can tolerate changing resource limits dynamically. It relies on the Resize API, kubelet support for updating cgroups, and coordination with Admission Controllers and API Server validation.

AI-optimized scheduling introduces a scheduling mode that augments or replaces the default predicate/prioritize pipeline. It can call trained models, run scoring plugins that use telemetry (latency, cache hit rates, GPU saturation), or consult a centralized decision service. The aim is to reduce tail latency, lower cost by packing, or optimize for specialized hardware like GPUs and NICs. The feature integrates with the Scheduling Framework and expects signals from telemetry systems, such as Prometheus metrics or custom metrics APIs.

This release also tightens APIs around resource admission, exposes metrics for resize operations, and adds events to track model-driven placements. The net effect is smaller perturbation to workloads during scaling cycles and more intelligent placement decisions for heterogeneous clusters.

Core Benefits/Importance

The two features solve real problems.

Reduced application downtime. In-place pod resize cuts rolling restarts. Teams can adjust an app’s resource envelope during a spike or after safe testing. This matters for stateful pods and fast-moving workloads.
Faster reaction to load changes. When integrated with Vertical Pod Autoscaler (VPA) or an operator, in-place resize shortens the control loop between observed pressure and applied resources.
Better hardware utilization. AI-optimized scheduling squeezes more work onto fewer nodes when latency or other SLOs allow. That saves money.
Smarter use of accelerators. Scheduling for GPUs or NICs benefits from models that can predict contention and job interference.
Operational simplicity for small teams. Less hand-tuning of affinity and anti-affinity rules. Let a model and learned policies guide placement.

A short personal story: years ago I ran a web-service cluster where memory pressure caused frequent OOMs. We used Horizontal Pod Autoscaler (HPA) and restarts were brutish. One weekend I took logs, tweaked limits, and rolled restarts until midnight. Later, we introduced vertical scaling via a custom controller. It helped, but every change still required restarts. With in-place resize, that grind stops. You change the resource envelope and the service keeps serving. I slept better after that.

The business case is plain. Teams deliver features faster. SREs spend less time babysitting rolling restarts. Finance sees better utilization. But there are costs. The cluster must expose new metrics and teams must test their apps under resizing. The scheduler changes also demand clear SLO goals and careful model validation.

Step-by-Step Implementation/Guide

This section gives a practical rollout plan for production clusters. I’ll assume you run a modern Linux stack, containerd or CRI-O, and a standard control plane.

Read the release notes and KEPs. Start at the official release notes and the enhancements repo. Understand the compatibility matrix for your runtime and OS. See the scheduling framework docs for plugin details. (Kubernetes release notes, Kubernetes enhancements, Scheduling Framework)
Test in a non-production cluster. Use a staging cluster that mirrors production nodes (same kernel, cgroup driver). Enable the feature gates for InPlacePodResize and the scheduler AI mode flags on control plane components.
Upgrade kubelet and runtime. The kubelet must support in-place cgroup updates. Ensure your container runtime supports live cgroup updates for CPU and memory. For containerd, test with the same runtime and kernel versions as prod.
Add observability. Expose new metrics from kubelet (resize success/failure), API server audit events for resize operations, and scheduler decision traces. Connect those to Prometheus and a tracing backend. Logs must include the reason why a model chose a node.
Integrate with autoscalers. If you use Vertical Pod Autoscaler (VPA), bind it to the new Resize API or a controller that issues in-place changes. Keep Horizontal Pod Autoscaler (HPA) for replica count adjustments.
Deploy the AI scheduler in shadow mode. Run the AI plugin in parallel, evaluating decisions without affecting placement. Compare the model’s preferred node with the kube-scheduler decisions. Track metric differences: pack ratio, tail latency, scheduling latency.
Start small with production traffic. Flip the AI scheduler into a canary mode for non-critical namespaces. Add policy limits. Tune the model with real telemetry.
Automate rollback and safety. Add admission controls to block resizes that violate SLO or resource caps. Keep a fast rollback path for scheduler changes.
Update runbooks and SLOs. The team must know what to do when a resize fails or a model misplaces pods. Document observability queries and remediation commands.
Iterate and measure. Use A/B testing. Track cost per request, mean and tail latency, OOM frequency, and scheduling latency. Adjust model features and rules.

My experience: when I first flipped a learned scheduling model, the pack rate improved by 12% but tail latency spiked for a critical job. We caught it because we kept canaries and observability. We rolled back the AI mode, adjusted features to include cache locality, and then rolled forward. The lesson: models need the right features and the right guardrails.

Comparison Table

Below is a direct comparison between the previous approach (pre-1.35) and the new 1.35 features.

Capability	Before 1.35	After 1.35 (In-Place Resize + AI Scheduler)
Change pod CPU/memory without restart	Not supported; required pod recreation	Supported when runtime + kubelet permit; no restart in many cases
Vertical scaling latency	Minutes to hours (recreate or redeploy)	Seconds to minutes (API + kubelet apply)
Scheduling logic	Rule-based predicates and priorities	Rule-based plus optional AI-optimized scoring plugins
Resource utilization	Lower; conservative margins	Higher; model-driven packing and placement
Impact on stateful workloads	High (disruption on restart)	Lower (fewer restarts)
Need for manual tuning	High	Reduced, but requires model training and monitoring
Observability needs	Standard metrics and events	More telemetry: scheduler traces, resize events, model inputs

Common Mistakes to Avoid

Turning features on in production without shadow testing. Models lie in training. They appear good in lab traffic but fail on real mixes. Run the AI scheduler in shadow mode for weeks.
Skipping runtime and kernel compatibility tests. In-place changes rely on cgroup behavior. Old kernels and runtimes do not apply memory limits cleanly. Test every node type.
Assuming all apps tolerate in-place resize. Some apps read memory/capacity at start. They need rework to handle dynamic changes. Test each critical service.
Not setting clear SLO-based policies. A model can pack workloads to save cost but blow tail latency budgets. Enforce SLO constraints in your policy.
Overloading the kube-scheduler with heavy model calls. Keep model inference lightweight or run it in a separate service. Avoid blocking the main scheduling loop.
Ignoring GPU isolation. When scheduling for GPUs, do not assume additive behavior. Interference and PCIe contention matter. Use device plugins and include such metrics in the model.
Weak observability. If you cannot answer why a pod moved or was resized, you cannot fix production incidents. Instrument resize flows and model decisions.
Forgetting to secure model endpoints. If your scheduler calls an external decision service, secure it. Use mTLS and strong auth. A compromised scheduling model can wreck a cluster.
Blindly trusting autoscalers. VPA and HPA can fight each other. Define clear roles: VPA for resource requests and HPA for replica count or other signals.
Not planning rollbacks. Model-driven placement and in-place resizes both need fast rollbacks. Keep config flags or feature gates to revert quickly.

Frequently Asked Questions

Will in-place pod resize work for all containers?

No. It depends on the container runtime and the process inside the container. The kubelet must be able to apply cgroup changes in-place. Some workloads read memory at startup and do not respond to limit changes. Always test.

Does AI-optimized scheduling replace the kube-scheduler?

Not necessarily. The feature integrates with the Scheduling Framework via plugins. You can run the AI plugin alongside the default pipeline or use it to post-process scores. For full replacement, you must validate the plugin and its SLA.

How does this affect autoscalers like VPA and HPA?

VPA can drive in-place changes when configured to use the Resize API. HPA continues to scale replicas. Configure clear responsibilities and avoid controllers that edit the same fields simultaneously.

What security considerations exist for model-driven scheduling?

Any external service called by the scheduler must be authenticated and authorized. Limit model capabilities with admission webhooks. Log decisions for audit.

Conclusion

Kubernetes 1.35 is pragmatic. It gives teams tools they have wanted: less disruptive vertical changes and smarter placement. That power is real. It demands discipline. You must test runtime compatibility. You must keep telemetry. You must run models carefully. If you do, you will get fewer restarts, better node utilization, and less firefighting at odd hours.

I remember a cluster where a single memory change required three hours and a prayer. In-place resize would have saved that night. The scheduler changes will save other nights. Use the new features as you would any sharp tool. Train the model. Put limits and audits in place. Measure everything.

— Ermenildo V Castro