Kubernetes and Containers for Enterprise Data and AI

Project status, prices and support dates in this article are as of 23 September 2026. INVISOM has no commercial relationship with any vendor or project named here.

Kubernetes won the AI platform argument. That is not a reason to adopt it.

The evidence that Kubernetes has become the common runtime for production AI is now strong. In the CNCF's annual cloud native survey, published with Linux Foundation Research in January 2026, 82% of container users run Kubernetes in production, up from 66% in 2023. Among organizations that host generative AI models themselves, 66% use Kubernetes for some or all of their inference. The open source serving and training projects that matter (KServe, Kubeflow, vLLM, Ray, llm-d) are built to run on it, and every major cloud sells a managed version.

The same survey contains the counterweight. The most cited obstacle to cloud native adoption was not technology but culture within development teams (47%), followed by lack of training and security (36% each) and complexity (34%). Only 7% of respondents deploy AI models daily; 47% do so occasionally. And 44% do not yet run AI or machine learning workloads on Kubernetes at all.

Our argument is simple. Kubernetes is the right substrate for data and AI when you have a portfolio of workloads that benefit from sharing one platform, and a team whose job is to run that platform. It is the wrong choice when it is adopted to look modern, to host one model, or to avoid a managed service that would do the job with fewer people. The adoption numbers describe where the industry has landed; they do not describe your cost of getting there.

Where Kubernetes earns its keep for data and AI

Four workload families make the case. Each has a mature open source path, and each has a managed alternative worth pricing first.

Model serving. This is the strongest fit. KServe, which began inside Kubeflow in 2019 as a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon, was accepted as a CNCF incubating project in September 2025. It gives a standard way to deploy, scale and route models, with vLLM as a common engine for large language models. vLLM itself has been a PyTorch Foundation project since May 2025. For larger models, llm-d, contributed to the CNCF as a sandbox project in March 2026 by IBM Research, Red Hat and Google Cloud, adds cache-aware routing and lets the prefill and decode phases of inference scale separately. If you serve several open-weight models to several internal teams, a shared Kubernetes serving layer is usually cheaper and more controllable than one endpoint per team. Whether you should be self-hosting models at all is a separate question, covered in our analysis of RAG, fine-tuning and custom models.

GPU scheduling. GPUs are the most expensive line in most AI budgets, and idle GPUs are the most common waste. Kubernetes improved here materially in version 1.34 (September 2025), when Dynamic Resource Allocation (DRA) became generally available. DRA lets a workload describe the device it needs and leaves the scheduler to allocate it, replacing much of the older, coarser device-plugin model. The CNCF's Certified Kubernetes AI Conformance programme, launched in November 2025, had grown from 18 to 31 certified platforms by March 2026, and its v1.35 requirements now cover in-place pod resizing and workload-aware scheduling for distributed training. Conformance does not guarantee good utilization, but it does make GPU behaviour more consistent across vendors.

Distributed data processing and training. Spark has supported Kubernetes as a generally available scheduler since Spark 3.1 in March 2021, and the community-maintained Kubeflow Spark Operator took over Google's original operator in April 2024. Ray, the distributed compute framework used for data processing, training and reinforcement learning, moved to the PyTorch Foundation in October 2025 and runs on Kubernetes through KubeRay. Running these on a shared cluster makes sense when the same GPUs and nodes can be reused across batch, training and serving.

ML pipelines and lifecycle. Kubeflow, created at Google in 2017, graduated in the CNCF in August 2026. The CNCF lists Bloomberg, NVIDIA, Red Hat, LinkedIn and Spotify among its adopters. It brings pipelines, notebooks, distributed training and integration with KServe and the Kueue job-queueing system.

Notice the governance column in that figure. Every core project now sits in a neutral foundation, CNCF or PyTorch Foundation. That matters for selection: on 30 July 2026, the GPU cloud provider Nscale agreed to acquire Anyscale, the main commercial company behind Ray. Ray itself was not part of the deal, because it already belonged to the PyTorch Foundation. Foundation governance does not remove risk, but it means a change of ownership at a vendor does not change the licence or roadmap control of the project you depend on. It is one of the tests in a vendor-neutral selection process.

What it actually costs

The control plane is cheap. Amazon EKS, for example, charges $0.10 per cluster per hour for a cluster on a Kubernetes version in standard support. The expensive parts are elsewhere.

The upgrade treadmill. The Kubernetes project maintains only the three most recent minor releases, each with about a year of patch support, and ships a new minor release roughly every four months. Version 1.34, the release that made DRA generally available, reaches end of life on 27 October 2026. Managed services follow the same rhythm with their own windows, and falling behind is priced: EKS charges $0.60 per cluster per hour for a version in extended support, six times the standard rate. Every add-on in the AI stack (GPU operator, serving framework, ingress, policy engine) has its own release cycle that has to be tested against each upgrade.

Overprovisioning. In a CNCF FinOps microsurvey (a small sample, just over 100 responses, collected June to November 2023), 49% said Kubernetes had increased their cloud spend, 28% said it was unchanged and 23% said it fell. Seventy percent named overprovisioning as a cause of overspend, and only 19% had accurate cost information; the rest relied on estimates or had no monitoring at all. With GPUs, the same habit is far more expensive.

People. This is the line most business cases leave out. A production platform for data and AI needs people who can run clusters, upgrades, networking, identity, GPU drivers, observability and security, around the clock. The CNCF respondents' top obstacles, culture and training, are a direct description of that cost. INVISOM's view is that a shared data and AI platform on Kubernetes needs a dedicated platform team before it needs a second cluster. If that team cannot be funded, the platform will degrade into a set of unmaintained clusters, which is the worst of both worlds. The organizational side of this choice is covered in our enterprise AI operating model.

Managed, self-managed, or not Kubernetes at all

The decision is less "Kubernetes or not" than how much of the stack you want to own. There are roughly four positions.

1. Fully managed AI and data services: a cloud's model endpoints, managed Spark, or a data platform's built-in ML serving. Lowest operating burden, least control over GPUs and cost per token, highest switching cost.
2. Managed Kubernetes with open source on top: EKS, AKS, GKE or similar, with KServe, vLLM, Kubeflow or Ray. The provider runs the control plane; you run everything above it. This is the sensible default for most enterprises that do need Kubernetes.
3. Enterprise Kubernetes distributions, such as Red Hat OpenShift, on premises or across clouds. Useful when you need one supported platform across private data centres and public clouds, at the cost of subscription fees and a vendor-shaped stack.
4. Self-managed upstream Kubernetes. Justified mainly for specialist providers or very large platform teams with unusual hardware or isolation needs.

When not to use Kubernetes. Do not adopt it to host a single model or a single pipeline; a managed endpoint or managed Spark service will be cheaper in people. Do not adopt it if your AI use is entirely through hosted model APIs; there is nothing for a cluster to schedule. Do not adopt it for a data warehouse workload that a SQL platform already serves well; the platform choices in our warehouse and lakehouse comparison will matter far more. And do not adopt it without a named owner for upgrades.

Containers are still worth using even where Kubernetes is not. A model server or pipeline packaged as a standard container image can run on a managed container service today and move to Kubernetes later, or to another cloud. Containerize by default; orchestrate with Kubernetes when the portfolio justifies it.

Security: AI frameworks assume a trusted network

The security record of the last eighteen months has one consistent lesson. Much of the AI tooling on Kubernetes was designed for trusted research networks and is being deployed into enterprise environments that are not.

• ShadowRay. Ray's Jobs API allows unauthenticated code execution (CVE-2023-48022). Ray's maintainers dispute that this is a vulnerability, because Ray is meant to run inside a strictly controlled network. In November 2025, Oligo Security reported an active campaign using it to turn exposed clusters into cryptomining and data-theft botnets, and counted more than 200,000 Ray servers exposed to the internet.
• IngressNightmare. In March 2025, Wiz disclosed a chain of flaws in the widely used Ingress NGINX controller, rated CVSS 9.8, allowing unauthenticated remote code execution and access to all cluster secrets. Wiz estimated that over 40% of cloud environments were vulnerable. In November 2025 the Kubernetes project announced the controller's retirement, noting that development had depended on one or two people working in their spare time; maintenance stopped in March 2026. Clusters still running it no longer receive security fixes.
• NVIDIAScape. In July 2025, Wiz disclosed a container escape in the NVIDIA Container Toolkit (CVSS 9.0), exploitable with a three-line Dockerfile, affecting the toolkit up to v1.17.7 and the GPU Operator up to 25.3.1. On shared GPU nodes, a malicious container could gain root on the host.

The controls that address these are well understood, but they must be applied deliberately:

• Pod security. Pod Security Admission has been stable since Kubernetes 1.25. Enforce the restricted profile by default, and grant baseline or privileged only to named namespaces with a reason.
• Network isolation. Default-deny network policies; no Ray dashboards, notebook servers or admission webhooks reachable from the internet; authentication in front of every model endpoint.
• Isolation for GPUs. Treat shared GPU nodes as a multi-tenant boundary. Keep the GPU operator and container toolkit patched, and separate untrusted workloads onto dedicated nodes or sandboxed runtimes.
• Supply chain. Pin and sign images, and keep an SBOM for every image, model server and operator you run. The Kubernetes project signs its own release artifacts with Sigstore cosign and publishes SBOMs for each release, so the practice is easy to ask of others. Regulation is moving the same way: the EU Cyber Resilience Act's reporting obligations for actively exploited vulnerabilities apply from 11 September 2026, with its main obligations, including documented component inventories, from 11 December 2027.
• Dependency health. Track the maintainer base of every add-on, not only its CVEs. Ingress NGINX shows that a popular component can be retired because too few people maintain it.

The Gulf view: GPUs and regions first

For organizations in Saudi Arabia and the UAE, two practical constraints come before the architecture. First, managed Kubernetes is only as local as the cloud region, and GPU instance types are not offered in every region. Microsoft has said its Saudi Arabia East region will open in November 2026 and AWS has announced its Saudi region for December 2026, so Saudi in-Kingdom options for managed Kubernetes with GPUs should be re-checked during 2027. Second, the CNCF has said it intends to add sovereign AI requirements, focused on sandboxing and data privacy, to its AI conformance programme later in 2026. For regulated workloads, a portable, conformant Kubernetes stack is one of the more credible ways to keep the option of moving between a hyperscaler region and an in-country provider.

What to do in the next 90 days

1. List the workloads, not the technology. Count the models you serve, the pipelines you run and the GPU hours you use. If the list is short, price a managed service first.
2. Name the platform owner and fund the team before approving a cluster. Write down who upgrades, who patches, and who is on call.
3. Start on managed Kubernetes with KServe and vLLM for serving, and add Kubeflow or Ray only when there is a training or pipeline workload to justify them.
4. Put a version calendar on the wall. Map every cluster's Kubernetes version against end-of-life dates, and budget the extended-support fee as the cost of slipping.
5. Measure GPU utilization and cost per inference from the first month, by team, so overprovisioning is visible.
6. Close the four doors: enforce restricted pod security, default-deny networking, no internet-exposed Ray or admission endpoints, and patched GPU toolkits.
7. Require SBOMs and signed images from internal teams and suppliers, and find any remaining Ingress NGINX controllers.

Frequently asked questions

Is Kubernetes necessary for enterprise AI?

No. It is the most common platform for self-hosted inference: 66% of organizations that host generative AI models use it for some or all inference, according to the CNCF's 2025 survey. But if you consume models through hosted APIs or run one or two models, a managed service is usually cheaper once staff costs are counted.

Should we run Spark on Kubernetes?

Spark has supported Kubernetes as a generally available scheduler since version 3.1 in 2021. It makes sense when you already run a shared Kubernetes platform and want batch, training and serving to share capacity. If Spark is your only workload, a managed Spark or lakehouse service is usually simpler.

What changed for GPUs on Kubernetes?

Dynamic Resource Allocation became generally available in Kubernetes 1.34 in September 2025, giving a standard way to request and allocate GPUs and other devices. The CNCF's AI conformance programme also sets common expectations for AI workloads across 31 certified platforms as of March 2026.

What are the main security risks of AI on Kubernetes?

Tools designed for trusted networks exposed to untrusted ones. Recent examples include over 200,000 internet-exposed Ray servers targeted by the ShadowRay campaign, the IngressNightmare flaws in Ingress NGINX, and a container escape in the NVIDIA Container Toolkit. Pod security, network isolation, patching and SBOMs address most of this.

Sources and further reading

• Kubernetes established as the de facto operating system for AI, CNCF, 20 January 2026; full report, CNCF and Linux Foundation Research; survey coverage with response breakdowns, Cloud Native Now, 26 January 2026.
• Cloud Native FinOps microsurvey, CNCF, 20 December 2023.
• Kubernetes releases and support dates, Kubernetes project; Amazon EKS pricing, AWS.
• Kubernetes v1.34: DRA has graduated to GA, Kubernetes, 1 September 2025.
• CNCF nearly doubles certified Kubernetes AI platforms, CNCF, 24 March 2026.
• Spark Release 3.1.1, Apache Spark; Announcing the Kubeflow Spark Operator, Kubeflow, April 2024.
• KServe becomes a CNCF incubating project, CNCF, 11 November 2025; Kubeflow graduation, CNCF, 17 August 2026.
• vLLM joins the PyTorch Foundation, May 2025; Ray joins the PyTorch Foundation, Linux Foundation, 22 October 2025; Donating llm-d to the CNCF, IBM Research, 24 March 2026; Nscale acquires Anyscale, Nscale, 30 July 2026.
• Pod Security Admission and Verify signed Kubernetes artifacts, Kubernetes documentation.
• IngressNightmare, Wiz, 24 March 2025; Ingress NGINX retirement, Kubernetes, 12 November 2025; NVIDIAScape, Wiz, 17 July 2025; ShadowRay 2.0, Oligo Security, 18 November 2025.
• Cyber Resilience Act, European Commission.
• Saudi regions: Microsoft Saudi Arabia East announcement, Microsoft, 31 August 2026; AWS Saudi region announcement, Saudi Press Agency, September 2026.