Kubernetes on Oxide: How customer needs shaped our integrations
· /
使用社群帳號登入
晨露未晞，花已上路
使用社群帳號登入
Oxide 计算机公司近日在其官方博客上详细介绍了 Kubernetes 集成方案的设计与演进历程。文章指出，这一集成并非一蹴而就，而是在与早期客户的持续互动中逐步打磨成型。Oxide 的硬件平台以"云计算机"为核心理念，将计算、存储与网络资源整合在一套可独立部署的机架系统中，而 Kubernetes 作为容器编排领域的事实标准，自然成为其软件生态中需要优先打通的关键环节。
根据博客内容，客户反馈在多个层面直接影响了集成方案的技术路线。例如，早期客户在裸金属环境中部署 Kubernetes 时，对存储卷的动态供给和网络策略的实现方式提出了具体需求，这促使 Oxide 团队重新审视其底层 API 的设计，使其能够更贴合 Kubernetes 控制平面的调用习惯。博客还提到，部分客户希望将 Oxide 的硬件特性——如基于 NVMe 的本地存储和可编程网络——以声明式的方式暴露给 Kubernetes 用户，而非要求运维人员手动配置，这一诉求最终推动了相应控制器与自定义资源定义（CRD）的开发。
从时间线来看，Oxide 的 Kubernetes 集成经历了从"最小可用"到"深度整合"的转变。初期版本侧重于基础的节点注册与工作负载调度，确保 Kubernetes 集群能够在 Oxide 硬件上正常运行；随后，团队根据客户在生产环境中遇到的实际问题，逐步补充了存储编排、负载均衡以及故障恢复等能力。博客强调，这一过程中客户提出的"非理想场景"——例如节点异常宕机后的数据持久性保障——往往比理想路径更能暴露集成方案的薄弱环节，也因此成为迭代优化的主要驱动力。
值得注意的是，Oxide 在文中并未将这一集成定位为"托管 Kubernetes 服务"，而是强调其与公有云托管方案的区别：Oxide 提供的是运行在客户自有数据中心内的基础设施，Kubernetes 集群的创建与管理仍由客户掌控，Oxide 的职责在于让底层硬件能力以最自然的方式融入 Kubernetes 的运维模型。这一立场与 Oxide 一贯主张的"本地部署、云体验"产品哲学保持一致。原文未提供该集成方案的具体版本号或发布时间表，但博客的发布本身表明相关功能已进入可公开讨论的成熟阶段。
13 Aug 2026
图片 图片 图片 Matthew Sanabria Solutions Software Engineer In late 2024, customers and prospects were eager to run Kubernetes on Oxide, but we had no supported integrations to help them do it.
Kubernetes and Oxide are a natural fit. Kubernetes defines the infrastructure behavior it expects through standard extension points, while Oxide exposes the primitives needed to implement that behavior through APIs. The foundation for integration was there. What was missing was the software and an understanding of which integrations customers actually needed.
That was the situation when I joined Oxide as its first Solutions Software Engineer, [1] focused on building software to solve customer problems. My first assignment was to make it easier to deploy and operate Kubernetes on Oxide.
In my first week, I was handed two resources to help me get started:
A customer-submitted pull request for a Rancher node driver
An early draft of RFD 493 Initial Kubernetes Integrations
What began with those two resources grew into a team effort shaped by a feedback loop. Rather than design integrations in the abstract, we followed the problems customers encountered as they moved from provisioning clusters to operating workloads.
This post follows those problems across the Kubernetes lifecycle rather than in strict chronological order. Different provisioning workflows led us to Rancher, Omni, and Cluster API. Running clusters required infrastructure reconciliation, exposing applications revealed networking gaps, and stateful workloads exposed storage constraints. At each stage, customer workflows exposed the next gap, shaping both the integrations we built and the platform work still ahead.
The first gap we tackled was provisioning. Our immediate goal was to unblock the customer who had submitted the Rancher node driver pull request. Working through their use case would also give us firsthand experience creating Kubernetes clusters on Oxide and help us uncover the next problems to solve.
No single provisioning approach fit all customers' workflows, so we ended up publishing three integrations.
Before we could maintain the customer-submitted integration, we needed to understand the workflow it supported. I had never used Rancher or worked with a node driver, so reviewing the contribution meant learning both.
A Rancher node driver is an executable plugin that teaches Rancher how to create and manage virtual machines on a particular infrastructure platform. The Oxide Rancher node driver translates those operations into Oxide API requests. Once installed in Rancher, it lets customers provision Oxide instances as nodes in Rancher-managed Kubernetes clusters.
Testing confirmed that the customer’s implementation worked. I merged the pull request, added CI/CD and documentation improvements, and published the initial release. Oxide officially had its first Kubernetes integration—and a customer was already using it successfully in production!
If you’re a Rancher shop looking to run Kubernetes on Oxide, see our Rancher guide to get started.
Customers expressed interest in using Sidero Labs' Omni to provision Kubernetes clusters running Talos Linux. Omni connects to infrastructure platforms through infrastructure providers, programs that create Talos Linux instances and register them with Omni.
With KubeCon North America 2025 a few months away, we saw an opportunity to partner with Sidero Labs to build and showcase an Oxide infrastructure provider for Omni. We had seven weeks to complete it before our Oxide+Sidero event. [2] Building against a second provisioning platform would also test Oxide’s APIs across distinct customer workflows.
The integration work uncovered several issues across Omni and Talos Linux. I brought those issues to Sidero Labs in siderolabs/omni#1633, where their team was eager to work with us—a lovely reminder of RFD 68 Partnership as Shared Values.
The most memorable issue was siderolabs/talos#11948. Oxide uses a FAT12 filesystem for cloud-init user-data, not ISO 9660, but Talos’s filesystem probe only attempted to read an ISO 9660 superblock from the NoCloud configuration disk. When that read failed, the probe stopped instead of trying other formats such as VFAT or MS-DOS. As a result, Talos never read the Oxide user-data containing the configuration needed to join Omni. The fix would not be released in time for KubeCon, leaving us with a rather funny workaround.
The workaround right now is to pad the user-data with comments to increase its size enough that it uses an ISO 9660 superblock.
KubeCon arrived and we hosted an Oxide+Sidero event to showcase the Oxide infrastructure provider for Omni. Customers could now use this infrastructure provider to provision Oxide instances running Talos Linux as nodes in Omni-managed Kubernetes clusters.
If you’re an Omni or Talos Linux shop looking to run Kubernetes on Oxide, see our Omni guide to get started.
We knew we wanted to build an infrastructure provider for Kubernetes Cluster API (CAPI) when we first wrote RFD 493 Initial Kubernetes Integrations . Cluster API offered something our first two integrations did not—an upstream, provider-extensible API for managing clusters without requiring a third-party platform like Rancher or Omni.
CAPI lets operators declaratively create, scale, upgrade, and delete Kubernetes clusters through Kubernetes custom resources. Infrastructure providers handle the platform-specific work, such as creating and deleting virtual machines. Building one is a significant investment. At the time, customer demand and engineering capacity did not yet justify that investment, so the project was deferred.
Eventually, both changed. Customers began asking for a CAPI provider, and the Solutions Software Engineering team grew. My teammates Josh and Brandon took ownership of the work and released Cluster API Provider Oxide (CAPOx) , giving customers a Kubernetes-native way to provision clusters on Oxide.
The Cluster API workflow also exercises several of our other integrations, allowing us to dogfood [3] the end-to-end cluster workflow. The Kubernetes Image Builder uses our Packer plugin to create CAPI-ready Oxide VM images, which CAPOx uses when provisioning instances. Clusters provisioned with CAPOx also use the separately installed Oxide cloud controller manager (CCM) to integrate Kubernetes with Oxide at runtime.
If you want to provision Kubernetes clusters on Oxide with Cluster API, see our Cluster API guide to get started.
Provisioning integrations create and manage Oxide instances, but they do not reconcile those instances with Kubernetes Node objects. Without that reconciliation, a cluster could not reliably determine whether an unreachable Kubernetes node was temporarily unavailable or whether its backing Oxide instance had been deleted.
We needed a component that ran in each cluster, spoke to the Oxide API, and continuously reconciled Oxide infrastructure with Kubernetes state. Kubernetes provides a standard extension point for this purpose: the cloud controller manager (CCM). A CCM lets infrastructure-specific controllers integrate Kubernetes resources with an infrastructure provider’s API without adding provider-specific code to Kubernetes itself.
We built the Oxide cloud controller manager to connect Kubernetes with Oxide. Its node controller keeps Kubernetes Node objects synchronized with their backing Oxide instances, recording details such as instance IDs and network addresses, and reporting whether each instance is running, shut down, or no longer exists. Kubernetes uses this information to initialize nodes and safely remove them when their backing instances are deleted.
The CCM does not create instances or provision clusters. That remains the job of provisioning integrations such as the Rancher node driver , the Omni infrastructure provider, and CAPOx. Instead, it provides a runtime integration shared across those provisioning workflows.
Importantly, building the CCM gave us a durable extension point inside each cluster. As Oxide evolves, we can add new infrastructure-aware controllers to the CCM rather than update every provisioning integration.
With that runtime extension point in place, we could address another layer of the Kubernetes experience: exposing applications. The CCM architecture also defines a service controller for Kubernetes LoadBalancer services, giving us a place to address the next customer problem.
One of the capabilities customers expect from cloud-integrated Kubernetes is support for Service objects of type LoadBalancer . When a user creates one, Kubernetes asks the cloud provider’s service controller to provision the necessary infrastructure and publish its address in the Service status. There was just one problem: Oxide did not yet offer a native load balancer.
Oxide did, however, have floating IPs. Floating IPs are addresses from a rack’s external IP pools that can be attached to and detached from instances, making those instances reachable from outside their VPCs. Using floating IPs offered a way to unblock LoadBalancer services. A floating IP would deliver traffic to a single Kubernetes node, and the Kubernetes Service dataplane could distribute that traffic to the appropriate pods.
Making that work required accounting for how Oxide floating IPs appear to an instance. They are transparent to the guest in two important ways. First, Oxide translates the destination address of inbound traffic to the instance’s internal IP before sending the traffic to the instance. Second, the instance has no network interface configured with the floating IP.
The resulting traffic flow looks like this:
Traffic flow to a LoadBalancer service using floating IPs. ┌────────────────────────────────────────────────────────────┐ │ Client │ │ Request to floating IP: 45.154.216.233:80 │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Oxide networking │ │ Translates destination to internal IP: 172.30.0.5:80 │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Kubernetes node │ │ Packet arrives at internal IP: 172.30.0.5:80 │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Kubernetes Service dataplane │ │ Selects a Service endpoint │ └────────────────────────────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────┐ │ Pod │ │ Receives traffic on its target port │ └────────────────────────────────────────────────────────────┘ That address translation created a subtle integration problem. The Kubernetes Service dataplane needed to treat the node’s internal IP as a Service frontend because that was the destination address packets actually carried when they reached the guest. The service controller therefore publishes two entries in status.loadBalancer.ingress : [4]
The attached floating IP in Proxy mode
The node’s internal IP in VIP mode
The status entries look like this:
status : loadBalancer : ingress :
$ kubectl get service nginx NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE nginx LoadBalancer 10.106.122.233 45.154.216.233,172.30.0.5 80:30605/TCP 37h Users see both the floating IP and the node’s internal IP in the EXTERNAL-IP column, even though only the floating IP is externally reachable. This is an imperfect abstraction, but it allows us to support a common Kubernetes workflow while waiting for a native Oxide load balancer.
This implementation currently supports externalTrafficPolicy: Cluster , [5] which allows the selected node to forward traffic to a Service endpoint anywhere in the cluster. If that node disappears, the CCM moves the floating IP to another eligible node and updates the internal address in the Service status.
When Oxide introduces a native load-balancing service, we can update the service controller to use it without changing the Kubernetes interface. Customers will continue creating the same LoadBalancer services and only the infrastructure behind them will change.
To install the Oxide CCM on your cluster, see our CCM guide to get started.
With clusters provisioned, reconciled with Oxide, and reachable from outside their VPCs, storage for stateful workloads became the next layer to address. Kubernetes users request persistent storage through PersistentVolumeClaim objects and expect a Container Storage Interface (CSI) driver to create, attach, and mount the underlying volumes. Oxide had disks, but Kubernetes had no native way to manage their lifecycle.
Without an Oxide CSI driver, customers could deploy a third-party Kubernetes storage system such as Longhorn. Longhorn provides its own CSI driver and replicates data across disks attached to Kubernetes workers. However, using Longhorn meant backing its replicas with Oxide distributed disks, which already store three replicas on distinct sleds.
Layering one replicated storage system on another can create substantial write fan-out. When a three-replica Longhorn volume is backed by three-way-replicated Oxide distributed disks, one application write can fan out to as many as nine disk writes. The exact physical write amplification depends on the workload and configuration, but customers wanted to avoid that duplicated replication.
The introduction of Oxide local disks provided a way to remove the second layer of replication. Local disks have no built-in replication and remain tied to their sled, making them well suited to systems such as Longhorn that replicate data across Kubernetes nodes. Our Rancher showcase uses this approach today. It avoids stacking two replicated storage systems, though Longhorn still manages the storage lifecycle rather than a native Oxide integration.
For a native integration, my teammate Luiz wrote RFD 595 Oxide CSI Plugin. The workflow seemed straightforward on paper. When a user creates a PersistentVolumeClaim , the CSI controller creates an Oxide distributed disk. After Kubernetes schedules the pod, the controller attaches that disk to the selected Oxide instance, and the CSI node plugin formats and mounts it for the pod. If the pod is rescheduled onto another node, the controller detaches the disk and reattaches it to the new node.
Prototyping that workflow immediately exposed a blocker. Oxide requires an instance to be stopped before attaching or detaching a disk. Kubernetes, however, expects a CSI driver to attach storage to a running worker after scheduling a pod. Stopping the worker would disrupt every other workload on the node and could trigger cascading scheduling and attachment operations.
Before we can release our CSI plugin, we need to add support for disk hot-plug throughout the Oxide stack, from the hypervisor all the way up to the API. What began as a Kubernetes integration has turned into a project spanning multiple layers of the Oxide software stack.
Disk hot-plug and the Oxide CSI plugin remain under active development as of this writing. In the meantime, customers can use software such as Longhorn with Oxide local disks for dynamically provisioned persistent storage without stacking two layers of replication. When the native CSI plugin ships, customers will be able to use familiar Kubernetes storage APIs backed directly by Oxide distributed disks with replication and durability built in.
The result is not a single Kubernetes integration but a growing ecosystem. Rancher, Omni, and Cluster API provide different paths for provisioning, while the Oxide CCM provides a shared runtime integration for node reconciliation and LoadBalancer services. Customers already use some of these integrations in production, and we dogfood several in our own production workloads. Together, they provide a solid foundation to build on.
Our next step is to expand our dogfooding with the newly released Cluster API provider. Using it to provision and operate more of our clusters will test how these integrations work together day to day.
We still have plenty to build and polish. Our near-term work includes completing disk hot-plug and shipping the CSI plugin, adding autoscaling support, and extending the CCM service controller to support external subnets. Longer term, as we ship resource tagging, OIDC support, and native load balancing, we’ll extend our Kubernetes integrations to take advantage of them.
Building these integrations showed how the architectures of Kubernetes and Oxide complement one another. Kubernetes gives infrastructure providers standard extension points, while Oxide exposes infrastructure primitives through APIs. Oxide’s hardware and software co-design lets us address integration blockers at the layer where they belong and carry the necessary changes through the full stack.
This work also lets us exercise our SDKs and APIs from our customers' perspectives and turn customer friction into product improvements. That feedback loop is how we will continue growing this ecosystem. Customer needs shaped each integration in this post, and they will shape the next one, too.
To see the Cluster API and cloud controller manager integrations in action, watch the video below, in which I deploy a Kubernetes cluster on Oxide.
Deploy Kubernetes on Oxide with Cluster API
1 There’s a team now! Check out the Oxide and Friends episode Solutions Software Engineering with Matthew Sanabria.
2 Our Oxide+Sidero event was November 12, 2025. Work on the Omni infrastructure provider began on September 24.
3 Dogfooding is the practice of using one’s own products or services. Oxide has a rack named dogfood in the office dedicated to, well, dogfooding.
4 Kubernetes uses ipMode to indicate whether traffic reaches the node with the load-balancer address as its destination ( VIP ) or after the destination has been translated ( Proxy ).
5 Supporting externalTrafficPolicy: Local would require the floating IP to follow nodes with local Service endpoints as pods are rescheduled, resulting in more attachment and detachment operations.