Kubernetes with Mixed CPU Architecture

Creating a Raspberry Pi Kubernetes cluster is straight forward, however it becomes more complicated when you want to mix and match kubernetes nodes of different CPU architectures.

I recently deployed a new Kubernetes cluster in my home lab which initially consisted of two Ubuntu nodes running kubeadm and are  VMs on VMware ESXi. Last night I tried to add a Raspberry Pi  and ran into a couple of issues which I resolved and describe further down to save you some time.

My Kubernetes Setup:Kubernetes Logo

Kubernetes: v1.11.2
Docker: ce-18.6.1
CNI: Flannel v0.10.0
CPU Architectures: AMD64 & ARM
OS: Ubuntu 18.04.1 LTS & Raspbian June 2018
NAS: NFS via FreeNAS

Note: If you’re looking for good instructions on setting up kubeadm on a Raspberry Pi, please take a look at Setup a Kubernetes 1.9.0 Raspberry Pi cluster on Raspbian using Kubeadm

Issue #1 – Kube-Proxy

If you start out with running Ubuntu on your master node, kube proxy will only be available for amd64 CPU architecture. In order to run kube proxy on a Raspberry Pi, you’ll need to deploy a separate daemonset for the ARM architecture. I created a yaml for this purpose and published it on my Github profile. Download kube-proxy-arm.yaml and deploy it with:

If you were successful, you should be able to see kube-proxy-arm running as a container.

 

Issue #2 – CNI Flannel

Flannel v0.10.0 provides two
kube-flannel.yaml files and both are referenced by the official Kubernetes documentation. When I followed the instructions, I didn’t read all the instructions and deployed the official flannel v0.10.0 yaml. However this version only support the amd64 CPU architecture. If you want to mix CPU architectures across your kubernetes clsuter, you want to deploy the following kube-flannel.yaml as it support ARM, ARM64, AMD64, PPC64LE and S390X.

Once you fix the two issues above, you should have all nodes operational and ready to run some containers. Be sure to deploy images which are ARM compatible as not all images are compatible.

 

5 Comments|Add your own comment below

  1. I’m thinking about doing something similar. Have you managed to create a deployment where the pods can run on any of the nodes?
    I’m seeing a problem in my mind where you’ll need a different image for each architecture so deployments will have to be targeted to nodes with labels or something. We would need something that picks an image depending on where it’s running.

    1. I haven’t been able to do this yet. I wonder if someone already has done it. Currently I run ARM and non-ARM nodes and manage them via tags to deploy either an ARM or a non-ARM image.

        1. You can run docker manifest inspect coredns/coredns:1.2.6 or a more readable output from docker run mplatform/mquery coredns/coredns:1.2.6 (coredns as an example, a bunch of images already offer this, I believe all official images).

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.