CNI overview

Managing a network where containers can interoperate efficiently is very important. Kubernetes has adopted the Container Network Interface(CNI) specification for managing network resources on a cluster. This relatively simple specification makes it easy for Kubernetes to interact with a wide range of CNI-based software solutions.

With Charmed Kubernetes, these networking 'plug-ins' are deployed as subordinate charms with each node running as a kubernetes-control-plane or kubernetes-worker, and ensure the smooth running of the cluster. It is possible to choose one of several different CNI providers for Charmed Kubernetes, which are listed below:

Supported CNI options

The currently supported base CNI solutions for Charmed Kubernetes are:

• Calico
• Canal
• Cilium
• Flannel
• Kube-OVN
• Tigera Secure EE

By default, Charmed Kubernetes will deploy the cluster using Calico. To choose a different CNI provider, see the individual links above.

The following CNI addons are also available:

• Multus
• SR-IOV

Migrating to a different CNI solution

As networking is a fundamental part of the cluster, changing the network on a live cluster is not straightforward. Currently it is recommended to create a new cluster with Charmed Kubernetes using the desired option. When federation becomes part of a future release of Kubernetes, such a migration should be manageable with no downtime.

CNI Requirements in a LXD

As they develop and mature, CNIs such as cilium and calico more frequently require specific kernel features which make them more difficult to operate from within the confinement of LXD. Charmed Kubernetes makes a best effort to operate CNIs, but admins should consider avoiding LXD as a choice to operate kubernetes-worker or kubernetes-control-plane since the CNIs require deeper access on the host machine's kernel.

If the choice is made to continue using LXD for nodes where a CNI is deployed, the LXD profile must be opened up for those units to adjust the host machine's kernel space. Consider a deployment where calico is deployed as a subordinate unit of kubernetes-control-plane on LXD. Those LXDs exist solely on machines where the primary app is ubuntu-control-plane-nodes with the ubuntu charm.

Both CNIs require access to the host /sys/fs/bpf and this can be exposed through the following profile adjustment

machine_app=ubuntu-control-plane-nodes
# Mount the host /sys/fs/bpf into the lxd containers of these machines
juju exec -a $machine_app -- 'sudo lxc profile edit default << EOF
devices:
  sysfsbpf:
    path: /sys/fs/bpf
    source: /sys/fs/bpf
    type: disk
EOF'

Every container in the machine should then be restarted to update its profile:

readarray units < <(juju status $machine_app --format yaml | yq ".applications.$machine_app.units | keys" )
for unit in "${units[@]}"; do
    juju exec -u $(echo $unit | yq .[0]) -- 'sudo lxc restart --all'
done

# wait for the cluster to settle
juju-wait