June 5th, 2020
This is outdated! k3s is now packaged in nixpkgs!
In part 1, you should have got k3s installed onto your nixos system. This part talks about running it as a service.
Firstly, remove your swap filesystem. Kubernetes is not intended to run with swap.
Next, disable the firewall. I had quite a bit of trouble running kubernetes along with the nixos firewall and it was easier for me to disable it completely as my VMs are running in an internal network (however, it probably could work with a bit of fiddling). When you disable the nixos firewall, it’ll also disable iptables, which kubernetes requires, so it has to be added to the system packages.
networking.firewall.enable = false;
environment.systemPackages = with pkgs; [
k3s iptables
];
Finally, the systemd service can be started. The following definition is for the server node. It is crucial that the ExecStartPre stanza activates all the defined kernel modules. kube-proxy depends on many of them and it tries to activate them itself. However, since nixos stores kernel modules in a different location, it will fail to activate them and it will result in many hard to debug errors.
The k3s command can be edited to include the desired cli flags. I disabled traefik and servicelb, as I prefer to use metallb and ingress-nginx.
systemd.services.k3s = {
# Unit
description = "Lightweight Kubernetes";
documentation = [ "https://k3s.io" ];
wants = [ "network-online.target" ];
# Install
wantedBy = [ "multi-user.target" ];
# Service
serviceConfig = {
Type = "notify";t
KillMode = "process";
Delgate = "yes";
LimitNOFILE = "infinity";
LimitNPROC = "infinity";
LimitCORE = "infinity";
TasksMax = "infinity";
TimeoutStartSec = "0";
Restart = "always";
RestartSec = "5s";
ExecStartPre = "${pkgs.kmod}/bin/modprobe -a br_netfilter overlay ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack";
ExecStart = "${pkgs.k3s}/bin/k3s server --tls-san 10.1.2.20 --no-deploy traefik --no-deploy servicelb --token-file /var/keys/k3s_token";
};
};
The systemd config for the agent node is quite similar, the only difference being the k3s command. Again, this can be edited to whatever is required, configuring it the same way you would normally configure k3s.
systemd.services.k3s-agent = {
# Unit
description = "Lightweight Kubernetes";
documentation = [ "https://k3s.io" ];
wants = [ "network-online.target" ];
# Install
wantedBy = [ "multi-user.target" ];
# Service
serviceConfig = {
Type = "exec";
KillMode = "process";
Delgate = "yes";
LimitNOFILE = "infinity";
LimitNPROC = "infinity";
LimitCORE = "infinity";
TasksMax = "infinity";
TimeoutStartSec = "0";
Restart = "always";
RestartSec = "5s";
ExecStartPre = "${pkgs.kmod}/bin/modprobe -a br_netfilter overlay";
ExecStart = "${pkgs.k3s}/bin/k3s agent --server https://10.1.2.20:6443 --token-file /var/keys/k3s_token";
};
};
The kubeconfig file can be found on the master node at /etc/rancher/k3s/k3s.yaml.