laurayka/garden
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 19 Wiki Activity
Files
14da5ef936635a7479c3633aa169ad2e7bb2d014
garden/KaraKeep/attachments/25d78444-1af2-4d98-8d56-7350aef62955-Making-dual-stack-ipv6-work.jpg
Lauren Kaviak 14da5ef936
Some checks failed
Deploy Quartz site to Pages / build (push) Successful in 2m43s
Details
Deploy Quartz site to Pages / deploy (push) Failing after 9s
Details
updated content
2026-04-23 20:31:22 -05:00

132 lines
20 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<div id="readability-page-1" class="page"><div><p>2021-07-27 - How to setup a working ipv4/ipv6 service on k3s<br>Tags: <a href="https://www.adyxax.org/tags/ipv6/">Ipv6</a> <a href="https://www.adyxax.org/tags/k3s/">K3s</a> <a href="https://www.adyxax.org/tags/kubernetes/">Kubernetes</a></p><h2 id="introduction">Introduction</h2><p>I have yet to write a lot about the kubernetes setup I use for pieces of my personal infrastructure, because I was not satisfied with what I had to show. Today I picked up k3s again which I like quite a lot for it being a light implementation. Consuming 800M of ram before you get any workload running is hardly lightweight, but it is the lightest I have experienced for kubernetes. An entry level virtual machine at ovh or hetzner having 2G of ram for 3€/month is sufficient to run it, that’s what I have been doing for the last year.</p><p>The main thing I was not satisfied was ipv6 support. I do not know what changed since last year when I tried and failed to make it work in k3s 1.19, but now with 1.21 and some effort it does work! Here is how.</p><h2 id="installation">Installation</h2><p>Let’s start with a freshly reinstalled ovh vps with Ubuntu 20.04. Make sure to properly configure ipv6 on it, for this ovh machine I configured a netplan that looks like this :</p><div><pre tabindex="0"><code data-lang="yaml"><span><span><span>network</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>version</span><span>:</span><span> </span><span>2</span><span>
</span></span></span><span><span><span> </span><span>ethernets</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>ens3</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>dhcp4</span><span>:</span><span> </span><span>true</span><span>
</span></span></span><span><span><span> </span><span>match</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>macaddress</span><span>:</span><span> </span><span>fa:16:3e:82:71:b7</span><span>
</span></span></span><span><span><span> </span><span>mtu</span><span>:</span><span> </span><span>1500</span><span>
</span></span></span><span><span><span> </span><span>set-name</span><span>:</span><span> </span><span>ens3</span><span>
</span></span></span><span><span><span> </span><span>dhcp6</span><span>:</span><span> </span><span>no</span><span>
</span></span></span><span><span><span> </span><span>addresses</span><span>:</span><span>
</span></span></span><span><span><span> </span>- <span>2001</span><span>:</span><span>41d0:401:3100:0:0:0:fd5/128</span><span>
</span></span></span><span><span><span> </span><span>gateway6</span><span>:</span><span> </span><span>2001</span><span>:</span><span>41d0:0401:3100:0000:0000:0000:0001</span><span>
</span></span></span><span><span><span> </span><span>routes</span><span>:</span><span>
</span></span></span><span><span><span> </span>- <span>to</span><span>:</span><span> </span><span>2001</span><span>:</span><span>41d0:0401:3100:0000:0000:0000:0001</span><span>
</span></span></span><span><span><span> </span><span>scope</span><span>:</span><span> </span><span>link</span><span>
</span></span></span></code></pre></div><p>After installation I just ran an <code>apt dist-upgrade</code> then installed <code>ipvsadm</code>. Afterwards it’s all k3s :</p><div><pre tabindex="0"><code data-lang="sh"><span><span><span>export</span> <span>INSTALL_K3S_VERSION</span><span>=</span>v1.21.3+k3s1
</span></span><span><span><span>export</span> <span>INSTALL_K3S_EXEC</span><span>=</span><span>"server --disable traefik --disable servicelb --disable metrics-server --disable-cloud-controller \
</span></span></span><span><span><span> --kube-proxy-arg proxy-mode=ipvs --cluster-cidr=10.42.0.0/16,fd42::/48 --service-cidr=10.43.0.0/16,fd43::/112 \
</span></span></span><span><span><span> --disable-network-policy --flannel-backend=none --node-ip=37.187.244.19,2001:41d0:401:3100::fd5"</span>
</span></span></code></pre></div><p>As you can see we need to disable quite a few k3s components, mainly flannel which does not support dual stack at all at this time (it has been coming soon© to flannel for quite some time) and servicelb (the internal component to k3s which allows to simply use the LoadBalancer service type). We are going to use Calico instead of flannel therefore we also disable k3s’ internal network policy system, and we are going to need to customize the ingress service so we also disable the integrated traefik. We will use metallb instead of servicelb and ingress-nginx instead of traefik.</p><p>If you are replicating this on your own setup make sure the node-ip addresses are the ones configured on your node, if the cluster-cidr and service-cidr do not conflict with your own you can keep those.</p><p>Once ready review the k3s installation script then run it :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>wget https://get.k3s.io -O k3s.sh
</span></span><span><span>less k3s.sh
</span></span><span><span>bash k3s.sh
</span></span></code></pre></div><p>With k3s installed you should be able to access the kubernetes cli with <code>kubectl get nodes</code> but basic services like coredns pod won’t start before calico is setup.</p><h2 id="calico">Calico</h2><p>Retrieve Calico’s manifests with :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>wget https://docs.projectcalico.org/manifests/calico.yaml
</span></span></code></pre></div><p>Edit this file and locate the <code>ipam</code> section of the ConfigMap. Change it to the following :</p><div><pre tabindex="0"><code data-lang="json"><span><span><span>"ipam"</span><span>:</span> <span>{</span>
</span></span><span><span> <span>"type"</span><span>:</span> <span>"calico-ipam"</span><span>,</span>
</span></span><span><span> <span>"assign_ipv4"</span><span>:</span> <span>"true"</span><span>,</span>
</span></span><span><span> <span>"assign_ipv6"</span><span>:</span> <span>"true"</span>
</span></span><span><span><span>}</span><span>,</span>
</span></span></code></pre></div><p>Then locate the <code>FELIX_IPV6SUPPORT</code> variable in the calico-node DaemonSet configuration and set it to <code>true</code>.</p><p>You can then apply this manifest :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>kubectl apply -f calico.yaml
</span></span></code></pre></div><p>From there for standard pods and services should start properly, give calico some time and check :</p><pre tabindex="0"><code>kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/local-path-provisioner-5ff76fc89d-5xvcg 1/1 Running 0 2m51s
kube-system pod/calico-node-dfwp5 1/1 Running 0 67s
kube-system pod/coredns-7448499f4d-ckzlk 1/1 Running 0 2m51s
kube-system pod/calico-kube-controllers-78d6f96c7b-m527n 1/1 Running 0 67s
</code></pre><p>You should have four pods running : coredns, two calico pods and k3s’ local path provisionner.</p><p>Since this is a cheap and self made infrastructure we are going to rely on metallb to provide us with external connectivity. Install it with :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>wget https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/namespace.yaml -O metallb-namespace.yaml
</span></span><span><span>wget https://raw.githubusercontent.com/metallb/metallb/v0.10.2/manifests/metallb.yaml -O metallb-0.10.2-manifest.yaml
</span></span><span><span>kubectl apply -f metallb-namespace.yaml -f metallb-0.10.2-manifest.yaml
</span></span></code></pre></div><p>Then create a metallb-config.yaml with content like this :</p><div><pre tabindex="0"><code data-lang="yaml"><span><span><span>apiVersion</span><span>:</span><span> </span><span>v1</span><span>
</span></span></span><span><span><span>kind</span><span>:</span><span> </span><span>ConfigMap</span><span>
</span></span></span><span><span><span>metadata</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>namespace</span><span>:</span><span> </span><span>metallb-system</span><span>
</span></span></span><span><span><span> </span><span>name</span><span>:</span><span> </span><span>config</span><span>
</span></span></span><span><span><span>data</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>config</span><span>:</span><span> </span><span>|</span><span>
</span></span></span><span><span><span> address-pools:
</span></span></span><span><span><span> - name: default
</span></span></span><span><span><span> protocol: layer2
</span></span></span><span><span><span> addresses:
</span></span></span><span><span><span> - 37.187.244.19/32
</span></span></span><span><span><span> - 2001:41d0:401:3100::fd5/128</span><span>
</span></span></span></code></pre></div><p>Don’t forget to replace the ipv4 and ipv6 addresses with the ones configured on your node. Then apply this manifest :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>kubectl apply -f metallb-config.yaml
</span></span></code></pre></div><p>Give it a minute then check that everything is ok :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>kubectl -n metallb-system get pods
</span></span><span><span>NAME READY STATUS RESTARTS AGE
</span></span><span><span>pod/controller-6b78bff7d9-szz78 1/1 Running <span>0</span> 86s
</span></span><span><span>pod/speaker-mx46m 1/1 Running <span>0</span> 86s
</span></span></code></pre></div><h2 id="ingress-nginx">Ingress-nginx</h2><p>From there we can setup our ingress-nginx, but it will require a bit of service customization :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.48.1/deploy/static/provider/baremetal/deploy.yaml <span>\
</span></span></span><span><span> -O ingress-nginx-0.48.1.yaml
</span></span></code></pre></div><p>Edit this file and locate the ingress-nginx-controller Service, which is by default of type NodePort. We are going to replace it with two services of type LoadBalancer, one for ipv4 and one for ipv6. Theoretically a single DualStack service should be supported but it does not work for me, the service only listens on its ipv6 address. So we are going to replace the whole ingress-nginx-controller Service with these two entries :</p><div><pre tabindex="0"><code data-lang="yaml"><span><span><span>apiVersion</span><span>:</span><span> </span><span>v1</span><span>
</span></span></span><span><span><span>kind</span><span>:</span><span> </span><span>Service</span><span>
</span></span></span><span><span><span>metadata</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>annotations</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>labels</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>helm.sh/chart</span><span>:</span><span> </span><span>ingress-nginx-3.34.0</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/name</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/instance</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/version</span><span>:</span><span> </span><span>0.48.1</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/managed-by</span><span>:</span><span> </span><span>Helm</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/component</span><span>:</span><span> </span><span>controller</span><span>
</span></span></span><span><span><span> </span><span>name</span><span>:</span><span> </span><span>ingress-nginx-controller-v4</span><span>
</span></span></span><span><span><span> </span><span>namespace</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span>spec</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>type</span><span>:</span><span> </span><span>LoadBalancer</span><span>
</span></span></span><span><span><span> </span><span>ipFamilies</span><span>:</span><span>
</span></span></span><span><span><span> </span>- <span>IPv4</span><span>
</span></span></span><span><span><span> </span><span>ports</span><span>:</span><span>
</span></span></span><span><span><span> </span>- <span>name</span><span>:</span><span> </span><span>http</span><span>
</span></span></span><span><span><span> </span><span>port</span><span>:</span><span> </span><span>80</span><span>
</span></span></span><span><span><span> </span><span>protocol</span><span>:</span><span> </span><span>TCP</span><span>
</span></span></span><span><span><span> </span><span>targetPort</span><span>:</span><span> </span><span>http</span><span>
</span></span></span><span><span><span> </span>- <span>name</span><span>:</span><span> </span><span>https</span><span>
</span></span></span><span><span><span> </span><span>port</span><span>:</span><span> </span><span>443</span><span>
</span></span></span><span><span><span> </span><span>protocol</span><span>:</span><span> </span><span>TCP</span><span>
</span></span></span><span><span><span> </span><span>targetPort</span><span>:</span><span> </span><span>https</span><span>
</span></span></span><span><span><span> </span><span>selector</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/name</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/instance</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/component</span><span>:</span><span> </span><span>controller</span><span>
</span></span></span><span><span><span>---</span><span>
</span></span></span><span><span><span>apiVersion</span><span>:</span><span> </span><span>v1</span><span>
</span></span></span><span><span><span>kind</span><span>:</span><span> </span><span>Service</span><span>
</span></span></span><span><span><span>metadata</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>annotations</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>labels</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>helm.sh/chart</span><span>:</span><span> </span><span>ingress-nginx-3.34.0</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/name</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/instance</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/version</span><span>:</span><span> </span><span>0.48.1</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/managed-by</span><span>:</span><span> </span><span>Helm</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/component</span><span>:</span><span> </span><span>controller</span><span>
</span></span></span><span><span><span> </span><span>name</span><span>:</span><span> </span><span>ingress-nginx-controller-v6</span><span>
</span></span></span><span><span><span> </span><span>namespace</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span>spec</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>type</span><span>:</span><span> </span><span>LoadBalancer</span><span>
</span></span></span><span><span><span> </span><span>ipFamilies</span><span>:</span><span>
</span></span></span><span><span><span> </span>- <span>IPv6</span><span>
</span></span></span><span><span><span> </span><span>ports</span><span>:</span><span>
</span></span></span><span><span><span> </span>- <span>name</span><span>:</span><span> </span><span>http</span><span>
</span></span></span><span><span><span> </span><span>port</span><span>:</span><span> </span><span>80</span><span>
</span></span></span><span><span><span> </span><span>protocol</span><span>:</span><span> </span><span>TCP</span><span>
</span></span></span><span><span><span> </span><span>targetPort</span><span>:</span><span> </span><span>http</span><span>
</span></span></span><span><span><span> </span>- <span>name</span><span>:</span><span> </span><span>https</span><span>
</span></span></span><span><span><span> </span><span>port</span><span>:</span><span> </span><span>443</span><span>
</span></span></span><span><span><span> </span><span>protocol</span><span>:</span><span> </span><span>TCP</span><span>
</span></span></span><span><span><span> </span><span>targetPort</span><span>:</span><span> </span><span>https</span><span>
</span></span></span><span><span><span> </span><span>selector</span><span>:</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/name</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/instance</span><span>:</span><span> </span><span>ingress-nginx</span><span>
</span></span></span><span><span><span> </span><span>app.kubernetes.io/component</span><span>:</span><span> </span><span>controller</span><span>
</span></span></span></code></pre></div><p>Note the metadata names with <code>-v4</code> and <code>-v6</code> suffixes, the <code>type: LoadBalancer</code> and the respective ipFamilies. You can now apply this manifest :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>kubectl apply -f ingress-nginx-0.48.1.yaml
</span></span></code></pre></div><p>Give it some time, then check that the two controller services each get the ipv4 or ipv6 address of your node :</p><div><pre tabindex="0"><code data-lang="sh"><span><span>kubectl -n ingress-nginx get pods,svc
</span></span><span><span>NAME READY STATUS RESTARTS AGE
</span></span><span><span>pod/ingress-nginx-admission-create-hcgdm 0/1 Completed <span>0</span> 52s
</span></span><span><span>pod/ingress-nginx-admission-patch-hl2vw 0/1 Completed <span>1</span> 52s
</span></span><span><span>pod/ingress-nginx-controller-5cb8d9c6dd-5692s 1/1 Running <span>0</span> 52s
</span></span><span><span>
</span></span><span><span>NAME TYPE CLUSTER-IP EXTERNAL-IP PORT<span>(</span>S<span>)</span> AGE
</span></span><span><span>service/ingress-nginx-controller-admission ClusterIP 10.43.244.41 &lt;none&gt; 443/TCP 37s
</span></span><span><span>service/ingress-nginx-controller-v4 LoadBalancer 10.43.139.251 37.187.244.19 80:31501/TCP,443:32318/TCP 37s
</span></span><span><span>service/ingress-nginx-controller-v6 LoadBalancer fd43::2a99 2001:41d0:401:3100::fd5 80:31923/TCP,443:30428/TCP 36s
</span></span></code></pre></div><h2 id="conclusion">Conclusion</h2><p>Now you can deploy your own services, personally I am going to migrate this blog then my privatebin and miniflux instances and see if it is reliable.</p></div></div>
Reference in New Issue View Git Blame Copy Permalink