qlyoung's wiki
    Trace:

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		Previous revision
network_simulation_with_k8s-topo_on_raspi_cluster [2023/09/09 00:59] – created qlyoungnetwork_simulation_with_k8s-topo_on_raspi_cluster [2024/01/06 05:24] (current) – tags qlyoung
Line 1: Line 1:
-This post covers how I set up [k8s-topo](https://github.com/networkop/k8s-topo) on a Raspberry Pi cluster to perform network simulations. ''k8s-topo'' is a sweet project that lets you spin up arbitrary network topologies on Kubernetes clusters. The router nodes can be cEOS, Quagga, or FRR. However, on ARM, the only supported router image is FRR since it's the only one that currently has ARM images available.+====== network simulation with k8s-topo on raspberry pi 3b+ cluster ====== 
 + 
 +This post covers how I set up [[https://github.com/networkop/k8s-topo|k8s-topo]] on a Raspberry Pi cluster to perform network simulations. ''k8s-topo'' is a sweet project that lets you spin up arbitrary network topologies on Kubernetes clusters. The router nodes can be cEOS, Quagga, or FRR. However, on ARM, the only supported router image is FRR since it's the only one that currently has ARM images available.
 Here's a picture of the cluster: Here's a picture of the cluster:
  
-![Raspberry Pi 3B+ cluster](/assets/images/picluster.jpg)+{{:picluster.jpg?600|Raspberry Pi 3B+ cluster}}
  
-If you're curious, the backplane is a [Bitscope Blade](https://bitscope.com/product/blade/?p=about). I'd like to write another post on how to set up and provision such a cluster when I find the time.+If you're curious, the backplane is a [[https://bitscope.com/product/blade/?p=about|Bitscope Blade]]. I'd like to write another post on how to set up and provision such a cluster when I find the time.
 You can run some pretty big topologies even on such a small cluster, such as You can run some pretty big topologies even on such a small cluster, such as
 this 40-node random topology: this 40-node random topology:
  
-![Undirected graph depicting example network topology](/assets/images/randomtopo.png)+{{:randomtopo.png?600|Undirected graph depicting example network topology}}
  
-Overall it's a capable, compact setup for experimenting with software routers. In my role as a maintainer for FRR it serves particularly well as a testbed for that project, especially for development related to our [Docker images](https://hub.docker.com/r/frrouting/frr).+Overall it's a capable, compact setup for experimenting with software routers. In my role as a maintainer for FRR it serves particularly well as a testbed for that project, especially for development related to our [[https://hub.docker.com|Docker images]].
  
-Fair warning: this involves a *lot* of source builds. The software industry - and this is especially true of the k8s / Docker communities - is largely an amd64 monoculture these days, and our target platform is ARM. Furthermore, Kubernetes is notoriously complicated and somewhat underdocumented and takes full advantage of semantic versioning to regularly break its APIs between versions. Similarly, the Go ecosystem is still a moving target and language features relied upon by tooling often require recent versions. Finally, anyone who's worked with Kubernetes before knows that there's really no such thing as Kubernetes; there's only Kubernetes implementations, each of which have their own little quirks that like to block you. k3s seems to be one of the better ones in this regard but there are still quirks.+Fair warning: this involves a //lot// of source builds. The software industry - and this is especially true of the k8s / Docker communities - is largely an amd64 monoculture these days, and our target platform is ARM. Furthermore, Kubernetes is notoriously complicated and somewhat underdocumented and takes full advantage of semantic versioning to regularly break its APIs between versions. Similarly, the Go ecosystem is still a moving target and language features relied upon by tooling often require recent versions. Finally, anyone who's worked with Kubernetes before knows that there's really no such thing as Kubernetes; there's only Kubernetes implementations, each of which have their own little quirks that like to block you. k3s seems to be one of the better ones in this regard but there are still quirks.
  
 Because of these factors, and because of the painfully slow build time of large software on Raspis, this project is a rather involved and will probably take at least a full day. Nevertheless, with perseverance, we shall prevail. Because of these factors, and because of the painfully slow build time of large software on Raspis, this project is a rather involved and will probably take at least a full day. Nevertheless, with perseverance, we shall prevail.
  
-Prerequisites +===== Prerequisites =====
--------------+
  
-- A k3s cluster of Raspberry Pis, or similar armv7l devices *with hard float +  * A k3s cluster of Raspberry Pis, or similar armv7l devices //with hard float support// 
-  support* +  * Raspbian 10 
-- Raspbian 10 +  * ''bash'' - all of my command lines are specific to ''bash'' but could be adapted to other shells. Also, they're all implied to be root shells. 
-- ''bash'' - all of my command lines are specific to ''bash'' but could be adapted +  * Docker, on the master node:
-  to other shells. Also, they're all implied to be root shells. +
-- Docker, on the master node:+
  
-  <code> +<code bash> 
-  apt install docker.io +apt install docker.io 
-  </code>+</code>
  
-- Patience+  * Patience
  
 I cannot emphasize hard float enough. Unfortunately nobody cares about soft float devices, especially Python. If you are on soft float do not waste your time. If you do attempt this project on soft float and succeed, I'd love to hear about it. I cannot emphasize hard float enough. Unfortunately nobody cares about soft float devices, especially Python. If you are on soft float do not waste your time. If you do attempt this project on soft float and succeed, I'd love to hear about it.
Line 38: Line 37:
 Also, before we go any further, add this line to the end of your ''.bashrc'': Also, before we go any further, add this line to the end of your ''.bashrc'':
  
-<code>+<code bash>
 export KUBECONFIG=/etc/rancher/k3s/k3s.yaml export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
 </code> </code>
Line 44: Line 43:
 A lot of tools look at this environment variable to know where the k8s global config is. A lot of tools look at this environment variable to know where the k8s global config is.
  
-Guide +===== Guide =====
------+
  
 Best to begin from the top. Best to begin from the top.
Line 52: Line 50:
  
  
-### Step 1: Install Dependencies+==== Step 1: Install Dependencies ====
  
 The main thing ''k8s-topo'' needs is a way to configure appropriate network devices in a Kubernetes-sanctioned way; something to sit on top of the "native" Kubernetes networking sludge and provide a playground to allow us to create interfaces accessible to our containers and wire them up. ''k8s-topo'' supports two methods of configuring the network resources it needs. The main thing ''k8s-topo'' needs is a way to configure appropriate network devices in a Kubernetes-sanctioned way; something to sit on top of the "native" Kubernetes networking sludge and provide a playground to allow us to create interfaces accessible to our containers and wire them up. ''k8s-topo'' supports two methods of configuring the network resources it needs.
Line 58: Line 56:
 The first is via a "CNI plugin" (Container Network Interface). When k8s is told that it needs to create some network resources, it first looks at a predetermined location on the node - typically ''/etc/cni/net.d/'' - for a config file(s). This config file tells it which CNI plugins are available and which one to call. In particular, the config file specifies the path of the plugin binary, some general k8s-related options, and some binary-specific options to pass. K8s then calls this binary, passes it the requested state of the cluster network in environment variables, and expects the binary to make it so. The binary itself typically interacts with the k8s API server to perform its actions, although it may do so indirectly by way of other tools, or even with other CNI plugins. The first is via a "CNI plugin" (Container Network Interface). When k8s is told that it needs to create some network resources, it first looks at a predetermined location on the node - typically ''/etc/cni/net.d/'' - for a config file(s). This config file tells it which CNI plugins are available and which one to call. In particular, the config file specifies the path of the plugin binary, some general k8s-related options, and some binary-specific options to pass. K8s then calls this binary, passes it the requested state of the cluster network in environment variables, and expects the binary to make it so. The binary itself typically interacts with the k8s API server to perform its actions, although it may do so indirectly by way of other tools, or even with other CNI plugins.
  
-The author of ''k8s-topo'', [@networkop](https://github.com/networkop), has built a CNI plugin called [meshnet-cni](https://github.com/networkop/meshnet-cni). There is a nice (but slightly outdated) writeup on it [here](https://networkop.co.uk/post/2018-11-k8s-topo-p1/). Tl;dr it creates ''veth'''s for connectivity between pods on the same node and runs a daemon on all the nodes to coordinate creation of VXLAN p2p links for cross-node pod connectivity. When I started on this project, it didn't have support for k8s 1.18, which is what I was using. After opening a GitHub issue asking dumb questions in which I mentioned this fact, he updated it to support 1.18 (thank you!). By that time I had already attempted to use the other supported method, [NetworkServiceMesh](https://networkservicemesh.io/). I got pretty far, but ultimately ran into some behavior that appears to be either a bug in NSM or a misconfiguration by me that I was not able to solve. For completeness, I'll cover my attempts on that front in case you want to try this method out. Maybe someone reading this will tell me what I did wrong.+The author of ''k8s-topo'', [[https://github.com/networkop|@networkop]], has built a CNI plugin called [[https://github.com/networkop/meshnet-cni|meshnet-ci]]. There is a nice (but slightly outdated) writeup on it [[https://networkop.co.uk/post/2018-11-k8s-topo-p1/|here]]. Tl;dr it creates ''veth'' 's for connectivity between pods on the same node and runs a daemon on all the nodes to coordinate creation of VXLAN p2p links for cross-node pod connectivity. When I started on this project, it didn't have support for k8s 1.18, which is what I was using. After opening a GitHub issue asking dumb questions in which I mentioned this fact, he updated it to support 1.18 (thank you!). By that time I had already attempted to use the other supported method, [[https://networkservicemesh.io/|NetworkServiceMesh]]. I got pretty far, but ultimately ran into some behavior that appears to be either a bug in NSM or a misconfiguration by me that I was not able to solve. For completeness, I'll cover my attempts on that front in case you want to try this method out. Maybe someone reading this will tell me what I did wrong.
  
 ''NetworkServiceMesh'' does sort of the same thing as ''meshnet-cni'', but in a different way. Since my end goal was really just to get my network sims going I didn't spend too much time researching what a "Service Mesh" is, but in NSM's case it seems to mean sidecar containers that deploy alongside your pods and provide network connectivity things. ''NetworkServiceMesh'' does sort of the same thing as ''meshnet-cni'', but in a different way. Since my end goal was really just to get my network sims going I didn't spend too much time researching what a "Service Mesh" is, but in NSM's case it seems to mean sidecar containers that deploy alongside your pods and provide network connectivity things.
Line 70: Line 68:
 </summary> </summary>
  
-*I was not able to get this working. I feel like I got pretty close, so if you're adventurous perhaps you can figure out the last piece of the puzzle, I would love to know about it.*+//I was not able to get this working. I feel like I got pretty close, so if you're adventurous perhaps you can figure out the last piece of the puzzle, I would love to know about it.//
  
 NSM is deployed with Helm, so we have to set up Helm first. Helm is like a package manager for k8s. NSM is deployed with Helm, so we have to set up Helm first. Helm is like a package manager for k8s.
Line 79: Line 77:
 (Actually shortly after I wrote this, NSM gained support for Helm 3. I didn't try it since Helm 2 works.) (Actually shortly after I wrote this, NSM gained support for Helm 3. I didn't try it since Helm 2 works.)
  
-<code> +<code bash> 
-wget https://get.helm.sh/helm-v2.16.7-linux-arm.tar.gz+wget "https://get.helm.sh/helm-v2.16.7-linux-arm.tar.gz"
 </code> </code>
  
 Extract the archive and "install" Helm: Extract the archive and "install" Helm:
  
-<code>+<code bash>
 tar xvzf helm-v2.16.7-linux-arm.tar.gz tar xvzf helm-v2.16.7-linux-arm.tar.gz
 cp linux-arm/helm /usr/local/bin/helm cp linux-arm/helm /usr/local/bin/helm
Line 92: Line 90:
 The Helm client is now installed. Next step is to install the backend, called ''tiller''.  k3s is a bit particular regarding permissions, so we need to give ''tiller'' its own k8s service account to do stuff with. The Helm client is now installed. Next step is to install the backend, called ''tiller''.  k3s is a bit particular regarding permissions, so we need to give ''tiller'' its own k8s service account to do stuff with.
  
-<code>+<code bash>
 kubectl -n kube-system create serviceaccount tiller kubectl -n kube-system create serviceaccount tiller
  
Line 101: Line 99:
  
 References: References:
-* <https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/> +  * https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account 
-* <https://rancher.com/docs/rancher/v2.x/en/installation/options/helm2/helm-init/>+  * https://rancher.com/docs/rancher/v2.x/en/installation/options/helm2/helm-init
  
 Now we start getting to the x86 monoculture stuff. Now we start getting to the x86 monoculture stuff.
Line 108: Line 106:
 The backend for Helm, ''tiller'', runs in a container in the k8s cluster itself. Despite the Rancher docs claiming you can simply ''helm init'', in reality, if you do this Helm downloads and deploys an amd64 ''tiller'' image. Obviously on ARM this won't work; it will just crash with ''exec format error''. We need an ARM image. As it turns out, Helm doesn't actually build these, because real computers are all amd64, right? The backend for Helm, ''tiller'', runs in a container in the k8s cluster itself. Despite the Rancher docs claiming you can simply ''helm init'', in reality, if you do this Helm downloads and deploys an amd64 ''tiller'' image. Obviously on ARM this won't work; it will just crash with ''exec format error''. We need an ARM image. As it turns out, Helm doesn't actually build these, because real computers are all amd64, right?
  
-Fortunately, [someone](https://hub.docker.com/u/jessestuart) looking out for us aliens builds ARM images of ''tiller'' and publishes them on his personal DockerHub registry. I could write another post railing against binary-only dependencies downloaded from random people on the internet, maybe another time. Hopefully his registry is still available when you're reading this document. If not you'll have to find out how to build ''tiller'' for ARM yourself.+Fortunately, [[https://hub.docker.com/u/jessestuart|someone]] looking out for us aliens builds ARM images of ''tiller'' and publishes them on his personal DockerHub registry. I could write another post railing against binary-only dependencies downloaded from random people on the internet, maybe another time. Hopefully his registry is still available when you're reading this document. If not you'll have to find out how to build ''tiller'' for ARM yourself.
  
 To initialize Helm with an appropriate backend image: To initialize Helm with an appropriate backend image:
  
-<code>+<code bash>
 helm init --service-account tiller --tiller-image=jessestuart/tiller:v2.16.7 helm init --service-account tiller --tiller-image=jessestuart/tiller:v2.16.7
 </code> </code>
  
-*Note the tag. If you downloaded a later version of Helm 2 earlier, you'll need to change the tag version to match. The backend version must match the client version (you can see what you installed with ''helm version'').*+//Note the tag. If you downloaded a later version of Helm 2 earlier, you'll need to change the tag version to match. The backend version must match the client version (you can see what you installed with ''helm version'').//
  
 Next we'll have to build NSM ourselves, because they don't provide images for anything except amd64. Next we'll have to build NSM ourselves, because they don't provide images for anything except amd64.
Line 124: Line 122:
 Install Go 1.13: Install Go 1.13:
  
-<code> +<code bash> 
-wget https://dl.google.com/go/go1.13.11.linux-armv6l.tar.gz+wget "https://dl.google.com/go/go1.13.11.linux-armv6l.tar.gz"
 tar -C /usr/local -xzf ./go1.13.11.linux-armv6l.tar.gz tar -C /usr/local -xzf ./go1.13.11.linux-armv6l.tar.gz
 export PATH=$PATH:/usr/local/go/bin export PATH=$PATH:/usr/local/go/bin
Line 132: Line 130:
 Clone the NSM repo: Clone the NSM repo:
  
-<code>+<code bash>
 git clone https://github.com/networkservicemesh/networkservicemesh.git git clone https://github.com/networkservicemesh/networkservicemesh.git
 cd networkservicemesh cd networkservicemesh
Line 139: Line 137:
 NSM has two forwarding plane implementations available. One is based on VPP (the default), the other uses the kernel. The VPP image, naturally, doesn't support our particular flavor of ARM: NSM has two forwarding plane implementations available. One is based on VPP (the default), the other uses the kernel. The VPP image, naturally, doesn't support our particular flavor of ARM:
  
-<code>+<code bash>
 root@clusterpi-master:/home/pi/vpp-agent# make prod-image root@clusterpi-master:/home/pi/vpp-agent# make prod-image
 IMAGE_TAG= ./docker/prod/build.sh IMAGE_TAG= ./docker/prod/build.sh
Line 155: Line 153:
 Patch 1: Patch 1:
  
-<code>diff+<code diff>
 diff --git a/forwarder/build.mk b/forwarder/build.mk diff --git a/forwarder/build.mk b/forwarder/build.mk
 index 0d24b89f..23c1cac4 100644 index 0d24b89f..23c1cac4 100644
Line 170: Line 168:
 Patch 2: Patch 2:
  
-<code>diff+<code diff>
 diff --git a/test/build.mk b/test/build.mk diff --git a/test/build.mk b/test/build.mk
 index 97354fbf..d1aff0d3 100644 index 97354fbf..d1aff0d3 100644
Line 185: Line 183:
 At this point we can build the project. This takes about an hour on my Pi 3B+. At this point we can build the project. This takes about an hour on my Pi 3B+.
  
-<code>+<code bash>
 make k8s-build make k8s-build
 </code> </code>
Line 191: Line 189:
 In order to work around some other stuff, we'll need raw tarballs of the Docker images we just built. There's a ''make'' target to save all those. In order to work around some other stuff, we'll need raw tarballs of the Docker images we just built. There's a ''make'' target to save all those.
  
-<code>+<code bash>
 make k8s-save make k8s-save
 </code> </code>
  
-Your built images are now in your local docker registry, but **k3s doesn't use that**, it has a private one. If you try to deploy the NSM Helm chart at this point, you'll end up pulling the amd64 images. These images will then be cached on every single one of your nodes, and will not be pulled again due to the pull policy in NSM's pod specs being ''IfNotPresent'', leading to much pain.+Your built images are now in your local docker registry, **but k3s doesn't use that**, it has a private one. If you try to deploy the NSM Helm chart at this point, you'll end up pulling the amd64 images. These images will then be cached on every single one of your nodes, and will not be pulled again due to the pull policy in NSM's pod specs being ''IfNotPresent'', leading to much pain.
  
 In case you already did this by mistake, you'll need to log into each node and run the following to delete the amd64 images from the cache: In case you already did this by mistake, you'll need to log into each node and run the following to delete the amd64 images from the cache:
  
-<code>+<code bash>
 k3s ctr images list | grep networkservicemesh | cut -d' ' -f1 | xargs k3s ctr images remove k3s ctr images list | grep networkservicemesh | cut -d' ' -f1 | xargs k3s ctr images remove
 </code> </code>
Line 207: Line 205:
 You could also create a local registry and serve your images from there; I tried this, ran into some TLS issues, canned it. You could also create a local registry and serve your images from there; I tried this, ran into some TLS issues, canned it.
  
-*Hacky way*+**Hacky way**
  
 All the images you saved in ''make k8s-save'' should be in ''build/images'' as ''tar'' archives. All the images you saved in ''make k8s-save'' should be in ''build/images'' as ''tar'' archives.
Line 215: Line 213:
 - Run this: - Run this:
  
-  <code> +<code bash> 
-  for file in ./*; do k3s ctr images import $file; done +for file in ./*; do k3s ctr images import $file; done 
-  </code>+</code>
  
 Note you'll have to do this for other images later on, so it's probably better to just use DockerHub. Note you'll have to do this for other images later on, so it's probably better to just use DockerHub.
  
-*Correct Way*+**Correct Way**
  
 Retag your built images and push them to your personal DockerHub repository. Retag your built images and push them to your personal DockerHub repository.
Line 227: Line 225:
 Now, at last, you can deploy NSM: Now, at last, you can deploy NSM:
  
-<code>+<code bash>
 SPIRE_ENABLED=false INSECURE=true FORWARDING_PLANE=kernel make helm-install-nsm SPIRE_ENABLED=false INSECURE=true FORWARDING_PLANE=kernel make helm-install-nsm
 </code> </code>
  
-At this point the containers deployed successfully, but the problem I ran into was some kind of communications between two of NSM's containers, similar to [this bug](https://github.com/networkservicemesh/networkservicemesh/issues/2160).+At this point the containers deployed successfully, but the problem I ran into was some kind of communications between two of NSM's containers, similar to [[https://github.com/networkservicemesh/networkservicemesh/issues/2160|this bug]].
  
 By this time ''meshnet-cni'' was ready to try again so I went in that direction again. By this time ''meshnet-cni'' was ready to try again so I went in that direction again.
Line 241: Line 239:
 Instead of NSM I ended up going with ''meshnet-cni'', and I recommend you go this route as well; it's lighter, simpler and created by the author of ''k8s-topo'' specifically for use with that project. Consequently, changes to one are less likely to break the other :) Instead of NSM I ended up going with ''meshnet-cni'', and I recommend you go this route as well; it's lighter, simpler and created by the author of ''k8s-topo'' specifically for use with that project. Consequently, changes to one are less likely to break the other :)
  
-#### Setting up meshnet-cni+=== Setting up meshnet-cni ===
  
 Just like with NetworkServiceMesh, we have to make a lot of adjustments, build custom images, etc. I've already done these and put them in a fork of ''meshnet-cni'', so rather than walk through the necessary changes you can just clone my branch: Just like with NetworkServiceMesh, we have to make a lot of adjustments, build custom images, etc. I've already done these and put them in a fork of ''meshnet-cni'', so rather than walk through the necessary changes you can just clone my branch:
  
-<code>+<code bash>
 git clone --single-branch --branch k3s-arm https://github.com/qlyoung/meshnet-cni.git git clone --single-branch --branch k3s-arm https://github.com/qlyoung/meshnet-cni.git
 </code> </code>
  
 Brief summary of changes made: Brief summary of changes made:
-- Change all binary downloads to fetch ARM versions +  * Change all binary downloads to fetch ARM versions 
-- Modify CNI config files for k3s Flannel +  * Modify CNI config files for k3s Flannel 
-- Remove node selectors that restrict to amd64 nodes +  * Remove node selectors that restrict to amd64 nodes 
-- Change docker images to point at ARM-compatible builds with above changes +  * Change docker images to point at ARM-compatible builds with above changes 
-- Modify CNI config paths to place them in the custom k3s locations+  * Modify CNI config paths to place them in the custom k3s locations
  
 The last one does require some manual patching. k3s does not use the standard ''/etc/cni/net.d'' location for CNI configs. It will only look at that directory if you disable the built-in CNI, Flannel. However, ''meshnet-cni'' works by layering additional network resources *on top* of resources created by a lower-level CNI referred to as the "delegate"; in this case Flannel. If we disable Flannel and replace it wholesale with ''meshnet-cni'', things will not work, but if we leave Flannel enabled, then ''/etc/cni/net.d'' is completely ignored. This is an unfortunate design choice by k3s that also precludes use of other CNIs that follow the "delegate" pattern, such as Multus. However, we can workaround this by simply installing our CNI to k3s's custom location, which is ''/var/lib/rancher/k3s/agent/etc/cni/net.d''. The last one does require some manual patching. k3s does not use the standard ''/etc/cni/net.d'' location for CNI configs. It will only look at that directory if you disable the built-in CNI, Flannel. However, ''meshnet-cni'' works by layering additional network resources *on top* of resources created by a lower-level CNI referred to as the "delegate"; in this case Flannel. If we disable Flannel and replace it wholesale with ''meshnet-cni'', things will not work, but if we leave Flannel enabled, then ''/etc/cni/net.d'' is completely ignored. This is an unfortunate design choice by k3s that also precludes use of other CNIs that follow the "delegate" pattern, such as Multus. However, we can workaround this by simply installing our CNI to k3s's custom location, which is ''/var/lib/rancher/k3s/agent/etc/cni/net.d''.
Line 262: Line 260:
 To find the GUID: To find the GUID:
  
-<code>+<code bash>
 ls /var/lib/rancher/k3s/data ls /var/lib/rancher/k3s/data
 </code> </code>
Line 268: Line 266:
 There should be a single directory there whose name is a long hash-looking string, in my case: There should be a single directory there whose name is a long hash-looking string, in my case:
  
-<code>+<code bash>
 #  ls /var/lib/rancher/k3s/data #  ls /var/lib/rancher/k3s/data
 ec54df8c1938fe49660230d16334b4c7e83888a93e6f037fd8552893e2f67383/ ec54df8c1938fe49660230d16334b4c7e83888a93e6f037fd8552893e2f67383/
Line 275: Line 273:
 To make sure ''meshnet-cni'' binaries make it into this location, ''cd'' into your ''meshnet-cni'' repo and run this, replacing ''YOUR_GUID'' with the GUID you just found: To make sure ''meshnet-cni'' binaries make it into this location, ''cd'' into your ''meshnet-cni'' repo and run this, replacing ''YOUR_GUID'' with the GUID you just found:
  
-<code>+<code bash>
 export K3S_CNI_GUID=<YOUR_GUID> export K3S_CNI_GUID=<YOUR_GUID>
 sed -i -e "s/YOUR_K3S_GUID/$K3S_CNI_GUID/g" manifests/base/meshnet.yml sed -i -e "s/YOUR_K3S_GUID/$K3S_CNI_GUID/g" manifests/base/meshnet.yml
Line 282: Line 280:
 Now you should be ready to deploy ''meshnet-cni''. To do this: Now you should be ready to deploy ''meshnet-cni''. To do this:
  
-<code>+<code bash>
 apt install -yqq build-essential apt install -yqq build-essential
 make install make install
Line 289: Line 287:
 If the installation was successful you should now have a ''meshnet-cni'' agent pod running on each node. Verify this: If the installation was successful you should now have a ''meshnet-cni'' agent pod running on each node. Verify this:
  
-<code>+<code bash>
 # kubectl -n meshnet get all # kubectl -n meshnet get all
 NAME                READY   STATUS    RESTARTS   AGE NAME                READY   STATUS    RESTARTS   AGE
Line 303: Line 301:
 With luck you should be good to go, and ''meshnet-cni'' is now ready to create the necessary VXLAN devices as directed by ''k8s-topo''. With luck you should be good to go, and ''meshnet-cni'' is now ready to create the necessary VXLAN devices as directed by ''k8s-topo''.
  
-### Step 2 - Disable load balancer +=== Step 2 - Disable load balancer ===
  
 Now that we have our overlay (''meshnet-cni'') deployed, we're ready to deploy ''k8s-topo''. Now that we have our overlay (''meshnet-cni'') deployed, we're ready to deploy ''k8s-topo''.
Line 317: Line 314:
 Anyway, we need to turn that off: Anyway, we need to turn that off:
  
-<https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller>+[[https://rancher.com/docs/k3s/latest/en/networking/#traefik-ingress-controller]]
  
-<https://github.com/rancher/k3s/issues/1160#issuecomment-561572618>+[[https://github.com/rancher/k3s/issues/1160#issuecomment-561572618]]
  
 Copied here for posterity: Copied here for posterity:
Line 353: Line 350:
 And finally, we can deploy ''k8s-topo''. And finally, we can deploy ''k8s-topo''.
  
-### Step 3 - Install k8s-topo+=== Step 3 - Install k8s-topo ===
  
 Now that we have our overlay (''meshnet-cni'') deployed, we're ready to deploy ''k8s-topo''. Now that we have our overlay (''meshnet-cni'') deployed, we're ready to deploy ''k8s-topo''.
Line 361: Line 358:
 Pull my ARM-compatible fork of ''k8s-topo'': Pull my ARM-compatible fork of ''k8s-topo'':
  
-<code>+<code bash>
 git clone --single-branch --branch k3s-arm https://github.com/qlyoung/k8s-topo.git git clone --single-branch --branch k3s-arm https://github.com/qlyoung/k8s-topo.git
 </code> </code>
  
-One notable change is that I've also added support for [FRR](https://github.com/frrouting/frr), which is a significantly upgraded fork of Quagga. The rest of the images will still pull amd64 versions and so at this time the only image you can choose for your topology simulations is the FRR image.+One notable change is that I've also added support for [[https://github.com/frrouting/frr|FRR]], which is a significantly upgraded fork of Quagga. The rest of the images will still pull amd64 versions and so at this time the only image you can choose for your topology simulations is the FRR image.
  
 Now you can deploy ''k8s-topo'' onto your cluster: Now you can deploy ''k8s-topo'' onto your cluster:
  
-<code>+<code bash>
 cd k8s-topo cd k8s-topo
 kubectl apply -f manifest.yml kubectl apply -f manifest.yml
Line 378: Line 375:
 Verify that the ''k8s-topo'' control pod and associated services have been created: Verify that the ''k8s-topo'' control pod and associated services have been created:
  
-<code>+<code bash>
 root@clusterpi-69 # kubectl get all root@clusterpi-69 # kubectl get all
 NAME                            READY   STATUS        RESTARTS   AGE NAME                            READY   STATUS        RESTARTS   AGE
Line 396: Line 393:
 That's it, we're done! That's it, we're done!
  
-### Step 4 - profit+=== Step 4 - profit ===
  
 Now you can log into the ''k8s-topo'' control pod and use the CLI tools there. Now you can log into the ''k8s-topo'' control pod and use the CLI tools there.
  
-<code>+<code bash>
 root@clusterpi-69 # kubectl exec -it deployment/k8s-topo -- sh root@clusterpi-69 # kubectl exec -it deployment/k8s-topo -- sh
  
Line 428: Line 425:
 </code> </code>
  
-### Usage Examples+=== Usage Examples ===
  
 Create a random FRR topology: Create a random FRR topology:
  
-<code>+<code bash>
 /k8s-topo # cd examples/builder/ /k8s-topo # cd examples/builder/
 /k8s-topo/examples/builder # ./builder 10 3 /k8s-topo/examples/builder # ./builder 10 3
Line 485: Line 482:
 Apply the topology: Apply the topology:
  
-<code>+<code bash>
 /k8s-topo/examples/builder # cd /k8s-topo/ /k8s-topo/examples/builder # cd /k8s-topo/
 /k8s-topo # k8s-topo --create examples/builder/random.yml /k8s-topo # k8s-topo --create examples/builder/random.yml
Line 505: Line 502:
 Log out of the container and verify that the pods have been created: Log out of the container and verify that the pods have been created:
  
-<code>+<code bash>
 root@clusterpi-69 # kubectl get all root@clusterpi-69 # kubectl get all
 NAME                            READY   STATUS        RESTARTS   AGE NAME                            READY   STATUS        RESTARTS   AGE
Line 524: Line 521:
 <code> <code>
 root@clusterpi-69 # kubectl exec -it pod/frr-192-0-2-2 -- sh root@clusterpi-69 # kubectl exec -it pod/frr-192-0-2-2 -- sh
-/ # ip addr+# ip addr
 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
Line 549: Line 546:
     inet6 fe80::2caf:91ff:fed4:c1b0/64 scope link     inet6 fe80::2caf:91ff:fed4:c1b0/64 scope link
        valid_lft forever preferred_lft forever        valid_lft forever preferred_lft forever
-/ # vtysh+# vtysh
 % Can't open configuration file /etc/frr/vtysh.conf due to 'No such file or directory'. % Can't open configuration file /etc/frr/vtysh.conf due to 'No such file or directory'.
  
Line 574: Line 571:
 Welcome to your simulation. Welcome to your simulation.
  
-### Notes+=== Notes ===
  
 I've extensively replaced various components of this setup with my own forks to which I've added ARM support. I plan to merge the ARM changes back into their respective upstream repos, but this is going to take some time, as the changes I've done currently are very quick-and-dirty. They need to be rewritten not just to work on ARM, but to generalize amd64-specific build options to work on any target architecture. I've extensively replaced various components of this setup with my own forks to which I've added ARM support. I plan to merge the ARM changes back into their respective upstream repos, but this is going to take some time, as the changes I've done currently are very quick-and-dirty. They need to be rewritten not just to work on ARM, but to generalize amd64-specific build options to work on any target architecture.
  
 I very much dislike making this setup depending on my personal GitHub forks and DockerHub registry, but doing so was the only way I could finish this project in a reasonable time frame. I very much dislike making this setup depending on my personal GitHub forks and DockerHub registry, but doing so was the only way I could finish this project in a reasonable time frame.
 +
 +{{tag>from_blog technology networking}}
Panorama theme by desbest
network_simulation_with_k8s-topo_on_raspi_cluster.1694221155.txt.gz · Last modified: 2023/09/09 00:59 by qlyoung
CC Attribution-Noncommercial-Share Alike 4.0 International Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 4.0 International