The Home Server Journey – 1: Motivation and Approach
Inspired by the recent Twitter/X situation here in Brazil (no strong opinions on Elon Musk and all this drama, but it surely sucks to lose access to an useful service) and by an important article by Ceadda of Mercia, I've decided to set an old project of mine in motion.
Truth is, we grow so used to many readily accessible (even if not completely free) platforms on the Internet that we end up taking this ability to communicate and inform ourselves for granted. We easily forget how liberty has to be conquered every day, and that has to be backed by power i.e. the capacity to do what we want to do
Anyway, there are plenty of speeches and commentary around that would do a much better job than me in convincing you of that reality, not to mention the risks of providing all your data directly to third-parties. So here I intend to concentrate where I can add something meaningful: give you some ideas and guidance on how you may own your Internet life a bit more
Living in reality is knowing where things come from and where they go to
Social media, messaging, videos, photos, e-mail, blogs like this one and every other thing that we access on the Internet are all hosted out there, on The Cloud. However, as the saying goes, ”'The Cloud' is just someone else's computer”. And why couldn't it be YOUR computer? Well, it could, if you're willing to make some sacrifices
You won't be able to run e.g. a personal Facebook or Instagram server from home, but if you care more about the functionality they provide, well, you may be able to actually own something close to it. Of course your friends would have to come over to your place, and many probably won't. That's part of what I meant by “make some sacrifices”: your personalized Web presence will be a bit lonely until it gets some traction
But, let's be real, if you're looking into building homeservers having almost no friends is probably the case already... I say it from experience
Ok, ok, maybe you DO have quite some friends [on the Internet] and don't want to give up interacting with them. You'd still find usefulness in having your own “cloud” storage/backup, or a less censorable place to publish your ideas
I'll try to show a bit of everything. Pick whatever suits you
The meat and potatoes of the issue
I don't plan on being condescending to the point of assuming you don't know the distinction between hardware as software. You might have already figured out that those personal software services would need dedicated hardware to be run, and it would be a good thing to have a relatively low-cost solution, so that affording a brand new Intel x86 PC (calm down, Apple M1 users) to keep turned on 24/7 is not a requirement
A common approach that I have adopted is resorting to ARM-based single board computers. They are cheaper, consume less power and take less space inside my modest house. Below I display my humble initial setup of an Hardkernel Odroid N2+ and a well-know Raspberry Pi 4:
(Size doesn't matter if you know how to use it, or so they say)
I have also attached one spare hard drive to each one of them using external USB 3.0 to SATA adapter cases, so that we have larger and more reliable storage. Besides, as you might have noticed, there is no screen, keyboard or mouse there, because I'll be managing everything software-wise from my desktop, over the local network (LAN)
Please make sure you have the correct/recommended power supply connected to each board, specially when using external devices powered through USB. I've just had some undervoltage issues with the Raspberri Pi for not using the ideal one
Both machines are running the minimal ARM version of Manjaro Linux. As a long-time user of Manjaro on my PC, I found it easier to let manjaro-arm-installer choose the right configuration for each board, download the image and write it to its respective SD card. There are plenty of options for operating systems, with it's own installation instructions, but generally you'll be taking and .img file and flashing it to the card with some tool like Etcher or dd. Take a look at what you like the most (preferably some Linux CLI version) or just use the same I did, you'll have to consult extra documentation regardless, as I can't give you all the details without making this article too long for my taste
Notes on networking
Needless to say, your server machines need Internet connection, preferably Ethernet, so that it can be detected without logging into each one locally and configuring Wi-Fi. Another [not so obvious] requirement is that at least one of them (your master) has to be exposed outside of your LAN, so that external devices can reach it (and you can imagine why that's not possible by default). If you have access to your router configuration (I DO hope you have), usually your NAT or Firewall settings, that's achieved by forwarding the ports used by the services you got running e.g. port 80/TCP for HTTP and 443/TCP for HTTPS (Web pages):
(If you're using standard IPv4, the machine's host address is valid exclusively inside your LAN, and only the router/gateway has also a global WAN IP. Even with all the IPv6 goodies enabled, reachability from outside might be limited for security reasons)
I'd also recommend reserving static IPs for each one of your server's network interfaces, so that we don't get any unpleasant surprises with changing addresses:
After redirecting incoming traffic from router WAN address to server machine LAN address on the given ports, it's time to make yourself easier to find. Passing your WAN IP around so that people may type it into the browser bar would already work, but that's both hard to remember and unreliable, as addresses attributed to your machine usually change from time to time (lucky you if a static global IP is available).
Then why not give your virtual place a proper memorable name, like “mybeautifulsite.com”, that everybody could know? Well, that what name service (DNS) is for.
Sadly domains are one aspect of Internet centralization that still cannot be trivially avoided (I know about blockchain DNS, but who supports it?), so you have to rely on a third-party service (e.g. Cloudflare) in order to propagate your hostname across the Web. Generally you have to pay a registrar for the right to use a certain name (it's a way to prevent the ambiguity from 2 hosts using the same one), but for testing purposes you may rely on free DNS services (such as No-IP or Dynv6) that provide you a somewhat uglier subdomain alternative:
(They were so kind to tell the entire world my router's IP)
If you don't manage to get your hands on a static WAN IP, it's important that your DNS service has support for being automatically updated about changes in your dynamic address. Your router itself might be able to notify the DDNS, but if the proper protocol option is not available you may run a refresh client from one of your computers:
(Again, there's too much variation across routers and services for me to cover it here)
Just don't go trying to reach your server via DNS right after naming it. Propagation takes a while, as many name servers across the world have to be informed. If getting anxious, you may try to check the state of that process using some Web service or nslookup command on your Linux system (also available under the same name on Windows, apparently, but who in his/her right mind still uses it?)
Becoming the helmsman
If you've ever tried using a non-mainstream Linux distro, you probably know the pain of looking for a package, not finding it even in a third-party repository, trying to compile it from source and failing miserably over and over (Been there. Done that). Manjaro, as an Arch-based OS, at least has the very welcome access to the AUR, a community repository for compilation scripts that automate the process for us mere humans, but not all are regularly maintained and many still break, for a plethora of reasons that you eventually have to figure out and fix in a case-by-case basis
A way to avoid those headaches would be welcome
For a long time, I've been the kind of person that sees Linux's Dependency Hell as a feature rather than a bug: “It's efficient for storage”, “It forces developers to keep their software updated and in sync with others”, “Library duplication is r*tarded”, I've been telling myself. I still find it somewhat true, and don't see sandboxed solutions like Flatpak, Snap and AppImage with good eyes, or even virtual environments for more than prototyping... Ok, maybe I'm still THAT kind of person, but I've found containers to be pretty neat
Having worked a bit with Docker in a previous job, I was convinced of how useful this level of isolation is for automated testing and deployment of applications, being able to change system settings and replicate them across countless [physical or virtual] machines. You really start visualizing containers as loosely coupled independent entities exchanging messages, which in turn naturally evolves into the main focus of my approach here: orchestration
And to cut it short, I'd guess in 9 of 10 cases that word would mean using one of the many implementations of Kubernetes (K8s for short). Not that there aren't other solutions, but I'm not knowledgeable enough to evaluate them properly and I went for what is more popular i.e. with more available documentation. Let me make it clear that this guide is a mish-mash of a number of tutorials and Stack Overflow threads, compiled, updated and adjusted by an amateur who is really stubborn about making all work together. So please bear in mind it might not be the most elegant solution
For tiny single-board computers, there is a lightweight implementation of K8s called K3s, which has worked well for me. If you're using Arch-based systems (assuming that you have actually gotten the machines set up), keep following my steps for K3s installation. On Debian-based systems like Ubuntu or Raspbian, you'd be probably better off following NetworkChuck's guide linked below. Otherwise, you'll have to do your own research:
As already mentioned, here I have accessed my servers from a desktop, via SSH, and installed K3s using yay, but you may use any other package manager with AUR access like pikaur or pamac:
$ ssh ancapepe@192.168.3.10
ancapepe@192.168.3.10's password:
Welcome to Manjaro ARM
~~Website: https://manjaro.org
~~Forum: https://forum.manjaro.org/c/arm
~~Matrix: #manjaro-arm-public:matrix.org
Last login: Wed Sep 11 18:41:48 2024 from 192.168.3.8
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S yay
[sudo] password for ancapepe:
resolving dependencies...
looking for conflicting packages...
Packages (1) yay-12.3.1-1
Total Download Size: 3.31 MiB
Total Installed Size: 8.88 MiB
:: Proceed with installation? [Y/n] y
:: Retrieving packages...
yay-12.3.1-1-aarch64 3.3 MiB 2.33 MiB/s 00:01 [#############################################################################] 100%
(1/1) checking keys in keyring [#############################################################################] 100%
(1/1) checking package integrity [#############################################################################] 100%
(1/1) loading package files [#############################################################################] 100%
(1/1) checking for file conflicts [#############################################################################] 100%
(1/1) checking available disk space [#############################################################################] 100%
:: Processing package changes...
(1/1) installing yay [#############################################################################] 100%
Optional dependencies for yay
sudo: privilege elevation [installed]
doas: privilege elevation
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S containerd fakeroot
...
(More of the same)
...
[ancapepe@ChoppaServer-1 ~]$ yay -S k3s-bin
AUR Explicit (1): k3s-bin-1.30.3+k3s1-1
...
(Too much to show here. Just follow along)
...
(Do that for each board. Teaching how to use a terminal is outside the scope of this guide)
Hopefully everything went well and you have K3s installed on all your servers, but it's still not doing anything. Kubernetes applies the concepts of master and worker nodes: masters coordinate workers so that all can work as a single system, namely a Cluster; both may dot the heavy lifting of running applications; Node is simply the abstraction used for each [physical or virtual] computer
[AFAIK] With K3s things are kept simple and only your first node will run as a master, or [K3s] server in this case (yes, terminology gets confusing). In order to initialize it as such when your main machine boots, open your K3s init file for modification with:
[ancapepe@ChoppaServer-1 ~]$ sudo systemctl edit k3s --full
(The file will be opened with the application defined in your $EDITOR environment variable, be it vim, emacs or even nano, a competition I don't want to be dragged into. Talking about undesired fights, I know systemd is not the only init system out there, but I haven't tried this with distros that use OpenRC or Upstart)
Edit the line starting with ExecStart to look like this:
ExecStart=/usr/bin/k3s server --write-kubeconfig-mode=644 --node-name <your chosen master node name>
And finally enable the initialization service with:
[ancapepe@ChoppaServer-1 ~]$ sudo systemctl enable containerd
[ancapepe@ChoppaServer-1 ~]$ sudo systemctl enable k3s --now
Not quite finally, actually. Before you restart your board to bring up K3s, you have to make sure that the cgroups feature, required by all containerized applications, is enabled and available in your system:
[ancapepe@ChoppaServer-1 ~]$ grep cgroup /proc/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
[ancapepe@ChoppaServer-1 ~]$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 51 1
cpu 0 51 1
cpuacct 0 51 1
blkio 0 51 1
memory 0 51 0
devices 0 51 1
freezer 0 51 1
net_cls 0 51 1
perf_event 0 51 1
net_prio 0 51 1
pids 0 51 1
rdma 0 51 1
(TL;DR: Cgroups system v2 running, memory cgroup disabled)
If v1 system or none at all appears, you're probably using an old kernel, so update it. If the memory cgroup is not enabled, you have to find the particular way in which you switch it on for your device (for Raspberry Pi, that means adding cgroup_enable=memory at the end of the single line in the /boot/cmdline.txt file)
Moreover, we must tell containerd, the daemon that manages your containers behind the curtains, to use the right cgroup for its main process, runc, by running the commands below:
# Create a containerd configuration directory
[ancapepe@ChoppaServer-1 ~]$ sudo mkdir -p /etc/containerd
# Write the default settings to a configuration file
[ancapepe@ChoppaServer-1 ~]$ sudo containerd config default | sudo tee /etc/containerd/config.toml
# Find and replace the given string inside the file
[ancapepe@ChoppaServer-1 ~]$ sudo sed -i 's/ SystemdCgroup = false/ SystemdCgroup = true/' /etc/containerd/config.toml
(The # character at the beginning of a line denotes a comment, which is not supposed to be typed. For more information on what you're doing there consult this article and the sed command documentation)
Now you may reboot the master node, and hopefully everything will start running properly. Your SSH connection will be terminated so it should be established again when the device finishes starting up:
[leonardo@ChoppaServer-1 ~]$ sudo reboot
[sudo] password for leonardo:
Broadcast message from root@ChoppaServer-1 on pts/1 (Sat 2024-09-14 20:31:07 -03):
The system will reboot now!
[leonardo@ChoppaServer-1 ~]$ Read from remote host 192.168.3.10: Connection reset by peer
Connection to 192.168.3.10 closed.
client_loop: send disconnect: Broken pipe
$ ssh leonardo@192.168.3.10
leonardo@192.168.3.10's password:
Welcome to Manjaro ARM
~~Website: https://manjaro.org
~~Forum: https://forum.manjaro.org/c/arm
~~Matrix: #manjaro-arm-public:matrix.org
Last login: Sat Sep 14 17:30:32 2024 from 192.168.3.8
[leonardo@ChoppaServer-1 ~]$
(At this point you may check the status of the enabled services using systemctl status)
With the master system set up, let's try connecting our worker nodes to it. That connection is established by sharing a common key or token generated by master and stored in a predetermined file:
# Displays content of the token file
[leonardo@ChoppaServer-1 ~]$ sudo cat /var/lib/rancher/k3s/server/token
[sudo] password for leonardo:
K10d4e8b232cbf03832752443a07cc6206e092733c8ae55c0e64407dcb2e1775f45::server:44a1a8b25c4a8f1d8a0d48164eff2303
Copy that file, guard it well, and repeat the same process (since the first ssh command) for each one of your workers, only changing the way the K3s service file is edited, as now it should contain:
ExecStart=/usr/bin/k3s agent --server https://<your master node IP>:6443 --token <your master token> --node-name <your chosen worker node name>
Take your time in repeating all that stuff, just make sure that you follow through. If everything works as intended, from any one of your nodes you'd be able to run the command below successfully:
[leonardo@ChoppaServer-2 ~]$ k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
odroidn2-master Ready control-plane,master 7d5h v1.30.3+k3s1
rpi4-agent Ready <none> 8d v1.30.3+k3s1
(Yay!)
That's it for now. I hope it wasn't very tiring
Next time we'll actually deploy applications and get a proper way to manage them