Becoming a man is no trivial matter, and I even suspect nobody really knows how to do it. The more I grew up, the more I realized our minds don't simply flip a switch at a certain age, and the same probably hasn't happened to the ones I used to see as adults by excellence. Regardless, we still face the external or self-imposed demand for manhood and its social status, and keep looking for a reference that we could follow
I know that sounds cliche, but we live at a time of extremes: old concepts of masculinity have been recklessly deconstructed on the pretext of social critic, promoting emasculation, and some new wave of overcompensated “retvrn” – as always happens when we get rid of something important out of convenience – has produced a caricature of masculinity that's little more than self-indulgence in a modern world that's supposedly not worth your care
If that's not clear already, I dislike both, in a visceral manner. I don't find answers in them, I don't even see a genuine attempt to find answers. There are just predetermined formulas to appear as either a nice guy or an alpha male to the rest of society – women, specially
For a long time, I have [truly] struggled to avoid that dichotomy, and seeing many young men, close ones even, falling for it teared me apart countless times. But as much as I wanted to provide some alternative, it always seemed pretentious to try that. I'm not wise, not even of average wisdom, and kept lagging behind people my age when it comes to maturity. Even so, there are things I wish to say, and a I have decided to at least leave them here for the record
I know there is kind of a third option, that seems more well-balanced: a religious life. I have my appreciation for religion (for not so “spiritual”, emotional or utilitarian reasons), but I sincerely find the typical approach of religious people extremely frustrating: if you open up about an issue, most of the time you won't find someone trying to understand, but looking constantly for an opportunity to start proselytizing. Your story is again just a hook for canned talking points
The problem is, for whoever comes across this and finds it worth a shot, that you will be reading from someone who failed. Almost 3 years ago i was already on shaky grounds, when the disastrous end of a relationship attempt and losing my father out of nowhere, in quick succession, honestly shattered most of the motivation I had in life. I don't want anyone to pity me, but the context is relevant for weighting the value of my advice. On the other hand, that lack of something to lose may help me being more honest, or maybe that's just a cop-out
Talking about cop-outs, I think the main theme of what I have to say is threading this fine line between seeing painful things for what they are and grasping something precious in them, without delusions. We (sorry if I'm sharing the blame with you unfairly) love the feeling of rubbing hard truths on the face of somebody else, but are we willing to accept our inconveniences without excuses or cynicism? It hurts, it may devour you from inside out, but the alternative is wasting the single chance you have (allegedly) living an untruthful life, the ultimate pointlessness
Even if life ultimately has no meaning, that would be a truth, and this is your only opportunity to find that out by yourself. If something makes sense, no time is wasted looking for it, if nothing does, there's no concept of “wasted time” for you to lament
Getting to the point, knowing who you are and understanding the situation you're in seem to me the way to move forward. That includes occasionally accepting, to some extent, that you're not hot stuff and not meant to be. I would even say that this potential series here is intended specially for you who suck in life, because that's my perspective
Don't take me wrong, I refuse to push yet another pseudo-self-derogatory rhetoric to convince people to take pride in mediocrity, with the resentful undertones of “I live in the real world, unlike those ideal guys”. You should feel genuinely bad for your shortcomings, but avoid selling your soul to pretend you're someone else, the price paid by both “evolved” and “red-pilled” men
Abusing Internet jargon, most of us are natural “betas”, as those are almost by definition a majority. We get the scraps because some have to, because there are no places at the table for everybody, because people are different and some will come out at the top. You either fight a losing battle against reality or make the most out of it without losing yourself, the only thing you get a chance to keep until the end
The latter is what I wish to figure out along the way. I hope we learn how to navigate those waters and find a tolerable place for our true selves in this world
One thing that bothers me when writing about self-hosting, to have greater control of my data, is that I don't apply those principles to the articles themselves. I mean, there is no immediate risk of Minds or Dev.to taking down my publications, but the mere fact that they could do that leaves me concerned
The Right Tool for the Job
First I've looked at the tools I was already familiar with. I have some old blog where I've posted updates during my Google Summer of Code projects. It uses Jekyll to generate static files, automatically published by GitHub Pages. It works very well when you have the website tied to a version-controlled repository, but it's cumbersome when you need to rebuild container images or replace files in a remote volume even for small changes
When looking for something more dynamic, I initially though about using Plume, since it's easy to integrate with some applications I plan to deploy later, but unfortunately it's not well maintained anymore. As Ghost or Wordpress seem overkill, I ended up opting for the conveniences of WriteFreely: it lets me create and edit posts in-place, with Markdown support and no need to upload new files. However, that comes with a cost: it requires a MySQL[-compatible] database
Reputation-related shenanigans aside, one great advantage of picking MariaDB is the ability to use a Galera Cluster. Similarly to what we did for PostgreSQL, I wish to be able to scale it properly, and Galera's replication works in an even more interesting manner, with multiple primary (read-write) instances and no need for a separate proxy!:
(Man, I wish PostgreSQL had something similar...)
Of course that requires a more complex setup for the database server itself, but thanks to Bitnami's mariadb-galeraDocker image and Helm chart, I've managed to get to something rather manageable for our purposes:
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-config
labels:
app: mariadb
data:
BITNAMI_DEBUG: "false" # Set to "true" for more debug information
MARIADB_GALERA_CLUSTER_NAME: galera
# All pods being synchronized (has to reflect the number of replicas)
MARIADB_GALERA_CLUSTER_ADDRESS: gcomm://mariadb-state-0.mariadb-replication-service.default.svc.cluster.local,mariadb-state-1.mariadb-replication-service.default.svc.cluster.local
MARIADB_DATABASE: main # Default database
MARIADB_GALERA_MARIABACKUP_USER: backup # Replication user
---
# Source: mariadb-galera/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: mariadb-secret
labels:
app: mariadb
data:
MARIADB_ROOT_PASSWORD: bWFyaWFkYg== # Administrator password
MARIADB_GALERA_MARIABACKUP_PASSWORD: YmFja3Vw # Replication user password
---
apiVersion: v1
kind: ConfigMap
metadata:
name: mariadb-cnf-config
labels:
app: mariadb
data: # Database server configuration
my.cnf: |
[client]
port=3306
socket=/opt/bitnami/mariadb/tmp/mysql.sock
plugin_dir=/opt/bitnami/mariadb/plugin
[mysqld]
explicit_defaults_for_timestamp
default_storage_engine=InnoDB
basedir=/opt/bitnami/mariadb
datadir=/bitnami/mariadb/data
plugin_dir=/opt/bitnami/mariadb/plugin
tmpdir=/opt/bitnami/mariadb/tmp
socket=/opt/bitnami/mariadb/tmp/mysql.sock
pid_file=/opt/bitnami/mariadb/tmp/mysqld.pid
bind_address=0.0.0.0
## Character set
##
collation_server=utf8_unicode_ci
init_connect='SET NAMES utf8'
character_set_server=utf8
## MyISAM
##
key_buffer_size=32M
myisam_recover_options=FORCE,BACKUP
## Safety
##
skip_host_cache
skip_name_resolve
max_allowed_packet=16M
max_connect_errors=1000000
sql_mode=STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY
sysdate_is_now=1
## Binary Logging
##
log_bin=mysql-bin
expire_logs_days=14
# Disabling for performance per http://severalnines.com/blog/9-tips-going-production-galera-cluster-mysql
sync_binlog=0
# Required for Galera
binlog_format=row
## Caches and Limits
##
tmp_table_size=32M
max_heap_table_size=32M
# Re-enabling as now works with Maria 10.1.2
query_cache_type=1
query_cache_limit=4M
query_cache_size=256M
max_connections=500
thread_cache_size=50
open_files_limit=65535
table_definition_cache=4096
table_open_cache=4096
## InnoDB
##
innodb=FORCE
innodb_strict_mode=1
# Mandatory per https://github.com/codership/documentation/issues/25
innodb_autoinc_lock_mode=2
# Per https://www.percona.com/blog/2006/08/04/innodb-double-write/
innodb_doublewrite=1
innodb_flush_method=O_DIRECT
innodb_log_files_in_group=2
innodb_log_file_size=128M
innodb_flush_log_at_trx_commit=1
innodb_file_per_table=1
# 80% Memory is default reco.
# Need to re-evaluate when DB size grows
innodb_buffer_pool_size=2G
innodb_file_format=Barracuda
[galera]
wsrep_on=ON
wsrep_provider=/opt/bitnami/mariadb/lib/libgalera_smm.so
wsrep_sst_method=mariabackup
wsrep_slave_threads=4
wsrep_cluster_address=gcomm://
wsrep_cluster_name=galera
wsrep_sst_auth="root:"
# Enabled for performance per https://mariadb.com/kb/en/innodb-system-variables/#innodb_flush_log_at_trx_commit
innodb_flush_log_at_trx_commit=2
# MYISAM REPLICATION SUPPORT #
wsrep_mode=REPLICATE_MYISAM
[mariadb]
plugin_load_add=auth_pam
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: mariadb-state
spec:
serviceName: mariadb-replication-service # Use the internal/headless service name
replicas: 2
selector:
matchLabels:
app: mariadb
template:
metadata:
labels:
app: mariadb
spec:
securityContext: # Container is not run as root
fsGroup: 1001
runAsUser: 1001
runAsGroup: 1001
containers:
- name: mariadb
image: docker.io/bitnami/mariadb-galera:11.5.2
imagePullPolicy: "IfNotPresent"
command:
- bash
- -ec
- |
exec /opt/bitnami/scripts/mariadb-galera/entrypoint.sh /opt/bitnami/scripts/mariadb-galera/run.sh
ports:
- name: mdb-mysql-port
containerPort: 3306 # External access port (MySQL's default)
- name: mdb-galera-port
containerPort: 4567 # Internal process port
- name: mdb-ist-port
containerPort: 4568 # Internal process port
- name: mdb-sst-port
containerPort: 4444 # Internal process port
envFrom:
- configMapRef:
name: mariadb-config
env:
- name: MARIADB_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mariadb-secret
key: MARIADB_ROOT_PASSWORD
- name: MARIADB_GALERA_MARIABACKUP_PASSWORD
valueFrom:
secretKeyRef:
name: mariadb-secret
key: MARIADB_GALERA_MARIABACKUP_PASSWORD
volumeMounts:
- name: previous-boot
mountPath: /opt/bitnami/mariadb/.bootstrap
- name: mariadb-data
mountPath: /bitnami/mariadb
- name: mariadb-cnf
mountPath: /bitnami/conf/my.cnf # Overwrite any present configuration
subPath: my.cnf
- name: empty-dir
mountPath: /tmp
subPath: tmp-dir
- name: empty-dir
mountPath: /opt/bitnami/mariadb/conf
subPath: app-conf-dir
- name: empty-dir
mountPath: /opt/bitnami/mariadb/tmp
subPath: app-tmp-dir
- name: empty-dir
mountPath: /opt/bitnami/mariadb/logs
subPath: app-logs-dir
volumes:
- name: previous-boot
emptyDir: {} # Use a fake directory for mounting unused but required paths
- name: mariadb-cnf
configMap:
name: mariadb-cnf-config
- name: empty-dir
emptyDir: {} # Use a fake directory for mounting unused but required paths
volumeClaimTemplates: # Description of volume claim created for each replica
- metadata:
name: mariadb-data
spec:
storageClassName: nfs-small
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
---
# Headless service for internal replication/backup processes
apiVersion: v1
kind: Service
metadata:
name: mariadb-replication-service
labels:
app: mariadb
spec:
type: ClusterIP
clusterIP: None
ports:
- name: mariadb-galera-service
port: 4567
targetPort: mdb-galera-port
appProtocol: mysql
- name: mariadb-ist-service
port: 4568
targetPort: mdb-ist-port
appProtocol: mysql
- name: mariadb-sst-service
port: 4444
targetPort: mdb-sst-port
appProtocol: mysql
publishNotReadyAddresses: true
---
# Exposed service for external access
apiVersion: v1
kind: Service
metadata:
name: mariadb-service
spec:
type: LoadBalancer # Let it be accessible inside the local network
selector:
app: mariadb
ports:
- port: 3306
targetPort: mdb-mysql-port
appProtocol: mysql
Incredibly, it works. My deployment has been running without issue for some time now:
$ kubectl get all -n choppa -l app=mariadb
NAME READY STATUS RESTARTS AGE
pod/mariadb-state-0 1/1 Running 2 (3d1h ago) 5d3h
pod/mariadb-state-1 1/1 Running 2 (3d1h ago) 5d3h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mariadb-replication-service ClusterIP None <none> 4567/TCP,4568/TCP,4444/TCP 5d3h
service/mariadb-service LoadBalancer 10.43.40.243 192.168.3.10,192.168.3.12 3306:31594/TCP 5d3h
NAME READY AGE
statefulset.apps/mariadb-state 2/2 5d3h
(The 2 restarts were due to a power outage that exceeded the autonomy of my no-break's battery)
Solving one Problem to Reveal Another
I just started typing my first self-hosted blog post to realize something was missing: images. On Jekyll I had a folder for that, but on Minds and Dev.to they are hosted somewhere else, e.g. https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v2221vgcikcr05hmnj4v.png
If complete self-hosting is a must, I now need some file server capable of generating shareable links, to be used in my Markdown image components. In summary, Syncthing is great for Dropbox-style backups, but can't share links, NextCloud is too resource-heavy and Seafile is interesting but apparently has proprietary encryption, which left me with the lightweight Filebrowser
I don't expect or intend my file server to ever deal with a huge number of requests, so I've ran it as a simple deployment with a single pod:
kind: PersistentVolumeClaim # Storage requirements component
apiVersion: v1
metadata:
name: filebrowser-pv-claim
labels:
app: filebrowser
spec:
storageClassName: nfs-big # The used storage class (1TB drive)
accessModes:
- ReadWriteOnce
#- ReadWriteMany # For concurrent access (In case I try to use more replicas)
resources:
requests:
storage: 200Gi # Asking for a ~50 Gigabytes volume
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebrowser-config
labels:
app: filebrowser
data: # Application settings file
.filebrowser.json: |
{
"port": 80,
"baseURL": "",
"address": "",
"log": "stdout",
"database": "/srv/filebrowser.db",
"root": "/srv"
}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: filebrowser-deploy
spec:
replicas: 1
strategy:
type: Recreate # Wait for the old container to be terminated before creating a new one
selector:
matchLabels:
app: filebrowser
template:
metadata:
labels:
app: filebrowser
spec:
# Run this initial container to make sure at least an empty
# database file exists prior to the main container starting,
# as a workaround for a know bug (https://filebrowser.org/installation#docker)
initContainers:
- name: create-database
image: busybox
command: ["/bin/touch","/srv/filebrowser.db"]
volumeMounts:
- name: filebrowser-data
mountPath: /srv
containers:
- name: filebrowser
image: filebrowser/filebrowser:latest
imagePullPolicy: IfNotPresent
ports:
- name: file-port
containerPort: 80
protocol: TCP
volumeMounts:
- name: filebrowser-readonly
mountPath: /.filebrowser.json
subPath: .filebrowser.json
- name: filebrowser-data
mountPath: /srv
volumes:
- name: filebrowser-readonly
configMap:
name: filebrowser-config
- name: filebrowser-data # Label the volume for this deployment
persistentVolumeClaim:
claimName: filebrowser-pv-claim # Reference volumen create by the claim
---
apiVersion: v1
kind: Service
metadata:
name: filebrowser-service
spec:
type: NodePort # Expose the service outside the cluster with an specific port
selector:
app: filebrowser
ports:
- protocol: TCP
port: 8080
targetPort: file-port
nodePort: 30080
(That's what I did here, make it makes the filebrowser.db file end up visible inside the root folder. It's probably a good idea to use subpaths and mount them separately e.g. srv/filebrowser.db and srv/data for root)
We can't upload or access the files from the Internet yet, but using NodePort an external port in the range 30000-32767 can be used to reach it locally. Use the default username admin and password admin to login and then change it in the settings:
Click on each file you wish to share and the option to generate links will appear on the top. In Markdown syntax, shared images may be annexed with the statement 
One Step Forward. Two Steps Back
All set to deploy WriteFreely, right? As you might guess, no
The application doesn't have an official Docker image, and the custom ones available are either too old or not available for the ARM64 architecture. The repository provided by karlprieb is a good base to build your own, but it lead to crashes here when compiling the application itself. In the end, I found it easier to create one taking advantage of Alpine Linux's packages:
Dockerfile
FROM alpine:3.20
LABEL org.opencontainers.image.description="Simple WriteFreely image based on https://git.madhouse-project.org/algernon/writefreely-docker"
# Install the writefreely package
RUN apk add --no-cache writefreely
# Installation creates the writefreely user, so let's use it
# to run the application
RUN mkdir /opt/writefreely && chown writefreely -R /opt/writefreely
COPY --chown=writefreely:writefreely ./run.sh /opt/writefreely/
RUN chmod +x /opt/writefreely/run.sh
# Base directory and exposed container port
WORKDIR /opt/writefreely/
EXPOSE 8080
# Set the default container user and group
USER writefreely:writefreely
# Start script
ENTRYPOINT ["/opt/writefreely/run.sh"]
Entrypoint script (run.sh)
#! /bin/sh
writefreely -c /data/config.ini --init-db
writefreely -c /data/config.ini --gen-keys
if [ -n "${WRITEFREELY_ADMIN_USER}" ] && [ -n "${WRITEFREELY_ADMIN_PASSWORD}" ]; then
writefreely -c /data/config.ini --create-admin "${WRITEFREELY_ADMIN_USER}:${WRITEFREELY_ADMIN_PASSWORD}"
fi
writefreely -c /data/config.ini
Here I've published the image to ancapepe/writefreely:lateston DockerHub, so use it if you wish and have no desire for alternative themes or other custom stuff. One more thing to do before running our blog is to prepare the database to receive its content, so log into the MariaDB server on port 3306 using you root user and execute those commands, replacing username and password to your liking:
CREATE DATABASE writefreely CHARACTER SET latin1 COLLATE latin1_swedish_ci;
CREATE USER 'blog' IDENTIFIED BY 'my_password';
GRANT ALL ON writefreely.* TO 'blog';
Now apply a K8s manifest matching previous configurations and adjusting new ones to your liking:
apiVersion: v1
kind: ConfigMap
metadata:
name: writefreely-config
labels:
app: writefreely
data:
WRITEFREELY_ADMIN_USER: my_user
config.ini: |
[server]
hidden_host =
port = 8080
bind = 0.0.0.0
tls_cert_path =
tls_key_path =
templates_parent_dir = /usr/share/writefreely
static_parent_dir = /usr/share/writefreely
pages_parent_dir = /usr/share/writefreely
keys_parent_dir =
[database]
type = mysql
username = blog
password = my_password
database = writefreely
host = mariadb-service
port = 3306
[app]
site_name = Get To The Choppa
site_description = Notes on Conscious Self-Ownership
host = https://blog.choppa.xyz
editor =
theme = write
disable_js = false
webfonts = true
landing = /login
single_user = true
open_registration = false
min_username_len = 3
max_blogs = 1
federation = true
public_stats = true
private = false
local_timeline = true
user_invites = admin
# If you wish to change the shortcut icon for your blog without modifying the image itself, add here the configmap entry generated by running `kubectl create configmap favicon-config --from-file=<your .ico image path>`
binaryData:
favicon.ico: <binary dump here>
---
apiVersion: v1
kind: Secret
metadata:
name: writefreely-secret
data:
WRITEFREELY_ADMIN_PASSWORD: bXlfcGFzc3dvcmQ=
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: writefreely-deploy
spec:
replicas: 1
selector:
matchLabels:
app: writefreely
template:
metadata:
labels:
app: writefreely
spec:
containers:
- name: writefreely
image: ancapepe/writefreely:latest
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 8080
name: blog-port
env:
- name: WRITEFREELY_ADMIN_USER
valueFrom:
configMapKeyRef:
name: writefreely-config
key: WRITEFREELY_ADMIN_USER
- name: WRITEFREELY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: writefreely-secret
key: WRITEFREELY_ADMIN_PASSWORD
volumeMounts:
- name: writefreely-volume
mountPath: /data/config.ini
subPath: config.ini
# Use this if you set the custom favicon.ico image above
- name: writefreely-volume
mountPath: /usr/share/writefreely/static/favicon.ico
subPath: favicon.ico
volumes:
- name: writefreely-volume
configMap:
name: writefreely-config
---
apiVersion: v1
kind: Service
metadata:
name: writefreely-service
spec:
publishNotReadyAddresses: true
selector:
app: writefreely
ports:
- protocol: TCP
port: 8080
targetPort: blog-port
(You may add your own favicon.ico to the image itself if you're building it)
Almost there. Now we just have to expose both our blog pages and file server to the Internet by adding the corresponding entries to our ingress component:
apiVersion: networking.k8s.io/v1
kind: Ingress # Component type
metadata:
name: proxy # Component name
namespace: choppa # You may add the default namespace for components as a paramenter
annotations:
cert-manager.io/cluster-issuer: letsencrypt
kubernetes.io/ingress.class: traefik
status:
loadBalancer: {}
spec:
ingressClassName: traefik # Type of controller being used
tls:
- hosts:
- choppa.xyz
- talk.choppa.xyz
- blog.choppa.xyz
- files.choppa.xyz
secretName: certificate
rules: # Routing rules
- host: choppa.xyz # Expected domain name of request, including subdomain
http: # For HTTP or HTTPS requests
paths: # Behavior for different base paths
- path: / # For all request paths
pathType: Prefix
backend:
service:
name: welcome-service # Redirect to this service
port:
number: 8080 # Redirect to this internal service port
- path: /.well-known/matrix/
pathType: ImplementationSpecific
backend:
service:
name: conduit-service
port:
number: 8448
- host: talk.choppa.xyz
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: conduit-service
port:
number: 8448
- host: test.choppa.xyz # Expected domain name of request, including subdomain
http: # For HTTP or HTTPS requests
paths: # Behavior for different base paths
- path: / # For all request paths
pathType: Prefix
backend:
service:
name: test-service # Redirect to this service
port:
number: 80 # Redirect to this internal service port
- host: blog.choppa.xyz
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: writefreely-service
port:
number: 8080
- host: files.choppa.xyz
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: filebrowser-service
port:
number: 8080
If everything went accordingly, you now have everything in place to log into your blog and start publishing. To get an idea of how your self-hosted articles will look like, pay a visit to the first chapter of this series that I'm starting to publish on my server as well:
As I have found out, like the Allies had for those river crossings at Operation Market Garden, stateful sets are not as trivial as they initially appear: most guides will just tell you what's their purpose and how to get them running, which leaves a false impression that synchronized data replication across pods happens automagically (sic)
Well, it doesn't
Crossing that River. No Matter the Costs
That special type of deployment will only give you guarantees regarding the order of pods creation and deletion, their naming scheme and which persistent volume they will be bound to. Anything else is on the application logic. You may even violate the principle of using only the first pod for writing and the other ones for reading
When it comes to more niche applications like Conduit, I will probably have to code my own replication solution at some point, but for more widely used software like PostgreSQL there are solutions already available, thankfully
I don't plan on covering Helm here as I think it adds complexity over already quite complex K8s manifests. Surely it might be useful for large-scale stuff, but let's keep things simple here. I have combined knowledge from the articles with the updated charts in order to created a trimmed-down version of the required manifests (it would be good to add liveliness and readiness probes though):
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-config
labels:
app: postgres
data:
BITNAMI_DEBUG: "false" # Set to "true" for more debug information
POSTGRESQL_VOLUME_DIR: /bitnami/postgresql
PGDATA: /bitnami/postgresql/data
POSTGRESQL_LOG_HOSTNAME: "true" # Set to "false" for less debug information
POSTGRESQL_LOG_CONNECTIONS: "false" # Set to "true" for more debug information
POSTGRESQL_CLIENT_MIN_MESSAGES: "error"
POSTGRESQL_SHARED_PRELOAD_LIBRARIES: "pgaudit, repmgr" # Modules being used for replication
REPMGR_LOG_LEVEL: "NOTICE"
REPMGR_USERNAME: repmgr # Replication user
REPMGR_DATABASE: repmgr # Replication information database
---
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-scripts-config
labels:
app: postgres
data:
# Script for pod termination
pre-stop.sh: |-
#!/bin/bash
set -o errexit
set -o pipefail
set -o nounset
# Debug section
exec 3>&1
exec 4>&2
# Process input parameters
MIN_DELAY_AFTER_PG_STOP_SECONDS=$1
# Load Libraries
. /opt/bitnami/scripts/liblog.sh
. /opt/bitnami/scripts/libpostgresql.sh
. /opt/bitnami/scripts/librepmgr.sh
# Load PostgreSQL & repmgr environment variables
. /opt/bitnami/scripts/postgresql-env.sh
# Auxiliary functions
is_new_primary_ready() {
return_value=1
currenty_primary_node="$(repmgr_get_primary_node)"
currenty_primary_host="$(echo $currenty_primary_node | awk '{print $1}')"
info "$currenty_primary_host != $REPMGR_NODE_NETWORK_NAME"
if [[ $(echo $currenty_primary_node | wc -w) -eq 2 ]] && [[ "$currenty_primary_host" != "$REPMGR_NODE_NETWORK_NAME" ]]; then
info "New primary detected, leaving the cluster..."
return_value=0
else
info "Waiting for a new primary to be available..."
fi
return $return_value
}
export MODULE="pre-stop-hook"
if [[ "${BITNAMI_DEBUG}" == "true" ]]; then
info "Bash debug is on"
else
info "Bash debug is off"
exec 1>/dev/null
exec 2>/dev/null
fi
postgresql_enable_nss_wrapper
# Prepare env vars for managing roles
readarray -t primary_node < <(repmgr_get_upstream_node)
primary_host="${primary_node[0]}"
# Stop postgresql for graceful exit.
PG_STOP_TIME=$EPOCHSECONDS
postgresql_stop
if [[ -z "$primary_host" ]] || [[ "$primary_host" == "$REPMGR_NODE_NETWORK_NAME" ]]; then
info "Primary node need to wait for a new primary node before leaving the cluster"
retry_while is_new_primary_ready 10 5
else
info "Standby node doesn't need to wait for a new primary switchover. Leaving the cluster"
fi
# Make sure pre-stop hook waits at least 25 seconds after stop of PG to make sure PGPOOL detects node is down.
# default terminationGracePeriodSeconds=30 seconds
PG_STOP_DURATION=$(($EPOCHSECONDS - $PG_STOP_TIME))
if (( $PG_STOP_DURATION < $MIN_DELAY_AFTER_PG_STOP_SECONDS )); then
WAIT_TO_PG_POOL_TIME=$(($MIN_DELAY_AFTER_PG_STOP_SECONDS - $PG_STOP_DURATION))
info "PG stopped including primary switchover in $PG_STOP_DURATION. Waiting additional $WAIT_TO_PG_POOL_TIME seconds for PG pool"
sleep $WAIT_TO_PG_POOL_TIME
fi
---
apiVersion: v1
kind: Secret
metadata:
name: postgres-secret
data:
POSTGRES_PASSWORD: cG9zdGdyZXM= # Default user(postgres)'s password
REPMGR_PASSWORD: cmVwbWdy # Replication user's password
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres-state
spec:
serviceName: postgres-service
replicas: 2
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
securityContext: # Container is not run as root
fsGroup: 1001
runAsUser: 1001
runAsGroup: 1001
containers:
- name: postgres
lifecycle:
preStop: # Routines to run before pod termination
exec:
command:
- /pre-stop.sh
- "25"
image: docker.io/bitnami/postgresql-repmgr:16.2.0
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
name: postgres-port
envFrom:
- configMapRef:
name: postgres-config
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: POSTGRES_PASSWORD
- name: REPMGR_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: REPMGR_PASSWORD
# Write the pod name (from metadata field) to an environment variable in order to automatically generate replication addresses
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
# Repmgr configuration
- name: REPMGR_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: REPMGR_PARTNER_NODES # All pods being synchronized (has to reflect the number of replicas)
value: postgres-state-0.postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local,postgres-state-1.postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local
- name: REPMGR_PRIMARY_HOST # Pod with write access. Everybody else replicates it
value: postgres-state-0.postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local
- name: REPMGR_NODE_NAME # Current pod name
value: $(POD_NAME)
- name: REPMGR_NODE_NETWORK_NAME
value: $(POD_NAME).postgres-service.$(REPMGR_NAMESPACE).svc.cluster.local
volumeMounts:
- name: postgres-db
mountPath: /bitnami/postgresql
- name: postgres-scripts
mountPath: /pre-stop.sh
subPath: pre-stop.sh
- name: empty-dir
mountPath: /tmp
subPath: tmp-dir
- name: empty-dir
mountPath: /opt/bitnami/postgresql/conf
subPath: app-conf-dir
- name: empty-dir
mountPath: /opt/bitnami/postgresql/tmp
subPath: app-tmp-dir
- name: empty-dir
mountPath: /opt/bitnami/repmgr/conf
subPath: repmgr-conf-dir
- name: empty-dir
mountPath: /opt/bitnami/repmgr/tmp
subPath: repmgr-tmp-dir
- name: empty-dir
mountPath: /opt/bitnami/repmgr/logs
subPath: repmgr-logs-dir
volumes:
- name: postgres-scripts
configMap:
name: postgres-scripts-config
defaultMode: 0755 # Access permissions (owner can execute processes)
- name: empty-dir # Use a fake directory for mounting unused but required paths
emptyDir: {}
volumeClaimTemplates: # Description of volume claim created for each replica
- metadata:
name: postgres-db
spec:
storageClassName: nfs-small
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
---
apiVersion: v1
kind: Service
metadata:
name: postgres-service
labels:
app: postgres
spec:
type: ClusterIP # Default service type
clusterIP: None # Do not get a service-wide address
selector:
app: postgres
ports:
- protocol: TCP
port: 5432
targetPort: postgres-port
As the Bitnami container runs as a non-root user for security reasons, requires a “postgres” administrator name for the database and uses a different path structure, you won't be able to mount the data from the original PostgreSQL deployment without messing around with some configuration. So, unless you absolutely need the data, just start from scratch by deleting the volumes, maybe doing a backup first
Notice how our ClusterIP is set to not have an shared address, making it a headless service, so that it only serves the purpose of exposing the target port of each individual pod, still accessible via <pod name>.<service-name>:<container port number>. We do that as our containers here are not meant to be accessed in a random or load-balanced manner
If you're writing your own application, it's easy to define different addresses for writing to and reading from a replicated database, respecting the role of each copy. But a lot of useful software already around assumes a single connection is needed, and there's no simple way to get around that. That's why you need specific intermediaries or proxies like Pgpool-II for PostgreSQL, that can appear to applications as a single entity, redirecting queries to the appropriate backend database depending on it's contents:
I bet most of it is self explanatory by now. Just pay extra attention to the NUM_INIT_CHILDREN, MAX_POLL and RESERVED_CONNECTIONS variables and the relationship between them, as their default values may not be appropriate at all for your application and result in too many connection refusals (Been there. Done that). Moreover, users other than administrator and replicator are blocked from access unless you add them to the custom lists of usernames and passwords, in the format user1,user2,user3,.. and pswd1,pswd2,pswd3,..., here provided as base64-encoded secrets
With all that configured, we can finally (this time I really mean it) deploy a useful, stateful and replicated application:
$ kubectl get all -n choppa -l app=postgres
NAME READY STATUS RESTARTS AGE
pod/postgres-state-0 1/1 Running 0 40h
pod/postgres-state-1 1/1 Running 0 40h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/postgres-service ClusterIP None <none> 5432/TCP 40h
$ kubectl get all -n choppa -l app=postgres-proxy
NAME READY STATUS RESTARTS AGE
pod/postgres-proxy-deploy-74bbdd9b9d-j2tsn 1/1 Running 0 40h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/postgres-proxy-service LoadBalancer 10.43.217.63 192.168.3.10,192.168.3.12 5432:30217/TCP 40h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/postgres-proxy-deploy 1/1 1 1 40h
NAME DESIRED CURRENT READY AGE
replicaset.apps/postgres-proxy-deploy-74bbdd9b9d 1 1 1 40h
(Some nice usage of component labels for selection)
Into the Breach!
You should be able to view your database you the postgres user the same way we did last time. After informing the necessary custom users to Pgpoll, now not only I can get my Telegram bridge back running (using the proxy address for the connection string), but also install WhatsApp and Discord ones. Although they're written in Go rather than Python, configuration is very similar, with the relevant parts below:
apiVersion: v1
kind: ConfigMap
metadata:
name: whatsapp-config
labels:
app: whatsapp
data:
config.yaml: |
# Homeserver details.
homeserver:
# The address that this appservice can use to connect to the homeserver.
address: https://talk.choppa.xyz
# The domain of the homeserver (also known as server_name, used for MXIDs, etc).
domain: choppa.xyz
# ...
# Application service host/registration related details.
# Changing these values requires regeneration of the registration.
appservice:
# The address that the homeserver can use to connect to this appservice.
address: http://whatsapp-service:29318
# The hostname and port where this appservice should listen.
hostname: 0.0.0.0
port: 29318
# Database config.
database:
# The database type. "sqlite3-fk-wal" and "postgres" are supported.
type: postgres
# The database URI.
# SQLite: A raw file path is supported, but `file:<path>?_txlock=immediate` is recommended.
# https://github.com/mattn/go-sqlite3#connection-string
# Postgres: Connection string. For example, postgres://user:password@host/database?sslmode=disable
# To connect via Unix socket, use something like postgres:///dbname?host=/var/run/postgresql
uri: postgres://whatsapp:mautrix@postgres-proxy-service/matrix_whatsapp?sslmode=disable
# Maximum number of connections. Mostly relevant for Postgres.
max_open_conns: 20
max_idle_conns: 2
# Maximum connection idle time and lifetime before they're closed. Disabled if null.
# Parsed with https://pkg.go.dev/time#ParseDuration
max_conn_idle_time: null
max_conn_lifetime: null
# The unique ID of this appservice.
id: whatsapp
# Appservice bot details.
bot:
# Username of the appservice bot.
username: whatsappbot
# Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty
# to leave display name/avatar as-is.
displayname: WhatsApp bridge bot
avatar: mxc://maunium.net/NeXNQarUbrlYBiPCpprYsRqr
# Whether or not to receive ephemeral events via appservice transactions.
# Requires MSC2409 support (i.e. Synapse 1.22+).
ephemeral_events: true
# Should incoming events be handled asynchronously?
# This may be necessary for large public instances with lots of messages going through.
# However, messages will not be guaranteed to be bridged in the same order they were sent in.
async_transactions: false
# Authentication tokens for AS <-> HS communication. Autogenerated; do not modify.
as_token: <same as token as in registration.yaml>
hs_token: <same hs token as in registration.yaml>
# ...
# Config for things that are directly sent to WhatsApp.
whatsapp:
# Device name that's shown in the "WhatsApp Web" section in the mobile app.
os_name: Mautrix-WhatsApp bridge
# Browser name that determines the logo shown in the mobile app.
# Must be "unknown" for a generic icon or a valid browser name if you want a specific icon.
# List of valid browser names: https://github.com/tulir/whatsmeow/blob/efc632c008604016ddde63bfcfca8de4e5304da9/binary/proto/def.proto#L43-L64
browser_name: unknown
# Proxy to use for all WhatsApp connections.
proxy: null
# Alternative to proxy: an HTTP endpoint that returns the proxy URL to use for WhatsApp connections.
get_proxy_url: null
# Whether the proxy options should only apply to the login websocket and not to authenticated connections.
proxy_only_login: false
# Bridge config
bridge:
# ...
# Settings for handling history sync payloads.
history_sync:
# Enable backfilling history sync payloads from WhatsApp?
backfill: true
# ...
# Shared secret for authentication. If set to "generate", a random secret will be generated,
# or if set to "disable", the provisioning API will be disabled.
shared_secret: generate
# Enable debug API at /debug with provisioning authentication.
debug_endpoints: false
# Permissions for using the bridge.
# Permitted values:
# relay - Talk through the relaybot (if enabled), no access otherwise
# user - Access to use the bridge to chat with a WhatsApp account.
# admin - User level and some additional administration tools
# Permitted keys:
# * - All Matrix users
# domain - All users on that homeserver
# mxid - Specific user
permissions:
"*": relay
"@ancapepe:choppa.xyz": admin
"@ancompepe:choppa.xyz": user
# Settings for relay mode
relay:
# Whether relay mode should be allowed. If allowed, `!wa set-relay` can be used to turn any
# authenticated user into a relaybot for that chat.
enabled: false
# Should only admins be allowed to set themselves as relay users?
admin_only: true
# ...
# Logging config. See https://github.com/tulir/zeroconfig for details.
logging:
min_level: debug
writers:
- type: stdout
format: pretty-colored
registration.yaml: |
id: whatsapp
url: http://whatsapp-service:29318
as_token: <same as token as in config.yaml>
hs_token: <same hs token as in config.yaml>
sender_localpart: SH98XxA4xvgFtlbx1NxJm9VYW6q3BdYg
rate_limited: false
namespaces:
users:
- regex: ^@whatsappbot:choppa\.xyz$
exclusive: true
- regex: ^@whatsapp_.*:choppa\.xyz$
exclusive: true
de.sorunome.msc2409.push_ephemeral: true
push_ephemeral: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: whatsapp-deploy
spec:
replicas: 1
selector:
matchLabels:
app: whatsapp
template:
metadata:
labels:
app: whatsapp
spec:
containers:
- name: whatsapp
image: dock.mau.dev/mautrix/whatsapp:latest
imagePullPolicy: "IfNotPresent"
command: [ "/usr/bin/mautrix-whatsapp", "-c", "/data/config.yaml", "-r", "/data/registration.yaml", "--no-update" ]
ports:
- containerPort: 29318
name: whatsapp-port
volumeMounts:
- name: whatsapp-volume
mountPath: /data/config.yaml
subPath: config.yaml
- name: whatsapp-volume
mountPath: /data/registration.yaml
subPath: registration.yaml
volumes:
- name: whatsapp-volume
configMap:
name: whatsapp-config
---
apiVersion: v1
kind: Service
metadata:
name: whatsapp-service
spec:
publishNotReadyAddresses: true
selector:
app: whatsapp
ports:
- protocol: TCP
port: 29318
targetPort: whatsapp-port
apiVersion: v1
kind: ConfigMap
metadata:
name: discord-config
labels:
app: discord
data:
config.yaml: |
# Homeserver details.
homeserver:
# The address that this appservice can use to connect to the homeserver.
address: https://talk.choppa.xyz
# The domain of the homeserver (also known as server_name, used for MXIDs, etc).
domain: choppa.xyz
# What software is the homeserver running?
# Standard Matrix homeservers like Synapse, Dendrite and Conduit should just use "standard" here.
software: standard
# The URL to push real-time bridge status to.
# If set, the bridge will make POST requests to this URL whenever a user's discord connection state changes.
# The bridge will use the appservice as_token to authorize requests.
status_endpoint: null
# Endpoint for reporting per-message status.
message_send_checkpoint_endpoint: null
# Does the homeserver support https://github.com/matrix-org/matrix-spec-proposals/pull/2246?
async_media: false
# Should the bridge use a websocket for connecting to the homeserver?
# The server side is currently not documented anywhere and is only implemented by mautrix-wsproxy,
# mautrix-asmux (deprecated), and hungryserv (proprietary).
websocket: false
# How often should the websocket be pinged? Pinging will be disabled if this is zero.
ping_interval_seconds: 0
# Application service host/registration related details.
# Changing these values requires regeneration of the registration.
appservice:
# The address that the homeserver can use to connect to this appservice.
address: http://discord-service:29334
# The hostname and port where this appservice should listen.
hostname: 0.0.0.0
port: 29334
# Database config.
database:
# The database type. "sqlite3-fk-wal" and "postgres" are supported.
type: postgres
# The database URI.
# SQLite: A raw file path is supported, but `file:<path>?_txlock=immediate` is recommended.
# https://github.com/mattn/go-sqlite3#connection-string
# Postgres: Connection string. For example, postgres://user:password@host/database?sslmode=disable
# To connect via Unix socket, use something like postgres:///dbname?host=/var/run/postgresql
uri: postgres://discord:mautrix@postgres-proxy-service/matrix_discord?sslmode=disable
# Maximum number of connections. Mostly relevant for Postgres.
max_open_conns: 20
max_idle_conns: 2
# Maximum connection idle time and lifetime before they're closed. Disabled if null.
# Parsed with https://pkg.go.dev/time#ParseDuration
max_conn_idle_time: null
max_conn_lifetime: null
# The unique ID of this appservice.
id: discord
# Appservice bot details.
bot:
# Username of the appservice bot.
username: discordbot
# Display name and avatar for bot. Set to "remove" to remove display name/avatar, leave empty
# to leave display name/avatar as-is.
displayname: Discord bridge bot
avatar: mxc://maunium.net/nIdEykemnwdisvHbpxflpDlC
# Whether or not to receive ephemeral events via appservice transactions.
# Requires MSC2409 support (i.e. Synapse 1.22+).
ephemeral_events: true
# Should incoming events be handled asynchronously?
# This may be necessary for large public instances with lots of messages going through.
# However, messages will not be guaranteed to be bridged in the same order they were sent in.
async_transactions: false
# Authentication tokens for AS <-> HS communication. Autogenerated; do not modify.
as_token: <same as token as in registration.yaml>
hs_token: <same hs token as in registration.yaml>
# Bridge config
bridge:
# ...
# The prefix for commands. Only required in non-management rooms.
command_prefix: '!discord'
# Permissions for using the bridge.
# Permitted values:
# relay - Talk through the relaybot (if enabled), no access otherwise
# user - Access to use the bridge to chat with a Discord account.
# admin - User level and some additional administration tools
# Permitted keys:
# * - All Matrix users
# domain - All users on that homeserver
# mxid - Specific user
permissions:
"*": relay
"@ancapepe:choppa.xyz": admin
"@ancompepe:choppa.xyz": user
# Logging config. See https://github.com/tulir/zeroconfig for details.
logging:
min_level: debug
writers:
- type: stdout
format: pretty-colored
registration.yaml: |
id: discord
url: http://discord-service:29334
as_token: <same as token as in config.yaml>
hs_token: <same hs token as in config.yaml>
sender_localpart: KYmI12PCMJuHvD9VZw1cUzMlV7nUezH2
rate_limited: false
namespaces:
users:
- regex: ^@discordbot:choppa\.xyz$
exclusive: true
- regex: ^@discord_.*:choppa\.xyz$
exclusive: true
de.sorunome.msc2409.push_ephemeral: true
push_ephemeral: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: discord-deploy
spec:
replicas: 1
selector:
matchLabels:
app: discord
template:
metadata:
labels:
app: discord
spec:
containers:
- name: discord
image: dock.mau.dev/mautrix/discord:latest
imagePullPolicy: "IfNotPresent"
command: [ "/usr/bin/mautrix-discord", "-c", "/data/config.yaml", "-r", "/data/registration.yaml", "--no-update" ]
ports:
- containerPort: 29334
name: discord-port
volumeMounts:
- name: discord-volume
mountPath: /data/config.yaml
subPath: config.yaml
- name: discord-volume
mountPath: /data/registration.yaml
subPath: registration.yaml
volumes:
- name: discord-volume
configMap:
name: discord-config
---
apiVersion: v1
kind: Service
metadata:
name: discord-service
spec:
publishNotReadyAddresses: true
selector:
app: discord
ports:
- protocol: TCP
port: 29334
targetPort: discord-port
Logging into your account will still change depending on the service being bridged. As always, consult the official documentation
We may forget for a while the multiple opened messaging windows just to communicate with our peers. That river has been crossed!
Hello again. It's been a while (in relative terms)
Having our own Matrix rooms feels amazing indeed, but left us too isolated, at least until we manage to bring people in. I know that in the beginning I've warned about making sacrifices, but it doesn't have to be that hard. In life some metaphorical burned bridges are better left unrepaired, but here let's try reconnecting to our old Internet life in an interesting way, right?
Notes on Infrastructure
I have waited a bit to resume publishing because there were 2 things bothering me about my setup: power reliability and storage coupling
I don't live near a big urban center, and power oscillations or outages are pretty common here. Having those causing frequent downtime on my server would be quite annoying, so I went after a nobreak to ensure more energy stability for my devices
With that out the way, as I have said on the previous article, ideally your K8s system would have an external and dedicated storage provider. I certainly wasn't satisfied with mixing K8s node and NFS server functionality in the same machines, but the ones I was using were the only units with high-bandwith USB 3.0 ports, desirable for external drives. So I've decided to invest a bit more and acquire an extra Raspberry Pi 5 for my cluster, leaving the RPi4 for data management
Not only the RPi5 gives me more performance, but it also is more equivalent to the Odroid N2+. As it comes with a beefier power supply (5V-5A), intended for video applications that I'm not doing, I've tried using it for the RPi4 connected to the current-hungry hard drives. However, its circuit is simply not made to use all that juice, so as a last piece of the puzzle I had to get an externally powered USB3 hub:
(The resulting Frankenstein is not a pretty sight)
If that mess of a wiring is too hard to understand (I bet it is), have a quick diagram of how it works below:
At first I was worried that using a single USB port for 2 disks would be a bottleneck, but even in the best case, sequential reading, HDDs cannot saturate the 5Gbps bandwidth the RPi4 can handle on that interface, and the 1Gbps Ethernet will be a greater constraint anyway
Keeping the State, Orchestrated Style
One thing that you notice when deploying all sorts of applications is how much they rely on databases, relational or non-relational, in order to delegate internal data management and remain stateless. That's true to the point that we can hardly progress here without having some DBMS running first
It's common to have databases running as off-node services just like NFS, but the ability to escalate them horizontally is then lost. Here, we'll resort to Kubernetesstateful features in order to keep data management inside the cluster. For that, instead of a Deployment we require a StatefulSet:
apiVersion: v1
kind: ConfigMap
metadata:
name: postgres-config
namespace: choppa
labels:
app: postgres
data:
POSTGRES_DB: choppadb # Default database
POSTGRES_USER: admin # Default user
---
apiVersion: v1
kind: Secret
metadata:
name: postgres-secret
namespace: choppa
data:
POSTGRES_PASSWORD: Z290eW91 # Default user's password
---
apiVersion: apps/v1
kind: StatefulSet # Component type
metadata:
name: postgres-state
namespace: choppa
spec:
serviceName: postgres-service # Do not give a service name to each replica
replicas: 2
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
spec:
containers:
- name: postgres
image: postgres:16.4 # Current stable version at 17.0, but let's be a bit conservative
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 5432
name: postgres-port
envFrom:
- configMapRef:
name: postgres-config
env:
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-secret
key: POSTGRES_PASSWORD
volumeMounts:
- mountPath: /var/lib/postgresql/data
name: postgres-db
volumeClaimTemplates: # Description of volume claim created for each replica
- metadata:
name: postgres-db
spec:
storageClassName: nfs-small
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
---
apiVersion: v1
kind: Service
metadata:
name: postgres-service
namespace: choppa
spec:
type: LoadBalancer # Let it be accessible inside the local network
selector:
app: postgres
ports:
- protocol: TCP
port: 5432
targetPort: postgres-port
The biggest difference from previous deployments is that we don't explicitly define a Persistent Volume Claim to be referenced by all replicas, but give the StatefulSet a template on how to request a distinct storage area for each Pod. Moreover, the serviceName configuration defines how stateful pods will get their specific hostnames from the internal DNS, in the format <pod name>.<service-name>:<container port number>
(I have tried using a stateful configuration for Conduit as well, but also wasn't able to increase the number of replicas due to some authentication issues)
Notice how those pods follow a different naming scheme. ReplicaSets with persisting data changes have to be created, deleted and updated/synchronized in a very particular order, therefore they cannot be as ephemeral as stateless ones, with asynchronously managed and randomly named containers. Those constraints are the reason we should avoid StatefulSets wherever possible
A Telegram from an Old Friend
As I've promised, today we're learning how to keep in touch with people in the centralized world from our new Matrix home. And that's achieved through the power of Mautrix bridges. As I'm a big user of Telegram, that's what I'll be using as an example, but the process is mostly the same for other services, like Discord or WhatsApp
Long story short, the bridge needs to be configured on 2 ends: how to interact with your third-party service account and how to register onto your Matrix server. The templates for those settings may be automatically generated using the same container image we'll me pulling into our K8s cluster:
# Create a directory for configuration files and enter it
$ docker run --rm -v `pwd`:/data:z dock.mau.dev/mautrix/telegram:latest
Didn't find a config file.
Copied default config file to /data/config.yaml
Modify that config file to your liking.
Start the container again after that to generate the registration file.
# Change the configuration file to match your requirements and preferences
$ docker run --rm -v `pwd`:/data:z dock.mau.dev/mautrix/telegram:latest
Registration generated and saved to /data/registration.yaml
Didn't find a registration file.
Generated one for you.
See https://docs.mau.fi/bridges/general/registering-appservices.html on how to use it.
Now you either manually add the contents of config.yaml and registration.yaml to a ConfigMap manifest file or automatically generate/print it with:
(Running as super user is required as the folder permissions are modified. --dry-run prevents the configuration from being directly applied, so that you may adjust possible mistakes, pipe the results to a YAML file or add the result to a bigger manifest)
Bridge configuration is quite long, so I'll reduce it below to only the most relevant parts. For information on all options, consult the official documentation:
apiVersion: v1
kind: ConfigMap
metadata:
name: telegram-config
labels:
app: telegram
data:
config.yaml: |
# Homeserver details
homeserver:
# The address that this appservice can use to connect to the homeserver.
address: http://conduit-service:8448
# The domain of the homeserver (for MXIDs, etc).
domain: choppa.xyz
# ...
# Application service host/registration related details
# Changing these values requires regeneration of the registration.
appservice:
# The address that the homeserver can use to connect to this appservice.
address: http://telegram-service:29317
# When using https:// the TLS certificate and key files for the address.
tls_cert: false
tls_key: false
# The hostname and port where this appservice should listen.
hostname: 0.0.0.0
port: 29317
# ...
# The full URI to the database. SQLite and Postgres are supported.
# Format examples:
# SQLite: sqlite:filename.db
# Postgres: postgres://username:password@hostname/dbname
database: postgres://telegram:mautrix@postgres-service/matrix_telegram
# ...
# Authentication tokens for AS <-> HS communication. Autogenerated; do not modify.
as_token: <same as token from registration.yaml>
hs_token: <same hs token from registration.yaml>
# ...
# Bridge config
bridge:
# ...
# Shared secrets for https://github.com/devture/matrix-synapse-shared-secret-auth
#
# If set, custom puppets will be enabled automatically for local users
# instead of users having to find an access token and run `login-matrix`
# manually.
# If using this for other servers than the bridge's server,
# you must also set the URL in the double_puppet_server_map.
login_shared_secret_map:
choppa.xyz: <your access token>
# ...
# ...
# Telegram config
telegram:
# Get your own API keys at https://my.telegram.org/apps
api_id: <id you have generated>
api_hash: <hash you have generated>
# (Optional) Create your own bot at https://t.me/BotFather
bot_token: disabled
# ...
registration.yaml: |
id: telegram
as_token: <same as token from config.yaml>
hs_token: <same hs token from config.yaml>
namespaces:
users:
- exclusive: true
regex: '@telegram_.*:choppa\.xyz'
- exclusive: true
regex: '@telegrambot:choppa\.xyz'
aliases:
- exclusive: true
regex: \#telegram_.*:choppa\.xyz
url: http://telegram-service:29317
sender_localpart: HyMPlWT_552RAGkFtnuy_ZNNpkKrSSaHDndS_nmb9VIZ4RiJLH0uiSas3fi_IV_x
rate_limited: false
de.sorunome.msc2409.push_ephemeral: true
push_ephemeral: true
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: telegram-deploy
spec:
replicas: 1
selector:
matchLabels:
app: telegram
template:
metadata:
labels:
app: telegram
spec:
containers:
- name: telegram
image: dock.mau.dev/mautrix/telegram:latest
imagePullPolicy: "IfNotPresent"
# Custom container initialization command (overrides the one defined in the image)
command: [ "python3", "-m", "mautrix_telegram", "-c", "/data/config.yaml", "-r", "/data/registration.yaml", "--no-update" ]
ports:
- containerPort: 29317 # Has to match the appservice configuration
name: telegram-port
volumeMounts:
- name: telegram-volume
mountPath: /data/config.yaml
subPath: config.yaml
- name: telegram-volume
mountPath: /data/registration.yaml
subPath: registration.yaml
volumes:
- name: telegram-volume
configMap:
name: telegram-config
---
apiVersion: v1
kind: Service
metadata:
name: telegram-service
spec:
publishNotReadyAddresses: true
selector:
app: telegram
ports:
- protocol: TCP
port: 29317 # Has to match the registration url
targetPort: telegram-port
After applying the changes, the bridge pod is deployed, but it keeps failing and restarting... Why?
Visualizing the logs give us a clearer picture of what's happening:
$ kubectl logs telegram-deploy-84489fb64d-srmrc --namespace=choppa ✔ default ⎈
[2024-09-29 15:28:34,871] [INFO@mau.init] Initializing mautrix-telegram 0.15.2
[2024-09-29 15:28:34,879] [INFO@mau.init] Initialization complete in 0.19 seconds
[2024-09-29 15:28:34,879] [DEBUG@mau.init] Running startup actions...
[2024-09-29 15:28:34,879] [DEBUG@mau.init] Starting database...
[2024-09-29 15:28:34,879] [DEBUG@mau.db] Connecting to postgres://telegram:password-redacted@postgres-service/matrix_telegram
[2024-09-29 15:28:34,946] [CRITICAL@mau.init] Failed to initialize database
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/mautrix/bridge/bridge.py", line 216, in start_db
await self.db.start()
File "/usr/lib/python3.11/site-packages/mautrix/util/async_db/asyncpg.py", line 71, in start
self._pool = await asyncpg.create_pool(str(self.url), **self._db_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 403, in _async__init__
await self._initialize()
File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 430, in _initialize
await first_ch.connect()
File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 128, in connect
self._con = await self._pool._get_new_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/asyncpg/pool.py", line 502, in _get_new_connection
con = await connection.connect(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/asyncpg/connection.py", line 2329, in connect
return await connect_utils._connect(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 991, in _connect
conn = await _connect_addr(
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 828, in _connect_addr
return await __connect_addr(params, True, *args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/asyncpg/connect_utils.py", line 876, in __connect_addr
await connected
asyncpg.exceptions.InvalidPasswordError: password authentication failed for user "telegram"
[2024-09-29 15:28:34,950] [DEBUG@mau.init] Stopping database due to SystemExit
[2024-09-29 15:28:34,950] [DEBUG@mau.init] Database stopped
If you pay attention to the configuration shown, there's a database section where you may either configure SQLite (local file, let's avoid that) or PostgreSQL usage, required by the bridge. I wouldn't go on a tangent about databases for nothing, right?
The problem is that the connection string defined, which follows the format postgres://<user>:<password>@<server hostname>/<database name>, is referencing a user and a database that don't exist (I wouldn't use the administrator account for a simple bot). So let's fix that by logging as the main user into the DBMS with a PostgreSQL-compatible client of your choice. Here I'm using SQLectron:
From there, use this sequence of SQL commands that are pretty much self-explanatory:
CREATE DATABASE matrix_telegram;
CREATE USER telegram WITH PASSWORD 'mautrix';
GRANT CONNECT ON DATABASE matrix_telegram TO telegram;
-- Personal note: ENTER THE DATABASE !!
GRANT ALL ON SCHEMA public TO telegram;
GRANT ALL ON ALL TABLES IN SCHEMA public TO telegram;
GRANT ALL ON ALL SEQUENCES IN SCHEMA public TO telegram;
-- Give priveleges for all newly created tables as well
ALTER DEFAULT PRIVILEGES FOR USER telegram IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO telegram;
(A bit overkill but, hey, works for me... If some commands fail you may need to wait a bit for the pods to synchronize or even reduce the number of replicas to 1 while making changes)
Errata: Once again we cannot use stateful features to the fullest. Replicating data across pods is not done automatically by the K8s system and dependends on the application. It took me a while to figure that out and I'll have to study how to do it properly for each case. For now we'll keep it at only one instance and address proper replication later
If you still get some log errors about an unknown token, go into the admin room of your Conduit server and send this message:
Finally, start a private chat with @telegrambot<your server name> (default name, may be changed in the config) and follow the login procedure:
Invites for your Telegram groups and private chats should start appearing, and now you can be happy again!
Time to forget about simple demo containers and struggle [not so much] with the configuration details of fully-featured applications. Today we not only put Kubernetes to production-level usage, but find out about some strengths and weaknesses of distributed computing
Notes on storage
Code nowadays, more than ever, works consuming and producing huge amounts of data, that usually has (or “have”, since “data” is plural for “datum”?) to be kept for later. It's no surprise then that the capacity to store information is one of the most valuable commodities in IT, always at risk of being depleted due to poor optimization
In the containerized world, where application instances can be created and recreated dynamically, extra steps have to be taken for reusing disk space and avoiding duplication whenever possible. That's why K8s adopts a layered approach: deployments don't create Persistent Volumes, roughly equivalent to root folders, on their own, but set their storage requirements via Persistent Volume Claims
Long story short, administrators either create the necessary volumes manually or let the claims be managed by a Storage Class, to provide volumes on demand. Like ingresses have multiple controller implementations, depending on the backend, storage classes allow for different providers, offering a variety of local and remote (including cloud) data resources:
Although local is an easy to use option, it quickly betrays the purpose of container orchestration: tying storage to a specific machine's disk path prevents multiple pod replicas from being distributed among multiple nodes for load balancing. Besides, if we wish to avoid cloud providers (like Google or AWS) to have full control of our data, we're left with [local] network storage solutions like NFS
If you remember the first article, my setup consists of one USB hard disk (meant for persistent volumes) connected to each node board. That's not ideal for NFS, as decoupling storage from the K8s cluster, using a dedicated computer, would prevent node crashes compromising data availability to the rest. Still, I can at least make the devices share disks with each other
In order to expose external drive's partition to the network, begin by ensuring that they're mounted during initialization by editing /etc/fstab:
# Finding the partition's UUID (assuming a EXT4-formatted disk)
[ancapepe@ChoppaServer-1 ~]$ lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
└─sda1 ext4 1.0 69c7df98-0349-4c47-84ce-514a1699ccf1 869.2G 0%
mmcblk0
├─mmcblk0p1 vfat FAT16 BOOT_MNJRO 42D4-5413 394.8M 14% /boot
└─mmcblk0p2 ext4 1.0 ROOT_MNJRO 83982167-1add-4b6a-ad63-3479493a4510 44.4G 18% /var/lib/kubelet/pods/ae500dcf-b765-4f37-be8c-ee20fbb3ea1b/volume-subpaths/welcome-volume/welcome/0
/
zram0 [SWAP]
# Creating the mount point for the external partition
[ancapepe@ChoppaServer-1 ~]$ sudo mkdir /storage
# Adding line with mount configuration to fstab
[ancapepe@ChoppaServer-1 ~]$ echo 'UUID=69c7df98-0349-4c47-84ce-514a1699ccf1 /storage ext4 defaults,noatime 0 2' | sudo tee -a /etc/fstab
# Check the contents
[ancapepe@ChoppaServer-1 ~]$ cat /etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.
# <file system> <dir> <type> <options> <dump> <pass>
PARTUUID=42045b6f-01 /boot vfat defaults,noexec,nodev,showexec 0 0
PARTUUID=42045b6f-02 / ext4 defaults 0 1
UUID=69c7df98-0349-4c47-84ce-514a1699ccf1 /storage ext4 defaults,noatime 0 2
Reboot and check if you can access the contents of the /storage folder (or whatever path you've chosen). Proceed by setting up the NFS server:
# Install required packages (showing the ones for Manjaro Linux)
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S nfs-utils rpcbind
# ...
# Installation process ...
# ...
# Add disk sharing configurations (Adapt to you local network address range)
[ancapepe@ChoppaServer-1 ~]$ echo '/storage 192.168.3.0/24(rw,sync,no_wdelay,no_root_squash,insecure)' | sudo tee -a /etc/exports
# Check the contents
[ancapepe@ChoppaServer-1 ~]$ cat /etc/exports
# /etc/exports - exports(5) - directories exported to NFS clients
#
# Example for NFSv3:
# /srv/home hostname1(rw,sync) hostname2(ro,sync)
# Example for NFSv4:
# /srv/nfs4 hostname1(rw,sync,fsid=0)
# /srv/nfs4/home hostname1(rw,sync,nohide)
# Using Kerberos and integrity checking:
# /srv/nfs4 *(rw,sync,sec=krb5i,fsid=0)
# /srv/nfs4/home *(rw,sync,sec=krb5i,nohide)
#
# Use `exportfs -arv` to reload.
/storage 192.168.3.0/24(rw,sync,no_wdelay,no_root_squash,insecure)
# Update exports based on configuration
[ancapepe@ChoppaServer-1 ~]$ sudo exportfs -arv
# Enable and start related initialization services
[ancapepe@ChoppaServer-1 ~]$ systemctl enable --now rpcbind nfs-server
# Check if shared folders are visible
[ancapepe@ChoppaServer-1 ~]$ showmount localhost -e
Export list for localhost:
/storage 192.168.3.0/24
(Remember that your mileage may vary)
Phew, and that's just the first part of it! Not only you have to do the same process for every machine whose storage you wish to share, but afterwards it's necessary to install a NFS provisioner to the K8s cluster. Here I have used nfs-subdir-external-provisioner, not as-it-is, but modifying 2 manifest files to my liking:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: nfs-storage
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
# 1TB disk on Odroid N2+
- name: nfs-client-provisioner-big
image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
volumeMounts:
- name: nfs-root-big
mountPath: /persistentvolumes # CAN'T CHANGE THIS PATH
env: # Environment variables
- name: PROVISIONER_NAME
value: nfs-provisioner-big
- name: NFS_SERVER # Hostnames don't resolve in setting env vars
value: 192.168.3.10 # Looking for a way to make it more flexible
- name: NFS_PATH
value: /storage
# 500GB disk on Raspberry Pi 4B
- name: nfs-client-provisioner-small
image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
volumeMounts:
- name: nfs-root-small
mountPath: /persistentvolumes # CAN'T CHANGE THIS PATH
env: # Environment variables
- name: PROVISIONER_NAME
value: nfs-provisioner-small
- name: NFS_SERVER
value: 192.168.3.11
- name: NFS_PATH
value: /storage
volumes:
- name: nfs-root-big
nfs:
server: 192.168.3.10
path: /storage
- name: nfs-root-small
nfs:
server: 192.168.3.11
path: /storage
---
# 1TB disk on Odroid N2+
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-big
namespace: choppa
provisioner: nfs-provisioner-big
parameters:
archiveOnDelete: "false"
reclaimPolicy: Retain
---
# 500GB disk on Raspberry Pi 4B
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-small
namespace: choppa
provisioner: nfs-provisioner-small
parameters:
archiveOnDelete: "false"
reclaimPolicy: Retain
(Make sure that all namespace settings match the RBAC rules. Also notice that, the way the provisioner works, I had to create on storage class for each shared disk)
If everything goes according to plan, applying a test like the one below will make volumes appear to meet the claims and the SUCCESS file to be written on each one:
Now that we have real storage at our disposal, some real apps would be nice. One of the first things that come to mind when I think about data I'd like to protect is the history of my private conversations, where most spontaneous interactions happen and most ideas get forgotten
If you're a old school person wishing to host your own messaging, I bet you'd go for IRC or XMPP. But I'm more of a late millennial, so it's easier to get drawn to fancy stuff like Matrix, a protocol for federated communication
“And what's federation?”, you may ask. Well, to this day, the best video about it I've seen is the one from Simply Explained. Even though it's focused on Mastodon and the Fediverse (We'll get there eventually), the same concepts apply:
As any open protocol, the Matrix specification has a number of different implementations, most famously Synapse (Python) and Dendrite (Go). I've been, however, particularly endeared to Conduit, a lightweight Matrix server written in Rust. Here I'll show the deployment configuration as recommended by its maintainer, Timo Kösters:
kind: PersistentVolumeClaim # Storage requirements component
apiVersion: v1
metadata:
name: conduit-pv-claim
namespace: choppa
labels:
app: conduit
spec:
storageClassName: nfs-big # The used storage class (1TB drive)
accessModes:
- ReadWriteOnce # For synchronous access, as we don't want data corruption
resources:
requests:
storage: 50Gi # Asking for a ~50 Gigabytes volume
---
apiVersion: v1
kind: ConfigMap # Read-only data component
metadata:
name: conduit-config
namespace: choppa
labels:
app: conduit
data:
CONDUIT_CONFIG: '' # Leave empty to inform that we're not using a configuration file
CONDUIT_SERVER_NAME: 'choppa.xyz' # The server name that will appear after registered usernames
CONDUIT_DATABASE_PATH: '/var/lib/matrix-conduit/' # Database directory path. Uses the permanent volume
CONDUIT_DATABASE_BACKEND: 'rocksdb' # Recommended for performance, sqlite alternative
CONDUIT_ADDRESS: '0.0.0.0' # Listen on all IP addresses/Network interfaces
CONDUIT_PORT: '6167' # Listening port for the process inside the container
CONDUIT_MAX_CONCURRENT_REQUESTS: '200' # Network usage limiting options
CONDUIT_MAX_REQUEST_SIZE: '20000000' # (in bytes, ~20 MB)
CONDUIT_ALLOW_REGISTRATION: 'true' # Allow other people to register in the server (requires token)
CONDUIT_ALLOW_FEDERATION: 'true' # Allow communication with other instances
CONDUIT_ALLOW_CHECK_FOR_UPDATES: 'true'
CONDUIT_TRUSTED_SERVERS: '["matrix.org"]'
CONDUIT_ALLOW_WELL_KNOWN: 'true' # Generate "/.well-known/matrix/{client,server}" endpoints
CONDUIT_WELL_KNOWN_CLIENT: 'https://talk.choppa.xyz' # return value for "/.well-known/matrix/client"
CONDUIT_WELL_KNOWN_SERVER: 'talk.choppa.xyz:443' # return value for "/.well-known/matrix/server"
---
apiVersion: v1
kind: Secret # Read-only encrypted data
metadata:
name: conduit-secret
namespace: choppa
data:
CONDUIT_REGISTRATION_TOKEN: bXlwYXNzd29yZG9yYWNjZXNzdG9rZW4= # Registration token/password in base64 format
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: conduit-deploy
namespace: choppa
spec:
replicas: 1 # Keep it at 1 as the database has an access lock
strategy:
type: Recreate # Wait for replica to be terminated completely befor creating a new one
selector:
matchLabels:
app: conduit
template:
metadata:
labels:
app: conduit
spec:
containers:
- name: conduit
image: ancapepe/conduit:next # fork of matrixconduit/matrix-conduit:next
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 6167 # Has to match the configuration above
name: conduit-port
envFrom: # Take all data from a config file
- configMapRef: # As environment variables
name: conduit-config
env: # Set environment variables one by one
- name: CONDUIT_REGISTRATION_TOKEN
valueFrom:
secretKeyRef: # Take data from a secret
name: conduit-secret # Secret source
key: CONDUIT_REGISTRATION_TOKEN # Data entry key
volumeMounts:
- mountPath: /var/lib/matrix-conduit/ # Mount the referenced volume into this path (database)
name: rocksdb
volumes:
- name: rocksdb # Label the volume for this deployment
persistentVolumeClaim:
claimName: conduit-pv-claim # Reference volumen create by the claim
---
apiVersion: v1
kind: Service
metadata:
name: conduit-service
namespace: choppa
spec:
selector:
app: conduit
ports:
- protocol: TCP
port: 8448 # Using the default port for federation (not required)
targetPort: conduit-port
One thing you should know is that the K8s architecture is optimized for stateless applications, which don't store changing information within themselves and whose output depend solely on input from the user or auxiliary processes. Conduit, on the contrary, is tightly coupled with its high-performance database, RocksDB, and has stateful behavior. That's why we need to take extra care by not replicating our process in order to prevent data races when accessing storage
There's actually a way to run multiple replicas of a stateful pod, that requires different component types to create one volume copy for each container and keep all them in sync. But let's keep it “simple” for now. More on that in a future chapter
We introduce here a new component type, Secret, intended for sensible information such as passwords and access tokens. It requires that all data is set in a Base64-encoded format for obfuscation, which could be achieved with:
You might also have noticed that I jumped the gun a bit and acquired a proper domain for myself (so that my Matrix server looks nice, of course). I've found one available at a promotional price on Dynadot, which now seems even more like a good decision, considering how easy is to bend Cloudflare. Conveniently, the registrar also provides a DNS service, so... 2 for the price of one:
The CNAME record is basically an alias: the subdomain talk.choppa.xyz is simply redirected to choppa.xyz and just works as a hint for our reverse proxy from the previous chapter:
apiVersion: networking.k8s.io/v1
kind: Ingress # Component type
metadata:
name: proxy # Component name
namespace: choppa # You may add the default namespace for components as a paramenter
annotations:
cert-manager.io/cluster-issuer: letsencrypt
kubernetes.io/ingress.class: traefik
status:
loadBalancer: {}
spec:
ingressClassName: traefik
tls:
- hosts:
- choppa.xyz
- talk.choppa.xyz
secretName: certificate
rules: # Routing rules
- host: choppa.xyz # Expected domain name of request, including subdomain
http: # For HTTP or HTTPS requests
paths: # Behavior for different base paths
- path: / # For all request paths
pathType: Prefix
backend:
service:
name: welcome-service # Redirect to this service
port:
number: 8080 # Redirect to this internal service port
- path: /.well-known/matrix/ # Paths for Conduit delegation
pathType: ImplementationSpecific
backend:
service:
name: conduit-service
port:
number: 8448
- host: talk.choppa.xyz # Conduit server subdomain
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: conduit-service
port:
number: 8448
If your CONDUIT_SERVER_NAME differs from the [sub]domain where your server is hosted, other Matrix federated instances will look for the actual address in Well-known URIs. Normally you'd have to serve them manually, but Conduit has a convenient automatic delegation feature that helps your set it up with the CONDUIT_WELL_KNOWN_* env variables
Currently, at version 0.8, automatically delegation via env vars is not working properly. Timo has helped me debug it (Thanks!) at Conduit's Matrix room and I have submitted a patch. Let's hope it gets fixed for 0.9. Meanwhile, you may use the custom image I'm referencing in the configuration example
Update: Aaaaaand... it's merged!
With ALL that said, let's apply the deployment and hope for the best:
(Success !?)
With your instance running, you can now use Matrix clients [that support registration tokens] like Element or Nheko to register your user, login and have fun!
For starters, I apologize for setting up the wrong expectations: chapter 1 anticipated all that stuff about external IPs and name servers [like it would be immediately necessary], but event in chapter 2 we still only accessed applications inside the local network
In my defense, I wished to make all the prerequisites clear from the get-go, so that there where no surprises about what you were getting into. Anyway, let's address that today, sooner rather than later, shall we?
Your Virtual Home Receptionist
The reason why you can't simply type http://<my WAN IP>:8080 or http://mybeautifuldomain.com:8080 to view the test page you're running is due to K3s (and Kubernetes implementations in general) not exposing pods by default. And that's the sensible decision, considering that not every application is supposed to be reached from outside, like a database providing storage for your backend service
Those access restrictions are created with the container runtime using iptables to set network rules. I've learned it the hard way when K3s conflicted with the nftables system service I had already running (fixed by disabling the latter). Time to update to new tools, containerd!
Moreover, while it's obvious where external requests should go to when we have a single application, how to handle many containers exposed to Internet access? For instance, it's not practical or sometimes even possible to demand an explicit port to select different Web services (e.g. mydomain.com:8081 and mydomain.com:8082), being the standard to use the default 80 and 443 ports for HTTP and HTTPS, respectively
With such requirement, it's common to share a single IP address (host and port) using paths (e.g. mydomain.com/app1 and mydomain.com/app2) or subdomains (e.g. app1.mydomain.com and app2.mydomain.com) to address a particular process. To achieve that, messages have to be interpreted for rerouting [to a local port] before reaching their intended target, a task performed by what is known as a reverse proxy
In K8s terminology, the component responsible for that functionality is called Ingress. It's actually a common interface for different implementations named Ingress Controllers, of which it's usually recommended (if you're not doing anything fancy or advanced) to use the NGINX-based one, as it's officially maintained
I emphasize usually because K3s is different and comes with a Traefik-based ingress controller by default. Taking that into account, as much as I like NGINX outside the container's world, I'd rather keep things simple and use what's already in place
It's totally fine to use Ingress-Nginx, tough. Just get into the GitHub repository and follow the instructions. I've used it myself before running into problems with upgrades (but you're not that dumb, right?). Be aware that all components will be created in their own predefined namespace, valid system-wide
Now we may use the provided controller to set our ingress rules:
apiVersion: networking.k8s.io/v1
kind: Ingress # Component type
metadata:
name: proxy # Component name
namespace: test # You may add the default namespace for components as a paramenter
status:
loadBalancer: {}
spec:
ingressClassName: traefik # Type of controller being used
rules: # Routing rules
- host: choppaserver.dynv6.net # Expected domain name of request, including subdomain
http: # For HTTP or HTTPS requests (standard ports)
paths: # Behavior for different base paths
- path: / # For all request paths
pathType: Prefix
backend:
service:
name: welcome-service # Redirect to this service
port:
number: 8080 # Redirect to this internal service port
As now the ingress exposes the service to the external network, it's not required to set the configuration type: LoadBalancer for welcome-service, as done on the previous chapter, which changes its behavior to the default ClusterIP
Save it to a file, apply the manifest with kubectl, and in the case everything is correct (including the network instructions from the first article), the test page should be accessible via domain name OR you get something like this:
Which is just the browser being picky with non-secure access, using http:// instead of https:// or not having valid certificates (more on that later). If you want to check the contents of the page, just be a pro for now and use curl:
$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/<desired version>/cert-manager.yaml
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-cluster-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
(Quite a few pieces, huh?)
Wait for the manager pods to complete their initialization:
The manifest not only creates many standard K8s resources but also defines new custom ones, like the ClusterIssuer we have to manually add now for each environment (only one in our case):
apiVersion: cert-manager.io/v1 # API service created by cert-manager
kind: ClusterIssuer # Custom component type
metadata:
name: letsencrypt
namespace: cert-manager
spec:
acme:
# The ACME server URL
server: https://acme-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: <your e-mail here>
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: traefik # Ingress controller type
(As always, save the .yaml and apply. A ClusterIssuer is not namespace-scoped and can be used by Certificate resources in any namespace)
Here we're using the ACME challenge with an HTTP01 solver to get a valid result from the Let's Encrypt certificate authority, and I honestly can't explain to you properly what all that means. However, one thing that I know is that this particular solver requires the ability to open port 80 to receive requests. It's common for system ports (below 1024) to be restricted by your ISP, and if that's your case a DNS01 solver might be useful. Please consult the documentation for more
Finally, we can edit our ingress manifest to define the domains requiring a certificate:
apiVersion: networking.k8s.io/v1
kind: Ingress # Component type
metadata:
name: proxy # Component name
namespace: test # You may add the default namespace for components as a paramenter
annotations:
cert-manager.io/cluster-issuer: letsencrypt # Reference the utilized ClusterIssuer name
kubernetes.io/ingress.class: traefik # Ingress controller type (yes, again)
status:
loadBalancer: {}
spec:
ingressClassName: traefik # Ingress controller type
tls: # Certificate options
- hosts: # List of valid hosts
- choppaserver.dynv6.net
secretName: certificate # Secret component that will store the certificates
rules: # Routing rules
- host: choppaserver.dynv6.net # Expected domain name of request, including subdomain
http: # For HTTP or HTTPS requests
paths: # Behavior for different base paths
- path: / # For all request paths
pathType: Prefix
backend:
service:
name: welcome-service # Redirect to this service
port:
number: 8080 # Redirect to this internal service port
After a while, a secret with the name defined in secretName should appear in the namespace you're using:
And the browser should stop complaining about the lack of security in your Web page:
Congrats! Now you the actual world can hear you say “hello”. For more information on setting up cert-manager, see this guide (or this one for Ingress-Nginx)
I confess that not adding some application deployment to our first article left me a bit frustrated, but I was unsure about it's length, even more so for an author whose reputation alone wouldn't [yet] make people pay more attention that what's natural. But let's fix that right away and get something running.
Hello, World!
As we're talking about Internet servers, what better than a Web page? I've found a very fitting test container in jdkelley's simple-http-server, but there was an issue: no image available for the ARM architecture. That's the only reason why I had to build one and publish it to my own repository, I swear
If those concepts related to containers are still confusing to you, I recommend following Nana Janashia's tutorials and crash courses. She's an amazing teacher and I wouldn't be able to quickly summarize all the knowledge she provides. Her Docker lectures compilation is linked below:
With that out of the way (thanks, Nana), let's use our test image in our first Kubernetesdeployment, where containers are encapsulated as pods. From now on I'll be abusing comments inside the YAML-format manifest files as it's an easier and more compact way to describe each parameter:
apiVersion: v1 # Specification version
kind: ConfigMap # Static/immutable data/information component
metadata: # Identification attributes
name: welcome-config # Component name
data: # Contents list
welcome.txt: | # Data item name and contents
Hello, world!
---
apiVersion: apps/v1
kind: Deployment # Immutable application management component
metadata:
name: welcome-deploy # Component name
labels: # Markers used for identification by other components
app: welcome
spec:
replicas: 2 # Number of application pod instances
selector:
matchLabels: # Markers searched for in other components
app: welcome
template: # Settings for every pod that is part of this deployment
metadata:
labels:
app: welcome
spec:
containers: # List of deployed container in each pod replica
- name: welcome # Container name
image: ancapepe/http-server:python-latest # Image used for this container
ports: # List of exposed network ports for this container
- containerPort: 8000 # Port number
name: welcome-http # Optional port label for reference
volumeMounts: # Data storages accessible to this container
- name: welcome-volume # Storage name
mountPath: /serve/welcome.txt # Storage path inside this container
subPath: welcome.txt # Storage path inside the original volume
volumes: # List of volumes available for mounting
- name: welcome-volume # Volume name
configMap: # Use a ConfigMap component as data source
name: welcome-config # ConfigMap reference name
---
apiVersion: v1
kind: Service # Internal network access component
metadata:
name: welcome-service # Component name
spec:
type: LoadBalancer # Expose service to non-K8s processes
selector: # Bind to deployments with those labels
app: welcome
ports: # List of exposed ports
- protocol: TCP # Use TCP Internet protocol
port: 8080 # Listen on port 8080
targetPort: welcome-http # Redirect trafic to this container port
(Copy it to a text editor and save it to something like welcome.yaml)
Among other things, Kubernetes are a solution for horizontal scaling: if your application process is bogged down by many requests, you may instantiate extra copies of it (replicas) in order to server a larger demand, even dynamically, but here we'll work with a prefixed amount. Moreover, see how services not only centralize access to all pods of a deployment, but allow us to use a different connection port than the one defined in the Docker image
A concept you should have in mind when working with K8s is idempotence: manifests such as above don't describe a sequence of operations to set up your pods and auxiliary components, but the desired final state, from which the operations (to either create or restore the deployment) are automatically defined. That way, trying to re-apply an already [successfully] applied configuration will result in no changes to the cluster
The time has come to get our hands dirty. Surely you may submit your YAML files by copying each new version to one of your nodes and invoking k3s kubectl via SSH, but how about doing that from the comfort of your desktop, where you have originally edited and saved the manifests?
It is possible to install kubectl independently of the K8s cluster (follow the instructions for each platform), but it won't work out-of-the-box, as your local client doesn't know how to find the remote cluster yet:
$ kubectl get nodes
E0916 18:45:31.872817 11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.873268 11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.875167 11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.875837 11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0916 18:45:31.877524 11709 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
In order to get that information, log into one of your cluster machines and get the system configuration:
(Here authentication data is omitted for safety reasons. in order to show the complete configuration, add the --raw option to the end of the command)
Copy the raw output of the command, change the server IP (just the 127.0.0.1) to match your master node's one, and overwrite your desktop's kubeconfig file (on Linux, the default location is <user home directory>/.kube/confg) with these modified contents. Now you should be able to use the control client successfully:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
odroidn2-master Ready control-plane,master 9d v1.30.3+k3s1
rpi4-agent Ready <none> 9d v1.30.3+k3s1
Finally, apply our test deployment:
$ kubectl apply -f welcome.yaml
configmap/welcome-config created
deployment.apps/welcome-deploy created
service/welcome-service created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
welcome-deploy-5bd4cb78b-hvj9z 1/1 Running 0 75s
welcome-deploy-5bd4cb78b-xb99p 1/1 Running 0 75s
If you regret your decision (already?), that operation may be reverted using the same manifest:
$ kubectl delete -f welcome.yaml
configmap "welcome-config" deleted
deployment.apps "welcome-deploy" deleted
service "welcome-service" deleted
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
welcome-deploy-5bd4cb78b-hvj9z 1/1 Terminating 0 114s
welcome-deploy-5bd4cb78b-xb99p 1/1 Terminating 0 114s
# A little while later
$ kubectl get pods
No resources found in default namespace.
As you can see, pods of a given deployment get a random suffix attached to their names, in order to differentiate them. Also, all Kubernetes are organized in namespaces to help with resource management. If not informed as a kubectl command option or inside the file itself, the default namespace is used
In order to perform the same operation with a particular namespace, create it first (if not created already):
$ kubectl create namespace test
namespace/test created
$ kubectl apply -f welcome.yaml --namespace=test
configmap/welcome-config created
deployment.apps/welcome-deploy created
service/welcome-service created
$ kubectl get all --namespace=test
NAME READY STATUS RESTARTS AGE
pod/welcome-deploy-5bd4cb78b-5df7d 1/1 Running 0 16s
pod/welcome-deploy-5bd4cb78b-9cpdj 1/1 Running 0 16s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/welcome-service LoadBalancer 10.43.194.209 192.168.3.10,192.168.3.11 8080:31238/TCP 16s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/welcome-deploy 2/2 2 2 16s
NAME DESIRED CURRENT READY AGE
replicaset.apps/welcome-deploy-5bd4cb78b 2 2 2 16s
(See how namespaces allow to easily select all components that we wish to monitor)
With a successful deployment, our Web page should be displayed by accessing one of the external IPs from the LoadBalancer service (which are the node IPs) on any browser (generally you have to make the usage of HTTP over HTTPS explicitly):
(Opening the listed file works too!)
Congrats! Go add it to your list of LinkedIn's skills (just kidding)
Again, if you're interested in understanding all those concepts in more depth, in order to follow along this and the next chapters of our guide, Nana comes to the rescue again with her Kubernetes tutorial:
A nicer view
That's all well and good, but I bet you can imagine how, as you're developing and running more and more components, issuing commands and keeping track of stuff using kubectl gets quite repetitive. One may open multiple terminals running the get command with --watch as option for automatic updates, but switching between windows is not that great either
Thankfully, for whoever likes that approach, there are friendlier GUIs to help manage yours clusters and deployments. Rancher, developed by the same folks from K3s, is one of them, but honestly I've found too complicated to set up. In my previous job I came across an alternative that I consider much more practical: Lens
After installing it, if you're interested, there's not much to do to get it running for your cluster. Select the option to add a new one from a kubeconfig and paste the contents from your local kubectl configuration:
If everything is correct, you'll be able to access it, visualize your resources, edit manifest files, and even use the integrate terminal to create new components with kubectl (There doesn't seem to be a widget for creation, only modification or deletion of existing components):
(I don't know about you, but that's what I'll be using from now on)
That's it for now! Thanks for reading and let's start making some real-world stuff next time
Inspired by the recent Twitter/X situation here in Brazil (no strong opinions on Elon Musk and all this drama, but it surely sucks to lose access to an useful service) and by an important article by Ceadda of Mercia, I've decided to set an old project of mine in motion.
Truth is, we grow so used to many readily accessible (even if not completely free) platforms on the Internet that we end up taking this ability to communicate and inform ourselves for granted. We easily forget how liberty has to be conquered every day, and that has to be backed by power i.e. the capacity to do what we want to do
Anyway, there are plenty of speeches and commentary around that would do a much better job than me in convincing you of that reality, not to mention the risks of providing all your data directly to third-parties. So here I intend to concentrate where I can add something meaningful: give you some ideas and guidance on how you may own your Internet life a bit more
Living in reality is knowing where things come from and where they go to
Social media, messaging, videos, photos, e-mail, blogs like this one and every other thing that we access on the Internet are all hosted out there, on The Cloud. However, as the saying goes, ”'The Cloud' is just someone else's computer”. And why couldn't it be YOUR computer? Well, it could, if you're willing to make some sacrifices
You won't be able to run e.g. a personal Facebook or Instagram server from home, but if you care more about the functionality they provide, well, you may be able to actually own something close to it. Of course your friends would have to come over to your place, and many probably won't. That's part of what I meant by “make some sacrifices”: your personalized Web presence will be a bit lonely until it gets some traction
But, let's be real, if you're looking into building homeservers having almost no friends is probably the case already... I say it from experience
Ok, ok, maybe you DO have quite some friends [on the Internet] and don't want to give up interacting with them. You'd still find usefulness in having your own “cloud” storage/backup, or a less censorable place to publish your ideas
I'll try to show a bit of everything. Pick whatever suits you
The meat and potatoes of the issue
I don't plan on being condescending to the point of assuming you don't know the distinction between hardware as software. You might have already figured out that those personal software services would need dedicated hardware to be run, and it would be a good thing to have a relatively low-cost solution, so that affording a brand new Intel x86 PC (calm down, Apple M1 users) to keep turned on 24/7 is not a requirement
A common approach that I have adopted is resorting to ARM-based single board computers. They are cheaper, consume less power and take less space inside my modest house. Below I display my humble initial setup of an Hardkernel Odroid N2+ and a well-know Raspberry Pi 4:
(Size doesn't matter if you know how to use it, or so they say)
I have also attached one spare hard drive to each one of them using external USB 3.0 to SATA adapter cases, so that we have larger and more reliable storage. Besides, as you might have noticed, there is no screen, keyboard or mouse there, because I'll be managing everything software-wise from my desktop, over the local network (LAN)
Please make sure you have the correct/recommended power supply connected to each board, specially when using external devices powered through USB. I've just had some undervoltage issues with the Raspberri Pi for not using the ideal one
Both machines are running the minimal ARM version of Manjaro Linux. As a long-time user of Manjaro on my PC, I found it easier to let manjaro-arm-installer choose the right configuration for each board, download the image and write it to its respective SD card. There are plenty of options for operating systems, with it's own installation instructions, but generally you'll be taking and .img file and flashing it to the card with some tool like Etcher or dd. Take a look at what you like the most (preferably some Linux CLI version) or just use the same I did, you'll have to consult extra documentation regardless, as I can't give you all the details without making this article too long for my taste
Notes on networking
Needless to say, your server machines need Internet connection, preferably Ethernet, so that it can be detected without logging into each one locally and configuring Wi-Fi. Another [not so obvious] requirement is that at least one of them (your master) has to be exposed outside of your LAN, so that external devices can reach it (and you can imagine why that's not possible by default). If you have access to your router configuration (I DO hope you have), usually your NAT or Firewall settings, that's achieved by forwarding the ports used by the services you got running e.g. port 80/TCP for HTTP and 443/TCP for HTTPS (Web pages):
(If you're using standard IPv4, the machine's host address is valid exclusively inside your LAN, and only the router/gateway has also a global WAN IP. Even with all the IPv6 goodies enabled, reachability from outside might be limited for security reasons)
I'd also recommend reserving static IPs for each one of your server's network interfaces, so that we don't get any unpleasant surprises with changing addresses:
After redirecting incoming traffic from router WAN address to server machine LAN address on the given ports, it's time to make yourself easier to find. Passing your WAN IP around so that people may type it into the browser bar would already work, but that's both hard to remember and unreliable, as addresses attributed to your machine usually change from time to time (lucky you if a static global IP is available).
Then why not give your virtual place a proper memorable name, like “mybeautifulsite.com”, that everybody could know? Well, that what name service (DNS) is for.
Sadly domains are one aspect of Internet centralization that still cannot be trivially avoided (I know about blockchain DNS, but who supports it?), so you have to rely on a third-party service (e.g. Cloudflare) in order to propagate your hostname across the Web. Generally you have to pay a registrar for the right to use a certain name (it's a way to prevent the ambiguity from 2 hosts using the same one), but for testing purposes you may rely on free DNS services (such as No-IP or Dynv6) that provide you a somewhat uglier subdomain alternative:
(They were so kind to tell the entire world my router's IP)
If you don't manage to get your hands on a static WAN IP, it's important that your DNS service has support for being automatically updated about changes in your dynamic address. Your router itself might be able to notify the DDNS, but if the proper protocol option is not available you may run a refresh client from one of your computers:
(Again, there's too much variation across routers and services for me to cover it here)
Just don't go trying to reach your server via DNS right after naming it. Propagation takes a while, as many name servers across the world have to be informed. If getting anxious, you may try to check the state of that process using some Web service or nslookup command on your Linux system (also available under the same name on Windows, apparently, but who in his/her right mind still uses it?)
Becoming the helmsman
If you've ever tried using a non-mainstream Linux distro, you probably know the pain of looking for a package, not finding it even in a third-party repository, trying to compile it from source and failing miserably over and over (Been there. Done that). Manjaro, as an Arch-based OS, at least has the very welcome access to the AUR, a community repository for compilation scripts that automate the process for us mere humans, but not all are regularly maintained and many still break, for a plethora of reasons that you eventually have to figure out and fix in a case-by-case basis
A way to avoid those headaches would be welcome
For a long time, I've been the kind of person that sees Linux's Dependency Hell as a feature rather than a bug: “It's efficient for storage”, “It forces developers to keep their software updated and in sync with others”, “Library duplication is r*tarded”, I've been telling myself. I still find it somewhat true, and don't see sandboxed solutions like Flatpak, Snap and AppImage with good eyes, or even virtual environments for more than prototyping... Ok, maybe I'm still THAT kind of person, but I've found containers to be pretty neat
Having worked a bit with Docker in a previous job, I was convinced of how useful this level of isolation is for automated testing and deployment of applications, being able to change system settings and replicate them across countless [physical or virtual] machines. You really start visualizing containers as loosely coupled independent entities exchanging messages, which in turn naturally evolves into the main focus of my approach here: orchestration
And to cut it short, I'd guess in 9 of 10 cases that word would mean using one of the many implementations of Kubernetes (K8s for short). Not that there aren't other solutions, but I'm not knowledgeable enough to evaluate them properly and I went for what is more popular i.e. with more available documentation. Let me make it clear that this guide is a mish-mash of a number of tutorials and Stack Overflow threads, compiled, updated and adjusted by an amateur who is really stubborn about making all work together. So please bear in mind it might not be the most elegant solution
For tiny single-board computers, there is a lightweight implementation of K8s called K3s, which has worked well for me. If you're using Arch-based systems (assuming that you have actually gotten the machines set up), keep following my steps for K3s installation. On Debian-based systems like Ubuntu or Raspbian, you'd be probably better off following NetworkChuck's guide linked below. Otherwise, you'll have to do your own research:
As already mentioned, here I have accessed my servers from a desktop, via SSH, and installed K3s using yay, but you may use any other package manager with AUR access like pikaur or pamac:
$ ssh ancapepe@192.168.3.10
ancapepe@192.168.3.10's password:
Welcome to Manjaro ARM
~~Website: https://manjaro.org
~~Forum: https://forum.manjaro.org/c/arm
~~Matrix: #manjaro-arm-public:matrix.org
Last login: Wed Sep 11 18:41:48 2024 from 192.168.3.8
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S yay
[sudo] password for ancapepe:
resolving dependencies...
looking for conflicting packages...
Packages (1) yay-12.3.1-1
Total Download Size: 3.31 MiB
Total Installed Size: 8.88 MiB
:: Proceed with installation? [Y/n] y
:: Retrieving packages...
yay-12.3.1-1-aarch64 3.3 MiB 2.33 MiB/s 00:01 [#############################################################################] 100%
(1/1) checking keys in keyring [#############################################################################] 100%
(1/1) checking package integrity [#############################################################################] 100%
(1/1) loading package files [#############################################################################] 100%
(1/1) checking for file conflicts [#############################################################################] 100%
(1/1) checking available disk space [#############################################################################] 100%
:: Processing package changes...
(1/1) installing yay [#############################################################################] 100%
Optional dependencies for yay
sudo: privilege elevation [installed]
doas: privilege elevation
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[ancapepe@ChoppaServer-1 ~]$ sudo pacman -S containerd fakeroot
...
(More of the same)
...
[ancapepe@ChoppaServer-1 ~]$ yay -S k3s-bin
AUR Explicit (1): k3s-bin-1.30.3+k3s1-1
...
(Too much to show here. Just follow along)
...
(Do that for each board. Teaching how to use a terminal is outside the scope of this guide)
Hopefully everything went well and you have K3s installed on all your servers, but it's still not doing anything. Kubernetes applies the concepts of master and worker nodes: masters coordinate workers so that all can work as a single system, namely a Cluster; both may dot the heavy lifting of running applications; Node is simply the abstraction used for each [physical or virtual] computer
[AFAIK] With K3s things are kept simple and only your first node will run as a master, or [K3s] server in this case (yes, terminology gets confusing). In order to initialize it as such when your main machine boots, open your K3sinit file for modification with:
(The file will be opened with the application defined in your $EDITORenvironment variable, be it vim, emacs or even nano, a competition I don't want to be dragged into. Talking about undesired fights, I know systemd is not the only init system out there, but I haven't tried this with distros that use OpenRC or Upstart)
Edit the line starting with ExecStart to look like this:
ExecStart=/usr/bin/k3s server --write-kubeconfig-mode=644 --node-name <your chosen master node name>
And finally enable the initialization service with:
Not quite finally, actually. Before you restart your board to bring up K3s, you have to make sure that the cgroups feature, required by all containerized applications, is enabled and available in your system:
(TL;DR: Cgroups system v2 running, memory cgroup disabled)
If v1 system or none at all appears, you're probably using an old kernel, so update it. If the memory cgroup is not enabled, you have to find the particular way in which you switch it on for your device (for Raspberry Pi, that means adding cgroup_enable=memory at the end of the single line in the /boot/cmdline.txt file)
Moreover, we must tell containerd, the daemon that manages your containers behind the curtains, to use the right cgroup for its main process, runc, by running the commands below:
# Create a containerd configuration directory
[ancapepe@ChoppaServer-1 ~]$ sudo mkdir -p /etc/containerd
# Write the default settings to a configuration file
[ancapepe@ChoppaServer-1 ~]$ sudo containerd config default | sudo tee /etc/containerd/config.toml
# Find and replace the given string inside the file
[ancapepe@ChoppaServer-1 ~]$ sudo sed -i 's/ SystemdCgroup = false/ SystemdCgroup = true/' /etc/containerd/config.toml
(The # character at the beginning of a line denotes a comment, which is not supposed to be typed. For more information on what you're doing there consult this article and the sed command documentation)
Now you may reboot the master node, and hopefully everything will start running properly. Your SSH connection will be terminated so it should be established again when the device finishes starting up:
[leonardo@ChoppaServer-1 ~]$ sudo reboot
[sudo] password for leonardo:
Broadcast message from root@ChoppaServer-1 on pts/1 (Sat 2024-09-14 20:31:07 -03):
The system will reboot now!
[leonardo@ChoppaServer-1 ~]$ Read from remote host 192.168.3.10: Connection reset by peer
Connection to 192.168.3.10 closed.
client_loop: send disconnect: Broken pipe
$ ssh leonardo@192.168.3.10
leonardo@192.168.3.10's password:
Welcome to Manjaro ARM
~~Website: https://manjaro.org
~~Forum: https://forum.manjaro.org/c/arm
~~Matrix: #manjaro-arm-public:matrix.org
Last login: Sat Sep 14 17:30:32 2024 from 192.168.3.8
[leonardo@ChoppaServer-1 ~]$
(At this point you may check the status of the enabled services using systemctl status)
With the master system set up, let's try connecting our worker nodes to it. That connection is established by sharing a common key or token generated by master and stored in a predetermined file:
# Displays content of the token file
[leonardo@ChoppaServer-1 ~]$ sudo cat /var/lib/rancher/k3s/server/token
[sudo] password for leonardo:
K10d4e8b232cbf03832752443a07cc6206e092733c8ae55c0e64407dcb2e1775f45::server:44a1a8b25c4a8f1d8a0d48164eff2303
Copy that file, guard it well, and repeat the same process (since the first ssh command) for each one of your workers, only changing the way the K3s service file is edited, as now it should contain:
Take your time in repeating all that stuff, just make sure that you follow through. If everything works as intended, from any one of your nodes you'd be able to run the command below successfully:
[leonardo@ChoppaServer-2 ~]$ k3s kubectl get nodes
NAME STATUS ROLES AGE VERSION
odroidn2-master Ready control-plane,master 7d5h v1.30.3+k3s1
rpi4-agent Ready <none> 8d v1.30.3+k3s1
(Yay!)
That's it for now. I hope it wasn't very tiring
Next time we'll actually deploy applications and get a proper way to manage them