====== personal infrastructure ======
//tldr - docker compose, isolated servers//
Here I describe how my personal computing infrastructure is set up. These aren't rules or a framework or anything - just a description of how things have ended up.
====== Security ======
Seems like an odd place to start, but security considerations have dictated the biggest architectural decisions. My view of security contextualizes the rest of the setup.
I have some basic assumptions about software security, main ones relevant to my infra include:
===== Every application is vulnerable =====
Every time you stand up a service that can be reached from the public internet you're adding a huge amount of area to your attack surface. Even a single modern web application relies on so much software that it's reasonable to assume there's a security flaw in there somewhere. Stand up 10 of those and your surface area is quite large. For that reason I assume everything I run is vulnerable and can be hacked.
===== Docker based isolation isn't reliable =====
I use docker for everything, mostly for its ability to isolate software and the filesystem from the host. That said, I assume that the container sandbox can be broken out of. I choose to make the assumption that isolation begins at the VM level. VM sandbox escapes aren't in my threat model.
===== Can't break what you can't touch =====
//Or, network isolation is a strong security measure.//
All my compute infrastructure is organized into two separate spheres, public and private. The idea is that if my public infrastructure gets completely owned and everything in that sphere is exfiltrated or wiped, that would be annoying but would not be a disaster. I also make an effort to ensure that the public sphere is properly isolated, meaning you can't get to the private sphere from the public sphere.
Whenever I'm setting up a new piece of infrastructure, the first thing I think about is what sphere it goes in.
flowchart TD
pub((public))
priv((private))
data{stores sensitive data?}
onlyme{used only by me?}
data --> |yes| priv
data --> |no| onlyme
onlyme --> |yes| priv
onlyme --> |no| pub
There are other concerns but that's the general thought process.
====== Machines ======
===== Public =====
For the public sphere, I use cloud-based Linux VMs from one of the affordable providers. I run most of my stuff on a single shared CPU VM with 4 CPU cores, 4gb of RAM and 50gb of disk space (storage is a later section).
For things that need to be exposed in the internet I think cloud is the best choice. From a network isolation perspective serving things from your home means untrusted traffic will be flowing within your home network. Apart from security concerns there are other problems:
* Serving things on a residential connection is sometimes outside of the TOS of your service provider
* If your connection is asymmetric performance can be low, especially when serving large files
* Bandwidth always inferior to cloud
* Dynamic IP addresses makes access annoying
* Advertising your home IP address is a mild security concern
* can be mitigated if you rent an ultra cheap VPS and use it as a gateway; I did this initially but realized I can run everything on a cheap vps to begin with
* Weather / residential power outages impact uptime
The main advantages of hosting at home are that it's cheaper over the long run, you aren't subject to cloud provider TOS and pricing changes, and are unaffected by cloud outages. The overall balance is in cloud's favor for services that need to be served to the public internet.
===== Private =====
For my private sphere, I have a home server. This a normal mATX desktop computer although the board is a server board and supports ECC memory, which I use.
Build details:
* Case: Fractal Node 804
* Motherboard: [[https://www.asrockrack.com/general/productdetail.asp?Model=X570D4U#Specifications|Asrock RACK X570D4U]]
* RAM: 128gb ECC memory (4x [[https://www.kingston.com/dataSheets/KSM32ED8_32ME.pdf|KSM32ED8/32ME]])
* Storage:
* 4x WD Blue 3.5" 4tb HDD
* 4x Hitachi 3.5" 4tb HDD
* 1tb WD Black NVME
* CPU: AMD Ryzen 7 3800X
* GPU: RTX 3070
* Cooling: That noctua air cooler everyone likes
Comments:
* The system is headless, the graphics card is for GPGPU compute
* Very pleased with the Asrock RACK board; good IPMI and the section display for status codes is extremely useful
* Fractal Node build quality lives up to its rep, it's a great server case
{{:homeserver.jpg?400|photo of home server}}
8oz Red Bull for scale.
===== OS =====
On my home server, I run Proxmox. I don't have many VMs on it - in fact, like the public sphere, most of my services run on a single VM. Some of the reasons I use a hypervisor:
* Much better control and monitoring of hardware resources
* Full suite of hardware monitoring tools built into a clean web interface; disk diagnostics, memory / CPU / network dashboards, etc.
* Ability to easily partition storage resources among VMs
* Ability to set fine grained memory and CPU resource limits
* Full snapshots for tenant VMs
* Ability to snapshot the entire machine (VM) at the "hardware" level before making major changes is very valuable; makes risky operations like OS upgrades risk free
* Ability to create new machines whenever I want to try something
* E.g. I have a Windows VM that runs some Windows-only server software
* I can spin up a VM with resources of my choosing to try out an operating system or run some experiment without worrying about damaging any infrastructure I rely on
Of course, you don't really need a dedicated hypervisor to do any of this; you can do it all with KVM on a traditional bare metal server OS. But it's much easier and more convenient in a hypervisor and in practice, there's very few downsides to the hypervisor.
The main downside is that hardware passthrough can be tricky. I pass through the RTX 3070 to the VM, and then into docker containers, in order to get accelerated encoding for media related services. KVM GPU passthrough is annoying.
The OS that applications run on (ignoring Docker) is always a recent Ubuntu Server LTS with automatic security upgrades enabled. I find the Debian-based platform familiar, comfortable and stable. Ubuntu Server also has a very wide install base which means most problems can be resolved with a web search. This helps keep the maintenance overhead as low as possible. I enjoy building my infrastructure, but I do rely on it and dislike being forced to fix it, so choices in system software tend to be on the less exotic side.
====== Storage ======
For public infrastructure I use cloud storage as detailed in the next section.
For private infrastructure I have an 8x4tb HDD array.
{{:homeserver-drives.jpg?400|photo of home server drive racks}}
These drives form a ZFS pool-of-mirrors with two hot swappable spares:
pool: tank
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x50014ee266927a7d ONLINE 0 0 0
wwn-0x5000cca22bca7cbb ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
wwn-0x50014ee2113d2fa4 ONLINE 0 0 0
wwn-0x5000cca22be46d4b ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
wwn-0x50014ee2bbe84236 ONLINE 0 0 0
wwn-0x5000cca23de171c4 ONLINE 0 0 0
spares
wwn-0x50014ee2bf43a534 AVAIL
wwn-0x50014ee2bbe843cf AVAIL
Total usable storage space is 12tb. One disk in each mirror pair can fail with no data loss.
I carve out virtual disks from this pool and attach them to VMs as needed.
====== Applications ======
Over time I've gravitated towards
* web applications
* running in Docker containers
* served by nginx reverse proxy
flowchart TD
subgraph VM
nginx(nginx)
subgraph Docker
direction LR
app1(piwigo)
app2(akkoma)
app3(peertube)
appel(. . .)
end
end
inet((network)) --> nginx --> app1 & app2 & app3 & appel
Most of the self-hostable software I use is web based. Containers offer a way to run applications without contaminating the host, with (usually) easy upgrades and good lifecycle management and without reliance on the OS packaging system, which makes them platform agnostic. Serving everything via nginx reverse proxy simplifies web server configuration and simplifies TLS. nginx has great performance and security and since nginx reverse proxy to docker is a popular choice now, many applications now provide example nginx reverse proxy configurations, making configuration even easier.
===== Docker Compose =====
Every single application I run is a docker compose project. Docker compose makes it very easy to manage complicated multipart services as a unit. Many of the applications I run are complex web applications with an application component, a separate database (eg mysql or postgres), and sometimes additional components. Putting everything in docker compose means that bringing up the simplest to the most complicated of these applications is done the exact same way:
docker compose up
All dependencies, environment variables, mounts, port bindings, application settings etc. are all completely specified either in the ''docker-compose.yml'' file, the ''.env'' file, and the persistent data files on disk, and all of these files are contained in a single directory.
===== Disk layout =====
On my machines, I have a directory called ''apps'' in my home directory. Within this ''apps'' directory, there is a directory for each application I run, and within that directory there is a ''docker-compose.yml'', ''.env'' and sometimes data files.
The entire state of any application is contained within that application's directory. As long as I have that directory, I can bring the application back up with its previous state even if the host is completely destroyed.
On disk, the situation looks like this:
qlyoung@private ~> tree --prune -P "docker-compose.yml" -L 3 apps
apps
├── calibre-web
│ └── docker-compose.yml
├── gitea
│ └── docker-compose.yml
├── healthchecks
│ └── docker-compose.yml
├── invidious
│ └── docker-compose.yml
├── jellyfin
│ └── docker-compose.yml
├── linkding
│ └── docker-compose.yml
├── miniflux
│ └── docker-compose.yml
├── nextcloud
│ └── docker-compose.yml
├── onlyoffice
│ └── docker-compose.yml
├── photoprism
│ └── docker-compose.yml
├── quassel-core
│ └── docker-compose.yml
├── tandoor
│ └── docker-compose.yml
...
Data files and ''.env'' are omitted here but live alongside each ''docker-compose.yml''.
I set things up so that I only need a given application's directory in order to completely restore its state. Consequently, I don't use docker volumes since these are stored in ''/var/lib/docker''. I only use bind mounts. I think it might be possible to have ''local'' driver volumes be located in a custom directory, but since bind mounts have always worked well for me, I haven't tried it.
====== Application Storage ======
===== Private =====
Storage on my private infrastructure is easy. I have a 32tb ZFS pool in my home server; if my VM runs low on storage space, I just make the disk bigger. If 32tb is not enough, I will buy bigger disks.
===== Public =====
On my public infrastructure, disk space is constrained. My primary VM has 50gb of disk space. I used to have storage volumes provisioned in my cloud provder that were attached to the VM, with data files stored by applications symlinked to a location on the volume. However, while you can get as much storage as you want this way without needing to upgrade the VM instance size, cloud providers know that and thus tend to charge quite a lot for storage.
I use Backblaze B2 object storage, which is affordable and provides an S3 compatible API. When selecting applications to fill some infrastructure need that will need to run on my public infra, I only select applications that either have minimal disk requirements or support the S3 API. Each service has its own B2 bucket and its own access key that grants access only to that bucket.
flowchart TD
subgraph VM
akkoma
piwigo
peertube
appel(. . .)
end
subgraph B2
qlyoung-pleroma(qlyoung-pleroma 4.2gb)
qlyoung-piwigo(qlyoung-piwigo 6.9gb)
qlyoung-peertube(qlyoung-peertube 32gb)
qlyoung-foobar(. . .)
end
akkoma -- S3 API + akkoma key --> qlyoung-pleroma
piwigo -- S3 API + piwigo key --> qlyoung-piwigo
peertube -- S3 API + peertube key --> qlyoung-peertube
appel -- S3 API + ... key --> qlyoung-foobar
If possible, I also configure applications so that users are served files directly from B2 rather than having them proxied through my VM, which improves performance both in terms of load time and bandwidth usage. When you watch a video on my Peertube server, your browser is downloading the video directly from B2, instead of ''B2 → VM → You''.
====== DNS & HTTP & TLS ======
This is identical on both private and public. Grand overview:
flowchart TD
subgraph VM
nginx(nginx 0.0.0.0:443)
subgraph Docker
direction LR
piwigo(piwigo 127.0.0.1:8000)
akkoma(akkoma 127.0.0.1:8001)
peertube(peertube 127.0.0.1:8002)
appel(. . .)
end
end
photos.qlyoung.net & social.qlyoung.net & qtube.qlyoung.net & el(. . .) -- CNAME --> qlyoung.net -- A --> VM
nginx -- photos.qlyoung.net --> piwigo
nginx -- social.qlyoung.net --> akkoma
nginx -- qtube.qlyoung.net --> peertube
nginx -- ... --> appel
===== DNS =====
Each service is on its own subdomain. All subdomains ''CNAME'' to the main domain name (''qlyoung.net'') which has an ''A'' pointing to the public IP of the VM. In my private infrastructure, the ''A'' record instead points to a Tailscale IP - but more on that in the networking section.
===== HTTP =====
nginx runs on the host and binds host ports 80 and 443. All docker containers bind to ''(127.0.0.1, P)'' where ''P'' is a host port number of my choosing. Each service has its own subdomain and a corresponding nginx configuration:
root@public /e/n/sites-available# ls | sort
birdcam.conf
maloja.conf
piwigo.conf
pleroma.conf
qlyoung.conf
qtube.conf
wiki.conf
...
Each of these looks more or less the same, and contains configuration to reverse proxy to the corresponding Docker container. For example, here is ''piwigo.conf''. It reverse proxies requests to ''photos.qlyoung.net'' to ''127.0.0.1:8090'', which is where the container running Piwigo is bound:
server {
server_name photos.qlyoung.net;
location / {
proxy_pass http://127.0.0.1:8090;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-For $remote_addr;
}
listen [::]:443 ssl ipv6only=on;
listen 443 ssl;
<... tls details, misc settings omitted ...>
}
Note that Piwigo itself is bound to port 80 within the container, which is mapped to 8090 on the host:
qlyoung@public ~/a/piwigo> docker-compose ps
Name Command State Ports
------------------------------------------------------------------------------------------
piwigo_mysql_1 docker-entrypoint.sh --def ... Up 3306/tcp, 33060/tcp
piwigo_piwigo_1 /init Up 443/tcp, 127.0.0.1:8090->80/tcp
I find this setup to be clean and convenient. Service specific settings such as request body size are set in the nginx configuration and only need an nginx reload to be applied. Sites can be made accessible or not by enabling or disabling their nginx configurations without touching the application itself.
===== TLS =====
TLS is terminated by nginx. This makes TLS management very simple, since certificates can be issued and configured in nginx using Certbot with the nginx plugin. No container specific configuration is necessary. I think this is the correct separation of concerns - TLS configuration is a detail relevant to how services are exposed, but not to the services themselves. Having TLS configuration live in the web server with services unaware of it makes sense.
Certbot issues certificates using Let's Encrypt. Certbot automatically renews certificates so there is no maintenance overhead.
====== Networking ======
===== Public =====
Everything is served on public IP addresses. DNS is managed through my DNS provider (Namecheap).
===== Private =====
My home server and all my personal computers are part of a single Tailscale network (tailnet). The network topology is full mesh.
flowchart LR
subgraph Tailnet
laptop
desktop
server
pihole(odroid )
el(. . .)
laptop --- desktop & server & pihole & el
desktop --- server & pihole & el
server --- pihole & el
pihole --- el
end
For DNS, I have a small SBC [[https://www.hardkernel.com/shop/odroid-xu4-special-price/|odroid-xu4]] running Pihole which is also part of the tailnet (''odroid'' in the diagram above). The tailscale daemon on each device configures the device to use that DNS server when resolving ''*.qlyoung.net'' domains. Otherwise DNS is the same as previously described, except that the ''A'' record for the primary VM points to a Tailscale address.
flowchart TD
subgraph VM
tip(Tailscale IP) === nginx(nginx)
end
git.qlyoung.net & healthchecks.qlyoung.net & jf.qlyoung.net & links.qlyoung.net & el(. . .) -- CNAME --> private.qlyoung.net -- A --> tip
On the VM nginx is configured to only bind to the Tailscale address. This ensures applications are only accessible by devices explicitly joined to the tailnet, and not by e.g. a guest who has joined my home LAN.
==== Typical Deployment ====
1. Identify a need, e.g., "I want a personal recipe book and meal planner"
2. Search for Web-based projects that fill that need, select one based on presence of Docker support, features & community activity, e.g. [[https://tandoor.dev/|Tandoor]]
3. Log into server, clone, configure, and start application (pseudocode)
$ ssh server
# set up directories
$ mkdir -p apps/tandoor && cd apps/tandoor
$ wget docker-compose.yml .env
# make any changes; set port bindings, env variables to set credentials, etc
$ vim docker-compose.yml .env
# start service
$ docker compose up -d
4. Set up new ''CNAME'' record; ''CNAME recipes.qlyoung.net -> qlyoung.net''
5. Configure nginx, request and install TLS certificate
# create nginx configuration, usually by copying another one and tweaking it
$ vim /etc/nginx/sites-available/tandoor.conf
# enable nginx with chosen
$ ln -s /etc/nginx/sites-available/tandoor.conf /etc/nginx/sites-enabled/tandoor.conf
# Log into namecheap, create new CNAME record (recipes.qlyoung.net)
# Set up TLS
$ certbot --nginx -d recipes.qlyoung.net
6. ???
7. Profit
The deployment process is identical for both internal and external services.
====== Backups ======
Everything, private and public, is backed up with [[https://restic.net/|restic]] to offsite locations. It runs daily on a ''cron'' job.
====== Cost ======
Private runs for the cost of electricity.
Public bill:
* Compute: $24/mo
* B2 Storage (~200gb): ~$2/mo
* Total: ~$26/mo
{{tag>from_blog technology}}