Hacking:Servers

Note: This page documents how the Parabola project's servers are configured. This is not about how to configure servers.

1 DNS

1.1 Subdomains

Note:

2021-09 - the Newtron server is planned to replace the previous web server (Hacking:Servers/Proton), but is not yet in full operation - this 'Subdomains' section is a WIP, documenting the migration of the web services currently/temproarily hosted on Winston, from Winston to Newtron

TODO:

• check which of these are still registered and/or handled by the server
• we should re-evaluate the duplicated/confusing semantics/routing of {bugs,labs,projects,git,pur} and {conference,xmpp}

Newtron Sub-domains	Notes	TODO:
bugs.parabolagnulinux.org	legacy tracker?	?
conference.parabolagnulinux.org	no service (perhaps was alias for 'xmpp.')	reserve for mumble, jitsimeet, or similar - mumble is installed but not running
forum.parabola.nu	web forum	delete forums.parabola.nu?
forum.parabolagnulinux.org
forums.parabola.nu
labs.parabola.nu	issue tracker	?
labs.parabolagnulinux.org
licenses.parabolagnulinux.org	no service	?
lists.parabola.nu	mailman	delete list.parabolagnulinux.org?
lists.parabolagnulinux.org
list.parabolagnulinux.org
packages.parabola.nu	parabolaweb packages	?
packages.parabolagnulinux.org
projects.parabola.nu	git server and cgit (redirects to git.parabola.nu)	split? git.parabola.nu(git)/projects.parabola.nu(cgit) - or replace cgit with pagure and delete projects.parabola.nu, replacing with forge.parabola.nu, code.parabola.nu, or similar
projects.parabolagnulinux.org
pur.parabola.nu	no service	?
pur.parabolagnulinux.org
wiki.parabola.nu	mediawiki	?
wiki.parabolagnulinux.org
www.parabola.nu	parabolaweb	?
www.parabolagnulinux.org
xmpp.parabolagnulinux.org	no service (was prosody)	prosody is installed but not running

Winston Sub-domains	Notes	TODO:
git.parabola.nu	git server and cgit	replace cgit with pagure?
redirector.parabola.nu	redirector	OK
repo.parabola.nu	repo rsync	OK
repo.parabolagnulinux.org
repomirror.parabola.nu	?	?
repomirror.parabolagnulinux.org

1.2 Records

These are the DNS entries with he.net, as of 2016-06-18.

Record	Type	Value
parabola.nu.	NS	ns1.he.net.
parabola.nu.	NS	ns2.he.net.
parabola.nu.	NS	ns3.he.net.
parabola.nu.	NS	ns4.he.net.
parabola.nu.	NS	ns5.he.net.
parabola.nu.	A	80.87.131.232
parabola.nu.	MX	10 repo.parabola.nu.
_xmpp-client._tcp.parabola.nu.	SRV	0 0 5222 xmpp.parabolagnulinux.org.
_xmpp-server._tcp.parabola.nu.	SRV	0 0 5269 xmpp.parabolagnulinux.org.
labs.parabola.nu.	A	80.87.131.232
parabola.nu.	TXT	"v=spf1 ip4:80.87.131.232 -all"
parabola.nu.	SPF	"v=spf1 ip4:80.87.131.232 -all"
labs.parabola.nu.	TXT	"v=spf1 -all"
_xmpp-client._tcp.parabola.nu.	TXT	"v=spf1 -all"
_xmpp-server._tcp.parabola.nu.	TXT	"v=spf1 -all"
labs.parabola.nu.	SPF	"v=spf1 -all"
_xmpp-client._tcp.parabola.nu.	SPF	"v=spf1 -all"
_xmpp-server._tcp.parabola.nu.	SPF	"v=spf1 -all"
repo.parabola.nu.	A	80.87.131.232
wiki.parabola.nu.	A	80.87.131.232
www.parabola.nu.	A	80.87.131.232
lists.parabola.nu.	A	80.87.131.232
projects.parabola.nu.	A	80.87.131.232
repo.parabola.nu.	SPF	"v=spf1 ip4:80.87.131.232 -all"
repo.parabola.nu.	TXT	"v=spf1 ip4:80.87.131.232 -all"
lists.parabola.nu.	TXT	"v=spf1 ip4:80.87.131.232 -all"
lists.parabola.nu.	SPF	"v=spf1 ip4:80.87.131.232 -all"
projects.parabola.nu.	TXT	"v=spf1 -all"
wiki.parabola.nu.	TXT	"v=spf1 -all"
www.parabola.nu.	TXT	"v=spf1 -all"
projects.parabola.nu.	SPF	"v=spf1 -all"
wiki.parabola.nu.	SPF	"v=spf1 -all"
www.parabola.nu.	SPF	"v=spf1 -all"
localhost.parabola.nu.	A	80.87.131.232
localhost.parabola.nu.	TXT	"v=spf1 ip4:80.87.131.232 -all"
localhost.parabola.nu.	SPF	"v=spf1 ip4:80.87.131.232 -all"
pur.parabola.nu.	A	80.87.131.232
pur.parabola.nu.	TXT	"v=spf1 -all"
pur.parabola.nu.	SPF	"v=spf1 -all"
redirector.parabola.nu.	A	80.87.131.232
repomirror.parabola.nu.	A	80.87.131.232
winston.parabola.nu.	A	93.95.226.249
proton.parabola.nu.	A	80.87.131.232
git.parabola.nu.	CNAME	winston.parabola.nu.
parabolagnulinux.org.	NS	ns1.he.net.
parabolagnulinux.org.	NS	ns2.he.net.
parabolagnulinux.org.	NS	ns3.he.net.
parabolagnulinux.org.	NS	ns4.he.net.
parabolagnulinux.org.	NS	ns5.he.net.
parabolagnulinux.org.	A	80.87.131.232
projects.parabolagnulinux.org.	A	80.87.131.232
wiki.parabolagnulinux.org.	A	80.87.131.232
repo.parabolagnulinux.org.	A	80.87.131.232
licenses.parabolagnulinux.org.	A	80.87.131.232
lists.parabolagnulinux.org.	A	80.87.131.232
forum.parabolagnulinux.org.	A	80.87.131.232
parabolagnulinux.org.	MX	0 repo.parabola.nu.
_xmpp-server._tcp.parabolagnulinux.org.	SRV	0 0 5269 xmpp.parabolagnulinux.org.
_jabber._tcp.parabolagnulinux.org.	SRV	0 0 5269 xmpp.parabolagnulinux.org.
_xmpp-client._tcp.parabolagnulinux.org.	SRV	0 0 5222 xmpp.parabolagnulinux.org.
list.parabolagnulinux.org.	MX	0 repo.parabola.nu.
lists.parabolagnulinux.org.	MX	0 repo.parabola.nu.
conference.parabolagnulinux.org.	A	80.87.131.232
list.parabolagnulinux.org.	A	80.87.131.232
xmpp.parabolagnulinux.org.	A	80.87.131.232
parabolagnulinux.org.	TXT	"v=spf1 ip4:80.87.131.232 -all"
lists.parabolagnulinux.org.	TXT	"v=spf1 ip4:80.87.131.232 -all"
repo.parabolagnulinux.org.	TXT	"v=spf1 ip4:80.87.131.232 -all"
bugs.parabolagnulinux.org.	TXT	"v=spf1 -all"
conference.parabolagnulinux.org.	TXT	"v=spf1 -all"
forum.parabolagnulinux.org.	TXT	"v=spf1 -all"
licenses.parabolagnulinux.org.	TXT	"v=spf1 -all"
list.parabolagnulinux.org.	TXT	"v=spf1 -all"
packages.parabolagnulinux.org.	TXT	"v=spf1 -all"
projects.parabolagnulinux.org.	TXT	"v=spf1 -all"
wiki.parabolagnulinux.org.	TXT	"v=spf1 -all"
xmpp.parabolagnulinux.org.	TXT	"v=spf1 -all"
_xmpp-client._tcp.parabolagnulinux.org.	TXT	"v=spf1 -all"
_xmpp-server._tcp.parabolagnulinux.org.	TXT	"v=spf1 -all"
_jabber._tcp.parabolagnulinux.org.	TXT	"v=spf1 -all"
bugs.parabolagnulinux.org.	SPF	"v=spf1 -all"
conference.parabolagnulinux.org.	SPF	"v=spf1 -all"
forum.parabolagnulinux.org.	SPF	"v=spf1 -all"
licenses.parabolagnulinux.org.	SPF	"v=spf1 -all"
list.parabolagnulinux.org.	SPF	"v=spf1 -all"
lists.parabolagnulinux.org.	SPF	"v=spf1 ip4:80.87.131.232 -all"
packages.parabolagnulinux.org.	SPF	"v=spf1 -all"
parabolagnulinux.org.	SPF	"v=spf1 ip4:80.87.131.232 -all"
projects.parabolagnulinux.org.	SPF	"v=spf1 -all"
repo.parabolagnulinux.org.	SPF	"v=spf1 ip4:80.87.131.232 -all"
wiki.parabolagnulinux.org.	SPF	"v=spf1 -all"
xmpp.parabolagnulinux.org.	SPF	"v=spf1 -all"
_xmpp-client._tcp.parabolagnulinux.org.	SPF	"v=spf1 -all"
_xmpp-server._tcp.parabolagnulinux.org.	SPF	"v=spf1 -all"
_jabber._tcp.parabolagnulinux.org.	SPF	"v=spf1 -all"
www.parabolagnulinux.org.	A	80.87.131.232
labs.parabolagnulinux.org.	A	80.87.131.232
pur.parabolagnulinux.org.	A	80.87.131.232
pur.parabolagnulinux.org.	TXT	"v=spf1 -all"
pur.parabolagnulinux.org.	SPF	"v=spf1 -all"
www.parabolagnulinux.org.	TXT	"v=spf1 -all"
labs.parabolagnulinux.org.	TXT	"v=spf1 -all"
labs.parabolagnulinux.org.	SPF	"v=spf1 -all"
www.parabolagnulinux.org.	SPF	"v=spf1 -all"
repomirror.parabolagnulinux.org.	A	80.87.131.232

1.3 SPF

These are mail domains, i.e. ones listed in envelope From addresses or used for outgoing mail servers:

• parabola.nu
• lists.parabola.nu
• repo.parabola.nu

Mail domains have TXT and SPF records with the value "v=spf1 ip4:80.87.131.232 -all", i.e. allowing only mail sent from the parabola.nu server. All other domains have "v=spf1 -all": all mail from them is fake.

We use SPF to get less spam claiming to be from our domains and to discourage spammers from using fake sender addresses.

2 proton.parabola.nu

proton.parabola.nu was a VPS hosted somewhere in the UK, on a machine operated by user n1md4 in cooperation with his employer, Positive Internet.

It is no longer used by Parabola.

2.1 Resources

$ uname -m
x86_64
$ free -h
             total       used       free     shared    buffers     cached
Mem:          1.9G       1.8G       100M       191M        28M       538M
-/+ buffers/cache:       1.2G       667M
Swap:         1.0G       413M       608M
$ df -h | grep sda
/dev/sda1        98G   12G   82G  13% /
/dev/sda3       150G  125G   26G  83% /srv

2.2 Reboot

Once upon a time, reboots were hard to do because of a weird issue that could make shutdown take half an hour! That's a lot of downtime and praying!

Nowadays, it takes about 3 and a half minutes (measured at the Nov 8 1:00 GTM reboot):

shut down	00:01:36
BIOS	00:00:30
boot up	00:01:21
total	00:03:27

Still slower than we would like, but so are most things on Proton.

2.3 Public-facing network sockets/services

Note: IPv4/6 support is just what lsof says. Except that systemd .socket targets are both IPv4 and IPv6, but lsof reports them as just IPv6, even for established IPv4 connections. Weird. I'm pretty sure it's a bug in lsof, so I guess that means that some of the others might be wrong?

socket	IPv4	IPv6	protocol	daemon	other info
TCP:*:22	Yes	Yes	SSH	sshd.service
TCP:*:25	Yes	Yes	SMTP	postfix.service/master
TCP:*:80	Yes	Yes	HTTP	nginx.service
TCP:*:443	Yes	Yes	HTTPS	nginx.service
TCP:*:465	Yes	Yes	SMTPS	postfix.service/master
TCP:*:587	Yes	Yes	SMTP-MSA	postfix.service/master
TCP:*:655	Yes	Yes	tinc	tincd@lvpn.service
TCP:*:875	Yes*	Yes	rsync	rsync.socket
TCP:*:1863	Yes	Yes	SSH	sshd.service
TCP:*:5222	Yes	Yes	xmpp-client	prosody.service
TCP:*:5269	Yes	Yes	xmpp-server	prosody.service
TCP:*:9418	Yes*	Yes	git	git-daemon.socket

2.4 Inward-facing sockets

socket	protocol	unit
TCP4/6:localhost.localdomain:2812	HTTP	monit.service
TCP4:localhost.localdomain:5432	pgsql	postgresql.service
TCP4/6:localhost.localdomain:5582	telnet	prosody.service
unix:/run/mailman-fcgi.sock	FastCGI	mailman-fcgi.socket
unix:/run/uwsgi/labs.sock	uwsgi	uwsgi@labs.socket
unix:/run/uwsgi/parabolaweb.sock	uwsgi	uwsgi@parabolaweb.socket
unix:/run/uwsgi/projects.sock	uwsgi/modifier1=9	uwsgi@projects.socket
unix:/run/uwsgi/repo.sock	uwsgi/modifier1=14	uwsgi@repo.socket
unix:/run/dovecot/...	misc	dovecot.service
unix:/var/spool/postfix/private/auth	Dovecot SASL	dovecot.service
unix:/var/spool/postfix/private/*	misc	postfix.service
System stuff
unix:/run/dbus/systemd_bus_socket	D-Bus	dbus.socket
unix:/run/lvm/lvmetad.socket	???	lvm2-lvmetad.socket
unix:/run/udev/control	???	systemd-udev-control.socket
unix:/run/systemd/...	misc	misc
unix:/run/user/${UID}/{bus,systemd/{notify,private}}	misc	user@${UID}.service

2.5 Other running services of note

• dovecot.service
• mailman.service
• parabolaweb-reporead-inotify.service

2.6 Nginx "servers"

server_name	HTTP	HTTPS
Simple redirects
*	return 301 https://$host$request_uri;	return 301 https://www.parabola.nu/404;
parabolagnulinux.org	N/A	return 301 https://www.parabola.nu$request_uri;
*.parabolagnulinux.org	N/A	return 301 https://$subdomain.parabola.nu$request_uri;
list.parabolagnulinux.org	N/A	return 301 https://lists.parabola.nu$request_uri;
parabola.nu	N/A	return 301 https://www.parabola.nu$request_uri
Websites
www.parabola.nu	N/A	Serve /static/, /favicon.ico, /robots.txt,and /img/ statically, redirect /https to /, and hand everything else off to uWSGI
labs.parabola.nu	N/A	Redirect / to /projects; use uWSGI
lists.parabola.nu	N/A	Redirect / to /mailman/ Redirect /mailman/ to /mailman/listinfo Serve the mailman CGI programs at /mailman/* via fcgiwrap Serve the static mailman icons at /icons Serve /var/lib/mailman/archives/public at /pipermail
projects.parabola.nu	N/A	Serve cgit via uWSGI
repo.parabola.nu	N/A	Serve the union of /srv/repo/main and /srv/repo/http, using repoindex.php (via uWSGI) for indexes.
wiki.parabola.nu	N/A	Serve MediaWiki via uWSGI (https://lukeshu.com/blog/nginx-mediawiki.html)
pur.parabola.nu	N/A	TODO
redirector.parabola.nu	N/A	TODO
repomirror.parabola.nu	N/A	TODO

2.7 Mail configuration

Note: I (lukeshu) have no idea if this is still true for the current server. Ask fauno or mtjm.

Postfix's postscreen handles port 25 enforcing the pregreet test, checks SPF records via python2-postfix-policyd-spf, the mail is delivered via Dovecot deliver. Both deliver and the daemon need to be of the same version, so restarting the daemon after an update is needed.

Mailman handles the lists, with Postfix integration via a virtual address map. Use /usr/lib/mailman/bin/newlist to add a list.

2.8 Configuration things

There used to be several symlinks added to /var to keep things in sane places, but they have been replaced by bind mounts in /etc/fstab, because the symlinks confused pacman.

/var/lib/mailman   ->  ../../srv/mailman
/var/lib/mysql     ->  ../../srv/sql/mysql
/var/lib/postgres  ->  ../../srv/sql/postgres
/var/spool/cron    ->  ../../etc/cron.spool

That is, nothing of consequence and needing to be backed up should live anywhere but /srv or /etc. Maybe /home for personal stuff.

3 winston.parabola.nu

winston.parabola.nu is A VPS server which is hosted for free by 1984, a Hosting Company in Iceland.

$ uname -m
x86_64

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           4.4G         98M        2.2G        692K        2.0G        4.0G
Swap:          4.3G          0B        4.3G

$ df -h | grep ^/dev
/dev/vda3        94G  3.2G   86G   4% /
/dev/vda4       375G  5.7M  355G   1% /srv

Pardon the sloppy formatting (<code> instead of {{ic|), I wrote this in markdown and used pandoc to convert it to mediawiki.

3.1 configuration philosophy

Avoid editing files owned by a package if possible; insert things into new files. If a file must be edited, try to avoid having to change any lines; try to only add new lines.

Everything worth backing up should be in /etc, /srv, or /home.

3.2 infrastructure/management

3.2.1 base setup

Packages installed:

• group:base except for linux-libre
• linux-libre-lts
• haveged
• irqbalance

Files affected:

• /etc/hostname : winston.parabola.nu
• /etc/locale.conf : LANG=en_US.UTF-8 LC_COLLATE=C
• /etc/locale.gen : en_US.UTF-8 UTF-8
• /etc/machine-info : LOCATION=1984 Hosting Company, Iceland
• /etc/vconsole.conf : KEYMAP=us CONSOLEMAP=8859-1 FONT_MAP=8859-1_to_uni
• /etc/systemd/system/multi-user.target.wants/getty@tty1.service
• /etc/systemd/system/multi-user.target.wants/haveged.service
• /etc/systemd/system/multi-user.target.wants/irqbalance.service

3.2.1.1 bootloader

Packages installed:

• grub

Files affected:

• /etc/default/grub

The grub config has two overrides added to it:

GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_TERMINAL_OUTPUT=console

3.2.1.2 filesystems

Packages installed:

• systemd-swap

Files affected:

• /etc/fstab
• /etc/systemd-swap.conf
• /etc/systemd/system/local-fs.target.wants/systemd-swap.service
• /etc/systemd/system/multi-user.target.wants/remote-fs.target

TODO: document

fstab is set up similarly to proton.parabola.nu.

3.2.1.3 networking

Files affected:

• /etc/udev/rules.d/80-net-setup-link.rules
• /etc/netctl/eth0-static
• /etc/systemd/system/netctl@eth0\x2dstatic.service (auto-generated by netctl enable)
• /etc/systemd/system/multi-user.target.wants/netctl@eth0\x2dstatic.service

/etc/udev/rules.d/80-net-setup-link.rules is a symlink to /dev/null, which disables new-style predictable network interface names (enp0s3? ens3?), and causes it to fall back to the old-style names (eth0).

The netctl profile eth0-static just has the network information from the 1984 VPS control panel.

3.2.1.4 Pacman

Files affected:

• /etc/pacman.conf : Enable [pcr]
• /etc/pacman.d/mirrorlist : Use repo.parabola.nu directly
• /etc/pacman.d/.gitignore : /gnupg/

3.2.1.5 timedate

Files affected:

• /etc/localtime : -> /usr/share/zoneinfo/UTC
• /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service

timesyncd is an NTP client daemon.

3.2.2 etckeeper

Packages installed:

• pristine-etc-keeper

Files affected:

• /etc/systemd/system/multi-user.target.wants/pristine-etc-keeper.timer
• /etc/systemd/system/multi-user.target.wants/etckeeper.timer
• /etc/.gitignore : append !/mtab /machine-id /group- /gshadow- /passwd- /shadow- /resolv.conf
• /etc/udev/.gitignore : /hwdb.bin

etckeeper is configured to use git to keep track of changes in /etc. The systemd unit etckeeper.timer is enabled, which makes a commit (if necessary) daily. It will also run before and after pacman via libalpm hooks.

In addition to etckeeper keeping track of the current configuration, lukeshu's pristine-etc-keeper maintains a branch of what /etc would be like if we never made any changes from the default files. The clean version of etc is available as the clean remote's master branch. Doing a git diff clean/master master should be helpful in investigating how things have been configured.

3.2.3 users

See also: keys user and group

3.2.3.1 Parabola hackers

Packages installed:

• parabola-hackers-nshd
• openssh

Files affected:

• /var/lib/hackers-git/
• /etc/ssh/sshd_config
• /etc/nsswitch.conf
• /etc/pam.d/passwd
• /etc/pam.d/system-auth
• /etc/pam.d/system-login
• /etc/systemd/system/sockets.target.wants/nshd.socket
• /etc/systemd/system/dbus.service.wants/nshd.service (temporary systemd bug workaround)
• /etc/nshd/shadow

sshd is configured to force the use of keys (no password-based login), and to use parabola-hackers ssh-list-authorized-keys in addition to checking ~/.ssh/authorized_keys. ssh-list-authorized-keys returns the authorized keys from the hackers.git checkout in /var/lib/hackers-git (the path to the checkout is configured in /etc/parabola-hackers.yml).

NSS and PAM have been configured to use the ldap modules that are part of nss-pam-ldapd. However, instead of running the normal nslcd LDAP client daemon, the system has ben configured to run the parabola-hackers-nshd nshd daemon, which reads user infomation from the same hackers.git checkout (configured the same way). This way we dn't have to worry about keeping /etc/passwd in sync with hackers.git. To this end, PAM has also been configured to create a users home directory when they log in if it doesn't already exist. Because hackers.git doesn't store any password information, nshd stores password hashes in /etc/nshd/shadow.

Sometimes after something fails in PAM, you get a "User not known to the underlying authentication module" message. Like, the pam_ldap.so failed because you typed your password wrong, but it acts like it failed because it didn't "own" the user. I think that it's just a bug in PAM's message selection. But (TODO) we should actually track it down.

3.2.3.2 emergency user

Files affected:

• /etc/passwd
• /etc/shadow
• /etc/sudoers.d/99-emergency
• /home/emergency/

In case anything should ever go wrong with hackers.git, the user "emergency" has been set up. "emergency" has authorized (statically via ~/.ssh/authorized_keys) the keys of serveral (not publicly disclosed) Parabola hackers. /etc/sudoers.d/99-emergency grants special privileges in case PAM or NSS get screwed up.

3.2.3.3 other

Files affected:

• /etc/sudoers.d/00-wheel
• /etc/systemd/system/shadow.service.d/sort.conf

/etc/sudoers.d/00-wheel gives sudo access to everyone in the wheel group.

The shadow.service has been extended to sort the files if they otherwise are ok. This makes dealing with pacman updates and such easier.

3.2.3.4 pbot

Files affected:

• /etc/group
• /etc/gshadow
• /etc/passwd
• /etc/shadow
• /etc/systemd/system/pbot.service <-- this script is coppied from the pbot git repo
• /etc/systemd/system/multi-user.target.wants/pbot.service <-- created by systemctl enable

3.2.4 SSH

Packages installed:

• openssh

Files affected:

• /etc/ssh/sshd_config
• /etc/systemd/system/multi-user.target.wants/sshd.service
• /etc/ssh/.gitignore: /ssh_host_*_key /ssh_host_*_key.pub

See above for how authentication and users are set up.

sshd is also configured to listen on both ports 22 and 1863. We may turn of port 22 in the future. Not using port 22 isn't security through obscurity, it is security through keeping-the-the-logs-useful-by-keeping-noise-down.

3.2.5 SSL

Packages installed:

• certbot

Winston uses the certbot ACME client to get certificates from Let's Encrypt.

All domains handled by the server are shoved in as Subject Alternative Names in a single certificate. This makes configuring nginx easier.

3.2.5.1 keys user and group

Files affected:

• /etc/passwd
• /etc/shadow
• /etc/group
• /etc/gshadow
• /etc/letsencrypt
• /var/lib/letsencrypt
• /var/log/letsencrypt

In order to run certbot as a non-root user, the keys user and group have been created:

useradd --system --user-group --no-create-home --home-dir /etc/ssl --shell /usr/bin/nologin keys
chown -R keys:keys /etc/letsencrypt /var/log/letsencrypt /var/lib/letsencrypt
chmod 750 /etc/letsencrypt/archive /etc/letsencrypt/live

The associated keys group allows users to read the (private) keys in /etc/letsencrypt/live.

3.2.5.2 issuance, renewal, and installation

Files affected:

• /etc/ssl/misc/certbot-get
• /etc/ssl/misc/certbot-hook
• /etc/sudoers.d/10-certbot
• /etc/systemd/system/certbot-renew.service
• /etc/systemd/system/certbot-renew.timer
• /etc/systemd/system/timers.target.wants/certbot-renew.timer
• /etc/nginx/snippets/ssl.conf

Unlike acmetool, certbot doesn't have an easy way of saying "please add this domain as a Subject Alternative Name". You have to re-run the same (long) command to get the cert, but with the domain added. So, I've encapsulated this into the script /etc/ssl/misc/certbot-get. Edit the array of domains at the top of the script, then run it.

Renewal, however, is very simple. It is handled by certbot-renew.service (triggered by the associated .timer). It runs certbot renew with a couple of flags.

Both certbot-get and certbot-renew.service prove ownership of the domain via the http-01 challenge. /etc/nginx/nginx.conf includes /etc/nginx/snippets/ssl.conf, which has a server{} block that handles ACME http-01 challenges.

Both certbot-get and certbot-renew.service have been written to run sudo /etc/ssl/misc/certbot-hook after certificates have been updated, and sudo has been configured to allow the keys user to do this without a password. Right now certbot-hook just runs systemctl reload nginx.service.

3.2.5.3 other

Files affected:

• /etc/nginx/nginx.conf
• /etc/nginx/snippets/ssl.conf
• /etc/ssl/private/dhparam-2048.pem
• /etc/ca-certificates/.gitignore : /trust-source/blacklist/ /trust-source/anchors/ /extracted/
• /etc/ssl/.gitignore : /certs/*.0 /certs/*.1 /certs/*.pem /certs/ca-certificates.crt /certs/java/cacerts /.ssh/

nginx.conf includes snippets/ssl.conf, which is primarily based on the output of Mozilla Security's recommended web server configuration generator. It has had the main SSL information promoted to be directly into the http{} block, instead of having to be in each server{} block. The HTTP->HTTPS redirector has had an exception added to it to have it respond to ACME http-01 challenges.

Because certbot is only configured to use http-01 challenges, the all challenges happen over pain HTTP, which means that the configurations for each subdomain (which only serve over HTTPS/HTTP2) do not need to include anything about ACME or SSL (other than mentioning ssl in the listen directive).

ssl.conf needs to refer to a dhparam PEM file. This has been generated with the command

openssl dhparam -out /etc/ssl/private/dhparam-2048.pem 2048

3.2.6 HTTP

Packages installed:

• nginx

Files affected:

• /etc/nginx/nginx.conf
• /etc/nginx/fastcgi.conf, /etc/nginx/{fastcgi,scgi,uwsgi}_params
• /etc/nginx/mime.types
• /etc/nginx/snippets/ssl.conf
• /etc/nginx/sites/
• /etc/ssl/misc/certbot-hook
• /etc/systemd/system/multi-user.target.wants/nginx.service

See the #SSL for how nginx has been configured to do HTTPS.

fastcgi.conf, fastcgi_params, scgi_params and uwsgi_params have been edited to pass 127.0.0.1 as the client IP address to worker processes, to protect user privacy.

The log_format has been customized to redact the client IP and referrer, again to protect user privacy. Unfortunately, the libre version of nginx does not allow for customizing the format of the error log, so user IPs may still end up in the error log.

mime.types has had xz, gzip, bzip2, tar, PGP (.sig), and bittorrent types added to it. This list was based on the unmatched file extensions that find turned up on repo.parabola.nu.

Per-subdomain configurations live in /etc/nginx/sites/*.conf (which is the exact wildcard used with include at the end of nginx.conf).

"Special" rules in sites/*.conf should have the alias- or meta- prefix. For example, right now there is only meta-unknown-domain.conf which is a fallback for when no other file handles a domain (it redirects to https://www.parabola.nu/404). Normal domain handlers should be named server-${domain}.conf

Every sites/server-*.conf file should be of the format (replace subdomain with the subdomain):

/etc/nginx/sites/server-subdomain.parabola.nu.conf

# -*- Mode: nginx; nginx-indent-level: 8; intent-tabs-mode: t -*-

server {
    server_name subdomain.parabola.nu;
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    error_log  /var/log/nginx/nginx.http.subdomain.parabola.nu.error.log  error;
    access_log /var/log/nginx/nginx.http.subdomain.parabola.nu.access.log specific;

    ...
}

nginx.conf sets root /srv/http/$server_name, which means that most of the time root won't need to be set in each individual server{} block. Unfortunately, error_log and access_log don't support variable expansion, so they do need to be set each time.

The individual sites/server-*.conf files won't be documented in this section; see the section for the service that it provides.

TODO: document the rest of the settings in nginx.conf

3.2.6.1 process management

Packages installed:

• uwsgi

Files affected:

• /etc/systemd/system/uwsgi@.service.d/socket.conf
• /etc/systemd/system/uwsgi@.socket.d/owner.conf
• /etc/uwsgi/vassal.ini
• /etc/systemd/system/sockets.target.wants/uwsgi@vassal.socket

Wherever possible, we should use uWSGI for process management of our HTTP services. This allows for much more consistent management and configuration than the hodge-podge of PHP-FPM, manage.py, fcgiwrap, et c. that we used to have on Proton.

uWSGI is the program, uwsgi is the protocol it speaks with nginx. A pool of workers is called a vassal, and is configured in /etc/uwsgi/vassal.ini, and activated by uwsgi@vassal.socket; a socket speaking the uwsgi protocol is created at /var/run/uwsgi/vassal.sock.

We use systemd socket activation rather than a uWSGI emperor because they provide largely the same functionality; the only real advantage that a uWSGI emperor would provide over systemd socket activation is if you ran it in tyrant mode, it lets you do some cool things with unpriveleged access, which would be useful for a shared web host. We aren't a shared web host, and have no reason to run emperor in tyrant mode.

Since the uwsgi@.service vassal unit is written to support socket-activated or non-socket-activated use, it is normally possible to accidentally start it without the associated .socket unit; which is an error with how our vassal configurations are written. To fix this, uwsgi@.service.d/socket.conf overrides the unit a bit to disable non-socket-activated use.

The ownership and permissions for the socket are configured in uwsgi@.socket.d/owner.conf, which sets the owner to http:http and the mode to 0600.

uWSGI supports thread pools in addition to process pools, but many of the actual workers you'll want to use aren't thread safe, so stick to process pools unless you specifically know that your worker is thread-safe (for example, PHP, at least with the modules needed for MediaWiki, is not thread-safe).

Individual vassal configurations are documented in the section for the service that they provide, not here.

3.2.7 email

Packages installed:

• postfix

Files affected:

• /etc/systemd/system/multi-user.target.wants/postfix.service
• /etc/postfix/.gitignore : *.db

Configuration

• /etc/postfix/main.cf edited to set make it accept, and locally deliver, mails to <user>@winston.parabola.nu.

Operation

• pbot gets emails delivered qmail-style in $HOME/Maildir/ to allow him to report bug tracker changes
• currently postfix aliases has not been set up. where should mails addressed to e.g. postmaster@winston.parabola.nu go? at the moment they get locally delivered to root user

3.3 services

You know, the things that the server is actually for.

3.3.1 System Information

Packages installed:

• uwsgi-plugin-cgi

Files affected:

• /srv/http/winston.parabola.nu/sysinfo.cgi
• /etc/uwsgi/winston.ini
• /etc/nginx/sites/server-winston.parabola.nu.conf
• /etc/systemd/system/sockets.target.wants/uwsgi@winston.socket

Serves a simple system information page at https://winston.parabola.nu/. The source of the (very simple) script is at https://winston.parabola.nu/sysinfo.cgi.

3.3.2 ParabolaWeb

Packages installed:

• uwsgi-plugin-python2
• parabolaweb-utils
• parabolaweb-dev

Files affected:

• /etc/passwd
• /etc/shadow
• /srv/http/www.parabola.nu/

Winston isn't running ParabolaWeb (yet), but has it installed because other pages symlink to the branding files in it.

useradd --comment ParabolaWeb --home-dir /srv/http/www.parabola.nu/ --create-home --system --gid http parabolaweb

3.3.3 Bazaar

Packages installed:

• bzr

Files affected:

• /etc/passwd
• /etc/systemd/system/bzr.service
• /etc/systemd/system/multi-user.target.wants/bzr.service
• /srv/bzr/

TODO: IDK, someone else set this up.

3.3.4 Git

Packages installed:

• git
• cgit
• uwsgi-plugin-cgi

Files affected:

• /etc/passwd
• /etc/sudoers.d/10-git
• /etc/cron.spool/git
• /srv/git/

The "git" user has been modified:

usermod --comment "Parabola git" --shell /usr/bin/git-shell git

The git repositories live in /srv/git/ and are owned by the git user.

Metadata about all of the git repositories, as well as most of the setup, lives in the git-meta.git repository. The git user has a cron-job to update this repository every hour.

git is configured (core.hooksPath such that it looks in ~/.config/git/hooks/ for hooks, instead of /path/to/repo.git/hooks/. All of the server-side hooks exist: they all do the "same" thing: use run-parts to run all of the scripts in ~/.config/git/hooks/hook-name.d/, and then run the normal hook file /path/to/repo.git/hooks/hook-name.

Right now there are only two "real" hooks:

• /srv/git/.config/git/hooks/post-receive.d/agefile : see #transport:_HTTPS
• /srv/git/hackers.git/hooks/post-receive : update the hackers.git checkout (see #Parabola_hackers), then run sudo systemctl reload nshd.service to activate changes to system users (sudo has been configured to allow this to be done without a password). TODO: see the hook file itself for more details, but it should also activate the changes to email aliases with postfix, as well as rebuild the parabola-keyring package with autobuilder.

3.3.4.1 transport: git protocol

Files affected:

• /etc/systemd/system/socket.target.wants/git-daemon.socket

3.3.4.2 transport: SSH

Files affected:

• /srv/git/git-shell-commands/

This is the only method for which push is enabled. Only the git@ user may push; even though if you have access to git, you probably have access to ssh into yourname@winston, you can't push from that account. As far as git is concerned, all users are equal (TODO: perhaps we should implement some more granular access control. Safety rails and all that). Access to git@ is based on membership in the "git" group in hackers.git (see #Parabola_hackers).

If you need to set up custom git hooks, then ssh in to your user, and use sudo -u git to do it.

3.3.4.3 transport: HTTPS

Files affected:

• /etc/uwsgi/git.ini
• /etc/systemd/system/socket.target.wants/uwsgi@git.socket
• /etc/nginx/sites/server-git.parabola.nu.conf
• /srv/http/git.parabola.nu/
• /srv/git/.config/git/hooks/post-receive.d/agefile

The git web interface is cgit, which is managed by uWSGI speaking uwsgi/modifier1=9, which is the variant of the uwsgi protocol for CGI requests.

uwsgi/git.ini sets CGIT_CONFIG=/srv/http/git.parabola.nu/cgitrc.

A post-receive hook that runs for all repositories updates the agefile, so that cgit has a better idea of when a repository was last updated.

The files for cgit are in /srv/http/git.parabola.nu/, which shadows /usr/share/webapps/cgit/:

# mirror /usr/share/webapps/cgit/
cgit.cgi    -> /usr/share/webapps/cgit/cgit.cgi
cgit.css    -> /usr/share/webapps/cgit/cgit.css # customized
cgit.png    -> /usr/share/webapps/cgit/cgit.png
favicon.ico -> /usr/share/webapps/cgit/favicon.ico /srv/http/www.parabola.nu/web/sitestatic/favicon.ico
robots.txt  -> /usr/share/webapps/cgit/robots.txt
# added files
cgitrc
archnavbar/ -> /srv/http/www.parabola.nu/web/sitestatic/archnavbar/
header.html              # archnavbar
syntax-highlighting.sh   # based on some old version of /usr/lib/cgit/filters/syntax-highlighting.sh

3.3.5 pbot

pbot, the Parabola IRC bot, lives at pbot.git, and runs as the user pbot

Home dir at /home/pbot/.

• This contains all the pbot files from the git repo. Not in a subdir, just right there.
• Also contains dir logs/ TODO add to git repo
• Also contains 600 permissions file passord.secret containing IRC password
• pbot writes data to info/ and announcements/. these are put in a tarball every day by cron job so they can be downloaded elsewhere if wanted (e.g. for backup).

Deps installed with pacman

• moreutils
• w3m
• recode
• inotify-tools

3.3.6 Redmine

Packages installed:

• ruby
• ruby-bundler
• rubygems
• uwsgi-plugin-rack

Files affected:

• /etc/nginx/sites/server-labs_parabola_nu.conf
• /etc/systemd/system/labs-restart.socket
• /etc/systemd/system/labs-restart.timer
• /etc/systemd/system/multi-user.target.wants/uwsgi@labs.service
• /etc/systemd/system/socket-user.target.wants/uwsgi@labs.socket
• /etc/systemd/system/timers.target.wants/labs-restart.timer
• /etc/systemd/system/uwsgi@labs.service.d/psql.conf
• /etc/uwsgi/labs.ini

See also: DeveloperWiki:Web_Admin

3.3.7 Rsync

Packages installed:

• rsync

Files affected:

• /etc/rsyncd.conf
• /etc/systemd/system/rsyncd.socket
• /etc/systemd/system/sockets.target.wants/rsyncd.socket

Installed by coadde or Emulatorman during migration of repo.parabola.nu from Proton. I resumed configuration of the server and enabled it to accommodate mirrors after the DNS change. Port number (873 -> 875) and modules ( [repos]:/srv/repo/main and [abs]:/srv/abslibre) were copied verbatim from Proton.

4 beefcake.parabola.nu

Beefcake is a virtual machine occupying roughly half of LukeShu's KGPE-D16 in Indianapolis. It runs no publicly accessible services, but is intended to be usable as a build machine for Parabola developers. It has a large data disk which archives all repos since April 2018; which are exposed to the librechroots as an HTTP service.

$ uname -m
x86_64

$ free -h
              total        used        free      shared  buff/cache   available
Mem:           29Gi       725Mi       250Mi        13Mi        28Gi        28Gi
Swap:         7.4Gi       4.0Mi       7.4Gi

$ df -h | sed -n '1p;/^\/dev/p'
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         10G  5.6G  4.1G  59% /
/dev/vdb        9.8G  6.8G  2.6G  73% /home
/dev/vdc        4.6T  4.6T  644K 100% /mnt/data
/dev/vdd        100G   42G   58G  43% /var/lib/archbuild

4.1 Login in Beefcake

4.1.1 Proxy via Winston

To login in Beefcake using Winston as a proxy, you need to first login in winston and forward Beefcake IP address locally:

ssh user@parabola.nu  -L localhost:1863:192.168.9.164:1863

You can then login to Beefcake with the following command:

ssh user@localhost -p 1863

Alternatively, you can also use the information from above as a 'ProxyCommand' in your SSH configuration file(s) in ~/.ssh/config or /etc/ssh/ssh_config.

More information is also available on a mailing list post.

4.1.2 LibreVPN

This is more complicated to setup; but allows accessing Beefcake and other parabola servers, without relying on any one of them as a proxy. LibreVPN also makes it fairly simple (once setup) to add any rare hardware, or useful services to the network, for other Parabola devs to access.

TODO: this is currently not well-documented

4.2 Using your local gpg-agent to sign files on beefcake

If you're building packages on beefcake, obviously you'll need to PGP-sign them when you release them. But copying your GPG secret key to Beefcake seems risky & silly. Instead, you should set up forwarding so that Beefcake can use your local gpg-agent that has access to your local secret key, rather than running gpg-agent on Beefcake.

4.2.1 Step 1: Stop Beefcake from automatically starting gpg-agent for you

user@beefcake $ mkdir -p ~/.config/systemd/user
user@beefcake $ ln -sT /dev/null ~/.config/systemd/user/gpg-agent.socket

4.2.2 Step 2: Configure SSH to forward your local gpg-agent

Unfortunately, this requires knowledge of where GPG will look for the gpg-agent socket. Which is under-documented, and relies on undocumented heuristics for magically trying to guess the appropriate place, instead of just letting the user configure it.

If /run/user/$UID/ exists (as it will on systemd systems (and elogind systems?); systemd keeps track of that directory as that user's $XDG_RUNTIME_DIR), then you insert the following in to the appropriate section of your SSH config:

user@local:~/.ssh.config

Host beefcake
	RemoteForward /run/user/1015/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra
	#                       ^^^^                             ^^^^
	#                your UID in hackers.git             your local UID

	# Whatever DISPLAY or TTY name the remote `gpg` sends to gpg-agent will be on Beefcake,
	# not on the local host.  So we need to tell gpg-agent the correct DISPLAY/TTY info. 
	PermitLocalCommand yes
	LocalCommand gpg-connect-agent UPDATESTARTUPTTY /bye

If /run/user/$UID/ doesn't exist (as it won't OpenRC systems without elogind), then GnuPG puts the socket somewhere else:

user@local:~/.ssh.config

	RemoteForward /run/user/1015/gnupg/S.gpg-agent /home/luke/.gnupg/S.gpg-agent.extra
	#                       ^^^^                   ^^^^^^^^^^
	#                your UID in hackers.git     your local HOME

5 Notes

TLS certs are used by:

• nginx
• dovecot (only used by postfix; does it really need TLS?)
• postfix
• prosody

6 TODO

• murmurd? *DONE*
• Use NSD (or PowerDNS) to manage DNS zones from the server.
• get proton using nshd *n/a*
• migrate services from proton to winston *DONE*
• set up autobuilder on winston *DONE ('your-freedom' and 'parabola-keyring' git hooks)*
• switch most A records to CNAME records
• ...

7 IN PROGRESS

• re-provision proton services (DONE)
• update the above linked articles
• increase nesting of In-Progress-2019-11 (below) to replace this heading

7.1 Categorized List of Services:

7.1.1 web services

native

• cgit

PHP

• mediawiki

python

• mailman
• parabolaweb (packages, hackers, news)
• pipermail (email archives)

ruby

• redmine (issue tracker, forum)

7.1.2 essential public services

• postfix (SMTP)
• repo HTTP access
• repo rsync access

7.1.3 essential private services

• nshd (all boxes will need this)
• tinc (at least one reliable box needs this)

7.1.4 essential internal services

• autobuilder (hackers=>parabola-keyring/nshd, blacklist=>your-freedom)
• duplicity (backups)
• pbot
• upstream repo importer

7.1.5 essential storage

• backups (duplicity cross-mirrored 60-100GB per mirror)
• git (1GB)
• packages (~80GB per arch - currently 3 arches - 2 more planned)
• ISOs (5-50 GB - depending how we slice them)

7.1.6 non-essential public services

• bazaar
• docecot (@parabola.nu IMAP)
• git HTTP access
• mumble (VOIP)
• prosody (XMPP)

7.1.7 notes

most of the "non-essential public services" are rarely or never used; and are disabled

the ones that are most tightly coupled are autobuilder<->git-storage, and postfix<->mailman - everything labeled as "essential" is probably best kept on the same box, and perhaps replicated on another for emergency use - all of those essentials were on winston, while the web and non-essentials were split between proton and winston

lukeshu explained to me that the separation between proton and winston was such that all clients of mysql were on one box and all clients of pg were on the other - when we migrated everything to winston, lukeshu was leery that mixing these could cause performance problems; so that is something to consider with regards to re-provisioning