Yeah na, put your home services in Tailscale, and for your VPS services set up the firewall for HTTP, HTTPS and SSH only, no root login, use keys, and run fail2ban to make hacking your SSH expensive. You’re a much smaller target than you think - really it’s just bots knocking on your door and they don’t have a profit motive for a DDOS.

From your description, I’d have the website on a VPS, and Immich at home behind TailScale. Job’s a goodun.

+1 for the main risk to my service reliability being me getting distracted by some other shiny thing and getting behind on maintenance.

I love this idea (of just picking something I’m loving each month), it would help me overcome my decision paralysis about who to support.

Yes, a few. Signal (daily use), LetsEncrypt & Certbot (EFF). It’s not enough.

One day I decided I’d spend $x every January (when I do all my other donations) on open source stuff I depend on, and roughly in the proportions I depend on them. It quickly became impossible - I can’t just fund Debian (which I use a lot of in VMs), I’d need to think of all their dependencies, same with NGINX, Node etc etc. The mind boggles.

I need something like a Spotify subscription for open source to assuage my guilt of the great value I extract for my personal use of open source.

I started as more “homelab” than “selfhosted” as first - so I was just stuffing around playing with things, but then that seemed sort of pointless and I wanted to run real workloads, then I discovered that was super useful and I loved extracting myself from commercial cloud services (dropbox etc). The point of this story is that I sort of built most of the infrastructure before I was running services that I (or family) depended on - which is where it can become a source of stress rather than fun, which is what I’m guessing you’re finding yourself in.

There’s no real way around this (the pressure you’re feeling), if you are running real services it is going to take some sysadmin work to get to the point where you feel relaxed that you can quickly deal with any problems. There’s lots of good advice elsewhere in this thread about bit and pieces to do this - the exact methods are going to vary according to your needs. Here’s mine (which is not perfect!).

• I’m running on a single mini PC & a Synology NAS setup for RAID 5
• I’ve got a nearly identical spare mini PC, and swap over to it for a couple of weeks (originally every month, but stretched out when I’m busy). That tests my ability to recover from that hardware failure.
• All my local workloads are in LXC containers or VM’s on Proxmox with automated snapshots that are my (bulky) backups, but allow for restoration in minutes if needed.
• The NAS is backed up locally to an external USB that’s not usually plugged in, and to a lower speced similar setup 300km away.
• All the workloads are dockerised, and I have a standard directory structure and compose approach so if I need to upgrade something or do some other maintenance of something I don’t often touch, I know where everything is with out looking back to the playbook
• I don’t use a script or Terrafrom to set those up, I’ve got a proxmox template with docker and tailscale etc installed that I use, so the only bit of unique infrastructure is the docker compose file which is source controlled on Forgejo
• Everything’s on UPSs
• A have a bunch of ansible playbooks for routine maintenance such as apt updates, also in source control
• all the VPS workloads are dockerised with the same directory structure, and behind NGINX PM. I’ve gotten super comfortable with one VPS provider, so that’s a weakness. I should try moving them one day. They are mostly static websites, plus one important web app that I have a tested backup strategy for, but not an automated one, so that needs addressed.
• I use a local and an external UptimeKuma for monitoring, enhanced by running a tiny server on every instance that just exposes a disk free and memory free api that can be consumed by Uptime.

I still have lots of single points of failure - Tailscale, my internet provider, my domain provider etc, but I think I’ve addressed the most common which would be hardware failures at home. My monitoring is also probably sub-par, I’m not really looking at logs unless I’m investigating a problem. Maybe there’s a Netdata or something in my future.

You’ve mentioned that a syncing to a remote server for backups is a step you don’t want to take, if you mean managing your own is a step you don’t want to take, then your solutions are a paid backup service like backblaze or, physically shuffling external USB drives (or extra NASs) back and forth to somewhere - depending on what downtime you can tolerate.

+1 for Syncthing. I run it on a server at home, then on my MacBook over Tailscale. For web access I run FileBrowser (also over Tailscale) against the same directory.

I switched from Copilot to Codeium after only a couple of months of Copilot use - just based on the cost since currently I’m just a hobby coder.

The main difference I’ve noticed is that Codeium doesn’t seem as smart about the local context as Copilot. Copilot would look at how I’m handling promises in a project, and stick to that, whereas Codeium would choose a strategy seemingly at random.

A second, and maybe more telling example, is that I do my accounts using ‘plain text accounting’ in VS Code. This is a very niche approach to accounting software and I imagine is hardly in the training sets at all - there certainly would not be a lot of public domain text accounts in the particular format (BeanCount) I use in public code repositories. Codeium doesn’t make any suggestions for entries as I’m entering transactions, whereas Copilot would see that the account names I’m using are present in another file in the project and suggest them, and very quickly figure out the formatting of transactions and suggest them correctly.

E. Jean Carroll could buy it off them for the lols.

I run two local physical servers, one production and one dev (and a third prod2 kept in case of a prod1 failure), and two remote production/backup servers all running Proxmox, and two VPSs. Most apps are dockerised inside LXC containers (on Proxmox) or just docker on Ubuntu (VPSs). Each of the three locations runs a Synology NAS in addition to the server.

Backups run automatically, and I manually run apt updates on everything each weekend with a single ansible playbook. Every host runs a little golang program that exposes the memory and disk use percent as a JSON endpoint, and I use two instances of Uptime Kuma (one local, and one on fly.io) to monitor all of those with keywords.

So -

• weekly: 10 minutes to run the update playbook, and I usually ssh into the VPS’s, have a look at the Fail2Ban stats and reboot them if needed. I also look at each of the Proxmox GUIs to check the backs have been working as expected.
• Monthly: stop the local prod machine and switch to the prod2 machine (from backups) for a few days. Probably 30 minutes each way, most of it waiting for backups.
• From time to time (if I hear of a security update), but generally every three months: Look through my container versions and see if I want to update them. They’re on docker compose so the steps are just backup the LXC, docker down, pull, up - probs 5 minutes per container.
• Yearly: consider if I need to do operating systems - eg to Proxmox 8, or a new Debian or Ubuntu LTS
• Yearly: visit the remotes and have a proper check/clean up/updates

The Debian thong made me laugh. Who is buying this? For themselves, their partners? I’m imagining Christmas morning when I’m trying to explain the value of this gift you’ve just opened.

My ‘good reason’ is just that it’s super convenient - for backups and painlessly moving apps around between nodes with all their data.

I would run plain LXCs if people nicely packaged up their web apps as LXC templates and made them available on LXCHub for me to run with lxc compose up, but they generally don’t.

I guess another alternate future would be if Proxmox added docker container supervision to their web interface, but you’re still not going to have the self-contained neat snapshot system that includes the data.

In theory you should be able to convert an OCI container layer by layer into an LXC, so I bet there’s projects out there that attempt this.

No answer, but just to say I run most of my services with this setup - Docker in a Debian LXC under Proxmox, and don’t have this issue. The containers are ‘privileged’, and I have ‘nesting’ ticked on, but apart from that all defaults.

There are a heap of general “Linux Administration” courses which will patch a lot of holes in the knowledge of almost all self-taught self hosters. I’d been using Linux for a while but didn’t know you could tab to complete file names in commands till I learned it on Udemy ¯_(ツ)_/¯

I routinely run my homelab services as a single Docker inside an LXC - they are quicker, and it makes backups and moving them around trivial. However, while you’re learning, a VM (with something conventional like Debian or Ubuntu) is probably advised - it’s a more common experience so you’ll get more helpful advice when you ask a question like this.

I’m on iOS. I’ve been testing a beta of Jello that looks really promising, but as a beta has a bit of distance to go. I’ll check out Feishin though - thanks for the recommendation.

I’d love Jellyfin to turn out to be the solution, but I suspect it’s not, at least yet.

I’ve got three of these little 1L HP’s, one for production, a spare, and one for development. But really, it’s a small load - that list would happily run on an old nuc. The constraint is really memory which I’ve mostly addressed by moving from VMs to LXCs. And I could be even more efficient by just running all the docker containers on one host if I had to.

Storage for media and backups is a Synology NAS.

how to access the NAS and HA separately from the outside knowing that my access provider does not offer a static IP and that access to each VM must be differentiated from Proxmox.

Tailscale, it will take about 5 minutes to set up and cost nothing.

Infrastructure:

• Proxmox VE - everything’s virtualised on Debian, mostly in docker inside LXC’s for neat backup/restore and moving between nodes
• NGINX Proxy Manager - in front of most of my homelab services so they have https certificates
• Tailscale - access everything, everywhere, including on phone, securely
• Uptime Kuma - monitoring, with ntfy notifications
• apt cacher NG - unnecessary caching of apt updates

Apps:

• AudioBookShelf - Audio books (download them from here into BookPlayer (iOS) to listen in the car)
• Kavita - eBooks.
• BOINC - curing childhood cancer
• Forgejo - git repository. I do a bit of dev, but everyday self-hosters should probably still learn git to manage their configs
• Jellyfin - TV/Movies
• Syncthing - I replaced dropbox with Syncthing and FileBrowser, and Evernote with Syncthing and SilverBullet

Currently in testing on the dev server:

• neko - virtualised browser. Been experimenting with this in a container with a VPN for really simple secure browsing - ie launch it, do your online banking and then destroy the container.
• Dashy - I go through periods of wanting a pretty home page with all my services, set it all up, then fail to actually use it and eventually delete it, then hear about another cool one…
• Sharry - securish file sharing. I don’t love just emailing my accounts off to the accountant.
• LimeSurvey - survey software (like Survey Monkey) - just something I’m testing for work
• Omada controller - I’ve got a TP-Link switch and WAP that don’t really need centrally controlled, but you know, can be.
• A couple of development environment LXCs I use VS Code in

I still have not landed on a music system. I’ve put some of my library on Jellyfin, and tried a couple of apps with, but haven’t hit on a good combination yet. [edit:formatting}

This, or two turnbuckles joined at the top point with a couple of links of chain.

I read somewhere that GoPros and other action cameras are one of the least used purchases, so I figured “that should mean there’s plenty on eBay”. So grabbed up second hand bargain, played around with it for a couple of weeks, bought some extra batteries and other accessories, and since then it’s sat in the cupboard except for a single occasion.

Turns out you don’t need an action cam if you’re not getting any action.