This used to be exactly what I said too, I still run bash as my terminal so when I remote it works the same way. I'm the girl everyone asks when they need a one liner, I read through the sed/awk man pages for fun, and I can skim a script and tell if it's posix compliant. But I finally realized I already know that stuff. When I'm developing locally I should be as productive as possible. When I'm running stuff remotely I can worry about whether the environment is gnu, bsd, or busybox.
You gave some options
TPM 2 based disk encryption. This is basically what bitlocker does, but it isn’t great. It uses an encryption key stored on your TPM chip, that shouldn’t ever be accessible to be exported. This means the disk should only be decryptable in the machine it’s in. That in conjunction with secure boot can give you some guarantees that the only way to access data is through the the computer itself (no pulling the disk first). The issue is there are many potential vulnerabilities that could subvert this, logoFAIL being the most recent.
You could setup a proper KVM. The two gotos are PiKVM and TinyPilot. Jeff Geerling did a good video on these. It’ll cost a few 100 bucks but can definitely be worth it. You might consider a motherboard with a builtin KVM in your next build too.
Setup NBDE (Network Bound Disk Encryption). This is pretty new, but what I’m planning to move to. Redhat has an implementation with Tang & Clevis (server and clients). You might be able to eventually use Clevis with other alternative backend too.