I also use fetchmail via POP, so I don't have my mail accessible potentially to attackers on remote servers. But I wait to flush the mail until after I run my backups locally, just to make sure I don't lose any messages.
If I need to access my mail, I do so via SSH, which requires my Nitrokey; my mailserver is inaccessible outside of the box that it's running on. Consequently, I use Gnus via that SSH session; I do not connect to my local IMAP server from my laptop, to limit attack surface a bit further.
This is also out of respect to people I correspond with, since any compromise of my system is a breach of their privacy as well.
Mike Gerwitz's personal Mastodon instance