Guide: Improving raspberry pi SD card lifetime for remote servers

 Having built two  rPI NAS boxes, I've had a chance to see how well they survive in the wild. The answer: each one has failed within a couple years. 

The first one failed because the power supply was undersized. It worked fine for a while, but eventually the powered hub that served the pi and the external hard drive could no longer support both, which killed the hard drive being used. Solution: but an external hard drive that comes with its own power brick. It had better be enough if they include it in the box. 

The 2nd one failed due to SD card corruption. This can be caused by power issues as well, but I don't think that's what happened. I think the issue is that it was too small, and too old. As with all flash devices there's a limited number of times each cell can be written before they start to fail. On a small SD card the number of cells is likewise small, so there's less space to spread the writes out. I was using a 2gb card, which had enough space to fit a lite pi OS install, but didn't have much spare space left over. I didn't need extra space for building a NAS, but that limited space means that there wasn't room to do write leveling.

Beyond that, it's worth trying to reduce the amount of writes to the SD card so that you don't even get into the point that significant write leveling is need. 

Reduce swapping

Some people think you should eliminate swap all together, but doing that risks causing the out of memory killer shutting software down indiscriminately. Instead, let's reduce the amount of swapping that happens. Linux swaps memory to disk opportunistically in order to maintain extra free memory for immediate use. This keeps your computer more responsive, but some of that swapping will be unnecessary. Let's reduce that as much as possible.

echo vm.swappiness = 0 >  /etc/sysctl.d/00-swappiness.conf

Note this file probably doesn't exist on your computer(yet); the default value is 60, which you can check with cat /proc/sys/vm/swappiness . Note that zero doesn't turn off swap entirely, just reduces opportunist swapping.  You might try running top to see how much swap is being used before you change this value. Don't expect the value to go down until after you reboot. 

RAMDISK logs or reduced logging

Logs can be a constant source of disk access. One option is to store them in ram instead, but then you lose them with each reboot (I like to reboot every week) and logs can be useful for catching configuration problems or failing hardware.

SSH is by far the biggest source of writes, at least on a world-accessible machine. Every 2-3 seconds it writes a new message reporting the latest attempt of somebody to log into your machine (as root, of course) via SSH. You can see this for yourself: 

watch -d ls -l /var/log

In theory this is an important thing to log and keep track of, because OMG, somebody is trying to hack your machine!!! But in reality it's all automated sniffing from botnets around the world and is meaningless noise unless you are building a honeypot. I certainly don't want to wear out my SD card keeping track of it.

Rather than putting all logs in ram, I suggest two less severe mitigations. One is security via obscurity: change your ssh port. You will get a lot less traffic with any port that's above 1024. Or, you can turn off logs of authentication attempts all together, which is my preference. You'd think that editing

/etc/ssh/sshd_config  to include  LogLevel QUIET

would do this, but you still get some messages; you also need to instruct rsyslog to reject all messages from sshd by adding 

 :programname, isequal, "sshd" stop  as the first rule in   /etc/rsyslog.conf

But you are not done; you still get failed logins recorded to a binary file called /var/log/btmp. btmp exists outside the syslog framework, so you can't block it that way. The activity comes from openssh and can only be turned off by recompiling openssh! Yikes, what a dumb choice that is. There's a file system work-around though: you can make btmp immutable:

chattr +i /var/log/btmp

Finally, there's the systemd journal, which is the modern duplicate of syslog. That is to say both should have all the messages you care about in them, but you use different tools to search them. systemd journal has much less options for controlling what gets written to it, and I find that >90% of it is failed logins, even after the tweaks above.  Two options: make it volatile, (entirely stored in memory), or change the rate at which it's flushed to disk (default is every 5 minutes). These options are in /etc/systemd/journald.conf

It seems like by far the easiest option is to change your SSH port.

Stop recording access time

Linux records the last time files were accessed, so a regularly running system is going to incur some writes just from reading configuration files, etc.. There's already an /etc/fstab option that's the default on newer releases of PiOS to keep this to a minimum called noatime, but it's also suggested that adding norelatime would reduce the writes further. Might as well.