ch00f

Woke up today to the homeserver being unresponsive. Couldn't SSH, no video out when I connected a monitor, and even the reset button didn't do anything. Had to hold the power button to shut it down.

/var/log/syslog doesn't show anything interesting other than the issue happened at just after 4am. Log

2026-02-27T03:55:01.481794-08:00 blackbox CRON[1743418]: (www-data) CMD (/usr/bin/php8.3 /mnt/MONSTERDRIVE/pixelfeddata/pixelfed/artisan schedule:run >> /dev/null 2>&1)
2026-02-27T04:00:00.198504-08:00 blackbox smartd[2126]: Device: /dev/sdd [SAT], CHECK POWER STATUS spins up disk (0x81 -> 0xff)
2026-02-27T04:00:00.291853-08:00 blackbox systemd[1]: Starting sysstat-collect.service - system activity accounting tool...
2026-02-27T04:00:00.298344-08:00 blackbox systemd[1]: sysstat-collect.service: Deactivated successfully.
2026-02-27T04:00:00.298523-08:00 blackbox systemd[1]: Finished sysstat-collect.service - system activity accounting tool.
2026-02-27T04:00:00.299608-08:00 blackbox kernel: kauditd_printk_skb: 8 callbacks suppressed
2026-02-27T04:00:00.299613-08:00 blackbox kernel: audit: type=1130 audit(1772193600.298:798916): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
2026-02-27T04:00:00.299615-08:00 blackbox kernel: audit: type=1131 audit(1772193600.298:798917): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='unit=sysstat-collect comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
2026-02-27T04:00:01.923610-08:00 blackbox kernel: audit: type=1101 audit(1772193601.922:798918): pid=1744810 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:accounting grantors=pam_permit acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:01.923614-08:00 blackbox kernel: audit: type=1103 audit(1772193601.922:798919): pid=1744810 uid=0 auid=4294967295 ses=4294967295 subj=unconfined msg='op=PAM:setcred grantors=pam_permit,pam_cap acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:01.923615-08:00 blackbox kernel: audit: type=1006 audit(1772193601.922:798920): pid=1744810 uid=0 subj=unconfined old-auid=4294967295 auid=33 tty=(none) old-ses=4294967295 ses=50544 res=1
2026-02-27T04:00:01.923615-08:00 blackbox kernel: audit: type=1300 audit(1772193601.922:798920): arch=c000003e syscall=1 success=yes exit=2 a0=7 a1=7fff81d75200 a2=2 a3=0 items=0 ppid=2654 pid=1744810 auid=33 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=50544 comm="cron" exe="/usr/sbin/cron" subj=unconfined key=(null)
2026-02-27T04:00:01.923616-08:00 blackbox kernel: audit: type=1327 audit(1772193601.922:798920): proctitle=2F7573722F7362696E2F43524F4E002D66002D50
2026-02-27T04:00:01.924259-08:00 blackbox CRON[1744811]: (www-data) CMD (/usr/bin/php8.3 /mnt/MONSTERDRIVE/pixelfeddata/pixelfed/artisan schedule:run >> /dev/null 2>&1)
2026-02-27T04:00:01.924614-08:00 blackbox kernel: audit: type=1105 audit(1772193601.923:798921): pid=1744810 uid=0 auid=33 ses=50544 subj=unconfined msg='op=PAM:session_open grantors=pam_loginuid,pam_env,pam_env,pam_permit,pam_umask,pam_unix,pam_limits acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:01.925610-08:00 blackbox kernel: audit: type=1110 audit(1772193601.924:798922): pid=1744811 uid=0 auid=33 ses=50544 subj=unconfined msg='op=PAM:setcred grantors=pam_permit,pam_cap acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T04:00:02.357616-08:00 blackbox kernel: audit: type=1104 audit(1772193602.356:798923): pid=1744810 uid=0 auid=33 ses=50544 subj=unconfined msg='op=PAM:setcred grantors=pam_permit acct="www-data" exe="/usr/sbin/cron" hostname=? addr=? terminal=cron res=success'
2026-02-27T09:23:35.786375-08:00 blackbox systemd-modules-load[904]: Inserted module 'dm_multipath'

Would something like this be a direct hardware failure? Like a power supply hiccup or something? It happening at 4am coincides with my electric car starting to charge, but the server is on a dedicated 20A circuit and behind a battery backup. I also don't see any power issues on my Sense monitor at that time though it has limited resolution.

Mainboard is a Supermicro H13SAE-MF and I'm using ECC RAM.

I've been running this hardware for over a year and never had this issue, but I'm running out of places to look.

Might be time to finally get IPMI working.

I’m running funkwhale in docker. This consists of a half dozen docker containers one of which is postgres.

To run a backup, funkwhale suggests shutting down all of the containers and then docker compose running pg_dump on the postgres container. Presumably this is to copy the database when nobody is accessing it.

For some reason when I do this, I get an error like:

pg_dump: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
	Is the server running locally and accepting connections on that socket?

It would seem that postgres isn’t running. I see the same error with other commands such as psql.

If I fully boot the container and then try exec-ing the command, it works fine.

So it would seem that the run command isn’t fully booting the instance before running the command? What’s going on here?

The container is built from postgres:15-alpine

I'm moving my music library to a funkwhale instance, but I don't want to have to keep two copies of every song (one imported to Funkwhale, one on a local drive).

It looks like Funkwhale will let you download a single song at a time from your own library , but there doesn't seem to be a similar button for albums or playlists.

The files themselves are obfuscated in whatever indexing system it uses, so there's nothing to be done there.

Anyone know how this is possible?

Just got Whisper working on my local server so I can send it audio files via curl POST request and receive transcribed text.

Are there any keyboard plugins for phones that could be directed to a personal server running Whisper to replace functions like Siri/Google assistant voice transcription?

I'm hosting a few services using docker. For something like an openstreetmap tileserver, I'd like it to remain on my SSD because high speed improves performance, and the directory is unlikely to grow and fill the drive.

For other services like NextCloud, speed isn't as important as storage size, so I might want it on a larger HDD raid.

I know it's trivial to move the volumes directory to wherever, but can I move some volumes to one directory and some volumes to another?

You always hear about gun sales in the US, but you never hear about what happens to the guns at the end of their lifecycle. I assume guns wear out eventually, and I assume you can't just chuck them in the garbage when they do. What happens to them?

Upgrading a server for the first time in 10 years, so I’m a little out of the loop. I was surprised to find that the RAM I bought didn’t fit.

This is my first time dabbling in ECC RAM, so I figured there was some minor detail I missed when purchasing, but I eventually came across the data sheet for this stick, and the dimensions given don’t match the measurements I’m making. The tip of the caliper should be in the middle of the notch at 68.1mm.

What’s more is that the dimensions in the data sheet seem to match the dimensions on my motherboard. What’s going on here?

[SOLVED] I and Kingston are morons. I ordered RDIMM instead of UDIMM. The Kingston datasheet gives the wrong dimensions.

Since 2016, I've had a fileserver mostly just for backups. System is on 1 drive, RAID6 for files, and semi-annual cold backup.

I was playing with Photoprism, and their docs say "we recommend placing the storage folder on a local SSD drive for best performance." In this case, the storage folder holds basically everything but the pictures themselves such as the database files.

Up until now, if I lost any database files, it was just a matter of rebuilding them by re-indexing my photos or whatever, but I'm looking for something more robust since I'll have some friends/family using Pixelfed, Matrix, etc.

So my question is: Is it a valid strategy to keep database files on the SSD with some kind of nightly backup to RAID, or should I just store the whole lot on the RAID from the get go? Or does it even matter if all of these databases can fit in RAM anyway?

edit: I'm just now learning of ZFS caching which might be my answer.

I’m working on driving a very finicky lcd. I have it working now with an FPGA dev kit. I had to use an FPGA because some of the timing requirements are in the tens of nanoseconds.

At the end of the day, I wrote a block for a one shot/continuous clock with a programmable duty cycle and initial delay. This block was repeated six times for the various clocks with their specific values.

Moving to the final product, this feels like overkill. In the past, I’ve managed to make this kind of thing work with a Rube Goldberg collection of on-board timer/counters on the microcontroller.

I’d like to avoid that mess this time around. If I can generate the clocks externally, I can have the host MCU send the data quickly using DMA.

An FPGA works great, but they’re expensive and there’s the issue of licensing for FPGA and and CPLD software.

I’ve seen this problem solved with a lookup table, but there aren’t a lot of cheap/small rom/ram options for what I’m trying to do.

Basically, what I’m asking is is there a component that can be easily programmed to generate a number of clocks, doesn’t need any costly software licensing, and comes in a very small package? (Like wlcsp)