2025-02-27 - Another i/o Error for storage01

Another disk issue on storage01 cropped up, and I haven’t even finished documenting the fault that happened last February 21, 2025. This time, it’s sdb1 acting up. Thankfully, I was able to resolve the issue without any data loss, but it was a close call. Incident I/O Error on /dev/sdb1 during scrub operation. Severity High Impact Data corruption on /dev/sdb1, potential data loss sdb1 is one branch of the mergerfs volume served by storage01, losing access to sdb1 effectively halves the usable capacity of the Network File System Affected Systems storage01 Timeline 2025-02-27: I/O errors detected during snapraid scrub operation. 2025-02-27: XFS filesystem on /dev/sdb1 shuts down due to log I/O errors. 2025-03-01: Troubleshooting steps initiated to identify the root cause. Logs from dmesg [145729.843018] sd 1:0:0:10: [sdb] tag#84 Sense Key : Aborted Command [current] [145729.843019] sd 1:0:0:10: [sdb] tag#84 Add. Sense: I/O process terminated [145729.843021] sd 1:0:0:10: [sdb] tag#84 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00 [145729.843023] I/O error, dev sdb, sector 0 op 0x1:(WRITE) flags 0x800 phys_seg 0 prio class 2 [145729.850513] sd 1:0:0:10: [sdb] tag#201 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=0s [145729.850517] sd 1:0:0:10: [sdb] tag#201 Sense Key : Aborted Command [current] [145729.850518] sd 1:0:0:10: [sdb] tag#201 Add. Sense: I/O process terminated [145729.850519] sd 1:0:0:10: [sdb] tag#201 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00 [145729.850540] I/O error, dev sdb, sector 3909372392 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 2 [145729.850549] XFS (sdb1): log I/O error -5 [145729.850569] XFS (sdb1): Log I/O Error (0x2) detected at xlog_ioend_work+0x6e/0x70 [xfs] (fs/xfs/xfs_log.c:1378). Shutting down filesystem. [145729.850749] XFS (sdb1): Please unmount the filesystem and rectify the problem(s) Detection Alert from Discord

Resolving Reverse Proxy Host Header Issues

I was having an issue with my website. When I used my browser to navigate to https://alain.apigban.com, there was nothing displayed. The setup is like this: User > Frontend proxy (Caddy) > Backend HTTP Server (Caddy) The frontend proxy receives the request, and here’s one line from the frontend proxy access logs: {"level":"debug", "ts":1740849407.8355885,"logger": "http.handlers.reverse_proxy", "msg":"upstream roundtrip", "upstream": "backend_server_1_IP:10000", "duration": 0.01451143, "request": {"remote_ip": "client_IP", "remote_port": "56466","client_ip": "client_IP", "proto": "HTTP/2.0", "method": "GET", "host": "alain.apigban.com", "uri":"/posts/ansible/02/netbox-source-of-truth/", "headers": {"Priority":["u=0, i"], "Sec-Fetch-Mode": ["navigate"], "X-Forwarded-For":["client_IP"], "Sec-Fetch-User": ["?1"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Dest": ["document"], "X-Forwarded-Host": ["alain.apigban.com"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0"], "Accept-Language": ["en-US,en;$q=0.5"], "Sec-Fetch-Site": ["none"], "X-Forwarded-Proto": ["https"], "Te": ["trailers"], "Accept": ["text/html,application/xhtml+xml,application/xml;$q=0.9,*/*;$q=0.8"]},"tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "alain.apigban.com"}},"headers": {"Date": ["Sat, 01 Mar 2025 17:16:47 GMT"], "Content-Length":["0"], "Server": ["Caddy"]},"status":200}