The Storage Array
Storage in the lab is split between two systems: the FreeNAS box for bulk media over NFS, and Longhorn on the k8s nodes for application config and state. They serve different purposes and don’t overlap.
Contents#
The Split#
Media (movies, TV, anime, music, audiobooks, downloads) lives on the FreeNAS box and gets served to the k8s cluster over NFS. Application data (configs, databases, app state) lives on Longhorn volumes replicated across the k8s nodes’ local NVMe drives.
This means losing a k8s node doesn’t lose app data (Longhorn replicates), and the NAS being down doesn’t affect app functionality, just media playback.
FreeNAS / NFS#
The FreeNAS box is a SuperMicro chassis running a single ZFS pool (Arr1) with 8 data drives:
| Drive | Model | Size |
|---|---|---|
| da0 | HGST HDN726050AL | 4.55 TiB |
| da1 | HGST HDS724040AL | 3.64 TiB |
| da2 | WDC WD40EFRX-68N | 3.64 TiB |
| da3 | WDC WD4000F9YZ-0 | 3.64 TiB |
| da4 | Seagate ST14000VN0008 | 12.73 TiB |
| da5 | Seagate ST14000VN0008 | 12.73 TiB |
| da6 | WDC WD201KFGX-68 | 18.19 TiB |
| da7 | WDC WD201KFGX-68 | 18.19 TiB |
Pool is at 81% used with 6.98 TiB free. lz4 compression, no dedup. NFS exports mount into the k8s cluster as static PVs over NFSv3. Full details in The FreeNAS Box.
Longhorn#
Longhorn runs on all 5 k8s nodes and uses their local 1TB NVMe drives for replicated block storage. It’s the default StorageClass for the cluster. Every app that needs persistent config (Sonarr, Radarr, Lidarr, Prowlarr, Sabnzbd, etc.) gets a Longhorn PVC.
Longhorn replaced Rook-Ceph, which I tried first and ripped out within a couple weeks. Ceph wants dedicated disks and nodes with headroom to run OSD daemons. On mini PCs with a single NVMe that also need to run application pods, it was fighting for resources constantly. Longhorn is lightweight, works with a single disk per node, and handles replication without a dedicated storage cluster. More on that in the cluster build post.
Why Not Just One#
NFS is great for large files that multiple pods need to read (media streaming). It’s terrible for small random I/O (databases, app configs). Longhorn is great for app state with replication. It’s overkill for serving a 40GB movie file to Plex.
The split keeps each system doing what it’s good at.
Upcoming Changes#
The two oldest drives in the NAS (da2 and da3, both 4TB WDs) are getting replaced with 16TB drives. That adds roughly 24TB of usable capacity and buys significant runway before the pool fills up again.