New (Quiet!) Home Lab
My (now) previous home lab was an old IBM System x3650 M3 (2 x Xeon E5645 processors, 128GB of RAM, 2.2 TB of SAS 10K disks in RAID 5), but the menacing glances I got every time I turned it on meant it was time for a replacement…
The server hosted ESXi 6.7, and ran approx. 15 VMs directly, with a further 6 or so nested VMs across 3 virtualsed ESXi 6.7 hosts, all glued together with a VyOS VM for routing, TrueNAS for virtualised shared SAN storage across the 3 ESXi hosts, and a Palo Alto VM as a gateway to the outside world and my home network.
It ran fine, except for the noise/cost of running it (surprise surprise), and the slight lagginess when connecting to the various VM desktops, which I suspected was due to a combination of disk speed and having to remote into VMs over a 100M link. I also suspected the read/write performance of the nested VMs in particular was causing issues; their path after all was VM -> Nested ESXi datastore -> TrueNAS virtualised disks -> ESXi datastore -> physical disk.
I just finished building a workstation to replace this server as my new home lab, while also serving as a general purpose workstation (and maybe a bit of gaming once video cards become available again!). It’s running an i9-9900KF 3.6Ghz, 128GB DDR4 3200Mhz RAM, and a 2TB SSD dedicated to lab VM storage. It has a Scythe FUMA 2 CPU cooler, and housed in a Lian Li LANCOOL II Mesh case.
All VMs were migrated across, but are now running under a local install of VMware Workstation, instead of a remote ESXi host. Happy to say that the new PC has exceeded my expectations in terms of lab performance. It’s much snappier and responsive, with plenty of resources left to run additional VMs and programs running on my desktop, and probably most importantly, is absolutely whisper quiet.
I love the suspend feature of VMware Workstation; in my previous lab it could take 15-20 minutes for the server to boot up and all the VMs to auto-start in correct sequence. With Workstation Suspect/Resume, I’m back where I left off within 2-3 minutes. It also means I’m not continually shutting down/starting up VMs between lab sessions, which in itself causes problems.
The only drawback so far as been losing the virtual networking flexibility of the ESXi host; with vSwitches, it was possible to create VLANs and trunks to quite accurately simulate a real SMB network. With Workstation, not so much. It’s not possible to create VLANs (that I could find, bar some unwieldy workaround involving additional network interfaces that are VLAN-tagging capable), and instead has the notion of ‘LAN Segments’, which are essentially hubs. It works, in so far that it allows for segmentation of different networks, but it’s just not in any way relatable to configuring a real life VLAN-segmented network.
Some quick benchmarks below showing the performance differences, which are night and day. These are taken with PassMark, running on the same VM across both labs while running all VMs.
Old Lab PassMark:
New Lab PassMark:
Old Lab DiskMark:
New Lab DiskMark:
