I started this project out of a need to run a set of NAND dump/flash tools, which happened to be developed for early Windows. Since my personal computer has a slow i5 with integrated graphics, running/virtualizing Windows 8-11 was out of the picture. Eventually I decided on emulating Windows Vista SP2 which would give me the high-performance I was looking for on my old hardware as well as be able to run the applications I needed quickly.
Almost immediately I ran into issues with QEMU/VirtIO. Since I would like at the very least basic hardware acceleration, almost every virtualization stack was out the window. That left me with either VMWare or mVisor. While mVisor did look promising, I found VMWare to be better documented and more easily accessible.
These are the HW specs I decided to configure my VM with, I found this to be more than enough to run Vista, however as long as it meets minimum requirements it should run perfectly fine.
Memory: 4GB
Processors: 4 (1x4)
HDD (SCSI): 128GB
Display: Acceleration Enabled (8GB vRAM)
USB Controller: 1.1 Compatibility
Sound Card: Auto Detect
For my installation media, I grabbed a signed copy of Windows Vista SP2 from Archive.org, I very much recommend grabbing a copy with SP2 pre-installed as it will save you from having to install it manually later. Link to ISO.
If you have a low-powered integrated graphics computer like I do, you may get a pop-up error from VMWare claiming that your host does not support 3d Effects/Accelleration. If you get this error, power off your VM, close out of VMWare and add the following lines to your vmware configuration file (typically ~/.config/vmware).
mks.enableMTLRenderer = "0"
mks.enableGLRenderer = "1"
mks.gl.allowBlacklistedDrivers = "TRUE"
# Add this line as well if the error still occurs
mks.enable3d = "TRUE"
Another issue I ran into specifically with Vista & ALSA was sound either not working at all, or heavily skipping/stuttering. In order to fix this issue, I change/add the following lines of your audio configuration in your VMs .vmx file. (If any of these entries already exist, modify them but make sure not to create duplicate entries.)
sound.virtualDev = "es1371"
sound.present = "TRUE"
pciSound.playBuffer = "80"
sound.maxLength = "2048"
sound.smallBlockSize = "1024"
sound.highPriority = "TRUE"
sound.bufferTime = "100"
sound.autoDetect = "TRUE"
sound.fileName = "-1"
Once you have attached your Vista installation ISO to your VM, go through the typical Windows installation process and make sure to select Vista Ultimate. This will give you the highest level of feature support, which is highly recommended if you want to have the best Vista experience. Once Vista is installed and you have booted into your new installation, detach the vista image from your VM.
While VMWare Workstation does have an option to automatically mount VMWare tools, disregard this. The latest version provided by VMWare does not actually support Vista. You can download the latest supported version of VMWare Tools 11.0.6 from the VMWare website. Run the installer and reboot your VM.
As you may quickly notice, while VMWare tools does install the "VMWare SVGA 3D" device drivers, the performance leaves a lot to be desired. This is because there is a memory bug in the SVGA3D driver included in VMWare tools versions past v10.3.10. In order to install the latest (and best performing) SVGA3D drivers, you will want to download VMWare Tools 10.3.10.
In order to extract and install the graphics drivers from this ISO, we will have to detach VMWare Tools v11.0.6 from our VM and attach v10.3.10 instead. You may get a pop-up in your VM to run the 10.3.10 installer; disregard this and navigate to the disc in file explorer instead. From here take note of the disc letter (i.e. E:\ or D:\)
Open an Administrator CMD prompt, and run the following command to cd into the disc: cd /d D:\ . Once you are in the disc directory, run setup64 /a to run the setup executable in extract mode. This will open the graphical installer window, and it will prompt you to select a Network Directory. Personally, I create a folder in my Downloads called TEMP or something similar and extract the files to there. Once the driver is installed, we can delete this folder.
Once the VMWare Tools files have been extracted to your temporary directory, you are safe to detach the ISO. From here, open Device Manager and navigate to Display Adapters. You should see an installed device named VMware SVGA 3D. Right click this device and click Uninstall, if the uninstaller gives you the option to "Delete the driver software for this device.", make sure to check this option. After the driver is deleted, refresh Device Manager and it should be replaced with a generic display adapter. Right click this adapter and select Update Driver. You can then select "Browse my Computer for Driver Software" and navigate to your new temp directory. From here, follow this path to the video_wddm folder and select it. "VMware/VMware Tools/VMware/Drivers/video_wddm/Vista".
From here, Vista should auto-detect the driver files contained within the folder and install them. You will have to reboot your VM once the drivers are completed installing in order to apply the changes. Once rebooted, you are safe to delete the temporary directory you created and all graphics acceleration capabilities should be enabled. You can run wglinfo to confirm OpenGL is in fact enabled and drivers are loaded.
Before installing Legacy Update and Extended Kernel, I recommend downloading and installing the latest Windows Server 2008 updates KB4474419 and KB971512. These updates download as executables, each one will restart the VM and go through the typical Windows update process. Once both of these updates are installed, download and install Windows Legacy Update in order to re-enable system updates and fix many of the broken applications included with Vista.
Once Legacy Update has been installed and you have installed all the Windows updates you'd like, download the Extended Kernel update for Windows. This installer will add extra kernel support to the Windows Vista kernel, allowing you to run many newer applications that do not work on SP2 by default.
If you have completed all the steps of this guide you should have a fully-featured
and performative Windows Vista installation. That said, there are some applications I recommend
downloading if you want to have a more modern user experience:
• r3dfox is a modern port of Firefox optimized for Windows Vista. It is much more
secure and performative than IE.
• 7zip v2501 is the latest version of 7zip with full Vista support.
I can guarantee this will be helpful, especially when downloading modern archives.
• Winamp v295 This should be self explanatory.
• mIRC while not my favorite irc client, is still fully featured on
windows Vista, and provides a solid user experience.
If you haven't already, you can enable the aero theme under Customization in Control Panel. Additionally, if you ran into any issues with features such as sound or IO not working after any of these steps, try running "sfc /scannow" in an Administrator cmd prompt.
While this project was originally inspired out of necessity to run legacy software, it quickly turned into a challenge to get the most out of my Vista VM. I will likely make a follow-up post in the near future going over the same installation process with mVisor instead, but I wouldn't hold your breath on it :P. While I haven't made any posts recently, I do still have many projects constantly in the works which I hope to share. I hope this guide provided some help, and as always I am always available to contact if you run into any issues :))