Inference is 3x Faster in Linux than in Windows [Video]

Recently, a developer shared how he switched from Windows to Linux after 30 years and saw remarkable results for AI-specific tasks. The person, who goes by the name Inevitable-Start-653, mentioned that he had six 24GB graphics cards, pushing the limits of what’s typically used in consumer-grade setups. 

As more GPUs were added to the system, the performance hit on Windows became increasingly noticeable. Despite using top-notch inferencing software like Oobabooga’s Textgen, the Windows operating system’s overhead proved to be a significant bottleneck. 

Attempts to mitigate this issue using Windows Subsystem for Linux (WSL) with DeepSpeed and even upgrading to PyTorch 2.2 failed to yield noticeable improvements in inferencing speeds. Once he transitioned to a dual-boot setup with Ubuntu Linux, the user reported a dramatic improvement in performance. 

Inferencing speeds increased by approximately 3x, more VRAM became available for context, and the overall system responsiveness improved significantly. 

This performance boost not only …