For those of you using a recent beta of Windows Vista, including the just-released beta 2, you may notice severe problems when attempting to use VirtualDub with Aero Glass enabled -- specifically, that the display keeps resetting over and over.
The problem is that the way that VirtualDub displays video is incompatible with the Vista Desktop Window Manager (DWM), and the DWM disables Aero Glass in response. Unfortunately, some semi-recent changes to the DWM have made the problem much worse: when the DWM attempts to disable Aero Glass, it resets the display and that also causes VirtualDub to reset its display mechanisms, tripping the DWM again, and resulting in a loop of resets. Click on another window to escape the loop.
The workaround is to go to Options > Preferences > Display and either turn off DirectX or enable Direct3D. The technical issue is that VirtualDub uses the same window handle for GDI and DirectDraw rendering, which the DWM really doesn't like.
Credit:
VirtualDub.org