@trixster: funny thing is that those tweaked vertical parameters are somewhat invalid as I noted in a follow-up post, and I think on older firmwares that would have resulted to DE being active during VSYNC. While you had it working, did the TV report the mode as 576p or 800×600?
@coderkind: Does your display support 240pX4? The signal should be very close to 480iX4 and 480pX2. I guess 1280×960 is not widely supported on TVs, you might have luck tweaking relevant modes to generate 1280×1024 output.