I don’t know why – but it seems like i only have this problem, while using the modified ossc_0.77-aud-wextmclk firmware from ‘borti4938’ for feeding the MCLK directly into the IT6613E. With this firmware the 256×240/320×240 optimized modes wont work either (there’s just a black screen with an occasional flash of the video signal).
But the ossc_0.77-aud works fine though:
The optimized modes are working too. There’s just some jitter… but (like it’s written in the wiki) adjusting the sampling phase fixes it.