Could you try to remove MODE_PLLDIVBY2 in video_modes.c from the optimized modes to see if its more reliable?
Here is a method for auto phase from TI. http://e2e.ti.com/support/data_converters/videoconverters/f/376/p/284760/993535#993535
Basically though there are 2 steps…
1) Sweep the AVID start/stop so that the starting position starts sampling the video before the expected location then monitor the captured frame buffer until you see a pixel value other than black(ish) appear in the first column. Another way to effectively do this is to start capturing early then analyze the captured frame buffer to look for the first non black(ish) pixel, then subtract this offest from the start/stop registers so that the first non-black pixel then appears at the start location.
2) Sweep the sampling phase through all values whilst looking at the first non-black pixel. You should see the captured pixel value cycle through from black to the max pixel value. Do the same on the last pixel so that you find the corresponding max value to black. Then you will have a phase profile showing all phases and corresponding samples. You then set the sample phase to the setting which corresponds to the mid point of the max/min value (remember that this is basically a circle)