The horizontal samplerate should be higher than active pixels because it includes sync and blanking. You can calculate the samplingrate with (dot clock / line count / refresh rate) so in this case (1260/77 * 1000000 / 261 / 60.14) = ~1042.5
So with line2X mode the H.samplerate should be around 1042. H.active 852 or 640 if you want to crop the overscan. You need to change H.synclen and H.backporch to center the image.
With line4X mode at generic 4:3, H.samplerate needs to be doubled so around 2085. H.active 1704.
With ‘Allow TVP HPLL2x’ On the sampling phase will be unstable so you’ll want to keep it off for pixel perfect sampling rates.