I know this thread is nearly a year old, but I’d like to see this as well. If at least some level of automatic mode detection can be implemented, it opens the door to automatic input switching. Please forgive me if I’m off-base with any of this.
Assuming that the OSSC can determine whether or not it has a valid sync given a sync type, I think a good starting point would be to set a per-input default for muxed sync, then start checking the H-sync/C-sync, V-sync (AV3 only), and G/Y lines on the current input for sync signals. In order of priority: if sync on H-sync and V-sync, assume RGBHV (AV3 only); if sync only on H-sync/C-sync, assume RGBS; and, if sync on G/Y, assume RGsB/YPbPr.
 Triggered by holding Button 0 for 1-2 seconds?
 Select between RGsB or YPbPr; system defaults could be YPbPr for AV2, RGsB for AV1 and AV3. (YPbPr over SCART and HD15 are less common, yes?)
 Perhaps tapping Button 0 when using auto-detect could toggle between RGsB and YPbPr? Remote could still manually set mode if needed.
Getting into automatic input switching, if the OSSC doesn’t find a sync signal after a timeout period (1 second?), it could start sequentially checking each input until it finds something.
Sure, it’s not elegant; but it’d be something until a better solution comes along.
As for trying to autodetect YPbPr or RGsB, would it be possible to check for quirks in the voltages? I’m not quite sure which standards are in use, and I’m probably reading this PDF incorrectly, but it looks like if the B/Pb or R/Pr voltages center on +350mV during the sync tip (or go negative during color data), the signal is probably YPbPr.