Which RGB mod do you have? My N64 has the simple RGB amp mod with luma sync cable. Do you connect it directly to the scart input of the OSSC?
After I found the best sharpness settings I tested various games. Mario 64, Cruis’n USA, Waverace, Zelda Majora’s mask, Starfox and the results stayed the same.
Can you upload a picture that shows the correct sharpness and the bleed that you get?