I’d suggest trying a couple of different RGB-capable systems to rule out any issues with the OSSC.
Have a go at adjusting the Video LPF setting, Analog sync LPF setting and (if necessary) you can try playing about with the Sampling Phase setting to see if that fixes anything.
It could also be as simple as using a different SCART cable for the N64. Is it an NTSC or PAL (french NUS-001) modded console? NTSC ones can use sync-on-luma cables, whereas the PAL ones can’t (IIRC).