That cable is not suitable for your console. Its for NTSC consoles and has a sync stripper inside which converts composite video to csync. Its important to know how the N64RGB is configured. If you can’t open up the console then ask your modder about the J1-J3 jumpers and if CS75 or CS# is connected and to which pin on the multi-out.
The post that I linked to said he used shorter wires for the digital lines and removed the fine pitch adapter and that solved the color problem so the RGB mod would have to be redone.
3. Maybe some PAL games use the full 288 lines in which case you need to set V.active to 288 via the Advanced timing tweaker.