1. Can you share a link to the exact RGB cable that you are using? There are a lot of different csync cables. On the N64RGB board, are J1,J2, J3 open or closed and is CS75 or CS# used? Its probably a mismatch between cable and csync output.
2. The strange colors has something to do with the wires for the digital lines. Look here: https://shmups.system11.org/viewtopic.php?p=1266455#p1266455