Have you already tried the settings here? This should help at least in the case of games using interlaced modes. OTOH, I see your mode seems to be 313p, which is progressive.
Are your Mega Drive and Master System unmodded systems? What games do the jitters occur with? Can you perhaps provide a Youtube video of what you’re seeing?