The problem is pixel change between frames even with a static image. Within a frame (inside the linebuffer) they are stable unless I introduced a bug with the read/write addressing and enables I haven’t found yet.
For example, I added a filter that outputs a red pixel if the compare fails and a green pixel if the compare succeeds. With the TVP input there are pixels that toggle between green and red on a static image. With a generated input done within the RTL (input is written into the linebuffer the same way as the TVP input) the image is stable and no pixels change from green to red.