I saw my Z80 was still glitching, and I found the NMI input was floating... Hard to keep track of every single pin on a breadboard. Presumably, spurious interrupts messed it up, simply pull it high has solved the problem.
I polished the test a bit as well. Finally, my Z80 has passed the complete memtest, full 16 KiB (*) of RAM is now operational! Next step is adding a TMS9928A video output.
(*) it only shows 16368 bytes, 16 bytes of RAM was reserved for the test routine itself.