Here is a better demo of teensy v3.6 (top) vs ESP32 direct wiring (bottom)
the demo mostly works, you can see a bit of ghosting on the white font (strangely no ghosting when I use 64x64 panel output instead of 64x32). Strangely too the first background is dark red by default with teensy, but full black with ESP32.
On the bright side, ESP32 is 99% working for me out of the box with direct wiring, so thank you for that 
I originally had some thoughts/questions about sync rate changing and how to not make it look so bad when filming, but I split that to a separate thread: Working around sync offsets and flicker when filming/taking pictures