An additional fix is needed to slow the matrix clock down at the higher speeds on the Teensy 3.5/3.6. I found that glitches start to show up at 140MHz, at 168MHz they're particularly bad, and at 180MHz they're a little worse than 140MHz. There's an option to stall a DMA channel for as few as 4 cycles after each read, which slows the transfer rate so it's a consistent 3.28us to transfer 32 pixels at 180MHz. There's already some code to enable this that's commented out, you can find this in SmartMatrix_Impl.h:
// for debugging - enable bandwidth control (space out GPIO updates so they can be seen easier on a low-bandwidth logic analyzer)
dmaClockOutData.TCD->CSR |= (0x02 << 14);
I'm seeing glitch free GIFs at 140, 168, and 180MHz with these two changes. I'll package this up unto the next release, hopefully shortly.