I can’t find my 3.5, but I just tried with a 3.6. I get similar results as you. 64x128 and 128x64 both worked but not 128x128. I tried lowering
kRefreshDepth to 24, which does make FeatureDemo work at 128x128, but just barely. I’m not sure exactly where the bottleneck is, but the Teensy 3.6 is having a hard time refreshing 128x128 pixels at once. The refresh rate is automatically lowered when the Teensy can’t keep up, and in my case, it got automatically lowered to 47FPS (default is 128FPS), and I see significant flicker. The 3.5 will probably fare worse than the 3.6.
I tried a simpler sketch - Bitmaps - and it refreshes at 96FPS with 24-bit color. I know the scrolling text layer isn’t as efficient as it could be, and the Bitmaps sketch doesn’t have that layer. The indexed layer might also be a bit inefficient as well.
There’s another hack you can do to squeeze a bit more out of the SmartMatrix Library:
comment out this line in SmartMatrix_Impl.h:
// also enable for now, until it can be selectively enabled for higher clock speeds (140MHz+) where the data rate is too high for the panel
//dmaClockOutData.TCD->CSR |= (0x02 << 14);
replace this line in MatrixHardware_KitV4 (halving the time waited for data to shift out):
#define PANEL_32_PIXELDATA_TRANSFER_MAXIMUM_NS (uint32_t)((3400 * 96000000.0) / F_CPU)
This hack may not work in all cases, e.g. if you’re using a SD card for AnimatedGIFs, there’s likely to be DMA conflicts that will show up as artifacts on the LED screen. With the Bitmaps sketch, I get 104FPS after making that change (using the 3.6).
Sorry this isn’t working. You’re on the cutting edge of SmartMatrix projects, I haven’t even tried these configurations myself. If you can share a bit about what you’re trying to do with your project, I can try to point you in the right direction. For this amount of pixels you might be better off driving from a Raspberry Pi with this shield, though I don’t know if 128x128 is supported.