Well I can verify that at least the 3.5 works. I hacked up a SmartMatrix Shield with female headers to space the Teensy off from the panel connector. This is a setup I’m using anyway (with 3.2) because I don’t want to lose an entire SmartMatrix should I somehow fry my Teensy.
I used Arduino 1.6.11 and Teensy Loader 1.30. Here’s some photos. Don’t judge me. 
Cheers,
-Vince