I did one more test: does interrupt driven IR reading work while updating neopixels and scrolling text on an RGBpanel.
The answer is “yes” 
I was a bit worried because the dirty ESP32 driver I contributed to GitHub - Arduino-IRremote/Arduino-IRremote: Infrared remote library for Arduino: send and receive infrared signals with multiple protocols does not use RMT but the same interrupt routine than the other chips. Thankfully it’s fast enough that it works most of the time (a few times you get back FFFFF instead of the key, but at least you don’t get random garbage back)
I then tested it on teensy 3.6, and it works fine there too.
So, the only sticking point are the interrupts from FastLED Neopixel vs SmartMatrix on teensy 3.6. That’s the only thing that doesn’t really work.