Well, that was months of work. Huge thanks to Louis for the work with his driver and animatedgifs that I heavily depended on.
Finally finished a page with more build info and the up to date code that runs on it: https://github.com/marcmerlin/NeoMatrix-FastLED-IR
Clubbing, EDM Festival and Burning Man LED Pants and Shirt v4 on ESP32 with RGBPanels and SmartMatrix::GFX:
More photos: https://photos.app.goo.gl/GAEQ9zHwUvF14vJH8