What the ESP32 API reports is misleading. It says you have 291772 bytes total in the heap, but some of that is 32-bit memory that can’t be used by SmartMatrix Library or most other libraries. Pay more attention to the 8-bit accessible memory, which you can consider “usable memory”.
SmartMatrix Library tries to allocate as much RAM as possible to refresh the panels. I’d call matrix.begin() as late as possible if you need to malloc other memory first.
If you switch to the circuit using the latch, you’ll double the efficiency of the RAM used for refreshing the panels. That may leave some more free RAM, or improve the refresh quality (brightness or color depth, can’t recall). I can’t recall if I added something to limit the amount of RAM that’s allocated for refreshing, seems like a good thing to add. e.g. matrix.begin(MAX_RAM)
There’s a THT version of the ESP32 SmartLED Shield that has the magic RAM-reducing Latch: SmartMatrix/extras/hardware/SmartLEDShield_ESP32_THT_V0_brd.pdf at teensylc · pixelmatix/SmartMatrix · GitHub
Long term, my fix for all these malloc issues isn’t in software, it’s to move refresh to a separate CPU. I’ve actually started this project. Trying to figure how to make the ESP32 receive SPI slave data (APA102 format) continuously is my current challenge.