Weird panel layout

[Louis]

Try inverting the clock with this option:
SMARTMATRIX_OPTIONS_ESP32_INVERT_CLK

Try adding 5v level shifters for the signals if you haven’t already

Try lowering the clock speed:

Monochrome 32*16 P10 HUB12 LED matrix control with ESP32

@BodoMinea you might want to lower the clock speed to see if that helps: Apologies if I already suggested this or you already tried this, I have limited time to help out with these things and can’t keep all the details in my head.