@netmindz I remember you from the FastLED group.
Ok, I know I’m pushing my stuff, but seriously, switch to
GitHub - marcmerlin/FastLED_NeoMatrix: Adafruit GFX compatible library for NeoPixel based Matrices using FastLED library 
It supports your XY function if it’s special, or will do the mapping for you otherwise.
You can have a look at
FastLED_NeoMatrix_SmartMatrix_LEDMatrix_GFX_Demos/FastLED at a98cb3e965433a8eebb8893d630735fda12469cb · marcmerlin/FastLED_NeoMatrix_SmartMatrix_LEDMatrix_GFX_Demos · GitHub
for example code using XY and direct fastled poking without using GFX and drawpixel.
Once you’ve switched to that, you can substitute for GitHub - marcmerlin/SmartMatrix_GFX: SmartMatrix: Allow using Adafruit::GFX as well as FastLED_NeoMatrix code with SmartMatrix backend, and your same code will just work.
All the magic is in https://github.com/marcmerlin/FastLED_NeoMatrix_SmartMatrix_LEDMatrix_GFX_Demos/blob/a98cb3e965433a8eebb8893d630735fda12469cb/neomatrix_config.hhttps://github.com/marcmerlin/FastLED_NeoMatrix_SmartMatrix_LEDMatrix_GFX_Demos/blob/a98cb3e965433a8eebb8893d630735fda12469cb/neomatrix_config.h
it’s a big file, but it’s mostly ifdefs that select the right backend.