Working around sync offsets and flicker when filming/taking pictures


#1

I split this into a separate thread because ultimately, it’s its own problem (originally in the ESP32 port thread SmartMatrix Library ESP32 Port )

So far I had actively avoided using RGBpanels because I knew from experience of seeing others, that they look bad when taking pictures/filming them. That said, neopixels which don’t suffer from those problems, are between hard and impossible to find in P5 or below, and expensive when you’re getting something like 96x64 or higher, so I thought I’d try RGBpanels after finding some flexible ones (it’s for a wearable).

Here is a demo of teensy v3.6 (top) vs ESP32 direct wiring (bottom)

This was taken with my cell phone. Another phone or another camera would probably have yielded slightly different results.

The artifacts when taking a video, are terrible on both :frowning:
I thought ESP32 refresh rate was not going to change with the code that was being run since one core shouldn’t affect the other one.
Sadly both demos clearly show a refresh sync changing depending on what’s being displayed and generally looking terrible on video, and also shows that the refresh rate with the ESP32 code, seems not to be quite constant either and depends on what code/calculations are being run, it seems.

I’m assuming I’m not the first one to see this. Are there are tips/workarounds around it, or do I have to accept that if I use RGBpanels, they’re going to look like crap when filmed?
(yes, I understand that I can tweak a proper camera to change the sync rate, but in a night club when I’m not the one taking pictures, or people are using cell phones (most common), or appearing on film, I have no control over that and Ideally it would “just work”)


#2

Try SpriteTMs example code, which refreshes full frames a lot faster than SmartMatrix. It uses a very different technique to generate color depth, using multiple frames instead of multiple passes through the same line Perhaps the other ESP32 library you found also does.

Also try increasing the SmartMatrix refresh rate by a lot, Eg 240 instead of 120. You can decouple the frame rate from the refresh rate to decrease CPU usage, I think I documented this in the README. (On my phone now and can’t easily look)


#3

Hi Louis. Just landed in Paris so I’ll have a bit of time to play with this after the family festivities. Best wishes to you and your family.


I never saw that, SmartMatrix does PWM by doing one line at a time before doing the next one? That would explain why I’m seeing this, even at 120Hz refresh (sadly cameras are much faster than human eyes and pick this up more easily). Indeed when I wrote my simpler 8x8 matrix PWM driver, I did multiple full frame refreshes http://marc.merlins.org/perso/arduino/post_2015-01-06_Driver-for-direct-driving-single-to-3-color-LED-Matrices-with-software-PWM.html
By the way if anyone reads this and wonders what it looks like in slow motion, see the https://youtu.be/9yGZLtewmfI?t=18 it shows a PWM refresh in slow motion and then sped up to normal speed (it’s much slower than Louis’ code though)

I think for SpriteTM example code, you meant https://github.com/pixelmatix/esp32_I2sParallelDmaLedMatrix If so, I’ll try it, as well as https://github.com/mrfaptastic/ESP32-RGB64x32MatrixPanel-I2S-DMA
will report back.

Thanks, Marc


#4

Quick report:

  1. I couldn’t get https://github.com/pixelmatix/esp32_I2sParallelDmaLedMatrix to build. I’m sure I could figure out why eventually but given that it doesn’t use the arduino environment, I couldn’t use it anyway

  2. https://github.com/mrfaptastic/ESP32-RGB64x32MatrixPanel-I2S-DMA works fine but only at that size. If I change to 64x64 it crashes (filed an issue with the author). Visually, it looks fine, but eventually when filming it also gets into a state where there are bars.
    https://www.youtube.com/watch?v=MZSxv4Knk0c
    You can tell by the artifacts that PWM is done on a per page refresh basis and not per line, just like you said.

  3. changing the refresh in your code, is that adding matrix.setRefreshRate(240) ? I tried to set it to 30, 60, 120, 240. I notice it changes the speed of scrolling text, so it seems to have some impact (slowing down the main code) but visually I’m not sure if I see the difference.
    Also, I’m confused by this comment that says it’s not supposed to work
    scrollingLayer.start(“Changing Refresh Rate is Currently Broken in ESP32”, 1);

The bars in video come and go depending on available lighting in the room (since it changes the shutter/exposure rate of the camera) and also depend on what’s being animated (which surprises me a bit more)
I’m starting to believe that maybe they are unavoidable given the scan/refresh technology and that I should either accept it or go back to neopixels.


#5

I forgot that this feature is broken. Change it manually here:

You might have to change minRefreshRate as well, I’m can’t remember what that does at the moment.


#6

Thanks @Louis changing both to 240 did noticeably help with the recording problem, at least when it’s dark around, forcing the shutter to stay open longer (which also helps of course). I think in full daylight, it would still fail, but I’m only going to use this in the dark, so I don’t care too much.

Now, I only need to resolve my passing matrix definitions by argument problem :slight_smile:


#7

Actually, given that refresh is done by the 2nd core on ESP32, is there a reason not to run it at 240 or even 480 or 960Hz by default in the code (whatever the ESP will allow)?


#8

given that refresh is done by the 2nd core on ESP32

Most everything is done on the 2nd core on the ESP32. There’s some system code that runs on the 1st core, but everything you do in your Arduino sketch - unless there’s some special multi-core library out there - runs on the 2nd core

is there a reason not to run it at 240 or even 480 or 960Hz by default in the code

At a high refresh rate, the display will be dimmer at higher color depth as it takes longer to shift out some of the data than the data is actually lighting the LEDs.

Also, at a high refresh rate, it may not be possible to display the full color depth as at a certain point the time to light the LEDs is shorter than one I2S module clock cycle on the ESP32. This is a problem the way SmartMatrix displays data a row at a time, but there might be a way to do it using the method Sprite_TM uses to do full frames at a time. I want to investigate this.


#9

Yes, FastLED on ESP32 runs on both cores I believe. Have a look at


and

Thanks for the reminder about higher refresh = dimmer. You did write that in your docs and I had forgotten about it.
Still, I think I’ll take 240 or 480Hz and dimmer as long as it looks ok on pictures and without the damaged frames that I’ve been having otherwise (for camera, not the naked eye which is much slower).