Working around sync offsets and flicker when filming/taking pictures


#1

I split this into a separate thread because ultimately, it’s its own problem (originally in the ESP32 port thread SmartMatrix Library ESP32 Port )

So far I had actively avoided using RGBpanels because I knew from experience of seeing others, that they look bad when taking pictures/filming them. That said, neopixels which don’t suffer from those problems, are between hard and impossible to find in P5 or below, and expensive when you’re getting something like 96x64 or higher, so I thought I’d try RGBpanels after finding some flexible ones (it’s for a wearable).

Here is a demo of teensy v3.6 (top) vs ESP32 direct wiring (bottom)

This was taken with my cell phone. Another phone or another camera would probably have yielded slightly different results.

The artifacts when taking a video, are terrible on both :frowning:
I thought ESP32 refresh rate was not going to change with the code that was being run since one core shouldn’t affect the other one.
Sadly both demos clearly show a refresh sync changing depending on what’s being displayed and generally looking terrible on video, and also shows that the refresh rate with the ESP32 code, seems not to be quite constant either and depends on what code/calculations are being run, it seems.

I’m assuming I’m not the first one to see this. Are there are tips/workarounds around it, or do I have to accept that if I use RGBpanels, they’re going to look like crap when filmed?
(yes, I understand that I can tweak a proper camera to change the sync rate, but in a night club when I’m not the one taking pictures, or people are using cell phones (most common), or appearing on film, I have no control over that and Ideally it would “just work”)


#2

Try SpriteTMs example code, which refreshes full frames a lot faster than SmartMatrix. It uses a very different technique to generate color depth, using multiple frames instead of multiple passes through the same line Perhaps the other ESP32 library you found also does.

Also try increasing the SmartMatrix refresh rate by a lot, Eg 240 instead of 120. You can decouple the frame rate from the refresh rate to decrease CPU usage, I think I documented this in the README. (On my phone now and can’t easily look)


#3

Hi Louis. Just landed in Paris so I’ll have a bit of time to play with this after the family festivities. Best wishes to you and your family.


I never saw that, SmartMatrix does PWM by doing one line at a time before doing the next one? That would explain why I’m seeing this, even at 120Hz refresh (sadly cameras are much faster than human eyes and pick this up more easily). Indeed when I wrote my simpler 8x8 matrix PWM driver, I did multiple full frame refreshes http://marc.merlins.org/perso/arduino/post_2015-01-06_Driver-for-direct-driving-single-to-3-color-LED-Matrices-with-software-PWM.html
By the way if anyone reads this and wonders what it looks like in slow motion, see the https://youtu.be/9yGZLtewmfI?t=18 it shows a PWM refresh in slow motion and then sped up to normal speed (it’s much slower than Louis’ code though)

I think for SpriteTM example code, you meant https://github.com/pixelmatix/esp32_I2sParallelDmaLedMatrix If so, I’ll try it, as well as https://github.com/mrfaptastic/ESP32-RGB64x32MatrixPanel-I2S-DMA
will report back.

Thanks, Marc


#4

Quick report:

  1. I couldn’t get https://github.com/pixelmatix/esp32_I2sParallelDmaLedMatrix to build. I’m sure I could figure out why eventually but given that it doesn’t use the arduino environment, I couldn’t use it anyway

  2. https://github.com/mrfaptastic/ESP32-RGB64x32MatrixPanel-I2S-DMA works fine but only at that size. If I change to 64x64 it crashes (filed an issue with the author). Visually, it looks fine, but eventually when filming it also gets into a state where there are bars.
    https://www.youtube.com/watch?v=MZSxv4Knk0c
    You can tell by the artifacts that PWM is done on a per page refresh basis and not per line, just like you said.

  3. changing the refresh in your code, is that adding matrix.setRefreshRate(240) ? I tried to set it to 30, 60, 120, 240. I notice it changes the speed of scrolling text, so it seems to have some impact (slowing down the main code) but visually I’m not sure if I see the difference.
    Also, I’m confused by this comment that says it’s not supposed to work
    scrollingLayer.start(“Changing Refresh Rate is Currently Broken in ESP32”, 1);

The bars in video come and go depending on available lighting in the room (since it changes the shutter/exposure rate of the camera) and also depend on what’s being animated (which surprises me a bit more)
I’m starting to believe that maybe they are unavoidable given the scan/refresh technology and that I should either accept it or go back to neopixels.


#5

I forgot that this feature is broken. Change it manually here:

You might have to change minRefreshRate as well, I’m can’t remember what that does at the moment.


#6

Thanks @Louis changing both to 240 did noticeably help with the recording problem, at least when it’s dark around, forcing the shutter to stay open longer (which also helps of course). I think in full daylight, it would still fail, but I’m only going to use this in the dark, so I don’t care too much.

Now, I only need to resolve my passing matrix definitions by argument problem :slight_smile:


#7

Actually, given that refresh is done by the 2nd core on ESP32, is there a reason not to run it at 240 or even 480 or 960Hz by default in the code (whatever the ESP will allow)?


#8

given that refresh is done by the 2nd core on ESP32

Most everything is done on the 2nd core on the ESP32. There’s some system code that runs on the 1st core, but everything you do in your Arduino sketch - unless there’s some special multi-core library out there - runs on the 2nd core

is there a reason not to run it at 240 or even 480 or 960Hz by default in the code

At a high refresh rate, the display will be dimmer at higher color depth as it takes longer to shift out some of the data than the data is actually lighting the LEDs.

Also, at a high refresh rate, it may not be possible to display the full color depth as at a certain point the time to light the LEDs is shorter than one I2S module clock cycle on the ESP32. This is a problem the way SmartMatrix displays data a row at a time, but there might be a way to do it using the method Sprite_TM uses to do full frames at a time. I want to investigate this.


#9

Yes, FastLED on ESP32 runs on both cores I believe. Have a look at


and

Thanks for the reminder about higher refresh = dimmer. You did write that in your docs and I had forgotten about it.
Still, I think I’ll take 240 or 480Hz and dimmer as long as it looks ok on pictures and without the damaged frames that I’ve been having otherwise (for camera, not the naked eye which is much slower).


#10

Hi there, I’ve been prototyping a new (?) refresh technique that degrades more gracefully as the camera shutter speed increases. Here are my test results shot in portrait mode with an iPhone 7:

Normal scan method, 240 Hz iPhone Slo-mo
New scan method, 240 Hz iPhone Slo-mo

Cheers,

Steve West


#11

(Couldn’t link this in the previous post)
New scan method, 30 Hz iPhone


#12

Very nice, thanks Steve.
Is that on teensy? ESP32?


#13

ESP32


#14

Great, would love to see your teensylc branch. I’m still working on my ESP32 smartmatrix shirt (64x96) and limiting artifacts for pictures would be awesome.


#15

Hi Marc, happy to share but my current prototype is limited by DMA_MAX for each bitplane. I also don’t (yet) have an external latch so haven’t tested that at all.


#16

I’m using direct wiring like you, so that’s not a problem :slight_smile:
What is the DMA_MAX limit? Number of pixels?


#17

Steve emailed me his code and gave me permission to share here:


#18

Yes Marc, the limit was 2046 direct wired pixels. But it’s okay, I’ve already fixed that limitation. Here is the updated source SmartMatrixLibrary-FlickerReducing-SteveWest-20190322.zip


#19

Thanks @seratosteve My big party where I’ll be trying my new 12,288 rgbpixel shirt (half the pixels are mirrored, so I only use half of that for memory), is tomorrow, so I’m trying to stabilize what I have, but I’ll definitely look at your code when I can.
May I recommend you put it on github so that it’s easier to see your progress as well as diff?
Also, for external latch, do you have a pointer to that setup so that I can read more about it/see how it works and why it’s more desirable?


#20

It’s the circuit that’s in the SmartLED Shield ESP32 V0 boards (and very close to the circuit that’s in SmartLED Shield V4). It uses an external flip flop to hold the ADDX lines, saving GPIO pins and RAM (data for each CLK edge can be stored in uint8_t instead of uint16_t)

I noticed that in pictures of your LED shirt you posted here and on your blog, there’s no noticeable scanning. How’d you work around that issue?