Working around sync offsets and flicker when filming/taking pictures

marcmerlin · December 23, 2018, 12:09am

I split this into a separate thread because ultimately, it’s its own problem (originally in the ESP32 port thread SmartMatrix Library ESP32 Port - #44 by marcmerlin )

So far I had actively avoided using RGBpanels because I knew from experience of seeing others, that they look bad when taking pictures/filming them. That said, neopixels which don’t suffer from those problems, are between hard and impossible to find in P5 or below, and expensive when you’re getting something like 96x64 or higher, so I thought I’d try RGBpanels after finding some flexible ones (it’s for a wearable).

Here is a demo of teensy v3.6 (top) vs ESP32 direct wiring (bottom)

This was taken with my cell phone. Another phone or another camera would probably have yielded slightly different results.

The artifacts when taking a video, are terrible on both
I thought ESP32 refresh rate was not going to change with the code that was being run since one core shouldn’t affect the other one.
Sadly both demos clearly show a refresh sync changing depending on what’s being displayed and generally looking terrible on video, and also shows that the refresh rate with the ESP32 code, seems not to be quite constant either and depends on what code/calculations are being run, it seems.

I’m assuming I’m not the first one to see this. Are there are tips/workarounds around it, or do I have to accept that if I use RGBpanels, they’re going to look like crap when filmed?
(yes, I understand that I can tweak a proper camera to change the sync rate, but in a night club when I’m not the one taking pictures, or people are using cell phones (most common), or appearing on film, I have no control over that and Ideally it would “just work”)

Louis · December 23, 2018, 6:20am

Try SpriteTMs example code, which refreshes full frames a lot faster than SmartMatrix. It uses a very different technique to generate color depth, using multiple frames instead of multiple passes through the same line Perhaps the other ESP32 library you found also does.

Also try increasing the SmartMatrix refresh rate by a lot, Eg 240 instead of 120. You can decouple the frame rate from the refresh rate to decrease CPU usage, I think I documented this in the README. (On my phone now and can’t easily look)

marcmerlin · December 24, 2018, 7:02pm

Hi Louis. Just landed in Paris so I’ll have a bit of time to play with this after the family festivities. Best wishes to you and your family.

I never saw that, SmartMatrix does PWM by doing one line at a time before doing the next one? That would explain why I’m seeing this, even at 120Hz refresh (sadly cameras are much faster than human eyes and pick this up more easily). Indeed when I wrote my simpler 8x8 matrix PWM driver, I did multiple full frame refreshes http://marc.merlins.org/perso/arduino/post_2015-01-06_Driver-for-direct-driving-single-to-3-color-LED-Matrices-with-software-PWM.html
By the way if anyone reads this and wonders what it looks like in slow motion, see the https://youtu.be/9yGZLtewmfI?t=18 it shows a PWM refresh in slow motion and then sped up to normal speed (it’s much slower than Louis’ code though)

I think for SpriteTM example code, you meant GitHub - pixelmatix/esp32_I2sParallelDmaLedMatrix: Building on Sprite_TM's ESP32 Parallel DMA example that refreshed an LED Matrix If so, I’ll try it, as well as GitHub - mrfaptastic/ESP32-HUB75-MatrixPanel-DMA: An Adafruit GFX Compatible Library for the ESP32, ESP32-S2, ESP32-S3 to drive HUB75 LED matrix panels using DMA for high refresh rates. Supports panel chaining.
will report back.

Thanks, Marc

marcmerlin · December 25, 2018, 11:27am

Quick report:

I couldn’t get GitHub - pixelmatix/esp32_I2sParallelDmaLedMatrix: Building on Sprite_TM's ESP32 Parallel DMA example that refreshed an LED Matrix to build. I’m sure I could figure out why eventually but given that it doesn’t use the arduino environment, I couldn’t use it anyway
GitHub - mrfaptastic/ESP32-HUB75-MatrixPanel-DMA: An Adafruit GFX Compatible Library for the ESP32, ESP32-S2, ESP32-S3 to drive HUB75 LED matrix panels using DMA for high refresh rates. Supports panel chaining. works fine but only at that size. If I change to 64x64 it crashes (filed an issue with the author). Visually, it looks fine, but eventually when filming it also gets into a state where there are bars.
https://www.youtube.com/watch?v=MZSxv4Knk0c
You can tell by the artifacts that PWM is done on a per page refresh basis and not per line, just like you said.
changing the refresh in your code, is that adding matrix.setRefreshRate(240) ? I tried to set it to 30, 60, 120, 240. I notice it changes the speed of scrolling text, so it seems to have some impact (slowing down the main code) but visually I’m not sure if I see the difference.
Also, I’m confused by this comment that says it’s not supposed to work
scrollingLayer.start(“Changing Refresh Rate is Currently Broken in ESP32”, 1);

The bars in video come and go depending on available lighting in the room (since it changes the shutter/exposure rate of the camera) and also depend on what’s being animated (which surprises me a bit more)
I’m starting to believe that maybe they are unavoidable given the scan/refresh technology and that I should either accept it or go back to neopixels.

Louis · December 25, 2018, 10:30pm

I forgot that this feature is broken. Change it manually here:

github.com

pixelmatix/SmartMatrix/blob/teensylc/src/SmartMatrixMultiplexedRefreshEsp32_Impl.h#L113


      
          template <int refreshDepth, int matrixWidth, int matrixHeight, unsigned char panelType, unsigned char optionFlags>
          void SmartMatrix3RefreshMultiplexed<refreshDepth, matrixWidth, matrixHeight, panelType, optionFlags>::setMatrixCalculationsCallback(matrix_calc_callback f) {
              setShiftCompleteCallback(f);
              matrixCalcCallback = f;
          }
          
          template <int refreshDepth, int matrixWidth, int matrixHeight, unsigned char panelType, unsigned char optionFlags>
          void SmartMatrix3RefreshMultiplexed<refreshDepth, matrixWidth, matrixHeight, panelType, optionFlags>::setBrightness(uint8_t newBrightness) {
          }
          
          template <int refreshDepth, int matrixWidth, int matrixHeight, unsigned char panelType, unsigned char optionFlags>
          void SmartMatrix3RefreshMultiplexed<refreshDepth, matrixWidth, matrixHeight, panelType, optionFlags>::setRefreshRate(uint16_t newRefreshRate) {
              if(newRefreshRate > MIN_REFRESH_RATE)
                  minRefreshRate = newRefreshRate;
              else
                  minRefreshRate = MIN_REFRESH_RATE;
          }
          
          template <int refreshDepth, int matrixWidth, int matrixHeight, unsigned char panelType, unsigned char optionFlags>
          uint16_t SmartMatrix3RefreshMultiplexed<refreshDepth, matrixWidth, matrixHeight, panelType, optionFlags>::getRefreshRate(void) {
              return refreshRate;

You might have to change minRefreshRate as well, I’m can’t remember what that does at the moment.

marcmerlin · December 28, 2018, 6:46pm

Thanks @Louis changing both to 240 did noticeably help with the recording problem, at least when it’s dark around, forcing the shutter to stay open longer (which also helps of course). I think in full daylight, it would still fail, but I’m only going to use this in the dark, so I don’t care too much.

Now, I only need to resolve my passing matrix definitions by argument problem

marcmerlin · December 29, 2018, 12:30am

Actually, given that refresh is done by the 2nd core on ESP32, is there a reason not to run it at 240 or even 480 or 960Hz by default in the code (whatever the ESP will allow)?

Louis · January 2, 2019, 3:16pm

given that refresh is done by the 2nd core on ESP32

Most everything is done on the 2nd core on the ESP32. There’s some system code that runs on the 1st core, but everything you do in your Arduino sketch - unless there’s some special multi-core library out there - runs on the 2nd core

is there a reason not to run it at 240 or even 480 or 960Hz by default in the code

At a high refresh rate, the display will be dimmer at higher color depth as it takes longer to shift out some of the data than the data is actually lighting the LEDs.

Also, at a high refresh rate, it may not be possible to display the full color depth as at a certain point the time to light the LEDs is shorter than one I2S module clock cycle on the ESP32. This is a problem the way SmartMatrix displays data a row at a time, but there might be a way to do it using the method Sprite_TM uses to do full frames at a time. I want to investigate this.

marcmerlin · January 2, 2019, 4:13pm

Yes, FastLED on ESP32 runs on both cores I believe. Have a look at
https://github.com/marcmerlin/NeoMatrix-FastLED-IR/blob/master/config.h#L74
and

Thanks for the reminder about higher refresh = dimmer. You did write that in your docs and I had forgotten about it.
Still, I think I’ll take 240 or 480Hz and dimmer as long as it looks ok on pictures and without the damaged frames that I’ve been having otherwise (for camera, not the naked eye which is much slower).

seratosteve · March 18, 2019, 9:47pm

Hi there, I’ve been prototyping a new (?) refresh technique that degrades more gracefully as the camera shutter speed increases. Here are my test results shot in portrait mode with an iPhone 7:

Normal scan method, 240 Hz iPhone Slo-mo
New scan method, 240 Hz iPhone Slo-mo

Cheers,

Steve West

seratosteve · March 18, 2019, 9:48pm

(Couldn’t link this in the previous post)
New scan method, 30 Hz iPhone

marcmerlin · March 18, 2019, 10:07pm

Very nice, thanks Steve.
Is that on teensy? ESP32?

seratosteve · March 18, 2019, 10:58pm

ESP32

marcmerlin · March 18, 2019, 11:00pm

Great, would love to see your teensylc branch. I’m still working on my ESP32 smartmatrix shirt (64x96) and limiting artifacts for pictures would be awesome.

seratosteve · March 22, 2019, 12:36am

Hi Marc, happy to share but my current prototype is limited by DMA_MAX for each bitplane. I also don’t (yet) have an external latch so haven’t tested that at all.

marcmerlin · March 22, 2019, 2:57am

I’m using direct wiring like you, so that’s not a problem
What is the DMA_MAX limit? Number of pixels?

Louis · March 22, 2019, 9:17am

Steve emailed me his code and gave me permission to share here:

seratosteve · March 22, 2019, 7:56pm

Yes Marc, the limit was 2046 direct wired pixels. But it’s okay, I’ve already fixed that limitation. Here is the updated source SmartMatrixLibrary-FlickerReducing-SteveWest-20190322.zip

marcmerlin · March 23, 2019, 5:37am

Thanks @seratosteve My big party where I’ll be trying my new 12,288 rgbpixel shirt (half the pixels are mirrored, so I only use half of that for memory), is tomorrow, so I’m trying to stabilize what I have, but I’ll definitely look at your code when I can.
May I recommend you put it on github so that it’s easier to see your progress as well as diff?
Also, for external latch, do you have a pointer to that setup so that I can read more about it/see how it works and why it’s more desirable?

Louis · March 25, 2019, 11:25am

It’s the circuit that’s in the SmartLED Shield ESP32 V0 boards (and very close to the circuit that’s in SmartLED Shield V4). It uses an external flip flop to hold the ADDX lines, saving GPIO pins and RAM (data for each CLK edge can be stored in uint8_t instead of uint16_t)

I noticed that in pictures of your LED shirt you posted here and on your blog, there’s no noticeable scanning. How’d you work around that issue?