Weird Blanking problem

1

I’m going to be very embarrassed if I’ve missed something already documented about this, but here goes.

I have a modification of FeatureDemo that I started playing with to work my way towards what I actually need to do. I’m using a single 32x64 small panel for testing at the moment. My problem is the display blanks out at odd times. For a particular compile, the blanking seems to be consistent. The problem is, that the oddest things make the blinking stop!

BTW, another tidbit (clue?), when the loop runs with the blanking problem, it takes longer than when there is no blanking, presumably the amount of time the display is blanked.

I’ve attached just about the smallest version of this program that shows the problem, although I’ve seen it a couple of times as I’ve been learning to use these displays. As my attached source code is now, it has the blanking problem. Here is a partial list of the things that ‘fix’ the problem, although it is certainly not complete, for instance, having the Serial.println() somewhere else in loop() might also work, I didn’t test all possibilities.

Having a Serial.println(“.”) at the top of loop() (or any number of Serial.print() lines), but if it’s in setup() it does not work. It is now commented out.

I have reduced the segments down to three at line 135, the last one is my own:
#define DEMO_DRAW_CHARACTERS 1
#define DEMO_MONO_BITMAP 1
#define CC_BITMAP 1
If any one or two of these is not ‘1’, the blanking goes away.

If “USE_ADAFRUIT_GFX_LAYERS” is not defined, the blanking stops, but I need the GFX layers eventually.

On line 96, in my function drawFullScreenBitmap(), I have a superfluous line as the code stands now, which is the “backgroundLayer.fillScreen({ 0,0,0 }); (I was eventually going to add transparency), removing that line makes the blanking stop, but the blanking is happening both when this routine is called, and when it isn’t, so it does not make sense.

An added note: I noticed before that after a few hours of running the program continuously while blinking (seeing the problem) it started doing some odd things like only showing every other line and ‘shimmering’ (fast blinking, line by line?). But I didn’t put that in the first version of this post, since I wanted to make sure it didn’t happen in the working version. I ran it all night with printing the temperature, which made it stable (not blinking). It ran all night with no problems, and BTW, the temperature of the chip seems to be stable between 136 and 138 degrees F, (~58.9C). When I commented out the serial printing of the temperature (but left in initializing the temp routines) it came up within a few (20 or so, consistently) minutes ‘shimmering’ again, but not every other line blanked.
If anyone is interested, the only difference between what is in the Zip file linked below and the shimmery version is:
#include <InternalTemperature.h>
and in setup()
InternalTemperature.begin(TEMPERATURE_NO_ADC_SETTING_CHANGES);
This library is pointed to on the PJRC web page for the Teensy4.1 and can be found at:
InternalTemperature Library

I’ve also changed all the hardware (teensy4.1, SmartLED Shield, and another panel with the same 32x64 but different chipset) and no change the results.

I really do not like having such an unpredictable problem, any help would be appreciated.

It’s a little long to quote here, and I can’t upload a zip, so here is a link to the files:
FeatureDemoMod

2

Correction, I didn’t read the chips properly before, but both panels I tested it on bought at different time and different pixel pitches were in fact the same part number. The chipset is ICN2028BP, if it makes a difference.

3

when the loop runs with the blanking problem, it takes longer

How much longer?

How are you powering the Teensy and panel?

Can you share some video of the blinking (and shimmering if it’s easy)?

If you lower the brightness, does it change the behavior?

4

It is a loop of about 33.9 seconds with no blanking, 38.16 seconds when blanking, (by stop watch, not the more accurate counting frames).

Tried two different power supplies, one was a PC power supply with a breakout at 20 amps, the other a 4 amp brick. No difference. I have the Teensy USB power jumper cut, smartLED shield powered from the same supply as the panel both ways of course.

I tried 100% and 50% brightness, no difference.

The only difference between these two programs in the videos is I added Serial.println(“.”); to loop().
With Blanking Problem
With Serial Println() added

But some further info: My target is an outdoor display, two 32x32 P6 MOD8SCAN, as opposed to the smaller 32x64 displays I was working with because they are easier to mount where I can see them and I don’t have to squint at how bright they are all the time for development. When I switched to the bigger MOD8SCAN, I was not able to reproduce the problem (yet). I can’t read the chipset on the chips since they are conformal coated.

It still makes me nervous that it could suddenly crop up when I don’t have time to mess with it.

5

Gut feel is it’s an overheating problem. The temperature you measured doesn’t seem nearly hot enough to cause the chip to shutdown though. Can you make the problem happen by heating up the chip externally? Or make it go away by cooling after it’s already started?

If you can tell if it’s blanking by something other than the panel (e.g. a printf or tracking the time to complete a sequence), does the blanking problem continue if the panel is removed?

I didn’t see anything in your code that jumped out as a problem.

6

I used a “canned air” upside down to freeze the processor and other chips on the Teensy 4.1, and got most of the chips on the panel. No difference. I then waited for it to start the slight flickering I keep seeing, and froze every single chip on the panel and the teensy. No change, it was still blanking, and flickering.

But I had little hope for that to work, because it’s such an on/off thing. If it works, it doesn’t change. I’ve left it running for hours at a time while I did other work (but in sight), and I left it running all night and the next morning, it was still fine. But if it’s blanking, it never works, but it does change with time! And there are such arbitrary things that turn the problem on or off, one line of code that is unassociated with the operation of the panel, most notably the single (or multiple) Serial commands, either input, output or “if( Serial.available() )” all make it work properly.

New information: a delay(1) at the top of loop(), or just after the

backgroundLayer.fillScreen(defaultBackgroundColor);
backgroundLayer.swapBuffers();

or in fact at the end of loop() or between any of the sections of code between the conditional compiles (DEMO_DRAW_CHARACTERS etc) all make the problem go away! As does digitalWriteFast(14, LOW); so I can’t easily measure the time of the loop without a panel hooked up, since I seemingly have no way to see the beginning of the loop!

I’m feeling better that I won’t run into this in a real-world application, since it seems so ‘delicate’ an issue I happened to hit on.

I’ll try to get time to put it on my scope later today or tomorrow. Unfortunately I only have 2 channel scopes, no four channel. I haven’t used it in a while, but I also have a 16 channel USB type logic analyzer, (if I can find it :slight_smile: ). I can guess about the functions of the pins, but if you could direct me towards a description of the pin functions and timing of the HUB75 panels?

Your gut feeling is heat, my gut feeling is it’s a compiler or even a microcode chip problem with the DMA code or microcode. Very hard to characterize and define. If I can show the problem on a scope trace without the panel interaction, I might send it to Paul S. to see if he has any bandwidth to look into it.

Corey

7

@easone do you have any thoughts on this?

8
9

I have seen a similar issue, but wasn’t able to reproduce it consistently. The screen would turn off every few seconds for about a second each time, but adding random Serial.print statements would sometimes prevent the issue which really made it difficult to troubleshoot… I also found that changing the compiler options (from faster to fastest) affected the issue, but not consistently…

I wonder, could it be related to this issue reported on the Teensy forums?

The last few posts in that thread found the cause was a bug in the Teensyduino startup code for the Memory Protection Unit, but I have not tried the bugfix to see if it helps with the display blanking issue.

10

I tried the change in startup.c, and it seems to ‘fix’ the problem, at least so far. However, when even adding writeDigitalFast() once in loop() fixes it, it’s pretty uncertain if it is fixed, or just hidden and will pop up later. In any case, I don’t believe it is in the SmatrMatrix library, but in the compiler. Therefore I added to the pjrc thread all the info, and links, I could so if anyone there wants to look at it they can.

Thank you @Louis and @easone for your time and effort. If any insight comes up from the pjrc forum thread, I’ll post it here.

11

The fix for the 1.7 sec freeze issue has been added to the latest Teensyduino 1.54 beta, to be included in the next stable release (TBD):

Interesting that they are also adding the FlexIO_t4 library into the next release. We should probably check that SmartMatrix plays nice with the new beta.

1 Like