Memory usage / reliability of a Teensy program


#1

Hey everyone,

I have designed a brewing controller that uses a Teensy 3.2 controller and the arduino environment to run.

The program was initially very simple, but over the last year or two of development, its now just over 140k in size and I want to start thinking about memory usage and reliability

Can anyone suggest where to start with monitoring the memory usage of the controller while the program is running, and secondly anything about implementing a watchdog style system on it for stability.


#2

This might help with the memory monitoring… https://sourceforge.net/projects/teensy-3-x-rammonitor/

For the watchdog, this looks like it might help… https://forum.pjrc.com/threads/25370-Teensy-3-0-Watchdog-Timer?p=44197&viewfull=1#post44197

(A little background info for those curious… Watchdog timers are usually a combined hardware/software feature, where the application software is supposed to regularly poke a heartbeat API, and if a timeout elapses since the last heartbeat, the watchdog trips and resets the hardware.)


#3

Hi Jason,
Does the IDE you use have a built-in usage monitor?


#4

I am using the generic arduino IDE. It shows the sketch size and the global variable sizes.

I’d like to avoid the below situation and dont fully understand how the various types of variables (int, byte, boolean, string, string arrays) use memory over the short vs long term.


#5

There is a call supported in the Arduino IDE that shows the current free memory (not sure if it works in the Teensy Arduino port). I have found this handy to show the free memory while you program is running. I have done this via print statements over serial monitor and via syslog messages over ethernet. Typically I’ll check the free memory while testing the code and comment it out once happy…

The memory usage shown in the IDE when you compile the code doesn’t allow for what you may use while your code is actually running.

See the free ram info here:


You may have already looked here as I think thats where the image you posted is from…


#6

What memory are you trying to monitor here? Are you doing dynamic memory allocation? If so, the first step would be to replace the default allocator with a custom one that will allow you to monitor both it’s how the heap is used and how fragmented it is.

On an embedded system (and other semi-reliable systems), generally it is a good idea to up-front allocate all your memory in one go and use that as the largest amount of heap that you can use. This can be as simple as declaring a large global variable - ie.

uint8_t g_heap[1 << 20];

I would then add your own implementation for sbrk. This is the function that will get called when the system needs memory for the heap. How you do that implementation specific though, depends on the c-library you are using. I think teensy uses newlibc.

To implement a watchdog, it may be a good idea to then write a ‘magic’ value at the end of the heap and periodically check if the value is still what it was. If not, then that means the stack has collided with the heap and you are boned.


#7

int is weird because its size is platform-dependent. On a Teensy it’s probably 32 bits, but you’re better off relying on explicit names for basic types (e.g. uint32_t for an unsigned 32-bit integer, values range from 0 to 232-1, or int32_t for a signed 32-bit integer, where values can range from -231 to 231-1)

For the rest, on Arduino byte and char are both 8 bits, but byte is unsigned (0 to 255) and char is signed (-128 to 127). Once again, if you have any doubt, use a size-specific explicit type rather than these generic names.

Strings, like any array, can be of any length when allocated (assuming you had enough memory) but can’t change their size*; you can figure out how much memory an array uses by multiplying the size of each element (e.g. 8 bits or 1 byte for char or byte values; 32 bits or 4 bytes for uint32_t values) by the size of the array. In C and many related languages, a string is just a char/byte array with an ASCII NUL character (code point 0) after the last character.

* if you’re storing stuff in an array and you run out of space, you can allocate a new array, copy the contents of the old one into it, then dispose of the old one… There are libraries to do most of the heavy lifting, but memory management can get complicated


#8

As for stack/heap…

It’s normal for the stack to grow when you use local variables, and call functions. If you use a lot of local variables, or use very large ones, or get into really deep call stacks (e.g. recursion), you’ll run out of stack faster. Normally when you return from a function, everything that was allocated on the stack in that invocation will be freed automatically.

The heap is used for dynamic memory allocation (the typical C keywords for using it are malloc and free). Unlike with stack allocation, it’s possible to allocate memory on the heap, and have it survive beyond the scope of the function where it’s allocated. This can be very useful, but memory management is hard, so memory leaks, where dynamically allocated heap doesn’t get freed, are one of the most common classes of bug.


#9

I think you’d probably be better off asking for help with specific bits of code… Sometimes these conceptual responses like mine are less helpful. :slight_smile:


#10

These responses are great, they help me get my head around how to approach coding in a less hobby-centric manner.

So the specific issue I am having is random controller freezing issue. Can be minutes, hours or days that it happens.

The program does create, edit and print strings all over the place to the GUI. I’ve started reading about the issues with strings and how they can fragment memory. Can I get some guidance on how to use (or avoid) strings?


#11

Do you have the freedom NOT to use the Dynamic (Heap) memory?
If you are forced to use it:
(1) Are you freeing the dynamically allocated memory (memory leaks)?
(2) Even if you have sufficient free memory in total, if it is not in Contiguous Blocks, it cannot be reallocated for the next object; it should return a NULL Pointer.
(3) Is it possible to move to C++ (have a destructor for every class) and use containers entirely such as a vector of strings (like a list) and leave the Dynamic memory headaches to the Compiler and Linker?

Moving to a processor with larger RAM may help.


#12

Can you post some snippets of what you are printing and how you are using strings? Are you in C or C++?


#13

Yep, so the program is a GUI and it constantly re-draws a menu screen as fast as it can, forever…

The way it is currently programmed (as an example)

void setup(){
}

void loop(){
String temp = “”;

 temp = "Main Menu";
 GD.cmd_text(240,150,OPT_CENTER, temp.c_str());

 temp = "New Program";
 GD.cmd_text(240,150,OPT_CENTER, temp.c_str());

 etc.....

}

So from reading recently, I am making a temporary variable (String temp) and then creating a bunch of extra string variables in dynamic memory every time I am running through the loop

Would it be better to structure it this way? I would reserve a space of memory at the start, then just reassign the memory slot with the new text before I want to print it to the screen? This way no dynamic memory assignments are needed?

void setup(){
char temp[50] ;
}

void loop(){
strcpy(temp, “Main Menu”);
GD.cmd_text(240,100,OPT_CENTER, temp);

 strcpy(temp, "New Program");
 GD.cmd_text(240,150,OPT_CENTER, temp);

 etc.....

}


#14

I annotated the likely dynamic memory allocations with the String below.*** I am not sure what the GD stuff is doing under the hood. May be worth looking into.

void setup(){
}

void loop(){
// Allocate a string object on stack.  Depending on implementation, this could be as small as
// a pointer to memory and a int tracking size.
// This will invoke assignment operator of string, which will likely cause a dynamic memory
// allocation here and then copy the null terminated string into that buffer, then set the pointer within
// the String object.
String temp = “”;

// Another dynamic memory allocation here of the size of "Main Menu" + 1, then copy the string
/// into the buffer
 temp = "Main Menu";
 GD.cmd_text(240,150,OPT_CENTER, temp.c_str());

// yet one more here.
 temp = "New Program";
 GD.cmd_text(240,150,OPT_CENTER, temp.c_str());

 etc.....
}

As for your second snippet. That is the right idea. You will need to put your char temp[50] either in loop() or make it a global variable. Placing it within setup will cause it to be stack allocated and then go out of scope once the setup() function finishes, making it not possible to access within the loop() function.

However, if you are not doing any formatting of the strings, then I would suggest the following:

void setup(){
}

void loop(){

 GD.cmd_text(240,150,OPT_CENTER, "Main Menu");
 GD.cmd_text(240,150,OPT_CENTER, "New Program");

 etc.....
}

The strings “Main Menu” and “New Program” already live in a section of your binary called TEXT. What the above code does is just pass a pointer to these strings directly to the cmd_text function without any copying or dynamic allocation.

If you need to do formatting, ie printf, then do something like this:

void loop(){

 GD.cmd_text(240,150,OPT_CENTER, "Main Menu");
 GD.cmd_text(240,150,OPT_CENTER, "New Program");

 // Allocate 64 bytes on stack
 char buffer[64];

 // Format your message into that memory
 snprintf(buffer, sizeof(buffer)/sizeof(buffer[0]), "Temperature1 %d", temperature1);
 // Print it
 GD.cmd_text(240, 150, OPT_CENTER, buffer);

 // Reuse that stack memory
 snprintf(buffer, sizeof(buffer)/sizeof(buffer[0]), "Temperature2 %d", temperature2);
 GD.cmd_text(240, 150, OPT_CENTER, buffer);

 etc.....
}

The above makes the allocations easy to see now. There is no abstraction around the character arrays and if we knew how cmd_text worked, we can fairly easily guesstimate how this function behaves from a memory standpoint.

*** String may also be implemented with a small string optimization where small strings are stored directly in the class instead of copied into a dynamically allocated buffer. Usually this is for strings less than 16 characters long.


#15

Awesome, thanks for clarifying this, super helpful. The function cmd_text(…) is from the Gameduino library, pasted and a picture below.

Essentially cmd_text is taking the char array and plotting it onto the LCD buffer (the LCD screen uses a EVE FT810 graphics chip, and communicates via spi with the teensy)

image

image

image


#16

For strings that are constant through the life of the program, you can do stuff like make them constants (so the compiler knows it doesn’t have to keep allocating/deallocating heap for them) or store them in flash using PROGMEM.

PROGMEM strings have the advantage that they are read directly from flash memory, which frees up that memory for your program to use.

https://www.arduino.cc/reference/en/language/variables/utilities/progmem/


#17

I wouldn’t worry too much about how the libraries work, I wouldn’t expect you to reimplement them. Too much work.

I’m not sure if PROGMEM is applicable to the Teensy. I believe it has a split instruction and data bus unlike the AVRs.

If I were you, I would try to instrument your heap with malloc/delete/free/new/etc hooks.

Edit: Oh fun, found out that newlib’s printf functions do allocate memory dynamically… aye aye aye… http://www.nadler.com/embedded/newlibAndFreeRTOS.html


#18

Edit: Oh fun, found out that newlib’s printf functions do allocate memory dynamically… aye aye aye… http://www.nadler.com/embedded/newlibAndFreeRTOS.html

So for the purposes of a program that is meant to run reliably for a few weeks, would you still suggest using snprintf functions with the buffer variable declared globally?

I have a .h file that all the global variables go into and I include this before the setup() function is run.


#19

It shouldn’t matter too much whether you have it declared globally or on the stack. Either are much much better than doing several dynamic allocations on each loop().

The malloc_r that newlib does on snprintf is only once when it first gets called, so that isn’t too big of a deal either. Just thought it was interesting.