Supervisor CPU Usage Spiking During CAN Testing

I have two RPI4s setup in a CAN test, one as a transmitter running canplayer and the other as a receiver running candump. (I based my work on this project but didn’t run the web service) The goal of my test was to understand the CPU utilization of the receiving PI running the CAN interface with a large amount of data on the bus.

I’m using MCP2515 CAN controllers in my test. The file I’m playing oncanplayer is a 30 minute dump of NMEA 2000.

During my testing I noticed the candump container CPU utilization didn’t exceed more than ~5% but what struck me as odd was the supervisor container CPU utilization get spiking up to 30% and then sometimes 50%.

I thought may this was related to sending the logs to the VPN so I tried disabling that utilization spikes stopped hitting 50% but still constantly hit 30%.

Both PIs are running balenaOS 2.65.0+rev1 and supervisor 12.2.11.

Thoughts,

Hi Darren,

Can you run htop on the receiving Pi, so we can understand where the CPU is spending its time? If you open a shell to the host OS of the device, you can run htop like so: balena run --rm -it --privileged --pid=host wrboyce/utils:htop. You can then open the SortBy menu using F6, where you can choose to sort by CPU usage. You can open the setup menu with F2, where you can use the arrow keys and space bar to configure htop. If you navigate to Display options in the setup menu, you can disable Hide kernel threads, which will show us if the CPU is spending a lot of time in the kernel.

@jakogut

Thanks for the help. I ran htop as you requested and it looks like the CPU is spending time in node /usr/src/app/dist/app.js the process called is journalctrl -a --follow -o json _SYSTEMD_UNIT=balena.service. This is what seems to fluctuating between 28% and 37% of the CPU.

I believe the journalctrl to logging and the high CPU utilization has to do with the fact that candump is writing to stdout. Should I send the data from candump to the null bucket for a better performance test?

Thanks,

Hi Darren,

Have you kept the same bit rate as is in the example repo of 10kps? Does modifying the bit rate and running can player again change the CPU usage at all?

Could you also try writing the can dump to a log file instead of stout ( can dump -f) and seeing if that makes a difference to the CPU usage?

@joehounsham

No, I didn’t use the default of 10kps. You will see in the docker-compose file that the bit rate is actually set to 500000 for the example. Which is typically for a vehicle. I choose 250000 as I’m looking at this for NMEA 2000 which is used on boats.

I will try the candump -f later this morning an report back.

Thanks for the help,

@joehounsham

Running candump -l can0 seems to have solved the problem as the CPU utilization via the balena cloud console is now showing ~6%.

Hello Darren

I’m glad that using candump to write to a log file instead of stdout has solved your issue.