Node-Red running with balena

@GregorR1, the Node-RED flows are created under /data folder, so that is normally the location where you need to copy your flow file to.

My github repository is currently not copying/deploying any flows - so that is why you can’t find them.

Note that the shared volume node_red_data is also mapped to the /data folder so you can access them also via samba (windows share) thanks to the samba service (see also docker compose file).

Hi @GregorR1,
Do you mean that your device is downloading the application update over and over again? If so, could you grant us support access to the device and let us know the uuid of the device?

Yeah that’s what it’s doing. I’ve granted support access and the UUID is f8c5a2f211ac5a4ef4015b3f93b234b8

Hi,
What kind of sd card are you using? There are indicators that the sd card might get/be corrupted. (see Diagnostic checks in the dashboard):

check_write_latency     Failed    Slow disk writes detected: mmcblk0: 6001.26ms / write, sample size112375 mmcblk0p2: 1292.69ms / write, sample size100 mmcblk0p3: 5099.6ms / write, sample size5 mmcblk0p5: 1367.93ms / write, sample size81 mmcblk0p6: 6010.03ms / write, sample size112166

It seems like these errors prohibit the downloaded image to be persisted on the disk. Could you try to change the sd card?

Its a Kingston 16gb MicroSDHC. Whats the best way to change them over? Do I just delete the device from balena dashboard and create a new one?

Yeah that is the best way.

Hi @afitzek even with the new SD card in it still appears to be doing the same thing of constantly repeating the download.

I’ll change what I last did back and see if it stops it.

Changing the dockerfile back to what it was before doesn’t seem to have made a difference to the constant updating - the new device is still “factory build” and it is repeatedly downloading the target, but not doing anything with it

Hi there,

I’ve tried to have a look at the device, but it’s currently showing as offline. Has the power to the device been disconnected? I do have another question, which is how are you powering the device? What you’re seeing might be related to an unstable power source.

If the device comes back, please let us know, and we’ll investigate what’s happening on-device!

Best regards,

Heds

Hi @hedss, We created a new device so we could swap the SD card. The new UUID is 31b53b5ef514d3897db450379aa714e2

Thanks,
Gregor

Hi @gregorr1 ,

I’ll take a look and let you know what I find!

Best regards,

Heds

Hi again,

A very quick initial look shows that’s the kernel’s having issues using the SD card:

[  771.299920] mmc0: Timeout waiting for hardware interrupt.
[  771.305417] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[  771.311961] mmc0: sdhci: Sys addr:  0x00000100 | Version:  0x00001002
[  771.318502] mmc0: sdhci: Blk size:  0x00007200 | Blk cnt:  0x000000f1
[  771.325043] mmc0: sdhci: Argument:  0x0089c342 | Trn mode: 0x0000002b
[  771.331587] mmc0: sdhci: Present:   0x1fef0106 | Host ctl: 0x00000017
[  771.338128] mmc0: sdhci: Power:     0x0000000f | Blk gap:  0x00000080
[  771.344668] mmc0: sdhci: Wake-up:   0x00000000 | Clock:    0x00000107
[  771.351209] mmc0: sdhci: Timeout:   0x00000005 | Int stat: 0x00000000
[  771.357750] mmc0: sdhci: Int enab:  0x03ff100b | Sig enab: 0x03ff100b
[  771.364291] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[  771.370832] mmc0: sdhci: Caps:      0x45ee6432 | Caps_1:   0x0000a525
[  771.377373] mmc0: sdhci: Cmd:       0x0000193a | Max curr: 0x00080008
[  771.383912] mmc0: sdhci: Resp[0]:   0x00000900 | Resp[1]:  0x00734f7f
[  771.390452] mmc0: sdhci: Resp[2]:   0x325b5900 | Resp[3]:  0x00000900
[  771.396993] mmc0: sdhci: Host ctl2: 0x0000808c
[  771.401505] mmc0: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0xded3a210
[  771.408036] mmc0: sdhci: ============================================
[ 2175.981947] INFO: task kworker/0:2:2182 blocked for more than 120 seconds.
[ 2175.988990]       Tainted: G         C        4.19.66 #1
[ 2175.995021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2176.003020] kworker/0:2     D    0  2182      2 0x00000028
[ 2176.008634] Workqueue: events_freezable mmc_rescan
[ 2176.013523] Call trace:
[ 2176.016032]  __switch_to+0xa8/0xe8
[ 2176.019510]  __schedule+0x254/0x850
[ 2176.023057]  schedule+0x38/0x98
[ 2176.026268]  __mmc_claim_host+0xb8/0x200
[ 2176.030255]  mmc_get_card+0x38/0x48
[ 2176.033791]  mmc_sd_detect+0x24/0x90
[ 2176.037442]  mmc_rescan+0xd0/0x370
[ 2176.040901]  process_one_work+0x1ec/0x458
[ 2176.044983]  worker_thread+0x48/0x430
[ 2176.048717]  kthread+0x130/0x138
[ 2176.052010]  ret_from_fork+0x10/0x1c

It took approximately 2 minutes for me to actually get a shell into this device, and whilst that usually indicates a very poor network connection, actually typing a command at the command prompt is very responsive. However, any execution of a command is incredibly slow, which also explains why it took so long for a shell prompt. This definitely suggests there’s an issue with SD card access as it tries to execute commands from the media. I know previously in the thread you’d mentioned you’d swapped cards? Have you been able to try an alternative make of card, instead of another of the same type? I think it’s well worth doing so to rule out any incompatibilities we might be seeing here.

Best regards,

Heds

Hi @hedss unfortunately I don’t have any other types of SD cards in the office. Could it be something I’ve done in my docker file that’s causing this? I can send you the files over if that would help?

Hi @gregorr1 ,

I don’t think there’s anything you could have done in your Dockerfile to cause this. The kernel error is pointing to an issue with the SD card, as does the lag in response from the host OS CLI. I’ve seen another couple of kernel errors pointing to the SD card since initially reporting it to you.

The only other possibility here might be the power source for the Pi4, as I mentioned previously. Is this running off a dedicated PSU, or is it being powered via a laptop USB port, or potentially a hub? I’ve personally seen issues with running a Pi4 from both laptop and hub (purposefully for testing, I might add!).

However, I’d really urge trying with another SD card if possible if you’re able to get your hands on one, as this could also be an issue with the model which we’ve never seen before (it’s extremely uncommon but could be a factor). As Andy mentioned previously, we personally recommend SanDisk Extreme Pro cards, as they’re tested very heavily and have a good wear rating.

Best regards,

Heds

Hi @hedss the Pi 4 is being powered by a dedicated 3A power supply so it shouldn’t be an issue. It has run the standard raspbian etc fine. Would it be of any help if I were to add you as a collaborator on the GitHub repository so that you can see what I’m trying to push? It may be that it’s something I have done that has caused this as I’ve been doing most things by trial and error since I’ve never used docker etc before.

Hi @gregorr1 ,

If you’d like me to have a look, I’m more than happy to have a quick inspection to make sure it’s nothing you’re doing. :slight_smile: My Github username is the same as my forum one (ie. hedss).

I’m still fairly confident it can’t be unless your service is trying to directly access the block device partitions, it shouldn’t have an impact on the SD card. I agree it sounds like the power definitely isn’t an issue here, though!

Best regards,

Heds

I’ve added you as a collaborator now so you can have a look. It should have sent you an invite I think.
Thanks, Gregor

Hi @gregorr1 ,

Thanks for the access. I’ve looked over the project, and I can’t see anything that would suggest what you’re doing would cause this. I’ve also very quickly run it on a Pi4 here to ensure that updates work, which they do with no issues.

I’m afraid this really does point to the SD card either being faulty (or possibly incompatible).

Best regards,

Heds

Just to confirm, I’ve also removed myself from access to your repository!

Best regards,

Heds

I’ll try a brand new one of the same model and see what happens. Failing that I’ll try and get a Sandisk one for tomorrow.

Also I’ve noticed that I can’t delete any of the devices that I’ve added? Why is this?
Edit: I was typing in the device name rather than the application name