Hi All,
I was running a Embedded Pi with a CM3 module.
This was running fine under: Resin OS 2.12.7+rev1.
Now i saw there was a update, so i updated my dev machine to the latest v2.15.1+rev2.
But after the upgrade the device is not coming online anymore via ethernet.
So i installed a clean img, but still is the device not reachable or getting online in the dashboard.
Any idea how to fix this?
Will try to upgrade a 2nd device and see if the same problem occurs.
Hmm, 2nd device bricked to. I do have to re flash it to get it working again. Somehow v2.15.1+rev2 won’t let me boot or let the ethernet work. Iam not sure.
The status led won’t even blink at all.
Anyone any idea what could be wrong?
I will downgrade to 2.12.7+rev1 to see if it i get the device working again.
Edit: flashed back to 2.12.7+rev1 and its working again. But i am wondering what can be the problem with v2.15.1+rev2.
Hi, one way for diagnosing this is to use a .dev image and a USB-serial cable to connect to your device from your laptop.
Since the issue is not existing in earlier versions alternatively you may bisect which version introduces this issue.
In v2.13.3 we switched the DHCP client used by NetworkManager to the internal one (provided by systemd) - this comes as a possible cause to my mind. If you create a resinstaging.io account, you will be able to access more OS versions (by choosing Show outdated versions in the download dialog). Can you please check whether this issue exist in v2.13.5+rev1 and if so then try v2.13.2+rev1 as well?
Hi, were you able to resolve the connectivity issue. I am working on a connectivity diagnostic tool that may help us in finding up more on this issue: https://github.com/resin-io-playground/nm-debug-logs
Please ping me in case you need more assistance on this.
Hi @majorz unfortunately i didnt yet find the time to look into it more.
For now i reverted to the working image and i still need to test from what image the connection breaks. This takes time since i have to re flash it with every next image.
Sounds good. I will be updating the tool today and will post more information here, so that whenever you have a bit of time to flash latest OS version you can try it out and send me back the debug logs for further investigation. Thanks a lot!
Hi @majorz,
I just got some new Embedded Pi devices. And installed the older: 2.12.7 img that worked before.
Now i see this is also not working anymore. I also tried the latest img: 2.27.0+rev1-dev-v8.3.5 but that was also not working. Embeddedpi noticed there was a problem with creating partitions on the CM module and MMC. At the bottom of this link:
Would this be something to do with my previous problem you think? Or would it be the network manager?
Are you using Etcher to flash the image on the compute module?
Yes i am. But it won’t boot. The led stays green.
Validation in etcher went fine.
The thing is i have multiple of the same devices. 8 of them won’t boot with exactly the same img. And 2 of them did start fine like the rest of the devices i already had.
Troubleshooting
For a small percentage of Raspberry Pi Compute Module 3s, booting problems have been reported. We have traced these back to the method used to create the FAT32 partition; we believe the problem is due to a difference in timing between the BCM2835/6/7 and the newer eMMC devices. The following method of creating the partition is a reliable solution in our hands.
$ sudo parted /dev/<device>
(parted) mkpart primary fat32 4MiB 64MiB
(parted) q
$ sudo mkfs.vfat -F32 /dev/<device>
$ sudo cp -r <files>/* <mountpoint>
Could this be the problem and how can we see if the Balena img will work again?
Will it boot Raspbian? This will determine better where the problem is. Also it is worth trying the the troubleshooting method above - probably with Raspbian first, and then our OS. What do you think?
Well yes it will boot the original img that’s delivered with the box.
The problem seems to be the partition alignment. That shoulds be shifted a bit since some CM modules have problems with the eMMC. But how to achieve to do this with the balena img? When the boot partition is made it should not start on the first sectors of the disk.
I talked with our Etcher maintainers and we would definitely like to know more about this. One possible solution would be to adjust our images to align these sectors in a more compatible way with those CM modules.
Etcher itself copies image data byte by byte and it is not involved with partition creation itself.
We will try recreating the boot partition of a flashed resin OS locally and let you know afterwards the commands so that you may try this out and see if it solves the issue.
@majorz the engineer of Embedded Pi told me that Resin used fat16 for the partition what actually should be fat32 as stated under the BCM283x Boot Process paragraph on: https://www.raspberrypi.org/documentation/hardware/computemodule/cm-peri-sw-guide.md
He had a way to realign the partition of the Resin img but this will take some time and commands to do this manually. So it would be much appreciated if the Etcher tool can do this for us.
Thanks for getting to the bottom of it. I can confirm this is the case indeed:
Raspbian boot partition:
/dev/mmcblk0p1 8192 98045 89854 43.9M c W95 FAT32 (LBA)
Resin OS boot partition:
/dev/mmcblk0p1 * 8192 90111 81920 40M e W95 FAT16 (LBA)
We will follow up on this soon.
@RDB I will send you now in a private message a link with a fat32 based image to try it out.
@RDB just pinging you to ask you whether you were able to check the image with the FAT32 boot partition.