Orange Pi devices official support request

[steelrat]

FIrst of all, I wanted to say thanks to balena team for such a beautiful ecosystem that allows to start an IoT project in half an hour with OS/software updates, remote debug, fast development, etc.

I know that there are several requests and bug reports regarding different Orange Pi models. Also, there is a community repo to support them. Unfortunately, this repo looks outdated and abandoned:

• Only 4 models are supported, e.g. super cool Orange Pi Zero 2 is not supported
• Old OS version (max 2.53.9)
• OS update not supported according to README
• Issues and PRs are active for years w/o any move

Why Orange Pi support is crucial:

• they are cheap and more performant. E.g. RPI 3 B+ (2018) costs 45$, while Orange Pi Zero 2 (2020) costs 25$ with even more performant CPU
• there are no supply problems. Official RPI distributors allow me to buy max 1-3 RPI 3 b+. Other models, like RPI 4 2GB can be simply unavailable (sold out). There are no problems with supply of Orange Pi device range
• it’s one more supplier (together with RPI) of cheap and multipurpose computers. Other devices that have official support are either expensive or tailored to specific needs

I want to ask balena team for an official support of Orange Pi devices, especially those models that are RPI competitors.

[dtischler]

Hi Paul, thanks for the detailed note and explanation of your request (and reasoning)!

We have primarily relied on community support for the Orange Pi devices, as most of our paid-customers go with more “enterprise” grade boards. However, I can definitely see the appeal of the Orange Pi products for the home enthusiasts. Let me ask this… Looking at their rather massive list of SKU’s, and knowing we likely can’t support all of them, which ones would you or want to see supported? What are the more important, higher priority models in your mind? Thanks!

[steelrat]

Thanks a lot for your support. For me the most important is Orange Pi Zero 2. I bought it as a cheap alternative to RPI 3 b+.

Community (on github and this forum) also request:

• Orange Pi One
• Orange Pi Zero
• Orange Pi Lite
• Orange Pi Plus 2E

There are also requests for Orange Pi Plus2, but looks like it’s no longer produced by manufacturer.

[dtischler]

Thanks @steelrat – we’ll have a look at what it would take to build OS’es for these. Some might be easier than others depending on the level of upstream support and the precise SoC on the board. But this helps to narrow down the list, so thanks for that!

[maggie0002]

Thanks for doing this background, you made a much better case than me.

I wonder though when some of the posts were from that you found requesting those other boards? Orange Pi Zero and Orange Pi One, and Plus 2 are already supported: (Run containers on embedded edge devices - balenaOS). Orange Pi Lite is quite a bit older, and looking at the spec in terms of performance and cost would be superseded by the Orange Pi Zero 2 so not sure how many people would go back to look at those today.

The Orange Pi Zero 2 is 64 bit which also moves it further forward in terms of future proofing and native software support (for example there is no official Node support for 32 bit Balena base images - Balena Documentation).

Orange Pi Zero 2, as you mentioned in your post supersedes the Orange Pi Zero which has been plagued with Wi-Fi issues for the community:

orange pi zero doesnt recover from wifi outaage

Hi @osde8info Thank you for the new information that you sent. I’ve taken this up with our product team. Considering what has been discussed so far, as well as other online discussions, it seems that the WiFi chip on the Orange Pi Zero presents many problems, see e.g. and Since this is outside of our control, it would not be of much use if you’d ship your own device to us, and there’s unfortunately not really something further we can do to assist you in this regard. Balena on this device …

orange pi zero doesnt recover from wifi outaage

No worries, just in case there is nothing evident in the logs, another suggestion would be to try and disable the on-board WLAN on the OPi, since there are reports of it being unstable and instead use an external USB adapter. Failing that, another workaround may be to wire up a watchdog type script to run periodically on the device, checking for network connectivity and manually reconnecting if detected offline.

wifi-connect - wpa_supplicant[848]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-95 retry=1

Hi team Balena, I am getting error messages every second in the journalctl logs that say: wpa_supplicant[848]: wlan0: CTRL-EVENT-SCAN-FAILED ret=-95 retry=1. I am running two containers both accessing the dbus, one is the wifi-connect provided by you guys, and the other is an container that just executes a start script and then sleeps forever. The issue appears to be intermittent, when I first booted the device it appeared. When I restarted the device it had gone. I have just restarted the con…

I would add that the Orange Pi Zero 2 also fills a bit of a gap in the market. If you look at its obvious alternative the Raspberry Pi there isn’t an equivalent, the leap from the Raspberry Pi Zero to the Raspberry PI 4 in performance is immense. Really the Raspberry Pi Zero is not a usable board anymore, it lacks the ability to connect to 5ghz wifi networks, is an older ARM version is really underpowered for the market and so forth. The Raspberry pi 1 - 3 is just an lower powered Raspberry pi 4 in terms of form factor, utility etc. And the Raspberry pi 4 is a larger more expensive board. So the Orange Pi Zero 2 really has its own class. Even if a new RPI Zero was released it would likely follow the same class as the previous RPI Zero, which I think is another good rationale for the Orange Pi Zero 2 to be supported.

Personally I intend to make the Orange Pi Zero 2 the primary recommended board for my fleets. I have been using the Orange Pi Zero in that role until now, but as it is buggy at the firmware level I am waiting the OPI Zero 2 support and treating it as a bug fix required for production.

The list could go on, OPI Zero 2 is USB-C powered as opposed to the original OPI Zero, has an aerial which in my tests has boosted wifi-range over the RPI 4, has USB ports unlike the RPI Zero etc. etc.

[the-real-kenna]

@maggie0002 @steelrat thank you so much for sharing this insight. I’m part of a team at balena working to make it easier for manufacturers, customers, and communities to support and maintain balenaOS on the hardware they use. I was really interested in reading what you had to say about the Orange Pi line in general, and the Zero 2 in specific, particularly the market gap it fills in the RPi lineup.

I’ve added that feedback to our list of boards for future support consideration, and also might reach out to the community to see if they would have interest in supporting our OS in addition to the other self-serve options they provide. If you happen to know anyone in the community who might be interested in that, please do let us know!

[the-real-kenna]

@maggie0002 I also was curious - you mentioned that you would recommend this hardware in the fleets you manage. We often see folks avoiding microSD card-based devices due to the higher potential for failure in the field and the problems that can cause for larger fleets. Would you mind telling me a bit more about the fleets you manage and if you see the same problem there?

If you feel comfortable sharing here broadly, I’m sure other folks looking for hardware could be curious to hear your answer. Else I’m happy to share my email address with you if some of your work is private and you’d prefer to share that way. And if you don’t want to share at all, I understand, but had to ask as I’m a curious mind!

[maggie0002]

Hi @the-real-kenna,

This is something that was really given a lot of thought, here were some of the factors that contributed and swayed the decision in the end:

• Cost per unit. Keeping the cost of a unit as low as possible has been key for us, and that tends to keep you confined to the micro-SD range. There are some low-ish cost entry level hardware with eMMC flash built in but it soon starts to add up. Especially when you require larger storage, getting up to 32GB eMMC.
• The units in the fleet work independently, so one going down does not have a ripple effect.
• Distribution networks. Orange Pi will post pretty much anywhere and the nature of the project requires reaching developing contexts which often do not have supply chains for many hardware lines with eMMC (supply chains do exist, but haven’t proven as ubiquitous which is why this is a contributing factor).
• Incredibly low cost of micro SD cards (can find starting at $5 USD for 32GB) makes a card burn out very cheap to replace (if you can afford the downtime).
• Configuring devices with eMMC is more complex than flashing to an SD card. It tends to require a greater level of expertise and we tend to encourage users to provision their own devices by downloading our Balena image and flashing it to their cards.
• Most of all, the real kicker was that SD card failure has not been a big issue for us. In fact I am yet to experience a failure. I have devices that have been running 24hrs 7 days with regular log and data writes for years using Raspbian OS and still tick along nicely. When combined with Balena OS read-only file systems, the risk gets lower. We always use SanDisk cards as they have proven the test of time for us. We also get a minimum of 16GB, usually 32 as the price difference isn’t that much more and we benefit from a larger area for wear-leveling where writes are spread out across the card to avoid hot-spots that create failures.
• Another area of confidence here is that we have been developing on these devices using SD cards, including flash and re-flashing, running build processes on the devices and everything in between for several years, and am yet to wear out a card. This environment is far far more damaging to a card than we can expect a device to experience in production.

As always, it comes down to use case. But in our experience, unless we really had mission critical hardware that we couldn’t afford any downtime on, couldn’t access easily for very long periods of time (years), or were running processes that do an awful lot of writing to the card (many projects are read-only, or close to it, and the risk of SD card corruption is reduced dramatically), the pros of eMMC or other alternatives (we experimented with USB drives for a while) are not great enough to outweigh the cons. We designed cases that enclose the SD card slot so we can even ship a device to a non-techy user without them having to worry about what does what (the lid still pops off and the board can be extracted for replacement, but they are built to not need to do this).

I should note that a lot of these pros/cons are a result of writing in 2021 when these devices and SD cards are at a new level of performance and usability. Only a few years ago (or maybe even a year ago) I would look at Orange, Banana, Raspberry and other SD card related devices and they just wouldn’t cut it. They were all 32bit while our software dependencies have shifted to 64bit (such as Node I mentioned above), things were not built for Arm arch as often as we would have liked (Docker multi-arch builds and Alpine Linux really transformed this) SD cards were expensive and small capacities, processing and network performance was very poor (such as the Raspberry Pi Zero which we barely want to support anymore, but also the <=Raspberry Pi 3B), USB2 not USB3, slower Ethernet ports, single core on occasion. Many before the RPI4 also shared a single channel to the SoC so for example SD card writes and Ethernet data transfer would share a single bandwidth and often limit the real world performance you would see. These days have largely gone now with the RPI 4 and others following suit (Raspberry Pi 4 specs and benchmarks — The MagPi magazine).

At that time, there were many reasons not to use SD card based devices, they just didn’t cut it performance and compatibility wise so weren’t a consideration. Now, it is only the SD card issue to consider, it hasn’t been a big issue, so it doesn’t add up for us to use anything else. I suspect we will see these devices used for even more things in the future, the days of them being limited to learning or as @dtischler suggests, for ‘home enthusiasts’ have gone.

The caveat to all this is the development side. I do struggle with building on these devices for development, the SD card write speeds are just too low for me to be able to recompile in reasonable time. Instead I have a docker development environment and then just build on devices for final testing stages. A small price to pay considering the benefits of these low-cost devices in production.

I would be interested to hear what sort of devices you find yourself supporting most among paid and unpaid customers, especially those working on smaller IOT device projects.

[the-real-kenna]

@maggie0002,

Thank you so much for sharing your thought process on this. We’ve actually been having a lot of internal discussions around SD card usage for a few reasons:

ONE
We are often asked by customers for hardware recommendations. It’s not a simple answer, even though it maybe seems that it should be. Whether on-board storage vs. SD cards should be used is one part of the recommendation, but the determining factors for that, as you suggest, are actually quite complex. My thoughts in the past have been something like the following, and seem to match a lot of the process you used to make the decision, which makes me feel even more confident in them:

• Final location of the device: plenty of customers would choose to avoid SD cards due to the remote nature / inaccessibility of the device. In those cases, I find that the cost of the device (its value to the end customer) is higher, so it’s possible to raise the cost of internal components which reduce failure / on-site service events and pass that cost along to the end-customer as part of the product’s overall value for uptime.

• End user capabilities: to your point, sometimes SD cards are something that end-users have familiarity with (because they’ve replaced them in their phones, cameras, etc. in the past) and so this is a reasonable thing for them to accomplish in case of card failure. In other cases, end-users may not be this savvy, and so on-board storage is used to avoid customers ever having to touch the device (i.e. if it has to be touched, it’s because it’s being replaced entirely).

• Factory/Imaging process: sometimes SD cards are a good fit because there isn’t a great imaging process for a customer and so they choose to send out an SD card + device with the installation tech, who can install them at the same time. I find this is typically only true when companies are first starting, and as they grow they do find ways to pre-image / pre-register devices. But sometimes folks are boot-strapping their product in the beginning and an SD card is a familiar and quick way to move forward while they focus on other parts of the business.

• Uptime requirements: this is one of the bigger reasons I see folks consider the cost of on-board storage. For some folks, the idea of having to wait for an SD card to be replaced, or troubleshoot if it’s the problem, just isn’t possible for their use-case. These folks usually have tight SLA’s scoped with us as well so that replacements happen quickly if needed.

• Case serviceability: I think this is an under-valued piece of the equation and was happy to see you mention it. A lot of folks worry about the ability for someone to tamper with hardware, but I find there are far more cases to be found where it would be nice for a tech to access the internal components than someone wanting to physically sabotage a device (that’s what the internet is for, right? lol). If it were easier to service the internal hardware (which of course, could add cost, so it has to be considered) there might be a case for using an SD card and saving some cost around that component.

• Cost to Repair vs. Cost to Replace: some folks determine that the cost to repair (sending someone on-site to fix something / replace an SD-card) is high enough that it would make more sense to keep on-board storage and replace the entire device as-needed. This blends into the Location and Serviceability components, but ultimately usually comes down to the almighty dollar and how expensive those things prove to be.

There is a lot of overlap in those various decision-points, obviously. However, that only makes it more obvious to me that the final decision around SD cards vs. on-board storage is one that’s worth a lot of thought, despite a lot of folks’ knee-jerk reaction one way or the other.

TWO
We actually experience a decent number of events in support where the root-cause of failure is the SD card. This is troubling for us, since it wastes time from both us and the customer going down the wrong path (thinking something is wrong with the balena components or their configuration). So we are brainstorming internally to determine if storage failure is something we can provide metrics around in the dashboard and/or have some method for more quickly and accurately determining SD card failure as root-cause early on in our troubleshooting.

To answer this question more exactly:

I would be interested to hear what sort of devices you find yourself supporting most among paid and unpaid customers, especially those working on smaller IOT device projects.

We have a very interesting mix, and it’s another reason we’re taking this so seriously right now. In the past, our customers have typically started smaller fleets with devices using SD-cards (especially because so many of them start by prototyping on something like the Raspberry Pi’s) and then eventually move away from them as their fleets have grown. But I recently worked with a customer to on-board over 15,000 devices which use SD cards and it made me start thinking deeply about what affect that would have on us in Support and what we could do to help improve troubleshooting experiences. It also turned on its head the idea I’d always had that large fleets didn’t use SD cards. Now, that’s not to say this fleet won’t learn a painful lesson - it may. But what if it doesn’t? What if this is absolutely the right configuration for them based on their user-base, service model, and cost structure? I’m interested to see how it goes, and in the meantime doing what I can to soften that blow on the balena side if possible.

Anyway, thank you so much for having this discussion with me. I’m curious what you think of the above.

[maggie0002]

This is all really interesting, thanks for sharing. A really big take away for me on this is your comment on the root-cause of issues being SD card failure and it being hard to diagnose, This is so true, I have seen this in the past and I think it deserves its own bullet point as a key consideration. Many hours lost in diagnosis of weird issues that are hard to attribute to the SD card without swapping it out first. It would be a major consideration for me in support.

I wouldn’t want to have to advise businesses on the best way to cost their hardware (SD card vs no card), it’s such a difficult task. If you are of the scale of 15,000 units, I would probably keep a box of SD cards pre-flashed and then if in doubt just swap it out. Harder to tell someone to do that though if they only have one device, they don’t want to pay an extra 5USD just on a hunch.

I have had similar issues with power supplies, if you get one without the right spec it can cause issues that will occupy you for days (network limitations, CPU throttling, the list goes on and doesn’t always provide sufficient indication). For that reason I tend to suggest people buy official ones (such as for the RPI) for a little bit more just to avoid those undiagnosable problems (“I picked up a USB cable from the market for 1USD and now my device does x or y”).

Another consideration is the data stored on the device. For many use cases, swapping an SD card for another one can work if there isn’t any important data stored locally. You just provision the device with the stock software and off it goes. But if there is local data stored in volumes on the device that can’t be replaced, such as offline collecting data from sensors or users in a database, then SD card may be a no for me. Too high risk of loosing irreplaceable data, plus if it is a device doing that sort of thing then the writes to the card are much higher (the writes being the bit that damages cards, not the reads).

I don’t envy your role having to advise on these things. There really are too many dimensions to be able to produce a useful guide (although I would love to be proved wrong), especially on the cost per unit debate. I guess if I was advising someone I would be sure that they had fully understood the risk in SD card to prevent me being as liable.

Very interesting to hear about the attempts to provide some metrics. I imagine it would be very possible though. I suspect something similar to what Apple did for their battery monitor on iPhones, it measures how many cycles it has gone through (writes in the case of an SD card) and then provides an indication of those cycles/writes. Like Apple’s battery monitor though, it would be very unpredictable. For some users though it could help, you could establish a policy to replace cards after every x writes, which would reduce failures and allow for much better long-term costing of servicing.

I suspect something to accompany that would be doing some stress tests on a bunch of SD cards. Writing to it over and over and get an average of how much data tends to be written before it fails. Much harder than it sounds of course, as you mention it’s not always clear when a card has failed, environmental factors and blind luck/bad luck with certain cards may play a part. Some indication is probably better than none though I guess, as long as it’s heavily caveated.

I think another area that might be worth exploring (which I haven’t yet) is external storage for running Balena. I have come across a bunch of cases where people boot off a USB pen to run Raspberry OS. It provides far quicker read/write speeds than an SD card, is less prone to corruption, cheap hardware, easy to swap out (you could leave hardware in place and just swap the USB pen), easy to flash, familiar to non-techy users, easy to expose the port on cases without worrying about tampering, could offer TB of data storage, I could probably think of more. Being able to provision a USB pen directly from my computer rather than hunting around for the card reader that never seems to be where I put it sounds appealing too.

A quick Google search and I’m sure you will find examples, but I think the idea in a Balena context would be an extremely extremely light Balena image on the SD card that does nothing but tells the device where to boot from (this card would have very little data on it and never have writes so life span would be improved considerably, or on at least some hardware you can write to the hardware the instruction to boot from USB and then not need the SD card in the device anymore). Then the main Balena OS on the USB pen which it would boot and run from. Come to think of it, this would also make those cheaper boards that contain tiny eMMC (like 2mb) to good use, maybe those could be flashed with the simple instruction to boot from the USB pen keeping the cost of hardware down (much cheaper than 32GB eMMC). I’m convincing myself of this more and more as I write, wow what a boon this would be for development! I would have an SD card in my devices that boots off a USB, then only ever have to change the software on my USB pen. No more slow flashes of SD cards, finding card readers, slow read and writes when doing complex build processes, that would transform my whole workflow! Plus I wouldn’t be running the risks of burning through corrupt cards in development. This may have been what I would have considered for your 15,000 unit customer (without knowing more details) as a middle ground (or eMMC as additional hardware RasPiKey: Plug And Play EMMC Module For Raspberry Pi | The Pi Hut, although with that many units I would take some convincing to pay out).

Making sure I circle all the way back around to the original topic of the user who posted this thread, seems to me there could be an equal number of use cases for SD card and fixed hardware for both hobbyists, small business and corporate, all dependent on their needs. I would encourage supporting both.

Would I be correct in thinking it was an SD card based OS that went to space? Not a bad advert for SD cards: Beyond the cloud: Docker containers in space

[maggie0002]

These two now complement each other nicely. Covers a vast array of use cases at low cost.

	Raspberry Pi Zero 2 W	Orange Pi Zero 2
Price	$15 (official price)	$24 (AliExpress)
CPU	Cortex-A53 (four cores)	Quad-core Cortex-A53 processor
CPU capabilities	32-bit and 64-bit capableARMv7 + ARMv8-A support	64-bit
CPU frequency	1 GHz, can be overclocked to 1.2 GHz+	1.5GHz
GPU	VideoCore IV	Mali G31 MP2
RAM	512 MB (LPDDR2), as part of the SiP	512MB/1GB DDR3 (Shared with GPU）
Network	802.11b/g/n	802.11 a/b/g/n/ac; Gigabit Ethernet
Bluetooth	Bluetooth 4.2 / BLE (Bluetooth Low Energy)	Supports BT5.0
HDMI port	Mini-HDMI Type C	Micro HDMI 2.0a
USB port	USB OTG microUSB	3*USB 2.0 HOST（Two of them are via 13pin interface board）
Composite video	As solder test point	Via 13 pin header
Reset pin	As solder test point
Camera port	Zero form factor MIPI CSI port
Power consumption	Max. 3W (0.6 A)
Power supply	microUSB	Type-C interface 5V2A input
misc. differences	advanced heat dissipation through copper layer for higher performance	Has 13pin header with 2*USB Host, IR pin, Tv-out、AUDIO (no MIC) and 3 GPIO ports.
form factor	Zero form factor (65mm × 30mm)	53mm×60mm
distribution	US, Canada, EU, UK and Hong Kong. Compliance for more countries is in the works (e.g. Australia, New Zealand are expected in November 2021).	worldwide

[maggie0002]

Just came across this useful blog post on Sd card technology. All about SD card / microSD card health on the Raspberry Pi — PiCockpit | Monitor and Control your Raspberry Pi: free for up to 5 Pis!

[jmakivic]

Thanks for sharing this blogpost @maggie0002 !

[maggie0002]

You guys are going to start thinking I am working for Orange Pi I bring it up so often.

But here is another one just released, Orange Pi 3 LTS: Orange Pi 3 LTS - Orangepi

It has 8gb emmc and is $35USD. Game changer for those worried about SD cards. The range is just too superior for the price for me not to keep plugging it.

[pranavpeshwe]

ha ha! No worries!
Thanks for sharing the LTS link. The emmc, other hardware specs, and supported OSes sound very good for that price!

Best regards,
Pranav

[radexter]

Hello friends,

thanks for all the work promoting the official support for the Orange Pi.

Are there any news or this device has been added to the roadmap? It would be a great addition to the lineup!

Best regards,
Renzo

[Jazzagi]

Hey @radexter

Which model of the Orange Pi is it you’re working with? We currently have support for the following:

Orange Pi One
Orange Pi Plus2
Orange Pi Zero

[stokes776]

Would just like to throw my hat (and my companies hat) in the ring for Orange Pi support, specifically OrangePiZero2.

Facing down the supply chain challenges, our hardware team has pivoted to readily available and suprisingly well built OrangePi boards. Now, on the software side we are being asked to get a fleet image ready for them and I was surprised there wasnt one for OrangePiZero2. I had already given my verbal that I thought Balena did offer some OrangePi support but I didnt look at specific models.

Do ANY of the current Balena images work for OrangePiZero2, even with drawbacks (like wifi not working)?

[maggie0002]

I have gone back and forth on the Orange Pi. For the most part, I still think it is an excellent board option. Overlooking one of the main IoT viable boards made in China is probably not very prudent as they are only going to get better and cheaper. That said, they are already very well made and have made real leaps in the last few years in particular. Distribution and supply is usually solid, especially considering the chip shortages, and their issuing of LTS (long-term support) boards really gives confidence that these are viable for production. An impending launch of the RPI Zero 2 raised a question mark over whether the OPI Zero 2 would still be in demand, but now we have seen the RPI Zero 2 launch we can see that they are very different devices, and the OPI Zero 2 still has it’s own place in the market that hasn’t really got any equivalent.

The new OPI’s are starting to use RockChips which is interesting (Orange Pi 4 LTS - Orangepi), that may be an easier board to build an image for as I think a RockChip Yocto layer already exists for Balena. But for the Orange Pi Zero 2 it is the Allwinner chip that seems to be the blocker? There is a maintained bunch of layers for Allwinner (more recent that the current submodule in the Balena Allwinner repo):

GitHub - linux-sunxi/meta-sunxi: Official sunxi OpenEmbedded layer for...

And interest from maintainers in Orange Pi Zero 2 if issues are encountered:

orange pi zero 2 - when will it be supported?

Perhaps someone from the Balena OS team could shed some light for us on what is required to be able to support the Orange Pi Zero 2 as a community image? I’m aware of the build instructions (balenaOS - Docs), and a Balena Allwinner layer for the H3 (GitHub - balena-os/balena-allwinner). Before getting lost in the trial and error rabbit hole, could someone highlight the blockers that will likely be encountered?

I haven’t tried any other images on the Orange Pi Zero 2, perhaps someone could shed some light on that too?

[alexgg]

Hi, thanks for your detailed message. Community board contributions are very welcomed, and if you are willing to attempt supporting Orange Pi boards we will strive to provide as much support as needed. The Balena devices/OS team keeps a queue of new boards to support which is driven on customer requests and our Custom Board Support service. The vision for the OS is to make new boards support as easy as possible, but an understanding of the hardware and BSP is usually required. I will ping the team members that have worked on the Orange Pis in the past see whether they have any feedback.