Balena Fleet Questions (GitHub, privacy and preloading)

[maggie0002]

A couple of Balena Fleet questions I haven’t been able to find answers to before migrating:

1. I see for Balena Blocks that GitHub submodules are not supported? I assume that is the case for Balena Fleet too?

2. Can you provide a GitHub URL that points to a specific branch, or will it always use whichever default branch is set on the repo?

3. I see there is an interactive map available of active devices. Can that be disabled? It is in my users interest to not have the countries, nor any more specific areas the devices are operating in public knowledge. Privacy is such a hot topic.

4. A lot of the fleets say: ‘Allow a few minutes for your device to download the application.’. Are the images not preloaded?

Assuming the pre-loading isn’t available, can we still create open fleets but preload an image with my app that is connected to the open fleet, and then distribute that image file myself?

If it isn’t preloading, what is the connection to the GitHub repo for? Merely to fetch the Balena.yml?

[maggie0002]

It may have answered the GitHub default repo question by itself? Doesn’t appear to pick up the balena.yml file: https://hub.balena.io/learner_s_block/lb-armv7hf

[maggie0002]

And another thing that has crept up. Hoping some of this feedback is helpful. I am offering multiple architectures (armv6hf, armv7hf etc.) to users for the same app. In Balena this means I have to create a different application for each and deploy to each individually due to the build processes having to run for each. This isn’t a major issue, but when it comes to using Balena Fleets, I then end up with three apps listed in your library, which from the fleet interface all do the same thing. If I do not list all three on the library, then they are not down as public and do not benefit from the unlimited devices being offered to open source non-commercial products.

This isn’t a big problem for me, but presumably your library is going to get a bit messy in time. Would be good to know if there is another way you would like me to go about it.

[maggie0002]

Turns out a mistake on my end. I thought adding the GitHub URL would read from the balena.yml file. Turns out you need to push the app first before it will read it.

[chrisys]

Hey @maggie0002 thanks for all of the questions and thanks for trying the open fleets functionality on balenaHub!

I’ll try to answer everything here individually but feel free to come back to me if you have any more questions. The open fleet functionality, along with the rest of hub, is in active development so now is a great time to let us know your needs and influence the road map.

I see for Balena Blocks that GitHub submodules are not supported? I assume that is the case for Balena Fleet too?

The important thing we’re trying to promote is that blocks are reusable containers or chunks of functionality, how they’re implemented is pretty much up to the developer. Blocks are thought of as not doing anything useful standalone and so someone is always going to have to integrate them with other functionality. In regards to that then it shouldn’t matter if submodules are used or not as they’re a separate aspect of source control that’s unrelated. We also have a blog post introducing blocks that has a bit more info. As for fleets, the same applies here; it doesn’t matter if submodules are used in GitHub because the full application source is pushed when you create a release.

A lot of the fleets say: ‘Allow a few minutes for your device to download the application.’. Are the images not preloaded?

The images are not currently preloaded, but dynamic preloading is a problem that we’re working on solving. It’s a hard problem to solve, as the preloading process takes time and so isn’t feasible to do on-demand, and simultaneously we don’t want to preload and store images for every supported device type for every release. There may be a concept of serving a file that contains both the OS image and the containers and then we apply them after download, e.g. within Etcher, or on the device itself.

Assuming the pre-loading isn’t available, can we still create open fleets but preload an image with my app that is connected to the open fleet, and then distribute that image file myself?

Absolutely!

If it isn’t preloading, what is the connection to the GitHub repo for? Merely to fetch the Balena.yml?

It is used in the ‘fork this fleet’ process to enable a user to deploy a copy of the code for themselves. Additionally, open fleets are only available for non-commercial, open-source projects, so this link should establish a link between the fleet and the open-source code that’s running on that fleet. We’re working on methods to synchronise a fleet with a GitHub repo, so that whenever a merge is made to the main branch, a release can automatically be created and update the fleet (if desired).

Can you provide a GitHub URL that points to a specific branch, or will it always use whichever default branch is set on the repo?

For the most part we’re just looking for a link to the repo, but it is possible to specify a direct path to a tarball using balena.yml that will come into play when a user uses the ‘fork this fleet’ or ‘deploy with balena’ functionality - read more.

I see there is an interactive map available of active devices. Can that be disabled? It is in my users interest to not have the countries, nor any more specific areas the devices are operating in public knowledge. Privacy is such a hot topic.

Currently it cannot be disabled, but I’ll record this request and put it on the roadmap as a configurable option to implement. It’s worth mentioning though that the location data is based on IP geolocation so isn’t super accurate, and then as a secondary measure, the reported latitude and longitude is limited in accuracy to 10 miles.

I am offering multiple architectures (armv6hf, armv7hf etc.) to users for the same app. In Balena this means I have to create a different application for each and deploy to each individually due to the build processes having to run for each.

You’re 100% correct in raising this as an issue; we’re working on enabling multi-architecture fleets to solve this very use case. The current thinking is that you’ll be able to enable builds for additional architectures within a single fleet, so any of your application’s supported device types will be able to join and download a release built for their architecture. As for today though, you’ve worked around it the only way we know how. You’ll see we did the same thing for the Rosetta app.

thought adding the GitHub URL would read from the balena.yml file. Turns out you need to push the app first before it will read it.

Yup you got it, the idea is that the balena.yml file is versioned along with the release that’s running on the fleet. If you’ve (for example) supported an additional device type with a new version of the code, you’d push that version and the new device type would appear. If there’s an issue which means you need to roll back and pin the fleet to an older version, then that device type would disappear again, as of course the older version doesn’t support it.

Thanks again for all of the feedback! Myself and some colleagues were really happy to see someone making use of this new functionality in the way that it’s intended, and we’d love for you to keep in touch and continue to let us know how we’re doing or if there’s anything else you need.

[maggie0002]

Thanks for this info.

To clarify it was the deploy button where I had seen that submodules were not supported: Deploy with balena button - Balena Documentation. Presumably then the clone option also doesn’t work with submodules?

All sound very promising. I will be keeping a close eye on that map disabling option. It doesn’t matter these days that your purpose is legit, people are still deterred by adding their locations like that (myself included). Especially when there is only a small number of devices as identifying users is that much easier.

[maggie0002]

On the auto deploy from GitHub after a commit and preload issue.

It seems rare I would ever need to auto push a commit from my repo to a live server. Other services do these sort of things, like Netlify, but they are web services which produce a demo of your commit when you create a pull request to confirm changes. Doing this for Balena would be demanding on the systems, but without would make it hard to enable an auto commit.

I think my preference would be to use GitHub actions. Their systems are free for open source projects and powerful. Here is an example of mine, although is more complicated than it needs to be right now: learners-block/deploy-release.yml at main · LearnersBlock/learners-block · GitHub. It may be worth considering maintaining a Balena GitHub/Git action instead/as well. Would allow a lot of customisation on under what circumstances to deploy, and can be a one click install.

Stripped down example that would run when pushing a tag to GitHub vX.X.X:

name: "Deploy Balena Release"

on:
  push:
    branches:
      - "main"
jobs:
  deploy-release:
    name: "Deploy Balena Release"
    runs-on: "ubuntu-latest"

    steps:
      - uses: actions/checkout@v2

      - name: Create build context
        env:
          BALENA_CLI: https://github.com/balena-io/balena-cli/releases/download/v12.44.16/balena-cli-v12.44.16-linux-x64-standalone.zip
        run: |
          sudo apt-get update && sudo apt-get install -y \
              curl \
              unzip
          cd /opt/ && \
            sudo curl -O -sSL $BALENA_CLI && \
            sudo unzip balena-cli-*-linux-x64-standalone.zip && \
            sudo ln -s /opt/balena-cli/balena /usr/bin/ && \
            cd $GITHUB_WORKSPACE

      - if: startsWith(github.ref, 'refs/tags/v')
        name: Create and deploy production code to Balena
        env:
          BALENA_API_TOKEN: ${{ secrets.BALENA_API_TOKEN }}
        run: |
          balena login --token ${BALENA_API_TOKEN} > /dev/null
          balena push ${BALENA_APP_NAME}

Variables like ${BALENA_API_TOKEN} use GitHub secrets: Encrypted secrets - GitHub Docs

For official actions, they are built using Docker: Creating a Docker container action - GitHub Docs. Then a user would use one section and just add variables:

  - name: Deploy to Balena
    uses: balena/balena-deploy-action
    with:
      api_token: BALENA_API_TOKEN
      app_name: MY_BALENA_APP_NAME

Another thing that has crept up to. It would be helpful if we could set the name that appears on the fleets page (https://hub.balena.io) from the .yml file. At the moment it seems to pull from the app name on Balena Cloud.

[chrisys]

We are working on ‘official’ GitHub actions as well, and are actually quite close to releasing something there, we are going the route of the Netlify ‘preview’ approach as well. A PR would create a draft release (effectively testing the build process) within the linked application, which you could then apply to specific devices to test it, then when that PR is merged, the draft release would be changed to a regular one and devices updated (if your release policy permits).

Another thing that has crept up to. It would be helpful if we could set the name that appears on the fleets page (https://hub.balena.io) from the .yml file. At the moment it seems to pull from the app name on Balena Cloud.

The approach we’re taking on this is to remove things from balena.yml unless it makes sense to have them versioned. The application name actually was set there until our last release party, where we took the decision to take it from the API instead. Consider a fleet to be analogous to a GitHub repo - there are certain settings that are separate from the code (such as the repo name). I expect ultimately that we’ll remove things such as the logo and description from balena.yml and move them to the cloud as well. I know it’s a transitional issue until we have multi-arch apps available but setting it in the yml file would also cause issues with multiple apps supporting different architectures as you’re doing currently. As I mentioned above though hopefully we’ll be able to rectify that particular problem soon.

[maggie0002]

@chrisys

I see there is an interactive map available of active devices. Can that be disabled? It is in my users interest to not have the countries, nor any more specific areas the devices are operating in public knowledge. Privacy is such a hot topic.

Currently it cannot be disabled, but I’ll record this request and put it on the roadmap as a configurable option to implement. It’s worth mentioning though that the location data is based on IP geolocation so isn’t super accurate, and then as a secondary measure, the reported latitude and longitude is limited in accuracy to 10 miles.

Would it be possible to have a tracking issue for this? It’s a real issue having it visible, especially as I really feel I need to explain it to users before they install the device and the message is very off putting to potential users. It would be helpful to have a tracking issue on GitHub to avoid having to keep checking back for updates on its implementation.

[chrisys]

Hey @maggie0002 currently balenaHub is closed source so I can’t add an issue there and share it, but I don’t think there’s any reason why it needs to be. Leave this with me and I’ll get back to you. I’ll try to see if there’s any way we can offer the tracking you’re asking for. That’s not to say we’re going to be working on it immediately as we’ll need to plan how this fits in with everything else, but the record of the request is there.

[maggie0002]

@chrisys did you come up with anything for the tracking? I could create a new forum thread if you think that would be easier, and if you were able to add the forum thread as a note in your tracker to update when it is available? Would mean I could get email notifications.

Seems odd to track this way I know, but the map thing is a real challenge for me. It’s not just the scaring away privacy conscious users but I’m not even sure where I stand on GDPR with it as it could be seen as use of personal data (location, even if it is within 10 miles because it is pretty accurate).

[chrisys]

@maggie0002 we do have this forum thread linked so we’ll definitely notify you when it’s available. There’s no update yet though, as we’ve been focussing efforts on getting other parts of the platform in line (notably the applications > fleets rename, release versioning efforts, image preloading and other stuff discussed in this thread). We haven’t yet had the discussion to see how we’d want to implement the proposed map option as there are a lot of other key bits of functionality we want to ship that are ahead in the queue at the moment.