Host Balena Os on AWS as a virtual device

Hello,

For our test and demo environments I’d like to be able to host some ‘virtual’(?) Balena devices in AWS. I would like to manage them through the regular Balena dashboard as much as possible. I just need a place where they’ll stick around without the need to organise a bunch of hardware.

I managed to get an instance going in VirtualBox. But AFAIK you can’t put VirtualBox on EC2 or in a Docker container. Otherwise I could make something manageable using Vagrant.

I tried to make Balena in container work under Kubernetes. But I couldn’t figure it out. It seemed like Kubernetes would not let Balena do al the mounting things and I could not find a way around that.

I found this pull request that looks perfect for me. I could just start an EC2 container with a Balena OS AMI. But the pull request looks like it has been idle for a while. And I don’t even know where to begin with that to contribute, looks like a really steep learning curve there.

Since I got it going on VirtualBox I’m going to try if I can import that somehow using AWS VM Import. But that feels like a bit of a long shot not to mention cumbersome. But if I manage to get EBS volumes for boot, state and data at the right mount points it might work right?

In conclusion; I’m kind of stuck with this. Does anyone have any practical tips on how I could get a few virtual Balena devices setup on AWS?

Small update. The AWS VM import doesn’t work. The AWS import process fails with ClientError: Unsupported GRUB configuration - Unable to determine kernel version

Hey there @ErikHH I definitely know some of my colleagues have been working on this but I’m not sure what the latest is or if their work has been written up anywhere. Let’s ping @ab77 who I think may be able to give an update and or some pointers when he gets a chance.

Hi @chrisys, thanks! I’ll try and be patient now.

Hi,

So, I’ve managed to produce a BalenaOS AMI for myself using the generate_ami.sh script.

I run the whole thing like this:

export WORKSPACE=$(pwd)
export AWS_ACCESS_KEY_ID="XXXXX"
export AWS_SECRET_ACCESS_KEY="XXXXXX"
export AWS_DEFAULT_REGION="eu-west-1"
export BALENA_PRELOAD_APP="one-of-my-balena-apps"
export S3_BUCKET="my-s3-bucket"
export IMAGE="${WORKSPACE}/genericx86-64-ext.v2.58.6_rev1.dev.img"
export AMI_NAME="onb-test-2.58.6_rev1-v11.14.0"
export BALENA_ENV="balena-cloud.com"
export BALENACLI_TOKEN="XXXXXX"
export BALENA_PRELOAD_SSH_PUBKEY="ssh-rsa XXXX <From AWS key pair>"

balena os download genericx86-64-ext --version "v2.58.6+rev1.dev" -o $IMAGE

docker run -it --rm  \
    --privileged  \
    --network host  \
    -v "${WORKSPACE}:${WORKSPACE}" \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
    -e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
    -e AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION}" \
    -e AWS_SESSION_TOKEN="${S3_SESSION_TOKEN}" \
    -e AMI_NAME="${AMI_NAME}" \
    -e S3_BUCKET="${S3_BUCKET}" \
    -e BALENARC_BALENA_URL="${BALENA_ENV}" \
    -e BALENACLI_TOKEN="${BALENACLI_TOKEN}" \
    -e BALENA_PRELOAD_APP="${BALENA_PRELOAD_APP}" \
    -e PRELOAD_SSH_PUBKEY="${BALENA_PRELOAD_SSH_PUBKEY}" \
    -e IMAGE="${IMAGE}" \
    -w "${WORKSPACE}" \
    resin/balena-push-env /bin/bash -c 'apt update \
      && apt install -y python3-pip \
      && pip3 install awscli \
      && ./generate_ami.sh'

And after a while I have a brand new Balena AMI in my AWS account!

Then I fire up a t3a.small instance (it’s roughly the same spec as our actual devices).
And then… sort of nothing happens at all. The AWS instance says it’s running, but nothing
shows up in the Balena dashboard. And there is really no way I can get to it for any trouble shooting. The EC2 serial console doesn’t show anything.

When I try SSH on port 22222 using the preload key the first attempt I get:

kex_exchange_identification: read: Connection reset by peer

Subsequent SSH attempts are rejected with a Connection refused until I reboot the instance.

Any idea on how I can begin to troubleshoot this? Why can’t I get in with SSH?

Hi,

Good to see that you made progress on running it with AWS instance. The kex_exchange_identification looks to me like a routing problem. Can you share a verbose output of ssh with ssh -v <instance-addr>?

Also, if I understand you correctly, you are able to see the device online on the dashboard. Can you try the console/terminal from the dashboard on that particular device?

Hi,

Here is the verbose SSH output:

OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Reading configuration data ~/.ssh/config
debug1: ~/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to 54.154.135.226 [54.154.135.226] port 22222.
debug1: Connection established.
debug1: identity file ~/fleet-key.pem type -1
debug1: identity file ~/fleet-key.pem-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.1
kex_exchange_identification: Connection closed by remote host

The main problem is that the machine does not show up in the Balena dashboard after the EC2 instance is started. There is no console or terminal for me on that end.

Hello, the image you are converting to an AMI must have your SSH key preloaded (e.g.) Configuration - Balena Documentation

You must have the private key corresponding to the public key you are inserting into the config. Otherwise your image will boot up, but you wont have any access into it. You can use balena-cli to configure your image or just mount the /boot partition from the image and update config.json manually. AWS SSH key-pairs have no effect what so ever in this configuration.

The other possibility, if you are using a configured images for our application, is that your EC2 instance running balenaOS, doesn’t have Internet access. This could be because it’s sitting in a private subnet without a NAT gateway.

I would recommend you start with a configured image (download from the dashboard). If you are using an anonymous balenaOS image, you’ll need to preload it with the correct application, which actually does the configuration and joins to balenaCloud. Our preload code isn’t currently public.

Hi @ab77,

Thanks for looking into this.

I’m really sure I tick all the boxes you mention.

  • It has internet connectivity. It’s a EC2 instance with public IP in the default VPC. I’ve double checked it by launching a regular AWS Linux in the same place.
  • I have the private key to that pair and it matches the public key I inserted in the config.
  • I have tried over the weekend with a pre configured image that I downloaded from the dashboard. Put my own ssh key into that. But the result in the end is exactly the same.

So at this point I don’t see this going anywhere soon. So I’m just going to call this a day. If someone has a bright idea I’ll give it another go. But I need to spend time on other aspects of our project now. We’ll just have to make do with the few real devices we have at our disposal.

The only other thing worth trying is to deploy a dev. instance of balenaOS (** ensuring SSH access is locked down in the AWS EC2 security group to your IP only**). That way you know it’s going to be listening on SSH port 2222 without username/password so you can terminal in and troubleshoot.

That’s a good idea. I’ll give that a go when I have some time.

Hi @erikhh Did you get chance to try this out yet? Let us know if you still require assistance.

Hello, I was also able to create an AMI with the generate_ami.sh scirpt. The instances created from that Image show up in the balena dashboard but the don’t start the provisioning.
With every reboot of my ec2 instance, a new device is created in balena.
I am also not able, to get shell access.
In a screenshot, I took from the ec2 console, there is an error from the resin flasher service.


Any Ideas?

Hi Stefan, can you try deploying a dev. instance of balenaOS (ensuring SSH access is locked down in the AWS EC2 security group to your IP only). That way you know it’s going to be listening on SSH port 2222 without username/password so you can terminal in and troubleshoot.