I am trying to make my openBalena stack stateless such that I can put multiple instances behind a load balancer / auto-scaling group in AWS. I was able to start up an RDS instance and connect openBalena to it, removing the need for the db to run within the openBalena stack. To my understanding the only data stored on my instance should be from the registry. I am trying to set up the registry to use an AWS S3 bucket instead of storing the data in the regisrty or s3 containers. I have updated the environment variables for the registry container to point to https://s3-us-west-2.amazonaws.com, as well as providing it AWS IAM credentials granting access to S3. Unfortunately it seems that these changes aren’t enough. Anyone have any ideas on how to accomplish this?
We added support for configuring registry to look into S3 just a few days ago, in version 2.0: https://github.com/balena-io/open-balena/blob/master/CHANGELOG.md#v200 – it’s now just a matter of passing the correct values via config/env.
I noticed the update the other day, it is actually what started me on this path. Is there any documentation as to how to properly configure openBalena to use S3?
Not really unfortunately. The quickstart will automatically configure the registry to use a private minio-based S3 instance. You can point it however anywhere you like by editing these keys after you’ve created an initial config/env: https://github.com/balena-io/open-balena/blob/master/scripts/make-env#L68-L72
I updated my make-env script with:
export OPENBALENA_S3_ACCESS_KEY=“aws_access_key”
export OPENBALENA_S3_BUCKETS=“my_s3_bucket_name”
export OPENBALENA_S3_ENDPOINT=“https://s3.us-west-1.amazonaws.com”
export OPENBALENA_S3_REGION=us-west-2
export OPENBALENA_S3_SECRET_KEY=“aws_secret_key”
Ran ./scripts/quickstart -c -U my-user-email -P my-password -d my-domain
./scripts/compose up -d
Then from a different machine, logged into the openbalena instance via the cli and attempted to deploy to an app. When the image was built, it would attempt to send it to openbalena, but stays at 0% then throws an internal server error 500.
Just noticed my endpoint was for the incorrect region, updated it and testing again now…
It worked!
Glad you got it running. It’s best if you run quickstart to get the config created and then edit the config/env file and change the values you need. This way you won’t run into issues updating openBalena in the future. (The config folder is ignored by git, but the scripts aren’t and any changes you make may cause conflicts if you git pull)
Thanks for the heads up on that.
I am not sure if this would be best put here or as an issue on the openBalena github page, but functionality to enable using an IAM Role attached to an EC2 instance instead of hard coding an IAM Users credentials would be great.
Generally a feature request would be better in the component repo. It allows the work to be triaged more easily and also for you to be able to follow the development process of the feature.
Managed to migrate existing openBalena deployment to S3. I couldn’t get the migration script to work so I just copied data across from registry containers data volume to S3 using s3cmd. Once that was completed add the S3 config as above and restarted.
Awesome stuff again from the openBalence open source team.
@dfunckt ,
Hello dfunckt, balena team,
We have configured registry as mentioned above to point it to S3 bucket.
It starts Pushing images to registry…continues to 13% but eventually fails after couple of tries
Folder structure getting created with some files created.
Any idea about this issue?
Openbalena version -3.2.1
Docker version 19.03.6, build 369ce74a3c
@ab77, @gelbal , Any tips regarding this issue?
We have configured registry as mentioned above to point it to S3 bucket. Getting below in debug logs as It starts Pushing images to registry…continues to 13% but eventually fails after couple of tries
[Info] Saving release…
[Error] Deploy failed
received unexpected HTTP status: 500 Internal Server Error
Error: received unexpected HTTP status: 500 Internal Server Error
at Stream. (/snapshot/versioned-source/node_modules/docker-progress/index.js:53:19)
at Stream.emit (events.js:315:20)
at Stream.EventEmitter.emit (domain.js:482:12)
at drain (/snapshot/versioned-source/node_modules/through/index.js:36:16)
at Stream. (/snapshot/versioned-source/node_modules/through/index.js:45:5)
at Parser.onToken (/snapshot/versioned-source/node_modules/JSONStream/index.js:132:18)
at Parser.write (/snapshot/versioned-source/node_modules/jsonparse/jsonparse.js:135:34)
at Stream. (/snapshot/versioned-source/node_modules/JSONStream/index.js:23:12)
at Stream.write (/snapshot/versioned-source/node_modules/through/index.js:26:11)
at IncomingMessage.ondata (_stream_readable.js:717:22)
at IncomingMessage.emit (events.js:315:20)
at IncomingMessage.EventEmitter.emit (domain.js:482:12)
at addChunk (_stream_readable.js:295:12)
at readableAddChunk (_stream_readable.js:271:9)
at IncomingMessage.Readable.push (_stream_readable.js:212:10)
at HTTPParser.parserOnBody (_http_common.js:132:24)
From previous event:
at awaitRegistryStream (/snapshot/versioned-source/node_modules/docker-progress/index.js:43:12)
at /snapshot/versioned-source/node_modules/docker-progress/index.js:416:16
at processImmediate (internal/timers.js:456:21)
at process.topLevelDomainCallback (domain.js:137:15)
From previous event:
at DockerProgress.push (/snapshot/versioned-source/node_modules/docker-progress/index.js:415:56)
at func (/snapshot/versioned-source/build/utils/compose.js:227:34)
at retry (/snapshot/versioned-source/build/utils/helpers.js:145:18)
at async Promise.all (index 1)
at async Promise.all (index 0)
Hey @smita, have you made any progress with this? Do you still experience issues with it?
If so, before pointing the registry to S3, did you have a running openBalena instance already?
Hi @gelbal ,
Thanks for checking with me.
Issue is not resolved yet.
Before pointing to s3 it was working successfully on openBalena with local default repository.
Regards,
Smita
Hi @smita, what’s the device type of your image? What’s the balenaOS version of the image? Finally, can you note the balena-cli version you are using for deployment?
I wonder if this error stems from a compatibility issue. Please see:
I also pinged my teammate maintaining openBalena internally as he might have more ideas to check here.
Openbalena version (mentioned in VERSION file) is-3.2.1
cat versions under compose folder has below entries -
export OPENBALENA_API_VERSION_TAG=v0.109.2
export OPENBALENA_DB_VERSION_TAG=v4.1.0
export OPENBALENA_MDNS_PUBLISHER_VERSION_TAG=v1.7.9
export OPENBALENA_REGISTRY_VERSION_TAG=v2.13.1
export OPENBALENA_S3_VERSION_TAG=v2.9.9
export OPENBALENA_VPN_VERSION_TAG=v9.16.1
and balena client version is - balena-cli-v12.37.1-linux-x64-standalone
Let me try upgrading balena-cli version as suggested and get back to you with outcome
Hello @gelbal ,
I have tried to deploy it with latest version of balena-cli (balena-cli-v12.40.4-linux-x64-standalone) But outcome is still the same.
Regards,
Smita
Have you verified with AWS CLI that you can list the contents of your S3 bucket, using the keys you set here?
The simple test is to create an identify in ~/.aws/credentials:
[test]
aws_access_key_id = ...
aws_secret_access_key = ...
Then aws s3 ls s3://${OPENBALENA_S3_BUCKETS} --profile test --region ${OPENBALENA_S3_REGION} should return some content.
If this is not working, then your will need to double check your bucket name, region and AWS credentials.
Hello, I just wanted to follow up on this, did you manage to solve the issue? Otherwise did you verify the S3 credentials are working with AWS CLI as suggested by my colleague?