Hey team Balena,
Your docs on environment and service variable doesn’t mention the maximum length of environment or service variables. I need to add an allow list feature and spend very little time doing it.
Is there a maximum char limit on the value side? Can I get away with variables in the low thousands?
There doesn’t seem to be a problem in my testing up to a couple hundred at least but I wanted to be sure before I push this to my sometimes connectivity challenged fleet.
-Thomas
Hey @tacLog, besides the maximum length, the byte size of the string is also a consideration. Low thousands of characters or less than 1MB shouldn’t be a problem. I could say that with confidence.
Then I don’t know the exact limit and I asked internally. I also opened this issue to document this:
Thank you for the ask. We’ll give an update to clarify the exact limit.
Cheers…
Hello.
While we currently rely on the (generous) database limits only, consider the fact that such variables are fetched on each poll that doesn’t return 304 and, as a result, can significantly increase your traffic usage.
If you’re storing some labels or metadata, you may consider tags for that purpose.
Hey @gelbal
Thanks for the update, it sounds like my use case of a couple hundred characters will be fine. I look forward to seeing what kind of limit you guys apply and always appreciate being able to follow issues in git.
While I am by no means new to balena OS or the balena engine, I am afraid your answer goes out of the depth of my knowledge.
I tired to answer my own questions with docs and this forum but would like you to confirm I got it right if that’s possible.
What is a poll?
I am assuming it is state update requested by the supervisor on the device from the balena backend.
How often does a poll occure by default?
I think this might be controlled by: “BALENA_SUPERVISOR_POLL_INTERVAL” which appears to default to 90000 ms so 15 minutes.
I couldn’t answer these:
When would a 304 return?
Would the polling interval of 15 minutes apply to when a 304 returns or would it try again sooner?
Is the polling interval of my entire fleet going to be in sync if there is a mass power cycle on location?
Thank you for your time,
-Thomas
As @pranasziaukas suggested using tags.
Which I could use because my devices already set and get their tags to report back statistics of how they are doing at their purpose.
Would it be better for this allow list of several hundred characters to be a tag?
It makes it slightly more difficult on my end because I have to handle getting the tag and failing to get the tag via the api. As my devices have to support full functionality in offline mode, I then also have to handle caching a allow list to the local file system, and deciding when to invalidate that cache.
However, unless I am misunderstanding things the benefit of putting this allow list in a tag is that the state poll from the supervisor won’t pull the list every 15 minutes and the only way the device will only pull the list when it requests the tag from the api.
Are their any other benefits? Because for my current list length of a couple hundred characters, I am still planning on doing this the easy way with an application variable that will always be present if set.
-Thomas
Hello @tacLog ,
That’s correct, at each BALENA_SUPERVISOR_POLL_INTERVAL the device will check if there were any changes (more info here and here) and retrieve the potentially big env var, which can cause significant bandwitdh use.
When a 304 is returned, it means that there were no changes (environment variables, releases, etc.), so in that case, the env vars would not be retrieved.
The device checks (and retrieves if something changed) for changes on every boot. So if there’s a mass power-cycle, all those devices will potentially re-download the env vars.
Hope this makes things a bit clearer but do let us know if you have any other questions.
Cheers,
Nico.
Hey @taclog, just a heads up that as a result of this thread we’ve updated the documentation to reflect the max length of environment variables. You can see the merged PR here for reference:
Take care!
Thanks for the update! 1 MB is an insane amount of data and should work great for us.
However, if my feedback is welcome. The documentation doesn’t not align with what I learning in this thread.
“Note however that variables are communicated to a device every time it checks in with the API, which may potentially result in a lot of [network traffic][bandwidth-control].”
This is not true in the way I read this line. See:
I read the docs as saying that my env var would be downloaded each time on the BALENA_SUPERVISOR_POLL_INTERVAL however they should only be downloaded in the cases that @ntzovanis outlined above. This would make a huge difference in how I interpret the bandwidth usage of a env variable value that’s even 20 KB.
I just wanted to point that out because I would have taken the documentation at it’s word and never posted this thread had it said that. It would have changed the outcome of our design decision and increased the cycles I had to put into this tiny feature. Env variables appear to be a safe way to enter a moderate amount of configuration data and are a great quality of life feature.
Hopefully that comes off as helpful more than overly semantic. I really apricate the top quality support and transparency you guys provide.
-Thomas
Thomas, your feedback is welcome and helpful! We’ll revisit that part of the documentation and make sure it reflects how often and under which circumstances the env var is actually downloaded. Thanks for pointing out the discrepancy!
Hey Thomas, thanks for the feedback. Would you consider a phrasing like:
Values can be up to 1MB (or approximately 1 million characters) in size each. Note that a device will re-download the variables every time its state changes in the API, which may potentially result in a lot of [network traffic]
To be more clear?
Hey @nazrhom,
This is more helpful to me as a experienced user that has a rough idea for what it means for a devices state to change in the API at this point. I worry about what I would think if I were reading that as a new user to Balena as I was last year.
It is certainly better all round.
I think the links provided in the original pull request for this fix would still help explain what a state is and what would cause a change.
The update interval can be configured with the [BALENA_SUPERVISOR_POLL_INTERVAL configuration variable][poll-interval].
Assuming that tag would link to this page:
Where this section has great information pertaining to what is a state change: BALENA_SUPERVISOR_POLL_INTERVAL
Again you guys are showing how awesome you are by even concerning yourself with such a fine detail in your docs.
Keep it up, the docs are what made our company choose Balena.
Thanks Thomas, glad we were able to clear it up. The recent PR from my colleague added the clarification to make it more clear and we also linked it to the right place - Clarify section about device env vars and bandwidth usage by nazrhom · Pull Request #1772 · balena-io/docs · GitHub
Please do reach out if you see any areas where more context could be helpful, happy to help!
Hi @vipulgupta2048 thank you for updating the docs on env variables.
I posted this elswhere (Openbalena CLI optimization - #11 by barryjump) but I noticed openbalena in particular does not permit creating envs with this kind of format while Balena Cloud does:
DD_TAGS: name:joe env:test another:true
My use case is for Datadog which expects their tagging mechanism to be set with multiple tags defined in a single value field of the DD_TAGS separated by a space.
Any thoughts on if that could be fixed or at least documented as invalid for the time being?
Hello,
Reading the thread you linked, did you try the method my colleague suggested on that thread. That should solve your problem. If not, it’s odd for openbalena to not support env variables in this format while balena Cloud does and we would need to investigate more into it. Can you check and let us know the error message that you get on that thread only. Thanks!
@vipulgupta2048 indeed you’re right, I was making a rookie mistake - thanks for assisting!