What are we working on? The Balena Notes

Hey @ashmastaflash that reminds me of the drone demo from last year’s Dockercon, when a thermal camera was enabled as a new piece of hardware on a drone, mid-flight (check the video here).

The Pi Zero + Enviro pHAT sounds good as enhancement demo, will definitely make some trial project around that. Pi Zero is a bit slow for demos sometimes (good enough for actual work, though), which would make me hesitate. Would work well with smaller audience or workshop, probably. A larger “brother” to that might be a Pi 3 + SenseHAT + battery pack combo.

The no-need-to-touch-the-SD-card is a very good angle, though, cheers!

Thank you, @sonya!

Whilst my ignite presentation was rhymed for the most, I think I’ll refrain for the rest of this post. :wink:

I’m currently working on several different bits of Resin, but some highlights from this week:

  • Looking into issues with resinOS discoverability on local networks, which has involved writing some C & BSD socket code for investigation (as an old embedded engineer, I secretly loved this!)
  • Updating our Development Environment to a new Linux kernel and Docker version, to allow engineers to work on some future Resin features
  • Working on internal processes to make the Resin workflow, from initial development through to deployment, as smooth as possible; at the moment I’m writing process bots that take Github webhooks and act on them (currently a versioning bot that modifies semver versions and Changelog files for a compoment based on merged PRs). We’re in the process of moving to TypeScript, so it’s been a great chance to use this to develop the process bots
  • Hoping if I get time today, to spend a couple of hours on my own hacks to get an RPi3 up and running as a serial programmer, via Resin, for some LPC microcontroller tinkering

And as the internal processes has me working very closely with him, I nominate @kostas for the next update! :smiley:

1 Like

Thank you @hedley !

What I’m currently working on:

  • Keeping track of roadmap items, prioritised projects and specification documents, which we often write to plan new features or address non-trivial issues. By the way, our plan is to give more information about our roadmap and ongoing specification document work to the community, which should happen in the following months.
  • Work on making our development and deployment processes more efficient. Currently this means moving forward with adopting conventions to automate changelog generation and versioning, with the ultimate goal being automated deployments that can account for inter-component dependencies.
  • Handle the logistics of other internal processes like our architecture calls
  • During this week I also plan to work on fixing some issues on rdt (https://github.com/resin-os/resin-device-toolbox) and resin sync (https://docs.resin.io/raspberrypi3/nodejs/getting-started/#using-resin-sync-to-develop-fast)

And I nominate @pablo !

1 Like

Thanks @kostas!

One of my main tasks is to maintain the Resin Supervisor, our agent on devices. I’ve recently been implementing a few bug fixes and adapting a few features to work correctly on ResinOS 2.0 (e.g. how we allow user apps to modprobe kernel modules).

A big thing I’m working on now is defining a re-architecture of the Supervisor code. As it grew organically when we developed features requested by users, some of the code is organized in ways that can be improved; the idea of the re-architecture is to optimize this so that the next features we have in mind (like multi-container as Shaun mentioned) are easier to implement and can be done in an elegant way. This should also allow us to progressively improve the Supervisor’s performance and resource usage, and we are thinking of a step-by-step transition to Rust. A challenge with this re-architecture is to do it without trying to rewrite the whole code, reusing as much as possible and avoiding a feature-freeze.

Besides this, I’m starting to implement a new version of the update locking mechanism that will solve some issues that are intrinsic to the current design - doing this will also hopefully imply some baby steps towards the new architecture.

I also always spend some time doing user support, or helping teammates do it, especially when there’s bugs or issues that may come from the Supervisor or its interaction with the host OS.

And now I nominate @jviotti for the next update!

1 Like

Thanks for the nomination @pablo!

I’ve been recently working on taking Reconfix to the next level (https://github.com/resin-io/reconfix/pull/6). Reconfix is a bidirectional configuration engine which with the help of some publisher-provided schemas, allows users to read the current device configuration, and re-configure it as needed, whatever “configuration” means for that device (wifi credentials, logging, etc), however I decided to take some time off from simplifying the schema languages to squash some pending accumulated bugs in order to prepare for the next release, as well as refactoring, increasing unit test coverage and other “cleanup” tasks.

Here’s a summary of what I did this week, in more detail:

  • I’ve been assigned on support for two slots per day, which kept me busy for half of my time.
  • I chased down a very subtle bug on etcher-image-write, Etcher’s backend which does the actual low-level writing, where errors coming from the first stream on a stream chain will go unhandled (https://github.com/resin-io-modules/etcher-image-write/pull/75)
  • I took some time to finish a couple of old PRs that re-implement NodeJS file-system readable and writable streams to retry I/O operations up to 10 times when EIO errors are encountered. Because of SDCard self-balancing mechanism, retrying failed operations (EIO) usually makes it work (https://github.com/resin-io-modules/etcher-image-write/pull/73 and https://github.com/resin-io-modules/etcher-image-write/pull/70)
  • I did some experimentation on adding node-gyp support to Etcher, so we can write certain critical pieces of functionality in C++ on the same repo, and have it compiled and loaded correctly
  • I debugged and fixed and bug in Etcher where the writer child process will send multiple state messages to the GUI in a short amount of time, causing the GUI IPC server to throttle them and attempt to parse them together, confusing the parser and eventually complaining with a JavaScript error (see https://github.com/resin-io/etcher/pull/997 and https://github.com/resin-io/etcher/pull/1008)
  • The Etcher GUI calls the Etcher CLI (which we’re working to publish as an standalone package) to actually perform the writing. We support a CLI option that we call “robot”, which causes the CLI to output state information in a way that can be easily parsed back by a machine (in this case, the GUI). Turns out that if an error happens during the initial option parsing stage of the CLI, the “robot” option argument will not be available yet, causing output to be printed in a normal CLI fashion, causing confusion for the GUI parser (which expects a certain structure out of the messages). In order to solve this problem, we allow the “robot” option to be specified as an environment variable, which we can access independently of the option parsing routines (see https://github.com/resin-io/etcher/pull/1010)
  • I chased down and fixed a weird configuration edge case where Windows will report multiple partitions of a single drive to belong to the same drive letter, confusing Etcher during the drive detection stage (https://github.com/resin-io-modules/drivelist/pull/130)
  • I carefully cleaned up some unneeded development dependencies from the npm-shrinkwrap.json file, who seems to have been added there accidentally long ago (https://github.com/resin-io/etcher/pull/1017)
  • I’m moving the GNU/Linux Etcher build system to run inside Docker containers in Travis CI, so we can more easily handle cross-architecture builds
  • I finished and merged a PR that provides support for Etcher unsigned snapshot builds. These builds will contain the short version of the git commit hash the refer to appended in the version (e.g: v1.0.0-beta.17+xxxxxxx) and will be published to some “development channel” very soon for testing purposes.
  • @kostas and I jumped into a call to so some pair debugging of a weird EBUSY error when configuring a resinOS image in OS X using the resin-device-toolbox CLI https://github.com/resin-os/resin-device-toolbox/issues/32, which we’re now working to fix

I think that’s all from me, and I’ll nominate @tim, who I know has been doing some interesting work on the Resin SDK.

1 Like

Thanks @jviotti!

I’ve been very busy too - right now we’re transforming the Resin SDK so you can use it in the browser :desktop:, not just in Node :pager:.

That means digging through all the modules we use in the SDK, and refactoring away everything that won’t work without access to the filesystem, for example. We’ve actually put out a first couple of beta releases of this up this week :champagne: so you can give it a go yourself right now if you like: npm install resin-sdk@6.0.0-beta2

The current step from here is updating everything that depends on the SDK to use this new version, which has had some necessary breaking API changes. That means larger projects like the resin cli, but also all sorts of smaller component libraries we use. The big exciting step that’s coming shortly though is to transform the Resin dashboard code to use the SDK too, rather than its own separately implemented Resin API interactions. This is going to nicely simplify the dashboard code, and should set us up well for some exciting new things we’ve got in the works over there.

And of course, all our work on this is open-source and discussed in the open on our Github repos, so you can dig and follow exactly what we’re doing and discussing day to day on Github for yourself if you like. Here’s a few recent more specific examples to check out:

Phew! I’ll nominate @Eugene.

1 Like

Thanks @tim!

Soo, actually Tim has stolen a good half of my occupation during the past week.
Migrating the Resin SDK from Node-only project to the universal codebase is a project I’ve started back in September.
Then different assignments, the Resin summit, and holidays have delayed it. Since Tim has joined the team things started moving faster, yet we’ve spent almost a month finalizing the code, stabilizing the tests, and simplifying and unifying the testing process across multiple modules.

I’ve been also currently working on improving our UX analytics. We collect some metrics to calculate how many users reach from registering to creating an application and from creating an application to deploying a real device. That helps us identify which parts of the UI may be slowing the users down. Recently we’ve also decided to start sending these events to Google Analytics to simplify some of our metrics calculations. So I’m working on refactoring some of our analytics-related code (which is also universal, running in Node.js and the browser), and experimenting with lerna a bit.

Finally, this week I’m doing 4 hours / day of support, following our support-driven development paradigm.

Aaaand I nominate @horia!

1 Like

Thank @Eugene.

For the last months I have been working on AutoHaT project, a project that aims testing the OS images on Single Board Computers. The test automation framework that we are using to write the test cases is Robot Framework.
The AutoHaT team and I have been writing test cases and keywords that are being tested in QEMU at first. We do have a first version of a rig that works with Raspberry Pi 3 but I am more excited to see the Mega Rig that will be able to test all of our supported boards.
Right now I am working on adding keywords and test cases that will make other’s life much easier to test and automate everything.

Our work is open-source and can be verified in Github and if you are interested, you can check our progress. Here are some examples:

For the next candidate, I nominate @florin

1 Like

For a while now I am finalizing the migration to Poky morty. This will start to be rolled out on boards in the following days, most likely on the Raspberry PI family. This means our host OS will have the new codebase from poky morty as the foundation, taking advantage of bugfixes / enhancements / security fixes in the newer packages versions as well as kernel upgrades for a few of the boards we support.

Aside from this, been working on getting a new board in the resin.io dashboard (to arrive in the following days, based on resinOS 2.X beta).

Also stay tuned for a new 1.26 release that will gradually take place end of this week / beginning of next week.

That’s pretty much it in a nutshell for me and I pass it on to @nghiant2710

1 Like

Thanks for the nomination @florin.

So let’s see what I’m having on my plate.

  • As usual, I’ve been working on a couple of updates for the Resin base images, we have some bug fixes, new python version and Alpine Linux 3.5 coming. Also, there are few issues need to be addressed: weird behavior of the base images on pure resinOS and rdt, new value for INITSYSTEM variable…

  • Besides the base images updates, I’m finalizing some new features for base images build script. The build script will now automatically store base images info (size, available tags…) on a Github repository, it’s really helpful for our users since they don’t need to find these info on the dockerhub anymore. Another coming feature is the automation test for base images which is the missing puzzle in our build process.

  • Aside from those, I’m playing with minideb (https://github.com/bitnami/minideb) a really cool base image introduced by my colleague @shaunmulligan. It’s a small base image designed for containers and hope that we can learn some awesome things to improve our base images from it.

And now, I nominate @ilias

1 Like

Thanks for the nomination @nghiant2710

So, I work as a data engineer developing methods for data analytics. We work with various data sources and at this time we are focusing on data integration and analysis. In particular, we are very interested in assessing data consistency between our data sources and our data warehouse. We are also testing some new services regarding data integration.

In general, we gather all the tools and scripts we write in a project that also provides visualizations, using some pretty cool Javascript libraries for charting. We have some methods for cohort analysis, regression and some basic prediction models.

Finally, I must say that, at this time, we are very happy because we recently welcomed a new data engineer in the team, so we are in the middle of the on-boarding process and we are trying to make him familiar with the the data and tools we use!

That’s the current status more or less and I pass it to @jack who helped me a lot recently with the setup of a new data warehouse.

1 Like

Thanks Ilias. I run DevOps at resin, so I tend to spend my time split between building and evaluating infrastructure/tooling, responding to issues or high-priority feature requests, and support (when I’m on rotation).

Lately, I’ve been experimenting with using Terraform, aiming to replace our current CloudFormation definition in our production and staging environments eventuall. I started by using it to set up the pieces Ilias needed for his data warehousing project. I’m really enjoying it, and find it so light-weight that It’d be nice to roll it out across all of our accounts (we have a bunch of accounts for different things, to isolate the potential for failure/mistakes to impact customers). I’m also in the early stages of working on porting the resin.io components to Kubernetes for an evaluation. Due to some of the stuff we do (our Git server, VPN, etc.) moving to Kubernetes isn’t as straightforward as it could be, and given the nature of what we do the value proposition isn’t as strong for us. We’ll see where that takes us.

I’ve been learning Rust, and wrote my first (ugly) little tool in it. It’s been a really fun, but occasionally frustrating, process. I enjoy the act of writing in Rust a lot. I have a strong Ruby background, and in many ways Rust reads and feels like a mashup of Ruby, JavaScript, and Go (in my opinion; feel free to disagree).

The resin team has been expanding rapidly, and so many of my day-to-day tasks have involved onboarding-related things. I don’t manage our IT, but I’m involved in getting people’s keys on the right hosts, troubleshooting Jenkins issues, etc. We’ve got a ton of projects in the pipe right now, so we’ve had fewer deploys since the holidays.

Of course, today’s Friday, so I’m thinking I’ll probably dive into one of my hack Friday projects. Lately I’ve been working on

  • a distributed temperature/humidity sensor network (my basement floods and has humidity problems)
    • this uses a DHT22/11 to take readings, and integrates with AWS IoT to push data
  • an OctoPrint on resin setup for my 3D printer (a PrintrBot Simple Metal)
  • a smart thermostat (ideally integrated with the temp/hum sensor network), as well as having motion detection, etc.

Additionally, I have a home-built CNC milling machine (http://www.thingiverse.com/thing:724999) I want to integrate with our microcontroller support. It uses grbl for its motion planning, but it would be nice to register the Arduino as a dependent device and be able to update it remotely.

Because I believe @joe has been doing more work with Rust lately (and I don’t think he’s already in this thread) I’m going to nominate him for the next post. Have a good weekend, everyone!

1 Like

Thanks @jack, here’s what I have been working on:

  • Improving resin-wifi-connect to add support for Network Manager which replaces Connman as the network configuration tool in ResinOS 2.0. This was a little challenging as we wanted to keep backwards compatibility to ensure our users can push the same code to a fleet of devices running different versions of ResinOS.

  • Started to work on a total re-write of resin-wifi-connect in Rust. We are going to be adding lots more features, including, support for 3G modems, configuration over Bluetooth and connection bridging. This work will be integrated into the Resin supervisor and made available as a standalone app for devices not running ResinOS.

  • Implementing dependent device (micro-controller) support on Resin, this brings all the great features of Resin (git push deploys, dashboard management etc.) to resource constrained low-power devices. I can’t say much more right now, other than that it’s coming soon and it’s got me very excited as a hardware guy.

  • I am also on support this week which takes a bit of time from normal day to day work. It’s a nice change and I find it really enjoyable to talk to our customers and have a play/debug other parts of our stack which I’m not normally involved with.

@benedict, you’re up next!

2 Likes

+1 for this one …

+1000 for this one … :heart_eyes: :heart_eyes: :heart_eyes:

As also being “a hardware guy” I can image what potential this will unleash! You guys are purely awesome!

3 Likes

Awesome, @joe. Looking forward to seeing more Rust.

I’m normally working on Etcher’s ‘front-end’, and today I’ve been working on a little refactoring of Etcher’s interface, swapping the first two steps of Etcher’s flashing process: the image selection will come first, and the drive selection second, rather than vice versa as it is now.

Otherwise, I also have two active PRs that I’m polishing, one for a full screen file dropzone, and another to make all drives visible with warnings, rather than having to enable unsafe mode to show them.

I nominate one of our Etcher back-end guys, @lurch!

Thanks @benedict,

Like @benedict I’m also part of the Etcher team (we started at around the same time), but I tend to be more involved in the ‘internals’.

Last year I was doing a lot of work on Etcher’s Makefile (I still have some pending features to clean up & merge into this in the pipleline - when I get time!).
Most recently, I’ve been helping out on support in the Etcher GitHub repo and Gitter channel, reviewing various pull requests, and helping plan out the extended image-metadata support we’ll be including into Etcher soon (to offer a much richer user experience - watch this space! :dark_sunglasses: )
I also like to sometimes jump into a random open resin.io issue, just to try to get a feel for how the rest of the resin ecosystem works and is evolving :sunflower: I occasionally manage to catch the odd spelling mistake here or there, thanks to my eye for detail :wink:

Next up, I nominate @stefanos, part of the great operations team :slight_smile:

1 Like

Thank you for the nomination @lurch! :grin:

Not as cutting edge here at the Operations team but we are in the middle of a few things.

Most of the projects we are involved in at the moment revolve around the process of automating as many procedures as possible as well as making each channel in the Resin.io ecosystem interact as smooth as possible with each other.

More specifically, one of the most exciting projects I am working on at the moment is the automation of our business travelling. Resin.io is expanding all around the globe and believe it or not we’ve got amazing people working with us remotely from 4 continents! Thus the need for a sophisticated and handy travelling service.

On the other hand, other than the normal everyday operations duties I am also updating our wiki pages on github, mostly those that are part of the operations procedures.

Not as exciting as the other stuff we’ve read but still I love what I do for Resin.io!

Who’s next? Who’s next? Ah… I know… Have you ever met a guy that his name meaning let’s say didn’t exactly represent him? Well… I nominate @aggelos (@okakosolikos) Aggelos as we all know him… (his name means “angel” in Greek)!!!

1 Like

Aggelos is out for a bit, so maybe I jump in here for a moment, if that’s okay, @stefanos :slight_smile:

I feel lucky to be able to work on some things that are at the intersection of fun and useful. Today my three main priorities are:

  • starting to spread the word of our upcoming hackathon in London (see this post), and work on the preparation as hardware orders, catering, partnerships, etc… :tools:
  • we are developing the ability to sign in to these forums with your resin.io login too (so that no separate username/password will be needed in the future). For that we’ll have to deploy a resin.io oAuth 2.0 provider, and I’m working on an internal spec on how to make that happen. When it works, it will also enable a bunch of resin.io-enabled applications as well, so I’m very excited to see what will you build with it later! :passport_control:
  • writing up the resin-electronjs project by @curcuz in a blogpost to help getting started, and showcase things that can be built with it! (if you have a project built with resin-electronjs, let me know, would love to include and spread the word! :tv:

Besides these I’m looking for events in London to represent resin.io at. I’m working from home today (I usually go to our office, where most of our hardware is), so I guess that’s another challenge to figure out how to use the time well there. :clock:

For tomorrow, I nominate @jhermsmeier :wink:

Thanks for the nomination @imrehg! :slight_smile:

Hello, hello! I’m a recent addition to the Etcher team, working from Berlin, and feeling very lucky to work full-time on open source software now. :tada:

I started off with looking after our resinOS application preload script, adding support for the AUFS storage driver used by resinOS 2.0 and turning it into an easier to use CLI utility.

Now I’m working on re-architecting the internal drive flashing engine, adding a streaming pipeline that can be changed on-the-fly, in preparation for some upcoming features (like flashing one image to multiple devices at the same time, or support for more verification strategies) while also trying to waltz through the backlog of writer component issues and (hopefully) making things even more reliable in the process.

Aaaand I’d like to nominate @shaunmulligan next :relaxed:

1 Like

Sweet, thanks @jhermsmeier. Sorry for the late response on this. The past few days I have been finalising the multi-container design and things are looking really great. Hopefully we will have more details to share with everyone in the coming weeks on timing for this feature as I know lots of people are excited about.

I also have been finalising the last couple of items we need to sort out before we can release resinOS 2.0 as production ready on resin.io. We are provisionally aiming to release promote resin 2.0 to production worthy on the 15th of March, so look forward to an announcement from us around then.

Other than those two large items, I have also been tearing through and trying to prioritise all the support tickets and bugs we have in our back log, trying to make sure we make as many people as happy as possible. The rest of my week will probably involve trying to refresh large portions of our documentation as well as add some of the new docs for resinOS 2.0. That about all for now!

Next up, lets hear from our awesome support lead @sonya

2 Likes