Hybrid cloud + local development workflow

ductape · May 14, 2020, 4:41pm

I’m not quite sure if this is a feature request or a support issue requesting help with a better workflow. When building a Balena project on Raspberry Pi Zero W with the application in Python. Local mode takes FOREVER to build the container because it is building locally on the RPi Zero. Every time I make changes to the container (e.g. editing Dockerfile, adding additional install_packages, changing my requirements.txt (Python is building all the libraries locally on the ARM architecture and it takes a many hours to build numpy wheels on RPi Zero)) the cycle time is hours. Maybe there is opportunity to optimize my Dockerfile to make better use of caching, but that’s a bit of a separate thread.

I really want a workflow where large builds are performed on the cloud and then downloaded locally but I can still attach to the local device ( balena push NNNNNNN.local) so that Livepush can function afterwards. Even if this is just the initial deploy built on the cloud and the current local mode is used after that?

ductape · May 14, 2020, 4:50pm

I might be missing something simple like the proper use of the cross build commands in my Dockerfile as mentioned here… Fastest way to dev on local device

ab77 · May 14, 2020, 4:52pm

Hi there, have a looked at the balenaCLI masterclass here: https://www.balena.io/docs/learn/more/masterclasses/cli-masterclass/#51-building-an-image-on-a-development-machine ?

ab77 · May 14, 2020, 4:53pm

You can use the cli to run an emulated build locally on your workstation and push it to the device running in local mode.

ductape · May 15, 2020, 3:58pm

@ab77 I found how to build locally and push locally in the balena CLI masterclass. Local build works great using balena build --arch rpi --deviceType raspberry-pi --emulated. And I see how to deploy locally using balena push 827b231.local– but that still tries to build on the Rpi Zero.

The CLI docs (--help for deploy or push) don’t seem to explain how to get the container I built on my computer onto my local device.

balena push --help
Usage: push

This command can be used to start a build on the remote balena cloud builders,
or a local mode balena device.

When building on the balenaCloud servers, the given source directory will be
sent to the remote server. This can be used as a drop-in replacement for the
“git push” deployment method.

When building on a local mode device, the given source directory will be
built on the device, and the resulting containers will be run on the device.
Logs will be streamed back from the device as part of the same invocation.

balena deploy --help
Usage: deploy [image]

Usage: deploy <appName> ([image] | --build [--source build-dir])

Use this command to deploy an image or a complete multicontainer project to an
application, optionally building it first. The source images are searched for
(and optionally built) using the docker daemon in your development machine or
balena device. (See also the balena push command for the option of building
the image in the balenaCloud build servers.)

push doesn’t let you specify a local, already built, container. And deploy doesn’t let you specify a local device… am I missing something obvious?

saintaardvark · May 15, 2020, 4:54pm

Hi @ductape – you’re right that balena push does build on the device. What you can try is:

Ensure your Dockerfile has the package installation steps (or any other steps that take a long time) close to the beginning of the file, so they make an earlier Docker layer
Run balena deploy <appName> <image> (to send your laptop-built image up to balena Cloud), or balena push <appName> (to have the balena Cloud builders build the image)
Switch the device out of local mode, and ensure the device pulls down that image
Switch back to local mode, and ensure that any additional changes come in subsequent Docker layers; this should end up re-using the earlier, cloud-built layers

Can you give that a try and let us know how it goes?

All the best,
Hugh

ductape · May 15, 2020, 9:35pm

Thanks @saintaardvark, that’s a much improved workflow! It looks like that will work. I tried it and looks like the cached version is used for the existing steps (it’s not trying to reinstall the OS dependencies or rebuild my Python dependencies now.

I can’t promise it works because it seems the local mode CLI is getting some node error (I’m using a Python container?) before the logging starts, but I believe that’s unrelated to this posting.

dtischler · May 15, 2020, 10:44pm

Excellent, glad to hear that is working well (other than the node error to chase down).

ntzovanis · May 15, 2020, 10:50pm

Hello @ductape (nice handle!),

What error are you seeing in the CLI?

Cheers,
Nico

ductape · May 17, 2020, 5:42pm

The error in the CLI (version 11.33.2) is:

[Build]   [main] Step 10/11 : LABEL io.resin.local.image=1
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> a9a80081a7eb
[Build]   [main] Step 11/11 : LABEL io.resin.local.service=main
[Build]   [main]  ---> Using cache
[Build]   [main]  ---> d4b8a1ae4ff5
[Build]   [main] Successfully built d4b8a1ae4ff5
[Build]   [main] Successfully tagged local_image_main:latest



#
# Fatal error in , line 0
# Check failed: result.second.
#
#
#
#FailureMessage Object: 0x7ffeefbf3a80
 1: 0x1000d565c node::NodePlatform::GetStackTracePrinter()::$_3::__invoke() [/usr/local/bin/node]
 2: 0x100a22f04 V8_Fatal(char const*, ...) [/usr/local/bin/node]
 3: 0x100377e8d v8::internal::GlobalBackingStoreRegistry::Register(std::__1::shared_ptr<v8::internal::BackingStore>) [/usr/local/bin/node]
 4: 0x10018d91d v8::ArrayBuffer::GetBackingStore() [/usr/local/bin/node]
 5: 0x1000631bc node::Buffer::New(node::Environment*, char*, unsigned long, void (*)(char*, void*), void*) [/usr/local/bin/node]
 6: 0x100062f53 node::Buffer::New(v8::Isolate*, char*, unsigned long, void (*)(char*, void*), void*) [/usr/local/bin/node]
 7: 0x10005ea8b napi_create_external_buffer [/usr/local/bin/node]
 8: 0x10469641e Napi::Buffer<char> Napi::Buffer<char>::New<Napi::Value FFI::WrapPointer<_ffi_type>(Napi::Env, _ffi_type*, unsigned long)::'lambda'(Napi::Env, char*)>(napi_env__*, char*, unsigned long, _ffi_type) [/usr/local/Cellar/balena-cli/11.33.2/libexec/lib/node_modules/balena-cli/node_modules/net-keepalive/node_modules/ffi-napi/build/Release/ffi_bindings.node]
 9: 0x1046916df FFI::FFI::InitializeBindings(Napi::Env, Napi::Object) [/usr/local/Cellar/balena-cli/11.33.2/libexec/lib/node_modules/balena-cli/node_modules/net-keepalive/node_modules/ffi-napi/build/Release/ffi_bindings.node]
10: 0x104694f26 __napi_Init(napi_env__*, napi_value__*) [/usr/local/Cellar/balena-cli/11.33.2/libexec/lib/node_modules/balena-cli/node_modules/net-keepalive/node_modules/ffi-napi/build/Release/ffi_bindings.node]
11: 0x10005e0be napi_module_register_by_symbol(v8::Local<v8::Object>, v8::Local<v8::Value>, v8::Local<v8::Context>, napi_value__* (*)(napi_env__*, napi_value__*)) [/usr/local/bin/node]
12: 0x100061ecb std::__1::__function::__func<node::binding::DLOpen(v8::FunctionCallbackInfo<v8::Value> const&)::$_0, std::__1::allocator<node::binding::DLOpen(v8::FunctionCallbackInfo<v8::Value> const&)::$_0>, bool (node::binding::DLib*)>::operator()(node::binding::DLib*&&) [/usr/local/bin/node]
13: 0x100061078 node::binding::DLOpen(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
14: 0x1001cda06 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/usr/local/bin/node]
15: 0x1001cd090 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
16: 0x1001cc89c v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
17: 0x100757f99 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/usr/local/bin/node]
18: 0x1006f0a98 Builtins_InterpreterEntryTrampoline [/usr/local/bin/node]
19: 0x1006f0a98 Builtins_InterpreterEntryTrampoline [/usr/local/bin/node]
Illegal instruction: 4

I see there is an update available for the CLI, so I’ll upgrade and try again later.

roman-mazur · May 18, 2020, 1:48pm

Hi, thanks for the stacktrace!

It looks like a crash caused by one of the module used by balena-cli (net-kepalive).
Please let us know if you reproduce this on the latest CLI version. If yes, we’ll ask our CLI maintainers to take a look.

ductape · May 19, 2020, 2:07pm

@roman-mazur it looks like the same error is still there in balena-cli v11.35.1. I see there is another update to the CLI, so I’ll try that and post back later.

Also for reference, I’m running macOS 10.15.4 using bash as shell and homebrew to install the Balena CLI. System is running Docker version 19.03.8 build afacb8b. There is expectedly no difference in result when using native terminal app or vscode terminal.

Lastly, here is the output of the scan:

sudo balena scan
Password:
Reporting scan results
- 
  host:          *******.local
  address:       ***.***.*.**
  dockerInfo: 
    Containers:        4
    ContainersRunning: 2
    ContainersPaused:  0
    ContainersStopped: 2
    Images:            14
    Driver:            aufs
    SystemTime:        2020-05-19T14:03:58.231277037Z
    KernelVersion:     4.19.75
    OperatingSystem:   balenaOS 2.48.0+rev1
    Architecture:      armv6l
  dockerVersion: 
    Version:    18.09.10-dev
    ApiVersion: 1.39

pdcastro · May 19, 2020, 8:36pm

@ductape, I’ve seen some stack traces like that before, with Node.js v14. The balena CLI is not yet compatible with Node.js v14. Also, homebrew is not an official distribution channel for the balena CLI – I understand it is maintained by some balena users. The supported installation methods and Node.js versions are documented at: https://github.com/balena-io/balena-cli/blob/master/INSTALL.md

Try one of the methods documented there, and/or downgrade Node.js to v12 (e.g. 12.16.3). Let us know if this solves the issue for you.

mhazley · May 21, 2020, 2:02pm

Is there any plan to introduce local push of an image built with balena build…?

That would be a workflow, especially when on devices with slower, smaller cores.

cmfcruz · May 21, 2020, 2:24pm

Hi @ductape,

That would be a really useful feature and we do have many CLI users asking for that feature. We are still considering it for future releases.

You may follow the development of that feature in this Github thread: https://github.com/balena-io/balena-cli/issues/1468

Cheers,
Carlo