Building Darknet on Balena-cloud vs on Jetson TX2

Product support
1

When I include in my Dockerfile.template:

RUN cd /usr/src/app/source/barnserv/docker/darknet/ \
    && mkdir -p obj \
    && make

I get a whole bunch of linking errors. See below

But when comment out that build command, push to the balena-cloud remote, wait for the service to start running on the device, and then connect to the service through the balena dashboard and execute the exact same commands… the build works without link errors.

Suggestions? First thoughts are to build on every boot, or a “run once” script.

https://dashboard.balena-cloud.com/apps/1366228/devices

Here is part of the output from make, I’ve stripped the [Service] tag at the beginning of each line and didn’t include every undefined symbol error:

Step 54/57 : RUN cd /usr/src/app/source/barnserv/docker/darknet/     && mkdir -p obj     && make
 ---> Running in e121b0a2a031
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/gemm.c -o obj/gemm.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/utils.c -o obj/utils.o
src/utils.c: In function ‘epoch_time_seconds’:
src/utils.c:26:3: warning: #warning "CLOCK_REALTIME and/or clock_gettime are not defined" [-Wcpp]
 # warning "CLOCK_REALTIME and/or clock_gettime are not defined"
   ^~~~~~~

gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/jeff.c -o obj/jeff.o
src/jeff.c: In function ‘sleep_for’:
src/jeff.c:42:3: warning: #warning "usleep is not defined" [-Wcpp]
 # warning "usleep is not defined"
   ^~~~~~~

gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/jarray.c -o obj/jarray.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/cuda.c -o obj/cuda.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/convolutional_layer.c -o obj/convolutional_layer.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/batchnorm_layer.c -o obj/batchnorm_layer.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/list.c -o obj/list.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/image.c -o obj/image.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/activations.c -o obj/activations.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/im2col.c -o obj/im2col.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/col2im.c -o obj/col2im.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/blas.c -o obj/blas.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/maxpool_layer.c -o obj/maxpool_layer.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/data.c -o obj/data.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/matrix.c -o obj/matrix.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/network.c -o obj/network.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/parser.c -o obj/parser.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/option_list.c -o obj/option_list.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/box.c -o obj/box.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/layer.c -o obj/layer.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/region_layer.c -o obj/region_layer.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/reorg_layer.c -o obj/reorg_layer.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/route_layer.c -o obj/route_layer.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/tree.c -o obj/tree.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/cmd_line_args.c -o obj/cmd_line_args.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/detector.c -o obj/detector.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/jmem.c -o obj/jmem.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/jclass.c -o obj/jclass.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/jstring.c -o obj/jstring.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/jimage.c -o obj/jimage.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/epoch_info.c -o obj/epoch_info.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/main.c -o obj/main.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast -std=c11 -c src/experiment.c -o obj/experiment.o
/usr/local/cuda/bin/nvcc -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=[sm_50,compute_50] -gencode arch=compute_52,code=[sm_52,compute_52] -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN --compiler-options "-Wall -Wno-unknown-pragmas -fPIC -Ofast" -c src/convolutional_kernels.cu -o obj/convolutional_kernels.o
/usr/local/cuda/bin/nvcc -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=[sm_50,compute_50] -gencode arch=compute_52,code=[sm_52,compute_52] -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN --compiler-options "-Wall -Wno-unknown-pragmas -fPIC -Ofast" -c src/activation_kernels.cu -o obj/activation_kernels.o
/usr/local/cuda/bin/nvcc -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=[sm_50,compute_50] -gencode arch=compute_52,code=[sm_52,compute_52] -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN --compiler-options "-Wall -Wno-unknown-pragmas -fPIC -Ofast" -c src/im2col_kernels.cu -o obj/im2col_kernels.o
/usr/local/cuda/bin/nvcc -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=[sm_50,compute_50] -gencode arch=compute_52,code=[sm_52,compute_52] -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN --compiler-options "-Wall -Wno-unknown-pragmas -fPIC -Ofast" -c src/col2im_kernels.cu -o obj/col2im_kernels.o
/usr/local/cuda/bin/nvcc -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=[sm_50,compute_50] -gencode arch=compute_52,code=[sm_52,compute_52] -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN --compiler-options "-Wall -Wno-unknown-pragmas -fPIC -Ofast" -c src/blas_kernels.cu -o obj/blas_kernels.o
/usr/local/cuda/bin/nvcc -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=[sm_50,compute_50] -gencode arch=compute_52,code=[sm_52,compute_52] -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN --compiler-options "-Wall -Wno-unknown-pragmas -fPIC -Ofast" -c src/maxpool_layer_kernels.cu -o obj/maxpool_layer_kernels.o
gcc -Isrc/ -DGPU -I/usr/local/cuda/include/ -DCUDNN -Wall -Wno-unknown-pragmas -fPIC -Ofast obj/gemm.o obj/utils.o obj/jeff.o obj/jarray.o obj/cuda.o obj/convolutional_layer.o obj/batchnorm_layer.o obj/list.o obj/image.o obj/activations.o obj/im2col.o obj/col2im.o obj/blas.o obj/maxpool_layer.o obj/data.o obj/matrix.o obj/network.o obj/parser.o obj/option_list.o obj/box.o obj/layer.o obj/region_layer.o obj/reorg_layer.o obj/route_layer.o obj/tree.o obj/cmd_line_args.o obj/detector.o obj/jmem.o obj/jclass.o obj/jstring.o obj/jimage.o obj/epoch_info.o obj/main.o obj/experiment.o obj/convolutional_kernels.o obj/activation_kernels.o obj/im2col_kernels.o obj/col2im_kernels.o obj/blas_kernels.o obj/maxpool_layer_kernels.o -o darknet.exe -lm -pthread -L/usr/local/cuda/lib64 -lcuda -lcudart -lcublas -lcurand -lcudnn -lstdc++
/usr/bin/ld: warning: libnvrm_gpu.so, needed by /usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld:
 warning: libnvrm.so, needed by /usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: warning: libnvidia-fatbinaryloader.so.28.2.0, needed by /usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so, not found (try using -rpath or -rpath-link)
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuDeviceAllocateMemory'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuTaskSchedulingGroupClose'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `fatBinaryCtl_Create'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elfLink_Finish'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuRegOpsSessionSetPowergateMode'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elf64_symbol_name'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elf64_string_at_offset'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuRegOpsSessionSetTimeoutMode'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuMappingClose'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuRegOpsSessionSetPcSamplingMode'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmSyncFreeSyncpoints'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `fatBinaryCtl_Delete'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elf_size'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elf64_section_contents'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elfLink_Load_Host_Object'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuTaskSchedulingGroupSetTimeslice'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmSyncClose'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuProfilerClose'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuDeviceSetSmDebugMode'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuClockGet'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuNvgpuGetAsFd'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuChannelKickoffPb'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmSyncGetSyncpoints'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuLibOpen'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elfLink_Delete'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmClose'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuProfilerReserveHwpm'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuRegOpsSessionSetSmpcContextSwitchMode'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmSyncFreeSemaphores'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuRegOpsSessionCreateForChannel'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elfLink_Add_Cubin'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elf64_file_header'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuChannelCycleStatsAttachSnapshot'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `fatBinaryCtl_PickCandidate'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuChannelGetInfo'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuTaskSchedulingGroupEventIdControl'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmSyncDup'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuChannelCycleStatsFlushSnapshot'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elf64_section_header'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmOpenNew'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `gpuInfoRunsOn'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmMemCacheSyncForCpu'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuChannelControl'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmSyncGetSemaphores'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuDeviceClearSmErrors'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `elf32_symbol_name'
/usr/lib/gcc/aarch64-linux-gnu/6/../../../aarch64-linux-gnu/libcuda.so: undefined reference to `NvRmGpuDeviceSetMmuDebugMode'
......

collect2: error: ld returned 1 exit status

Makefile:114: recipe for target 'darknet.exe' failed
make: *** [darknet.exe] Error 1
3

Hi Jason
Looks like the C code is missing some definitions. Maybe there is something that needs to be installed before you compile. Is the compile command the last action in your Dockerfile or are other packages being installed later ?
Regards
Thomas

7

Yes, the build of the c code is the last thing done in the dockerfile.template right before the CMD

8

@jason10, so we should look for what is different between the environment of the cloud builder, and the time when you login to the device:

  • Environment variables: try comparing the output of the /usr/bin/env command at build time (Dockerfile RUN /usr/bin/env) and when you login to the device.
  • During build time, typically fewer or no “background services” (daemons) will be running. At the moment I can’t think of why this would affect code compilation, but it’s good to keep in mind.
  • This is an ARM device, right? balena has native ARM servers that usually cover the vast majority of the build tasks, but from time to time the native ARM servers are not available (load, maintenance) and the cloud builders revert to QEMU emulated builds on x86 servers. Emulated builds are known to fail for some compilation tasks. I would suggest that you tried running the build on a Jetson TX2 device using the balena build or balena push commands providing the IP address of a development device that will run the image build:

In order to build on the device itself, the balena push command requires the device to be in local mode. For the balena build command, it is sufficient that device is running a development image (not needed to be in local mode).

Let us know the results!

9

Success but confusion. I built it using local build and it built successfully. Then I built it using the balena-cloud service again and it built successfully.

I will report on any future failures.

10

Hi ! This may be a case of emulated builders kicking in as Paulo mentioned.
One of the first lines of output of the push should tell you which builder is being used - you could see if there is a correlation between this and whether the build succeeds ? Thanks !

12

Just to followup, I haven’t seen any failed builds due to the compilation of darknet in a while. Most of the time the build machine is arm03.

1 Like
14

Thanks for letting us know, this helps :slight_smile: p.s. you can always use https://status.balena.io/ to check for builder status :slight_smile: