# Running Yolov8n on Jetson Nano with TensorRt **URL:** \ **Category:** Product support\ **Tags:** docker, nano\ **Created:** [May 21, 2024, 1:12pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549 "2024-05-21T13:12:48Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![MWLC](https://avatars.discourse-cdn.com/v4/letter/m/c68b51/32.png) [@MWLC](https://forums.balena.io/u/MWLC)\ **Post date:** [May 21, 2024, 1:12pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/1 "2024-05-21T13:12:48Z")
Hi all, I want to share with you a docker image that I’m using to run Yolov8n on my Jetson nano with TensorRt. I’m sharing it here to save the effort for anyone who is planning to do the same. This post came after the one I made before but it was limited to the one using `.pt` model instead of `.engine` These are the details of the Jetson nano it has been tested on: HOST OS VERSION: balenaOS 4.0.9+rev2 SUPERVISOR VERSION: 16.1.0 I made a build (3.82GB) with the docker image [here](https://hub.docker.com/r/mwlvdev/jetson-nano-ubuntu/tags) under the name `mwlvdev/jetson-nano-ubuntu:bionic-torch1.10-cp38-cuda10.2-TRT` Here are some metrics with the time difference ![onnx vs pt1](https://global.discourse-cdn.com/balena/original/2X/d/d7960911241f2c09571c4311f1f26c2826be90c5.jpeg) You can find the original docker file [here](https://github.com/MWLCDev/BYODR/blob/RFC0008_Following/docker/bionic-torch1.11-cp38-cuda10.2-TRT.Dockerfile) if you decided to build your own Regards, ---
**Author:** ![chrisys](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/chrisys/32/1731_2.png) [@chrisys](https://forums.balena.io/u/chrisys)\ **Post date:** [May 23, 2024, 12:38pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/3 "2024-05-23T12:38:06Z")
@MWLC thanks for sharing! I had a quick look at your BYODR project which looks very cool. Are you running these robots with balena? We’d love to hear more about your project and your mission with MWLC so feel free to share. 🙂 ---
**Author:** ![MWLC](https://avatars.discourse-cdn.com/v4/letter/m/c68b51/32.png) [@MWLC](https://forums.balena.io/u/MWLC)\ **Post date:** [May 23, 2024, 2:58pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/4 "2024-05-23T14:58:33Z")
Hi @chrisys, thanks for showing interest. We have been actively developing for quite sometime with Balena. All our robots are using it. Our mission is to make the logistics of farming (for now) easier. I would be happy to answer any questions you have regarding this. Feel free to reach on my [linkedIn](https://www.linkedin.com/in/ahmed-mahfouz-8005a017a/) Regards, ---
**Author:** ![thomaskleiven](https://avatars.discourse-cdn.com/v4/letter/t/ecc23a/32.png) [@thomaskleiven](https://forums.balena.io/u/thomaskleiven)\ **Post date:** [May 25, 2024, 1:12pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/5 "2024-05-25T13:12:16Z")
Hi @MWLC, thanks for sharing. In order to build the `.engine`, do you have any dependencies to shared object libraries located under `/usr/lib/aarch64-linux-gnu/tegra`? E.g., `libnvdla_compiler.so`? If so, how do you make sure these are available when running your container? ---
**Author:** ![MWLC](https://avatars.discourse-cdn.com/v4/letter/m/c68b51/32.png) [@MWLC](https://forums.balena.io/u/MWLC)\ **Post date:** [May 27, 2024, 7:31am UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/6 "2024-05-27T07:31:32Z")
Hi @thomaskleiven, I have encountered the same problem with `libnvdla_compiler.so`. This dependency is part of `onnxruntime_gpu`. This solution involves adding a dynamic linker/loader that can find the `/tegra` directory and use the shared libraries located in it. You can find more info here [ImportError: libnvdla\_compiler.so: cannot open shared object file: No such file or directory - Jetson & Embedded Systems / Jetson Nano - NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/importerror-libnvdla-compiler-so-cannot-open-shared-object-file-no-such-file-or-directory/161652) I exported the `.engine` directly from .pt using `model.export()` [Export - Ultralytics YOLOv8 Docs](https://docs.ultralytics.com/modes/export/) which is offered as part of Ultralytics library in Python. Some answers mentioned that I should export to .onnx first then to .engine, but I didn’t need to with the default yolov8n.pt. Regards, ---
**Author:** ![thomaskleiven](https://avatars.discourse-cdn.com/v4/letter/t/ecc23a/32.png) [@thomaskleiven](https://forums.balena.io/u/thomaskleiven)\ **Post date:** [May 27, 2024, 9:33am UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/7 "2024-05-27T09:33:23Z")
Thank you for your quick response. It makes sense, and I appreciate you sharing the resource links. Have you encountered any performance issues running models with `tensorrt` on BalenaOS compared to the default OS on the Jetson? In my tests, using the same containerized applications on both an off-the-shelf Jetson and one flashed with BalenaOS, I noticed higher CPU and GPU usage on the BalenaOS. If this hasn’t been your experience, I may need to investigate further to understand the cause of this behavior. ---
**Author:** ![MWLC](https://avatars.discourse-cdn.com/v4/letter/m/c68b51/32.png) [@MWLC](https://forums.balena.io/u/MWLC)\ **Post date:** [May 27, 2024, 10:53am UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/8 "2024-05-27T10:53:40Z")
I didn’t try to run it with the normal Jetson L4T. I would assume the higher CPU usage is because of the `balena_supervisor` that is running in the background all the time. Achieving 72ms while running yolov8 with a 640p stream on a legacy device (that its dev kit is EOL) is quite good actually. If you are looking for more performance improvement, I would recommend you to keep with the [PR -Docker image for jetson nano by Ahelsamahy · Pull Request #13100 · ultralytics/ultralytics (github.com)](https://github.com/ultralytics/ultralytics/pull/13100)) I made on ultralytics, maybe one of their devs will have a recommendation to it. There are more approaches you can take if you want to run the model faster, such as lower the input resolution, lower the model precision like fp16 or fp32 instead of INT64, more can be found [here](https://github.com/ultralytics/ultralytics/issues/3070#issuecomment-1666854080) or running the Nano on performance mode. Feel free to ask here if you have more questions😃. ---
**Author:** ![thomaskleiven](https://avatars.discourse-cdn.com/v4/letter/t/ecc23a/32.png) [@thomaskleiven](https://forums.balena.io/u/thomaskleiven)\ **Post date:** [May 27, 2024, 12:03pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/9 "2024-05-27T12:03:48Z")
I will investigate this issue further. Currently, we are using fp16 and cannot reduce it further due to the size of the objects we need to detect and the accuracy required. We are updating our fleet to Orin devices, which will allow us to simplify our Docker images to the one you suggested, even though they are quite similar. The GPU performance is still good and almost matches what we got with the OS that came with the device. My main concern is the unexplained increase in CPU usage. We had our app running smoothly on the original OS, but now we’re seeing CPU usage consistently at 20-30% higher with BalenaOS. I’ll keep digging to figure out what’s causing this. ---
**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\ **Post date:** [May 27, 2024, 12:16pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/10 "2024-05-27T12:16:36Z")
@thomaskleiven do you think you can share the services you are running when you see the CPU usage at 20-30% higher with balenaOS? Where do you see this data? running `top` or `systeminformation`? Thanks ---
**Author:** ![thomaskleiven](https://avatars.discourse-cdn.com/v4/letter/t/ecc23a/32.png) [@thomaskleiven](https://forums.balena.io/u/thomaskleiven)\ **Post date:** [May 27, 2024, 12:54pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/11 "2024-05-27T12:54:11Z")
We haven’t thoroughly investigated the problem yet. We noticed that BalenaCloud is reporting very high CPU usage (e.g., as seen in the attached image). This is also reflected in our Grafana dashboard, so we’ve just assumed this to be accurate for now. Our services are a bit specialized and rely on different hardware, so it’s tough to reproduce the issue without the same setup. I’ll try to narrow down the key services to make it easier to share with you. Do you expect the performance of multithreaded applications within containers to be comparable when running on the original OS versus running on BalenaOS? ![usage](https://global.discourse-cdn.com/balena/original/2X/f/fa27b4d9249b156c28f4e64acf1d371927d0a109.png) ---
**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\ **Post date:** [May 27, 2024, 1:13pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/12 "2024-05-27T13:13:41Z")
@thomaskleiven could you please try `top` on the Terminal? ---
**Author:** ![mpous](https://sea2.discourse-cdn.com/balena/user_avatar/forums.balena.io/mpous/32/4742_2.png) [@mpous](https://forums.balena.io/u/mpous)\ **Post date:** [May 28, 2024, 9:37am UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/13 "2024-05-28T09:37:26Z")
BTW @thomaskleiven @MWLC did you read this blogpost with some tricks? > **[Using NVIDIA Jetson NGC containers on balenaOS - balena Blog](https://blog.balena.io/jetson-ngc-catalog-containers/)** > > Leverage pre-built containers from NVIDIA's NGC Catalog and deploy them successfully on balenaOS. ---
**Author:** ![thomaskleiven](https://avatars.discourse-cdn.com/v4/letter/t/ecc23a/32.png) [@thomaskleiven](https://forums.balena.io/u/thomaskleiven)\ **Post date:** [May 28, 2024, 12:18pm UTC](https://forums.balena.io/t/running-yolov8n-on-jetson-nano-with-tensorrt/370549/14 "2024-05-28T12:18:03Z")
Thanks for the link to the blog post! I guess we might diverge a bit from the original issue in this post, if so, I guess we could continue the discussion elsewhere. Anyways, thanks a lot for your help so far. It looks like we’re using a pretty similar Dockerfile to the one you pointed out [here](https://github.com/balena-io-experimental/balena-jetson-catalog/blob/master/tensorrt/Dockerfile.jetson-tx2). For development purposes, we start from `nvcr.io/nvidia/l4t-tensorrt:r8.5.2.2-devel` and installs the BSP for `L4T` version `r35.4.1`, as you can see in the attached Dockerfile. From what I can tell, BalenaOS comes with `l4t-r35.4`, which matches up with the BSP we’re installing in the Docker image. The output from `uname -a` confirms this. The metrics reported in BalenaCloud seem to be accurate when we check `top`. When we use the resulting Docker image on Balena OS, we see a higher system load and CPU usage compared to running it directly on the OS shipped with the Jetson. For example, when we use `docker run --runtime nvidia --rm --device /dev/bus/usb private-docker-image:v0.2.0` directly on the Jetson OS our app uses around 70% CPU on average. But when we use the same image on BalenaOS (with the BSP bundled in as per the instructions), the CPU usage shoots up to over 90%, and the system load is much higher. Here’s the Dockerfile we’re using, and we push it with `balena push`: ```auto FROM private-docker-image:v0.2.0 # Don't prompt with any configuration questions ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && apt-get install -y --no-install-recommends \ lbzip2 \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* # Set paths ENV CUDA_HOME=/usr/local/cuda-11.4 ENV UDEV=1 # Download and install BSP binaries for L4T RUN \ cd /tmp/ && wget https://developer.nvidia.com/downloads/embedded/l4t/r35_release_v4.1/release/jetson_linux_r35.4.1_aarch64.tbz2 && \ tar xf jetson_linux_r35.4.1_aarch64.tbz2 && \ cd Linux_for_Tegra && \ sed -i 's/config.tbz2\"/config.tbz2\" --exclude=etc\/hosts --exclude=etc\/hostname/g' apply_binaries.sh && \ sed -i 's/install --owner=root --group=root \"${QEMU_BIN}\" \"${L4T_ROOTFS_DIR}\/usr\/bin\/\"/#install --owner=root --group=root \"${QEMU_BIN}\" \"${L4T_ROOTFS_DIR}\/usr\/bin\/\"/g' nv_tegra/nv-apply-debs.sh && \ sed -i 's/chroot . \// /g' nv_tegra/nv-apply-debs.sh && \ ./apply_binaries.sh -r / --target-overlay && cd .. \ rm -rf Linux_for_Tegra && \ echo "/usr/lib/aarch64-linux-gnu/tegra" > /etc/ld.so.conf.d/nvidia-tegra.conf && ldconfig CMD ["./build/run"] ``` Have you seen this behaviour before when comparing BalenaOS to the OS that the Jetson is shipped with?