Building a nanny bot

Idea

I have a 6 months old daughter and she loves our iRobot Roomba 980. We’re vacuuming just for fun with some toys sticked to it. Weird activity, I know, but those who have a child know - you’re going to do whatever keeps your child happy.

Let’s move this to another level. Nvidia started selling Jetson Nano Developer Kit quite recently and they have a very nice projects - like Hello AI World or a JetBot. Here’re some nice videos:

And here’s the idea I have - build a nanny bot to amuse my daughter:

  • bot should be able to find her,
  • bot should be able to play a sound (or do whatever to make her happy),

Optional features:

Future enhancements:

  • when she starts crawling, play with her, move a bit away from her, wait till she moves closer, move a bit away from her,
  • when she stops crawling, do whatever to catch her attention and to force her to crawling again,
  • etc.

Bill of materials

This part will be updated when l buy a new part. I’ll try to source all required parts from the Amazon, Pimoroni, …

Here’s the bill of materials for the Nvidia JetBot.

Don’t buy it now, because some parts wont be used. I have to test them and who knows how this will end. Wait till the project will be finished.

Nvidia

Pcs Price Link
1x $ 99.00 Jetson Nano Developer Kit

Electronic-Shop SARL - Luxembourg

Amazon UK

Amazon DE

Pcs Price Link
1x 22.99 € Intel Dualband Wireless AC 8265 NGW
1x 10.41 € DeLOCK dual band antenna

Pimoroni UK

Look and feel

I personally like how the JetBot looks like …

… but I’m afraid that my daughter wont like it. I’m going to build & paint a nice balsa wood body. I already have the chassis 3D printed …

… and I build everything on top of it.

Current status

  • Chassis 3D printed
  • Working Jetson Nano on my desk
  • Play mat at home


P.S. Buy this play mat even for yourself. I usually fall asleep on it when taking care of my daughter. This soft foam … :slight_smile:

Timeframe

I live in the Czech Republic and sourcing all required parts from abroad (so you can buy them as well) will take some time. Usually days, a week or two sometimes, be patient. My goal is to build this bot as soon as possible.

Next step

I’m at the same beginning. As a first thing, I’m going to build manually controlled moving part (wheels, motors, …).

Stay tuned! And if you have some ideas for this nanny bot, feel free to share them!

1 Like

@zrzka thanks for sharing, this is going to be an awesome project! Following along with interest! :smiley:

1 Like

Some parts already arrived. Decided to proceed with some steps in advance and test the HW.

Repository

HW setup & wiring

I did follow the Hardware Setup guide:

Some steps skipped as I still don’t have all the HW I need.

Tests

PiOLED

  • Check the oled folder
  • It’s a modified stats.py from the JetBot repository

IMG_1606

Camera

  • Check the camera folder
  • I tried to setup MJPEG stream, but didn’t succeed yet
  • Seems there’re some issues with the ffmpeg & 10-bit Bayer RGRG/GBGB format

I know that the camera is connected, sensor is recognised, …

root@407feda65a8a:/# v4l2-ctl --list-devices
vi-output, imx219 6-0010 (platform:54080000.vi:0):
	/dev/video0
root@407feda65a8a:/# v4l2-ctl -d /dev/video0 --list-formats
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
root@407feda65a8a:/#

… but not sure if the JetPack supports it. If not, first mistake and I’ll have to buy another sensor. We’ll see.

Motor driver

  • Check the motor folder
  • It’s basically a copy of the motor.py & robot.py from the JetBot repository
  • It simply tells the driver to go forward, backward, left, right and stop. Each command lasts for 10s.
  • I was just measuring if the driver was giving the right voltage (no motors yet)

Stop

IMG_1604

Forward

IMG_1603

Backward

IMG_1602

Battery life

Jetson Nano was up for almost 4hours …

root@1273841:~# uptime
 15:42:07  up   3:55,  1 user,  load average: 1.63, 1.68, 1.58
root@1273841:~#

… in the 10W mode …

root@1273841:~# nvpmodel -q
NV Power Mode: MAXN
0
root@1273841:~#

… and if you didn’t notice it on the first photo, power bank is at 77% capacity. Which is nice, because it was running balenaOS, I was building everything on the Jetson Nano, running on battery only, etc. I know, no motors, no camera stream, no AI for detection, … but it gives a promise that it can last for some reasonable time with all these features enabled.

Jetson Nano can also be switched to the 5W mode (recommended when running on battery) …

root@1273841:~# nvpmodel -m 1
root@1273841:~# nvpmodel -q
NV Power Mode: 5W
1
root@1273841:~#

… and you can switch it back with the nvpmodel -m 0 command.

Some fun today! Still waiting for other parts :frowning:

1 Like

A great start! Good to see things coming together!

Yeah, it’s coming, slowly, but coming. I had to call DPD today, because one of my parcels is sitting in the NL for three days. Hope it’s not lost :frowning: This one contains wheels, motors, …

Such a great idea! Can’t wait to see Nanny Bot in action.

P.S I think I need a robot vac (I have a 1 year old daughter)… Would you recommend the model you bought?

Wheels & motors arrived!

And they do work! Here’s the short video of a quick test.

The last missing piece is WiFi module & antennas. Should arrive next week.

1 Like

Yup, I’m happy with it. Can control it from the mobile phone, can create schedule when to clean, …

But I’d probably buy s9+ today, because it can empty itself on its own. Depends how big is the area for cleaning. I have to empty it once a week (cleaning twice a week). Not a big deal, but once you get used to it you want more from it.

Intel® Dual Band Wireless-AC 8265 & antennas arrived today. Will review all parts tomorrow just to check that I have everything I need. Excited!

And it works!

Intel® Dual Band Wireless-AC 8265

Hardware

I have a dual band antenna (2.4GHz & 5.X GHz). 5.X GHz is connected to the MAIN and 2.4 GHz is connected to the AUX. Here’s the Intel explanation why.

Software

Intel® Dual Band Wireless-AC 8265 is not officially supported. I’ve created the Missing AC8265 NGW support issue. There’s a workaround (how to make it working) inside it.

wlan0     Link encap:Ethernet  HWaddr A0:51:0B:4A:6F:05  
          inet addr:192.168.1.70  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::91cd:389a:7706:3a87/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:545 errors:0 dropped:0 overruns:0 frame:0
          TX packets:465 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:84943 (82.9 KiB)  TX bytes:84346 (82.3 KiB)

Photos

Can’t upload video, so, here’s the tweet with the short one.

Intel® Dual Band Wireless-AC 8265

Good news here. PR adding WiFi & BT support for this M.2 card was merged. We will officially support this chip for the Jetson Nano board soon.

Parts

Hopefully last package arrived today.

Caster ball

Later today, I’m going to build a special spacer for this caster ball, which will hold it & will fit all the holes on the chassis.

Amplifier & speaker

Not sure if if I use this speaker, it’s quite big, but at least I have something to test sound output with.

Jetbot

From the software point of view, I didn’t start with the development yet.

I’m trying to use the official Jetbot code on the balenaCloud device now. It’s a bit cumbersome to install all dependencies, but I’m getting there. Will share once done.

One important fact - use some fan if you’re doing heavy tasks on the Jetson Nano (like pytorch or tensorflow compilation). Otherwise it can get very warm, you can brake your SD card, etc. It’s just a precaution to protect it.

@zrzka oh wow, what’s the ambient temperature in your office? I’m enjoying following along, keep the updates coming! :grinning:

@chrisys 31 degrees of celsius outside, you can imagine what temperature is inside :slight_smile:

Evening exercise - caster ball mounted.

1 Like

These photos are from the work in progress. Forgot to take the final one. Will do tomorrow when I test how it moves :slight_smile:

Motors, wheels, caster ball & quickly hacked remote control test.

The code is available in the motor & cli folders. It’s a quick hack to see it moving with some degree of remote control. I do use Flask & simple HTTP requests. Next step will be to provide remote control via BT.

P.S. Don’t run it at full speed :slight_smile:

JetBot

Got some time yesterday evening, some small progress …

pytorch

JetBot depends on the pytorch package. I tried to build it on the Jetson Nano, but gave up after 8 hours. Decided to prebuild these big dependencies to speed up the whole process a bit. You can find the source here and binaries here. Don’t try to build pytorch, because it’s really big. The build folder has 10GB, it takes ages (even on our builders, ± hour), etc.

pip

The pytorch wheel size is 0.5GB (2GB unpacked) and pip (shipped with Ubuntu) can’t install it. It fails with MemoryError. You have to use pip >= 19.0. More info in this PR. Install it in this way:

apt-get install python3 python3-setuptools
apt-get remove python3-pip
python3 /usr/lib/python3/dist-packages/easy_install.py pip

Branch jetbot

I made a special branch for this JetBot experiment and it’s called jetbot (how surprising :slight_smile:). The Dockerfile is commented and you should be able to understand why it’s done in this way.

Focus on the README as well, because it’s not that straightforward as it sounds. I can’t provide JetPack SDK files for you (licensing) and you have to download them yourself.

And the last note - the resulting image has ±10GB - it’s still highly unoptimised. Anyway, you should end up with an image with all dependencies for the JetBot and the JetBot itself.


If you wanna play with all these tools, stick with 64GB SD card. 32GB will be fine for the final Nanny Bot, but not now.

JetBot II.

pytorch

Made a mistake last time and the pytorch wheel wasn’t working. The reason was that I forgot specify correct CUDA arch (missing TORCH_CUDA_ARCH_LIST=5.3). It’s fixed now and wheels are available for download at the jetson-nano-1.1 release page.

Don’t use the jetson-nano-1.0 release, it doesn’t work.

balena-nanny-bot/jetbot branch

Changes

  • Add missing GStreamer libraries (NVIDIA)
  • Download & install IMX219-160 Camera overrides
  • Install & properly configure Tegra EGL libraries
  • Utilize docker-compose.yml to be able to add SYS_RAWIO capability

All changes diff.

Deployment

  • Follow the README file instructions

NOTE: It’s still quite big image (not optimised yet).

[Info]     ┌─────────┬────────────┬────────────────────────┐
[Info]     │ Service │ Image Size │ Build Time             │
[Info]     ├─────────┼────────────┼────────────────────────┤
[Info]     │ jetbot  │ 8.57 GB    │ 29 minutes, 26 seconds │
[Info]     └─────────┴────────────┴────────────────────────┘
[Info]     Build finished in 36 minutes, 33 seconds

wander.py

JetBot repository contains wander.py application. Your JetBot just wanders and is trying to avoid collisions.

First terminal:

  • balena ssh $UUID --host
  • nvpmodel -m 1 (switch to 5W mode)
  • exit
  • balena ssh $UUID jetbot
  • nvargus-daemon (keep it running)
root@19c80a8:/usr/src/app# nvargus-daemon 
=== NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...

Second terminal:

  • balena ssh $UUID jetbot
  • Copy the best_model.pth from your computer (scp ...)
  • python3 jetbot/jetbot/apps/wander.py ./best_model.pth
root@19c80a8:/usr/src/app# python3 jetbot/jetbot/apps/wander.py ./best_model.pth 
Loading model...
Initializing camera...
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3280 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3280 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 3280 H = 2464 
   seconds to Run    = 0 
   Frame Rate = 21.000000 
GST_ARGUS: PowerService: requested_clock_Hz=37126320
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
Running...

It’s not user friendly, I know, I know, … but at least it works properly now!

Will also expose Python notebooks, so, we can train the JetBot and do everything what’s explained at the examples page.

Camera test pipeline

apt-get install gstreamer1.0-tools
xinit &
gst-launch-1.0 nvarguscamerasrc \
   ! 'video/x-raw(memory:NVMM),width=3280, height=2464, framerate=21/1, format=NV12' \
   ! nvvidconv flip-method=2 \
   ! 'video/x-raw(memory:NVMM), format=I420, width=3280, height=2464' ! nvoverlaysink