Log shows some errors.

Product support
1

hi, I followed step by step instructions to set up AWS via balena clouds for RPI from here:

I cant see the pyton script to return anything into cloud (like messages with 1,2,3,4 examples shows).
my log shows the following:

Traceback (most recent call last):
01.05.20 16:54:52 (+0300) main File “src/main.py”, line 57, in
01.05.20 16:54:52 (+0300) main mqtt_client.connect()
01.05.20 16:54:52 (+0300) main File “/usr/local/lib/python3.5/site-packages/AWSIoTPythonSDK/MQTTLib.py”, line 355, in connect
01.05.20 16:54:52 (+0300) main return self._mqttCore.connect(keepAliveIntervalSecond)
01.05.20 16:54:52 (+0300) main File “/usr/local/lib/python3.5/site-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py”, line 291, in connect
01.05.20 16:54:52 (+0300) main raise connectTimeoutException()
01.05.20 16:54:52 (+0300) main core.exception.AWSIoTExceptions.connectTimeoutException
01.05.20 16:55:10 (+0300) Service exited ‘main sha256:849e202f3bf194a01b7b1e0a646e82e66cceb8ce79a01c1e700f97991b9ce723’
01.05.20 16:55:36 (+0300) Restarting service ‘main sha256:849e202f3bf194a01b7b1e0a646e82e66cceb8ce79a01c1e700f97991b9ce723’
01.05.20 16:54:52 (+0300) main Traceback (most recent call last):
01.05.20 16:54:52 (+0300) main File “src/main.py”, line 57, in
01.05.20 16:54:52 (+0300) main mqtt_client.connect()
01.05.20 16:54:52 (+0300) main File “/usr/local/lib/python3.5/site-packages/AWSIoTPythonSDK/MQTTLib.py”, line 355, in connect
01.05.20 16:54:52 (+0300) main return self._mqttCore.connect(keepAliveIntervalSecond)
01.05.20 16:54:52 (+0300) main File “/usr/local/lib/python3.5/site-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py”, line 291, in connect
01.05.20 16:54:52 (+0300) main raise connectTimeoutException()
01.05.20 16:54:52 (+0300) main core.exception.AWSIoTExceptions.connectTimeoutException
01.05.20 16:55:49 (+0300) main MQTT Thing Starting…
01.05.20 16:56:00 (+0300) main Connect timeout.
01.05.20 16:56:08 (+0300) main Traceback (most recent call last):
01.05.20 16:56:08 (+0300) main File “src/main.py”, line 57, in
01.05.20 16:56:08 (+0300) main mqtt_client.connect()
01.05.20 16:56:08 (+0300) main File “/usr/local/lib/python3.5/site-packages/AWSIoTPythonSDK/MQTTLib.py”, line 355, in connect
01.05.20 16:56:08 (+0300) main return self._mqttCore.connect(keepAliveIntervalSecond)
01.05.20 16:56:08 (+0300) main File “/usr/local/lib/python3.5/site-packages/AWSIoTPythonSDK/core/protocol/mqttCore.py”, line 291, in connect
01.05.20 16:56:08 (+0300) main raise connectTimeoutException()
01.05.20 16:56:08 (+0300) main core.exception.AWSIoTExceptions.connectTimeoutException
01.05.20 16:56:16 (+0300) Service exited ‘main sha256:849e202f3bf194a01b7b1e0a646e82e66cceb8ce79a01c1e700f97991b9ce723’
01.05.20 16:56:32 (+0300) Restarting service ‘main sha256:849e202f3bf194a01b7b1e0a646e82e66cceb8ce79a01c1e700f97991b9ce723’
01.05.20 16:56:08 (+0300) main Traceback (most recent call last):

Please advice me something to fix the issue.

I wonder during a setup it is possible that I a bit messed up with certificate files. File names are different in instruction link and in real system. Might be this was an issue. I’m sure only in file which has a ‘private’ tag in a name.

2

Hi, thanks for your post. Blogs have an unfortunate habit of getting dated quickly but we are keen to understand what is wrong here so we can look into it further and update. We also have documentation here https://www.balena.io/docs/learn/develop/integrations/aws/ for the AWS integration that has received more frequent updates and there was a recent update to Fix error in values for AWS_PRIVATE_CERT and AWS_THING_CERT so that might indeed be the root of the issue. Can you have a quick check of this document and see if it resolves your problem?

4

Garethdavies, thank you for your reply and help. I’m sure we will solve this.

May I ask you when you followed the link of instructions you sent me last time?
Is it 100% working?
Was it tested by somebody?

I wonder there is still some mess with certificate files :
Private key file xxx.private.key, is actually file 34b76ca30a-private.pem.key when it gets saved at my PC.
Thing file xxx.cert.pem, is actually file 34b76ca30a-certificate.pem.crt when it gets saved at my PC.

The file root-CA.crt not exists at any page at all, no way, why?.
The last download link refers to another big page where there are 6-7 files with ‘pem’ extension, I saved them all here:

image
image745×293 12.6 KB

They all are with extension ‘pem’ but have ‘Root’ in name, but no ‘crt’ extension as described.
Instruction shows that ‘root-CA’ file has extension ‘crt’ but the only single file with ‘crt’ extension
is actually ‘thing’ file.
FFS, If file is wrong I cant return to same page. I need to delete all certificates and generate them again, and then start from the middle of instruction.
This is so mad.

Please explain by using my file list attached which files I need to use.
Also I remember the focus been at 4 files, is it that we look now for 3 files only, private file, thing file and very well hidden root-ca file ?

I cant see similar questions at the forum, so I wonder all this is clear for most people but looks not for me here.
I do apologies again but I do set it all up 2nd day now step by step and I never yet reached a demo.

Ahead there will be questions if I may edit and upload pyton scrip back to RPI as I want to read from Raspberry PI com port some info and send that info to AWS and other questions but I’m far from all these in a minute.

Alex

8

hey @AlexJeli

I can confirm that the project is working. Here are logs from a device running the project.

image
image727×148 16.9 KB

However, I do realise some of the steps need more clarifications and we will surely update that. Thanks for the feedback.

For now, here are the things you need to pay attention to

  1. When you create a thing in AWS, after you have downloaded the certificates, make sure you activate them.

image

  1. You will also need the root CA certificate for AWS IoT which must be downloaded separetely from here and I chose this one.

Note in the guide it says, root-CA.crt but when you download the above one I linked, you can name it AmazonRootCA1.pem

  1. make sure you are converting the certificates and keys to base64 string. You should have these environment variables:
    image

Please give it a try and let us know if it works.
We will be looking at ways to make the guide and running the project more intuitive.

Thanks

10

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello Rahul-thakoor,

Thank you for your reply and help.

I do apologies for the delay in my reply.

I have no chance to try your last advice yet.

But I will surely do this asap, so your time and help is not wasted!

I’m C++/C# embedded developer, quite busy, and all these “Cloud”

things are a bit new for me.

In a meanwhile I think I also seen an option when AWS IoT SDK was used

and license keys been copied first to Raspberry PI unit and been accessed

from some Python app at RPI.

The demo I try to achieve is not my final goal only.

Honestly, I don’t want precisely to learn AWS as I have lots of other things to do.

But I got a request if we may collect our customer’s data and send it to some cloud,

so I selected famous at this moment!

If my prototype project gets success, then we will look to have more business

and financial business with you and AWS by switching to different plans and etc.

My goal is actually to read serial data at Raspberry PI, parse it and send it

to AWS Cloud, so then later it would be possible to browse and report that data by somebody else.

I see Raspberry PI side could be done by using a coding at Python.

I never much used Python but I had a look and wrote a little app which reads serial port at RPI.

Now I need to send parsed data to the AWS Cloud for long storage there.

Might be now, based on all this info I sent you you may advice me best approaches

to achieve what I explained you here.

Alex

rahul-thakoor balena team

May 4hey @AlexJeli

I can confirm that the project is working. Here are logs from a device running the project.

However, I do realise some of the steps need more clarifications and we will surely update that. Thanks for the feedback.

For now, here are the things you need to pay attention to

  1. When you create a thing in AWS, after you have downloaded the certificates, make sure you activate them.

  2. You will also need the root CA certificate for AWS IoT which must be downloaded separetely from here and I chose this one.
    Note in the guide it says, root-CA.crt but when you download the above one I linked, you can name it AmazonRootCA1.pem

  3. make sure you are converting the certificates and keys to base64 string. You should have these environment variables:
    Please give it a try and let us know if it works.

We will be looking at ways to make the guide and running the project more intuitive.

Thanks

11

Hi @AlexJeli, first of all, yes please try out Rahul’s advice and let us know if you get stuck with any of the steps outlined in our docs.

As for your final goal. balena team and our community have couple of projects and blog posts about reading serial data and processing it on Raspberry Pi. Take a look at these as they might help you to start with your own project:


https://scrobotics.es/2020/02/03/gps-tracker-with-balenaos/

Sounds good to use Python to read serial data. I heard this library mentioned couple of times by our users that utilize serial ports:
https://pyserial.readthedocs.io/en/latest/pyserial.html

Let us know how it goes.

13

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello, Gelbal.

I had to edit this email as it returned back with note that it had more than 5 links.

14

Hi Alex,

Do you still need further information aside from the references provided by Gelbal?

Regards,
Carlo

16

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello Carlo,

thank you for reply.

Yes, I do need to know once I saved data from Raspberry Pi

to AWS into topic called demo-topic, how I may read that data

from the cloud system from PC.

Alex

cmfcruz balena team

May 12Hi Alex,

Do you still need further information aside from the references provided by Gelbal?

Regards,

Carlo

17

Hi Alex

In the document you are following, there is a section for testing that your app is working, including testing that your data is being recieved into AWS:

18

Could you use that section and see if your data appears.

21

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello Phil,

thank you for your reply.

Yes, sure, I may see the data

from Raspberry Pi at the topic once I publish it at AWS page MQTT client.

My question from this point is - how I use another PC 1,000 miles away

and grab that data from AWS server from that topic ?

This is the point of the experiment - to be able to use serial device plugged into

Raspberry Pi, which then sending data into AWS IoT and then another PC reads that data

from AWS IoT.

The point until data comes to AWS cloud is clear and works.

Do you have an example how to read the data programmatically from AWS IoT at another device,

different from which sent the data?

Alex

phil-d-wilson balena team

May 12 balena – 16 Jul 19

Use a Raspberry Pi to communicate with Amazon AWS IoT

When developing an IoT project, there are many things to take into account, from choosing the hardware and software to the infrastructure where you will host all the sensor data. Gladly managing a device fleet with balenaCloud is as easy as typing…

Could you use that section and see if your data appears.

22

Hey @AlexJeli,

In order to do that you will have to install an MQTT client to that PC, connect to the AWS IoT MQTT broker and finally subscribe to the topic to which RPI sends the data. While you can use any MQTT client, it is best to use the python SDK provided by AWS.

You can actually replicate the post’s project, to the remote PC by:

  1. Creating a VM in the PC (Guide)
  2. Then creating a virtual balenaOS device into that VM.
  3. Finally, installing the MQTT project into that device.

Best,
Odysseas

24

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello,
Thank you for your help.
I think I managed to read data from AWS Cloud by using MQTT client under my Windows machine.
I do use python SDK provided by AWS as I found codes for publish and subscribe quite close.
You help and suggestions are priceless! :wink:
I do apologies if I’m still asking questions here,
in general I’m mostly clear with the principle of the data exchange now.
May I know -
N1.
if you may suggest best example for MQTT Client this time running under **Windows using .Net C#**language ?

Please point also correct dll for .Net as I found it is important to have a right one for this.

Another little question I want to ask -

N2.

if you think there is a possibility to send binary data string from RPI, accept it by AWS

and then receive (read) exactly same binary data string at other end at the PC or another RPI ?

By other words, without a need to parse the data via JSON method

may I send say 19 bytes via Cloud and then receive exactly same 19 bytes

at the other at the client side ?

Basically, use a cloud transport as a very very long serial cable. :wink:

N3.

If so, what are the limitations for the data refresh period and the packet size ?

I do apologies for primitive example,

my goal is only to be sure that I’m understood with what I need.

Alex

odyslam balena team

May 13Hey @AlexJeli,

In order to do that you will have to install an MQTT client to that PC, connect to the AWS IoT MQTT broker and finally subscribe to the topic to which RPI sends the data. While you can use any MQTT client, it is best to use the python SDK provided by AWS.

You can actually replicate the post’s project, to the remote PC by:

  1. Creating a VM in the PC (Guide)
  2. Then creating a virtual balenaOS device into that VM.
  3. Finally, installing the MQTT project into that device.
    Best,

Odysseas

26

Hi Alex,

Have you looked into the AWS SDK for .NET? According to AWS the “package includes the code library, code samples, and documentation.” There are also some other .NET C# MQTT client libraries out there such as MqttDotNet, paho.mqtt.m2mqtt, and MQTTnet but I have not used any of them so I can’t make a specific recommendation. However if you look these up you should find some examples that will be helpful for you. (Maybe another user in our forum has some suggestions?)

For your question about sending/receiving binary data, I believe there are ways you can accomplish this. You might want to search for information about “serial over IP” to get started in that direction.

28

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello Alan,

Thank you very much for your time and help.

I’m going to look AWS SDK for .Net now.

I’m sure in general there will be same principle like it exists in Python code in your examples, I think.

Yes, I understand the suggestion for serial to IP. I do have some hardware solutions from

some company called Moxa. It sends serial data over Ethernet and then same type of the box at other end

reads Ethernet back to serial.

I thought we could do the same using cloud system, where real time is not vital.

Serial to IP is difficult to pass via all routers with open ports

and forward ports on sites for our customers.

Alex

alanb128 balena team

May 19Hi Alex,

Have you looked into the AWS SDK for .NET? According to AWS the “package includes the code library, code samples, and documentation.” There are also some other .NET C# MQTT client libraries out there such as MqttDotNet, paho.mqtt.m2mqtt, and MQTTnet but I have not used any of them so I can’t make a specific recommendation. However if you look these up you should find some examples that will be helpful for you. (Maybe another user in our forum has some suggestions?)

For your question about sending/receiving binary data, I believe there are ways you can accomplish this. You might want to search for information about “serial over IP” to get started in that direction.

30

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello Alan,
I have found that I have a little issue with python scripts
at publish side and at subscribe side as well.
**ISSUE N1.**My Raspberry PI publishes some data to AWS fine.
I could subscribe to the topic and
I see the actual data via aws web.
The issue is at PC side when I do run my python script sub.py,
which subscribes to the same RPI’s topic.
I do subscribe by using:
myAWSIoTMQTTClient.subscribeAsync(topic, 1, ackCallback=customSubackCallback, messageCallback=customOnMessage)
However, the only event which triggers is:
customSubackCallback

(Attachment publisher.py is missing)

(Attachment sub.py is missing)

35

Hi. Looks like it is working, and the issue is specifically related to the AWS SDK. Unfortunately, that’s out of scope for us. You might want to try contacting AWS support or opening a ticket in the aws-iot-device-sdk-python github repo.

37

Re: [balenaForums] [balenaCloud] Log shows some errors.
Hello Carol,

thank you for the reply.

How you seen my scripts?

The email returned back as attachments not passed a security check at mail system:

39

Hi Alex,

I haven’t seen your scripts as it was not attached to the forums but it seems to me that your problem is related to this Stackoverflow thread I found where a user was having problems publishing using the AWS IoT SDK MQTT client to an existing topic using a different client. Please click the link below to follow the discussion thread there:

I hope this helps you with your problem in publishing from your PC.

All the best,
Carlo