Receiving wrong bytes from co-processor over serial line

balenaFin
1

I’m at a point where I have flashed the pre-build firmata firmware from https://github.com/balena-io-playground/balena-fin-firmata-flash to the co-processor, but are having problems communicating with it. When sending a firmware version request over serial, it only sometimes answer.

I have therefore tried to dig deeper into what’s going on. I therefore managed to compile the firmware myself, from the source: GitHub - balena-io-hardware/fin-coprocessorFirmata-sw: firmata firmware for the BGM111 co-processor on the balenaFin

And flashed it on to the co-processor but the same error was present (it didn’t always respond). I therefore commented out almost everything in the main StandardFirmata.cpp so it just echoed whatever it received (with an extra byte, 25, added in front, so I could see if that at least came through correct):

int main(void)
{
balenaInit();

 Serial.begin(57600);

 while(1) 
 {
    while(Serial.available() > 0)
    {
        int inputData = Serial.read();
        if (inputData != -1) 
        {
             Serial.write((byte)25);
             Serial.write((byte)inputData);
         }
     }
 }

}

After flashing this on to the co-processor, it replied all the time. Sometimes with the correct echoed bytes (with the byte: 25 in front) and most of the times with rubish bytes. If I for example send over

194

it sometimes returned as expected:

25 194

but most of the times altered bytes like:

9 250
240 25
etc.

I therefore tried to alter the code, so it only sent byte 65 (character ‘A’) every 1 second, and didn’t have to read from the serial port. After flashing, I opened putty, connected to the serial line, and saw the same behavior. The character ‘A’ was sometimes coming through, but mostly rubish bytes.

Has anyone else experienced the same sort of behavior? I’m initializing the serial port on the PI to communicate with the co-processor using following settings:

Baud rate: 57600
Data bits: 8
Stop bit: 1
Parity: None
Handshake: None
DTR/RTS: Disabled

3

Hi @mad.dove,

Which version of the Fin are you using? v1.0 or v1.1?

Thanks,
Alex

5

Hi Alex,

Ahh yea, I of course should have mentioned that. It’s v1.1 - 16GB (so the BGM111 chip).

6

Ok, great :slight_smile: What were you specifically doing initially that made you realise it wasn’t communicating? Were you sending it custom firmata commands or trying to test the sleep functionality?

7

My target was to try and read the GPIO’s (digital and analog read), however to see if the connection worked, I just send over a firmware version request. I did this a couple of times and found out it only answered me a couple of times. Then i thought it could be something with the firmware version method, and tried sending some analog read request. But it only periodically replied to that as well. This was when i started trying to get the firmware compiled myself, so I could do some “debugging”.

So I have not tested anything concerning sleep no. Only firmware version and analog report requests.

8

Would you mind sharing the commands/script you used to test the firmware version request? I’ll try this on a local device and attempt to reproduce this behaviour.

Thanks!

9

I’m using C# mono and the SerialPort class and basically just create the SerialPort object (with /dev/serial1), call Open(), create a thread reading from the port and then start writing a byte array with the bytes { 0xF0, 0x0B, 0x00, 0xF7 } to the serial port followed by a sleep(5000), which is then looped 15 times. In this case, I would typically get 2 / 15 replies.

The code is currently in a large project for an application we are making for the fin, but I could create a new small project a build an executable you could run via mono v. 6?

10

I went ahead a created the small program, you can download the exe (zipped) from here: https://transfernow.net/897vl4y2oskf

To run it you need mono on the fin. To install it, run the following commands (assuming you are running Raspbian 9):

sudo apt install apt-transport-https dirmngr gnupg ca-certificates
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo “deb https://download.mono-project.com/repo/debian stable-raspbianstretch main” | sudo tee /etc/apt/sources.list.d/mono-official-stable.list
sudo apt update
sudo apt install mono-complete

Place the CoProcessor.exe file somewhere on the fin and fire the following command to run it:

mono CoProcessor.exe /dev/serial1 57600

This should open a connection to serial1 with baud rate 57600 and write firmware version requests every 3 seconds 15 times. For me, the co-proccessor typically replies the first time, but not for the following 14 requests.

11

That’s great :slight_smile: Could you also share the source for the exe?

Thanks!

12

Yes, of course. Since it’s not that much code, I’ll just post it in here for everyone to see:

using System;
using System.IO.Ports;
using System.Linq;
using System.Threading;

namespace CoProcessorTest
{
    class Program
    {
        private static SerialPort _serialPort;
        private static Thread _readThread;

        private static object _objReadWriteLock = new object();
        
        static void Main(string[] args)
        {
            try
            {
                if(args == null || args.Length == 0)
                {
                    Console.WriteLine("Please specify arguments: serial port name and baud rate (baud rate is optional).");
                    Console.ReadKey();
                    return;
                }

                int baudrate = 57600;

                if(args.Length > 1)
                    int.TryParse(args[1], out baudrate);

                Console.WriteLine("Opening serial port: {0} with baud rate: {1}.", args[0], baudrate);

                _serialPort = new SerialPort(args[0], baudrate);
                _serialPort.DataBits = 8;
                _serialPort.Parity = Parity.None;
                _serialPort.StopBits = StopBits.One;
                _serialPort.BaudRate = baudrate;
                _serialPort.Handshake = Handshake.None;
                _serialPort.DtrEnable = false;
                _serialPort.RtsEnable = false;

                _serialPort.Open();

                if (_readThread == null)
                {
                    _readThread = new Thread(processInput);
                    _readThread.IsBackground = true;
                    _readThread.Start();
                }

                var arr = new byte[] { 0xF0, 0x0B, 0x00, 0xF7 };
                for (int i = 0; i < 15; i++)
                {
                    Console.WriteLine("");

                    lock (_objReadWriteLock)
                    {
                        _serialPort.Write(arr, 0, arr.Length);
                        Console.WriteLine("DateTime: {0}, Wrote bytes: {1}", DateTime.Now.ToString("dd-MM-yyyy HH:mm:ss.fff"), string.Join(", ", arr.Select(x => x.ToString())));
                    }
                    System.Threading.Thread.Sleep(3000);
                }

                _serialPort.Close();

                Console.WriteLine("");
                Console.WriteLine("Closed serial connection, done calling co-processor. Press any key to exit.");
            }
            catch(Exception e)
            {
                Console.Write("An unexpected error occurred. {0}", e.Message ?? "");
            }

            Console.ReadKey();
        }

        public static void processInput()
        {
            while (_serialPort != null && _serialPort.IsOpen)
            {
                try
                {
                    var i = _serialPort.BytesToRead;
                    if (!_serialPort.IsOpen || i == 0)
                    {
                        System.Threading.Thread.Sleep(10);  //Avoid CPU burn
                        continue;
                    }

                    byte[] arr = new byte[i];

                    lock (_objReadWriteLock)
                    {
                        var read = _serialPort.Read(arr, 0, i);
                    }

                    Console.WriteLine("DateTime: " + DateTime.Now.ToString("dd-MM-yyyy HH:mm:ss.fff") + ", Read bytes: " + string.Join(", ", arr.Select(x => x.ToString())));
                }
                catch (Exception e)
                {
                    Console.WriteLine("Read error: {0}", e.Message ?? "");
                }

                Thread.Sleep(10);  //Avoid CPU burn
            }
        }
    }
}
14

Thanks! @mad.dove

17

@mad.dove just clarifying some details - you flashed firmata initial via a container provisioned by balenaOS but then you reflashed the board with a raspbian image and have been using that to test?

18

No, the balenaFin is running Raspbian and has always done so. I just executed the same commands in a terminal as the balena application do when installing the application (https://github.com/balena-io-playground/balena-fin-firmata-flash). So basically all commands in the Dockerfile.template file. Then I compiled and installed the kernal module gpio-pca953x.c, since it was missing (to get GPIO 510 up and running for co-processor reset). And hereafter I flashed the co-processor.

I can now also verify the serial problem on a 8GB v1.1 fin (we have 3 fins at the office, and I have now tested it on two of them).

19

Got it! Thanks for the clarification - I’ve tested this on a Fin using balenaCloud so the issue doesnt seem to be on the coprocessor itself but something to do with the serial interface. I’ll follow your described steps, along with compiling said modules and report back.

Thanks!

21

Ok, so what you are saying is that it works on a fin running balenaOS with whatever serial interface/driver that uses, but not on Raspbian with its serial interface/driver, right?

Also note, that the co-processor answers as expected basically every first time you open the serial port and send a single request. The problem occurs when you keep the serial port open after first successful request and start sending more requests.
I’m mentioning this because if the application you tested with running on balenaOS basically opens and closes the serial port for each request to the co-processor, it maybe wont hit the issue.

When you flash in Raspbian, remember to set GPIO 41 high before running flash.sh (I didn’t mention that in my precious description). :slight_smile:

23

Yes that’s right, it is working on a fin running balenaOS :slight_smile: I’ll set my fin up how you have described and try to reproduce. I did notice that your C# method for receiving bytes back from the coprocessor was causing received data to become segmented as the code reads from the buffer before a complete sysex message is returned by the coprocessor.

Ideally when reading from a firmata host, the client should wait until the sysex message returns 0xF7 to signify the end of the message. This might help address the issue?

28

The code is just a short “demo” demonstrating the co-processor is not answering 90% of the time, due to it receiving other bytes than the ones being sent to it from the PI. So yes, the message returned from the co-processor, when it answers, might get segmented, but that’s easy to handle. I just want the bytes received by the co-processor to be the actual bytes the PI send, as well as I want the bytes returned from the co-processor to the PI, to be the same that the co-processor actually wrote to the serial port.

An example:
If I send the byte, say 65 to the co-processor. Then sometimes it receives the byte 65, but most of the time it receives something else (like 226 or 11, etc - it seems random). Likewise, when the co-processor sends a byte to the PI, then I would like the PI to actually receive that same byte, and not something else (again what it receives seems random).

I hope the clarifies the problem I’m facing. :slight_smile:

As I wrote earlier, I have also flashed a version of the co-processor firmware, where it writes the byte 65 to the serial port every 1 second. If I then open Putty and connect to the serial port, then 1/10 times it receives the correct byte 65 and 9/10 times it receives some other random byte.

31

Thanks for your patience @mad.dove, I’ve got your executable running under your described scenario and I can confirm that I am also seeing this reported behaviour (coprocessor responds once/twice). I believe this issue relates to the Serial interface so am now examining how this is behaving under Raspbian vs BalenaOS.

34

Alright sounds good and perfect you can reproduce it. It would definitely also be interesting to see the output of:

stty -a -F /dev/serial1

On both BalenaOS and Raspbian, to make sure they are connecting using the same settings (to rule out a simple settings problem).

43

Hi @mad.dove,

Thanks for your patience. Finally worked out what was going on here, at least on my end so I wanted to confirm with you; do you have core_freq=250 set in your /boot/config.txt file? If not, please try to set this and let me know if it fixes your issue.

We discovered that the UART1 interface was sending mismatched data to the coprocessor which was behaving in a particularly strange way - the payload being sent was consistent but incorrect (offset hex sometimes with additional bytes). The problem turn out to be the clock used to generate the miniUART (UART1) baudrate. This inherits from the video processing unit clock frequency, which when not fixed can stretch the baudrate used by the UART1 interface.
We set the core_freq in the kernel under balenaOS, which is why this was working under balena and not our raspbian image.

Let me know how you get on with this!

Thanks!