Home
JAQForum Ver 20.06
Log In or Join  
Active Topics
Local Time 00:15 04 Dec 2021 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Serial comms fundamentals - I think I'm missing a piece of the puzzle

Author Message
thwill

Guru

Joined: 16/09/2019
Location: United Kingdom
Posts: 1943
Posted: 10:12pm 25 Nov 2021
Copy link to clipboard 
Print this post

Hi folks,

I've been merrily making progress on adding serial comms to MMB4L and everything has been working beautifully writing to and reading from the CMM2 console over the serial port (/dev/ttyS0) with Ubuntu 20 on my overpowered ThinkPad.

Tonight I played with the same code on the much slower Raspberry Pi and I'm receiving incomplete data. I tell the CMM2 to LIST ALL some file and I see it all scroll past on the VGA screen but only receive a fraction of the data over the serial port before my code on the Pi starts reporting there is no data available.

My working theory (which may be bollocks) is that the CMM2 is sending faster than I am reading and this is resulting in data being lost, does anyone know what happens (in general or specifically in Linux/C) when a buffer is filled and how this should be mitigated, it's obvious that the CMM2 isn't being blocked because the VGA screen shows the entire file.

I suspect I'm missing some fundamental piece of understanding that I wouldn't be if I had been more than a snotty nosed kid in the 1980s.

Thanks in advance,

Tom
CMM2 Welcome Tape, Creaky old text adventures
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 5460
Posted: 10:22pm 25 Nov 2021
Copy link to clipboard 
Print this post

Serial I/O on the Pi is crap. There is only one proper H/W UART and this is used for Bluetooth or something. Anything else is software and with an OS intervening you have no chance of keeping up with any reasonable baudrate
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 4475
Posted: 11:12pm 25 Nov 2021
Copy link to clipboard 
Print this post

Just to prove the point, try the CMM2 at a slow baud rate.
I am not sure how slow you will have to go for the Pi to keep up.

You might also be able to increase the Pi receive buffer size. If you can set it to 5-10k, you 'might' have better results.

Jim
VK7JH
MMedit   MMBasic Help
 
thwill

Guru

Joined: 16/09/2019
Location: United Kingdom
Posts: 1943
Posted: 11:55pm 25 Nov 2021
Copy link to clipboard 
Print this post

Thank you that makes sense and means I'm not scratching my head over some mysterious programming bug. And it explains why:

I don't see it AUTOSAVEing from the Pi to the CMM2 because the CMM2's serial is faster than the Pi.

I don't see it with XMODEM transfers because the sender waits for an ACK message after each packet which keeps them in synch.

I guess what I can do is not use LIST ALL but just use LIST and then the PRESS A KEY... messages and the act of sending the "key" will keep them in synch.

Thanks again,

Tom
CMM2 Welcome Tape, Creaky old text adventures
 
tgerbic
Newbie

Joined: 25/07/2019
Location: United States
Posts: 17
Posted: 12:29am 26 Nov 2021
Copy link to clipboard 
Print this post

thwill,

What is missing here is software (xon/xoff) or better yet hardware (RTS/CTS/DCD) flowcontrol. Someone needs to go into the serial support module and add it in. Without it, there is little hope of keeping up with fast or large file transfers, even if you use large buffers. As matherp points out a hardware UART (with flow control) would be the right answer. This would also allow really high bit rates for transfers.

Regards
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 4475
Posted: 12:45am 26 Nov 2021
Copy link to clipboard 
Print this post

  thwill said  
I guess what I can do is not use LIST ALL but just use LIST and then the PRESS A KEY... messages and the act of sending the "key" will keep them in synch.

Thanks again,

Tom

That's how I did it in MMEdit before we had the "LIST ALL" option.
Send "LIST"
wait for the prompt,
send <CR>
repeat as required.

Jim
VK7JH
MMedit   MMBasic Help
 
SimpleSafeName

Senior Member

Joined: 28/07/2019
Location: United States
Posts: 229
Posted: 02:10am 26 Nov 2021
Copy link to clipboard 
Print this post

  thwill said  I guess what I can do is not use LIST ALL but just use LIST and then the PRESS A KEY... messages and the act of sending the "key" will keep them in synch.

Thanks again,

Tom


So, serial handshaking involving the use of an actual hand? :)
 
scruss
Newbie

Joined: 20/09/2021
Location: Canada
Posts: 34
Posted: 03:28am 26 Nov 2021
Copy link to clipboard 
Print this post

  matherp said  Serial I/O on the Pi is crap. There is only one proper H/W UART and this is used for Bluetooth or something. Anything else is software and with an OS intervening you have no chance of keeping up with any reasonable baudrate

That's odd: I've been running 3d printers from Raspberry Pis over serial links for years, and they've performed flat-out flawlessly. Some are on GPIO serial, others are on USB. 115200 and up for multi-day long print jobs. No glitches.

You can reassign what serial ports are used for which. Details here: Configuring UARTs

Also, don't expect PC-style UARTs, which kind of forget the whole "asynchronous" thing and are usable for bit-banging. Linux serial ports are much more laid back.
 
tgerbic
Newbie

Joined: 25/07/2019
Location: United States
Posts: 17
Posted: 04:02am 26 Nov 2021
Copy link to clipboard 
Print this post

I think the problem with PIs is inbound data can overrun the serial function. The 3D printer obviously can input data at rates the PI can put out. It may also be that the PI is not sending out continuous data at high speed to the printer, perhaps in bursts. The printer might also have a very large data buffer.

Dumping data out of a processor is pretty easy, taking it in without loss is a lot more complicated.

The only real way to know what is happening is to monitor the serial traffic.

Regards
 
SimpleSafeName

Senior Member

Joined: 28/07/2019
Location: United States
Posts: 229
Posted: 04:03am 26 Nov 2021
Copy link to clipboard 
Print this post

  scruss said  "...for multi-day long print jobs. No glitches."


Ummm, just how big (or how slow) *is* your printer? The first would be awesome, the second would be more like what I've got. BTW, my new printer sports a 590mm cubed build area.

Getting back on topic, I agree with you that the Pi should be able to handle it.

The caveat being that if the Pi is off doing its own thing occasionally and if the serial port is neglected during that time then maybe...

But there would have to be some hefty dead spots in your processing to cause that to happen.

A simple way to see would be to drive a LED on the Pi when it receives new data from the serial port. Then send it a string over and over and see if the LED goes out. If so, see what that period is, and look at Cron jobs and other periodic processes to see if you can find the culprit.


John
 
robert.rozee
Guru

Joined: 31/12/2012
Location: New Zealand
Posts: 1824
Posted: 04:55am 26 Nov 2021
Copy link to clipboard 
Print this post

  matherp said  Serial I/O on the Pi is ****. There is only one proper H/W UART and this is used for Bluetooth or something. Anything else is software and with an OS intervening you have no chance of keeping up with any reasonable baudrate


brutal, but honest. but also misses the obvious solution: USB to serial bridges. these are a dollar or two each, and provide sterling service up to 1,000,000 baud and beyond.

the RPi's all have at least one USB port, and using a USB hub you can increase this to four, seven, or more USB ports. Linux and the hardware UART in each bridge does the heavy lifting for you.

you can even get SIX serial ports using this:
https://github.com/harrywalsh/pico-hw_and_pio-uart-gridge/tree/HW_and_pio_uarts
https://forums.raspberrypi.com/viewtopic.php?t=302684&hilit=harrywalsh

GFXterm64 runs fine on a RPi, using a 256k Rx ring buffer, and a 16k Tx ring buffer. serial I/O is handled by a separate thread, see the source code as an example.


cheers,
rob   :-)
Edited 2021-11-26 14:56 by robert.rozee
 
panky

Guru

Joined: 02/10/2012
Location: Australia
Posts: 896
Posted: 04:58am 26 Nov 2021
Copy link to clipboard 
Print this post

Tom,

This link, Linux Serial IO might be usefull. A bit beyond me but seems to address some of the low level serial io structures etc., also with some sample code at the end.

Hope it helps,
Doug.
... almost all of the Maximites, the MicromMites, the MM Extremes, the ArmMites, the PicoMite and loving it!
 
phil99
Senior Member

Joined: 11/02/2018
Location: Australia
Posts: 227
Posted: 05:30am 26 Nov 2021
Copy link to clipboard 
Print this post

If I understand correctly, what Tom needs is something sufficiently universal that it can be incorporated into MMB4L. Jim's solution of having MMB4L send <CR> after receiving the "Press any key" prompt does not require any specific hardware.
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 1196
Posted: 08:20am 26 Nov 2021
Copy link to clipboard 
Print this post

Remember that the number of USB sockets on a Pi is not the number of USB ports! They are not independent of each other and are all connected to a single USB connection on the SOC. They share the 480Mb/s between them. Until the model 4 the ethernet connection was on the same bus and shared the same bandwidth. Yes, you can plug many devices in simultaneously via hubs (I think the limit is 30), but they are not on separate "ports".
Edited 2021-11-26 18:22 by Mixtel90
-- Mick

Zilog Inside! nascom.info for Nascom & Gemini
 
robert.rozee
Guru

Joined: 31/12/2012
Location: New Zealand
Posts: 1824
Posted: 11:55am 26 Nov 2021
Copy link to clipboard 
Print this post

  Mixtel90 said  Remember that the number of USB sockets on a Pi is not the number of USB ports [...] They share the 480Mb/s between them [...] Yes, you can plug many devices in simultaneously via hubs (I think the limit is 30), but they are not on separate "ports"


valid point, i was referring (rather loosely) to physical 4-pin sockets available once you've connected together enough hubs to fit your needs (5x 4-port hubs will give you 16 sockets to plug devices into). these sockets will then all end up sharing bandwidth on the single D+/D- pair of wires that connect back to the SoC.
see: https://acroname.com/blog/how-many-usb-devices-can-i-connect

but for most practical purposes, provided you've not got something attached that will suck up the full 480Mb/sec, there should be more than enough bits-per-second spare to support a full compliment (30?) of serial ports running at their top practical speed.


cheers,
rob   :-)
Edited 2021-11-26 21:56 by robert.rozee
 
thwill

Guru

Joined: 16/09/2019
Location: United Kingdom
Posts: 1943
Posted: 12:49pm 26 Nov 2021
Copy link to clipboard 
Print this post

Hi folks,

Thanks for all the useful comments.

Upon reflection, and I haven't tested this yet, I'm pretty certain this is nothing to do with the performance or otherwise of the Pi's serial I/O. Instead the CMM2 (running LIST ALL which is a pure "C" implementation) is simply pumping data into the port faster than my MMBasic client program (which is in my maintainable though not necessarily efficient style) running under MMBL on the Pi can read it out. Buffering can mitigate, but at some point unless there is some form of flow control any buffer is going to overflow given a big enough file being LISTed.

The "missing piece of the puzzle" is that I didn't know or expect that serial comms would simply drop data (without obviously reporting any errors) rather than blocking when the buffers were full, now I do .

  TassyJim said  That's how I did it in MMEdit before we had the "LIST ALL" option.
Send "LIST"
wait for the prompt,
send <CR>
repeat as required.


Right, I suspected from past comments that you have made that I was propably re-inventing part of the MMEdit wheel.

As part of the development of serial support in MMB4L and because I want it for my personal workflow I have been writing an MMBasic command-line program called "gonzo" that provides a "kermit" like interface to allow me to send and receive files between MMB4L and a CMM2 or PicoMite without having to work directly at the AUTOSAVE and XMODEM level. I'm aware of other options but I like Linux, command-line tools, and not having to worry about Python dependencies.

  SimpleSafeName said  So, serial handshaking involving the use of an actual hand? :)


Not quite . I can automatically parse the data received from the CMM2 for that prompt and send a character (key press) in response. Alternatively I suppose I could suggest everyone gets one of those "drinking bird" toys as an accessory for MMB4L but that didn't go so well for Homer.

  robert.rozee said  GFXterm64 runs fine on a RPi, using a 256k Rx ring buffer, and a 16k Tx ring buffer. serial I/O is handled by a separate thread, see the source code as an example.


A timely reminder to take a look at your source-code. The initial MMB4L implementation is pretty primitive, but hopefully functional, just enough to meet my own needs. If it turns out MMB4L gets some actual users then I can improve it based on feedback.

  panky said  This link, Linux Serial IO might be usefull. A bit beyond me but seems to address some of the low level serial io structures etc., also with some sample code at the end.


Thanks for the link Doug, I've already read and digested several similar documents, e.g. Serial Programming Guide for POSIX Operating Systems and the Linux Serial HOWTO but it never hurts to read one more.

  phil99 said  If I understand correctly, what Tom needs is something sufficiently universal that it can be incorporated into MMB4L. Jim's solution of having MMB4L send <CR> after receiving the "Press any key" prompt does not require any specific hardware.


It's not my plan to incorporate "gonzo" into MMB4L initially, but I might do so in time considering my personal use-case for MMB4L as a development platform for CMM2 and PicoMite.

Thanks again folks,

Tom
Edited 2021-11-26 22:57 by thwill
CMM2 Welcome Tape, Creaky old text adventures
 
robert.rozee
Guru

Joined: 31/12/2012
Location: New Zealand
Posts: 1824
Posted: 02:35pm 26 Nov 2021
Copy link to clipboard 
Print this post

here is the (latest) linux serial code used:
linux_serial.zip

i've removed the (unused) non-threaded read and write routines, as they are likely to be just confusing. while it is in pascal, the names of functions called should match fairly well with C, and you can just ignore all the little counters, flags, calls to GetTickCount64, etc that are sprinkled throughout the code as these are just things to make GFXterm 'work more pretty'.

hopefully there is a C equivalent to the lazarus threading library, as this eases life considerably.


cheers,
rob   :-)
 
Print this page


To reply to this topic, you need to log in.

© JAQ Software 2021