 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : Serial comms fundamentals - I think I'm missing a piece of the puzzle
| Author | Message | ||||
thwill Guru  Joined: 16/09/2019 Location: United KingdomPosts: 3839 |
Hi folks, I've been merrily making progress on adding serial comms to MMB4L and everything has been working beautifully writing to and reading from the CMM2 console over the serial port (/dev/ttyS0) with Ubuntu 20 on my overpowered ThinkPad. Tonight I played with the same code on the much slower Raspberry Pi and I'm receiving incomplete data. I tell the CMM2 to LIST ALL some file and I see it all scroll past on the VGA screen but only receive a fraction of the data over the serial port before my code on the Pi starts reporting there is no data available. My working theory (which may be bollocks) is that the CMM2 is sending faster than I am reading and this is resulting in data being lost, does anyone know what happens (in general or specifically in Linux/C) when a buffer is filled and how this should be mitigated, it's obvious that the CMM2 isn't being blocked because the VGA screen shows the entire file. I suspect I'm missing some fundamental piece of understanding that I wouldn't be if I had been more than a snotty nosed kid in the 1980s. Thanks in advance, Tom Game*Mite, CMM2 Welcome Tape, Creaky old text adventures |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 8578 |
Serial I/O on the Pi is crap. There is only one proper H/W UART and this is used for Bluetooth or something. Anything else is software and with an OS intervening you have no chance of keeping up with any reasonable baudrate |
||||
TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 5899 |
Just to prove the point, try the CMM2 at a slow baud rate. I am not sure how slow you will have to go for the Pi to keep up. You might also be able to increase the Pi receive buffer size. If you can set it to 5-10k, you 'might' have better results. Jim VK7JH MMedit MMBasic Help |
||||
| thwill Guru Joined: 16/09/2019 Location: United KingdomPosts: 3839 |
Thank you that makes sense and means I'm not scratching my head over some mysterious programming bug. And it explains why: I don't see it AUTOSAVEing from the Pi to the CMM2 because the CMM2's serial is faster than the Pi. I don't see it with XMODEM transfers because the sender waits for an ACK message after each packet which keeps them in synch. I guess what I can do is not use LIST ALL but just use LIST and then the PRESS A KEY... messages and the act of sending the "key" will keep them in synch. Thanks again, Tom Game*Mite, CMM2 Welcome Tape, Creaky old text adventures |
||||
| tgerbic Newbie  Joined: 25/07/2019 Location: United StatesPosts: 40 |
thwill, What is missing here is software (xon/xoff) or better yet hardware (RTS/CTS/DCD) flowcontrol. Someone needs to go into the serial support module and add it in. Without it, there is little hope of keeping up with fast or large file transfers, even if you use large buffers. As matherp points out a hardware UART (with flow control) would be the right answer. This would also allow really high bit rates for transfers. Regards |
||||
| TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 5899 |
That's how I did it in MMEdit before we had the "LIST ALL" option. Send "LIST" wait for the prompt, send <CR> repeat as required. Jim VK7JH MMedit MMBasic Help |
||||
SimpleSafeName Senior Member  Joined: 28/07/2019 Location: United StatesPosts: 286 |
So, serial handshaking involving the use of an actual hand? :) |
||||
| scruss Regular Member  Joined: 20/09/2021 Location: CanadaPosts: 83 |
That's odd: I've been running 3d printers from Raspberry Pis over serial links for years, and they've performed flat-out flawlessly. Some are on GPIO serial, others are on USB. 115200 and up for multi-day long print jobs. No glitches. You can reassign what serial ports are used for which. Details here: Configuring UARTs Also, don't expect PC-style UARTs, which kind of forget the whole "asynchronous" thing and are usable for bit-banging. Linux serial ports are much more laid back. |
||||
| tgerbic Newbie Joined: 25/07/2019 Location: United StatesPosts: 40 |
I think the problem with PIs is inbound data can overrun the serial function. The 3D printer obviously can input data at rates the PI can put out. It may also be that the PI is not sending out continuous data at high speed to the printer, perhaps in bursts. The printer might also have a very large data buffer. Dumping data out of a processor is pretty easy, taking it in without loss is a lot more complicated. The only real way to know what is happening is to monitor the serial traffic. Regards |
||||
| SimpleSafeName Senior Member Joined: 28/07/2019 Location: United StatesPosts: 286 |
Ummm, just how big (or how slow) *is* your printer? The first would be awesome, the second would be more like what I've got. BTW, my new printer sports a 590mm cubed build area. Getting back on topic, I agree with you that the Pi should be able to handle it. The caveat being that if the Pi is off doing its own thing occasionally and if the serial port is neglected during that time then maybe... But there would have to be some hefty dead spots in your processing to cause that to happen. A simple way to see would be to drive a LED on the Pi when it receives new data from the serial port. Then send it a string over and over and see if the LED goes out. If so, see what that period is, and look at Cron jobs and other periodic processes to see if you can find the culprit. John |
||||
| robert.rozee Guru Joined: 31/12/2012 Location: New ZealandPosts: 2285 |
brutal, but honest. but also misses the obvious solution: USB to serial bridges. these are a dollar or two each, and provide sterling service up to 1,000,000 baud and beyond. the RPi's all have at least one USB port, and using a USB hub you can increase this to four, seven, or more USB ports. Linux and the hardware UART in each bridge does the heavy lifting for you. you can even get SIX serial ports using this: https://github.com/harrywalsh/pico-hw_and_pio-uart-gridge/tree/HW_and_pio_uarts https://forums.raspberrypi.com/viewtopic.php?t=302684&hilit=harrywalsh GFXterm64 runs fine on a RPi, using a 256k Rx ring buffer, and a 16k Tx ring buffer. serial I/O is handled by a separate thread, see the source code as an example. cheers, rob :-) Edited 2021-11-26 14:56 by robert.rozee |
||||
panky Guru Joined: 02/10/2012 Location: AustraliaPosts: 1094 |
Tom, This link, Linux Serial IO might be usefull. A bit beyond me but seems to address some of the low level serial io structures etc., also with some sample code at the end. Hope it helps, Doug. ... almost all of the Maximites, the MicromMites, the MM Extremes, the ArmMites, the PicoMite and loving it! |
||||
phil99 Guru Joined: 11/02/2018 Location: AustraliaPosts: 1781 |
If I understand correctly, what Tom needs is something sufficiently universal that it can be incorporated into MMB4L. Jim's solution of having MMB4L send <CR> after receiving the "Press any key" prompt does not require any specific hardware. |
||||
Mixtel90 Guru Joined: 05/10/2019 Location: United KingdomPosts: 5714 |
Remember that the number of USB sockets on a Pi is not the number of USB ports! They are not independent of each other and are all connected to a single USB connection on the SOC. They share the 480Mb/s between them. Until the model 4 the ethernet connection was on the same bus and shared the same bandwidth. Yes, you can plug many devices in simultaneously via hubs (I think the limit is 30), but they are not on separate "ports". Edited 2021-11-26 18:22 by Mixtel90 Mick Zilog Inside! nascom.info for Nascom & Gemini Preliminary MMBasic docs & my PCB designs |
||||
| robert.rozee Guru Joined: 31/12/2012 Location: New ZealandPosts: 2285 |
valid point, i was referring (rather loosely) to physical 4-pin sockets available once you've connected together enough hubs to fit your needs (5x 4-port hubs will give you 16 sockets to plug devices into). these sockets will then all end up sharing bandwidth on the single D+/D- pair of wires that connect back to the SoC. see: https://acroname.com/blog/how-many-usb-devices-can-i-connect but for most practical purposes, provided you've not got something attached that will suck up the full 480Mb/sec, there should be more than enough bits-per-second spare to support a full compliment (30?) of serial ports running at their top practical speed. cheers, rob :-) Edited 2021-11-26 21:56 by robert.rozee |
||||
| thwill Guru Joined: 16/09/2019 Location: United KingdomPosts: 3839 |
Hi folks, Thanks for all the useful comments. Upon reflection, and I haven't tested this yet, I'm pretty certain this is nothing to do with the performance or otherwise of the Pi's serial I/O. Instead the CMM2 (running LIST ALL which is a pure "C" implementation) is simply pumping data into the port faster than my MMBasic client program (which is in my maintainable though not necessarily efficient style) running under MMBL on the Pi can read it out. Buffering can mitigate, but at some point unless there is some form of flow control any buffer is going to overflow given a big enough file being LISTed. The "missing piece of the puzzle" is that I didn't know or expect that serial comms would simply drop data (without obviously reporting any errors) rather than blocking when the buffers were full, now I do  .Right, I suspected from past comments that you have made that I was propably re-inventing part of the MMEdit wheel. As part of the development of serial support in MMB4L and because I want it for my personal workflow I have been writing an MMBasic command-line program called "gonzo" that provides a "kermit" like interface to allow me to send and receive files between MMB4L and a CMM2 or PicoMite without having to work directly at the AUTOSAVE and XMODEM level. I'm aware of other options but I like Linux, command-line tools, and not having to worry about Python dependencies. Not quite  . I can automatically parse the data received from the CMM2 for that prompt and send a character (key press) in response. Alternatively I suppose I could suggest everyone gets one of those "drinking bird" toys as an accessory for MMB4L but that didn't go so well for Homer.A timely reminder to take a look at your source-code. The initial MMB4L implementation is pretty primitive, but hopefully functional, just enough to meet my own needs. If it turns out MMB4L gets some actual users then I can improve it based on feedback. Thanks for the link Doug, I've already read and digested several similar documents, e.g. Serial Programming Guide for POSIX Operating Systems and the Linux Serial HOWTO but it never hurts to read one more. It's not my plan to incorporate "gonzo" into MMB4L initially, but I might do so in time considering my personal use-case for MMB4L as a development platform for CMM2 and PicoMite. Thanks again folks, Tom Edited 2021-11-26 22:57 by thwill Game*Mite, CMM2 Welcome Tape, Creaky old text adventures |
||||
| robert.rozee Guru Joined: 31/12/2012 Location: New ZealandPosts: 2285 |
here is the (latest) linux serial code used: linux_serial.zip i've removed the (unused) non-threaded read and write routines, as they are likely to be just confusing. while it is in pascal, the names of functions called should match fairly well with C, and you can just ignore all the little counters, flags, calls to GetTickCount64, etc that are sprinkled throughout the code as these are just things to make GFXterm 'work more pretty'. hopefully there is a C equivalent to the lazarus threading library, as this eases life considerably. cheers, rob :-) |
||||
 Print this page |
