 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : Failed attempt at a FTP server.
Page 1 of 3   |
|||||
| Author | Message | ||||
TassyJim Guru  Joined: 07/08/2011 Location: AustraliaPosts: 6283 |
I have been trying to implement a FTP server for the CMM2 with limited success. Using the ESP32 at 115200 baud, everything works well but that's not fast enough for larger files. At higher baud-rates, I can send to the CMM2 OK but sending from the CMM2 has problems. Even with large time delays to make sure that I am not overrunning any buffers, In a 300k file there will be 2 or 3 dropped bytes. The bytes get dropped from varying locations in the middle of a packet. Then some extra bytes get added to the end to make up the size. These extra bytes are AT+ which is the start of a previous command. This is two examples of errors: The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick browox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The AT+quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps overhe lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. ATThe quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy dog. The ESP32 with the latest official AT firmware is only safe to use at 115200 baud. The only error checking is data length so these errors could go unnoticed. Not good. Next I tried the ESP8266 with the 1.7.4 firmware. Here the main problem is trying to receive data. With FTP, a second channel is opened for the data transfer. As soon as the data transfer is finished, the channel gets closed (by the sender which I have not control over). This is how you know that all data has been sent. On the ESP32, the remaining data is safely in a buffer and can be retrieved but on the ESP8266, as soon as the channel gets closed, I loose access to the remaining bytes in the buffer. Another problem. Normally, the ESP will send +IPD,1,1460 to indicate it has 1460 bytes available and I respond with AT+CIPRECVDATA=1,1460 to receive the data. Not with the ESP8266 Here I see: +IPD,1,1460 +IPD,1,2048 SEND OK AT+CIPRECVDATA=1,1460 +CIPRECVDATA,1460: Note the two +IPD lines! later I see: OK +IPD,1,2332 +IPD,1,2920 AT+CIPRECVDATA=1,2332 ERROR I have seen 3 +IPD lines at once. Sometimes, using the first data size in the request works, other times nothing seems to work. That makes them useless for FTP work. I will try with the ZiModem firmware and see if it plays nicely. The ZiModem has a bug in the NTP timezone routines. I have advised the author and hope that an official fix appears soon. It was easy to change. Jim VK7JH MMedit |
||||
jirsoft Guru Joined: 18/09/2020 Location: Czech RepublicPosts: 533 |
This was the reason, why I'm developing own FW. My NC over ESP8266 connected to Python UDP server is transferring ca 50kB/s  But I have again problems with TCP connection  Didn't you tried I2C or SPI? Maybe it will be faster and easier than serial... Jiri Napoleon Commander and SimplEd for CMM2 (GitHub), CMM2.fun |
||||
| jirsoft Guru Joined: 18/09/2020 Location: Czech RepublicPosts: 533 |
This was the reason, why I'm developing own FW. My NC over ESP8266 connected to Python UDP server is transferring ca 50kB/s But I have again problems with TCP connection Didn't you tried I2C or SPI? Maybe it will be faster and easier than serial... Jiri Napoleon Commander and SimplEd for CMM2 (GitHub), CMM2.fun |
||||
| mkopack73 Senior Member  Joined: 03/07/2020 Location: United StatesPosts: 261 |
So I see 2 potential points of issue/failure: There's the serial communications between the CMM2 and the Modem, and then the TCPIP communications between the Modem and the other device you're talking to. Is hardware flow control / handshaking an option for the serial comms? Based on what you described it sounds a lot like software flow control not keeping up properly. |
||||
| TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6283 |
I will try handshaking on the ESP32. It is definitely a strong contender. I did try very short data length with long delays between without any success. On the ESP8266, the real show stopper is it's loosing access to the final packet on any transfer that has more than one TCPIP packet. No amount of flow control will fix that. I tend to agree with Jiri. It needs better firmware. I really would like to be able to use the common ESP-01 module, so no I2C. Not sure about handshaking lines. Jim VK7JH MMedit |
||||
| frnno967 Senior Member Joined: 02/10/2020 Location: United StatesPosts: 104 |
Jim Drew has a lot of insight into this issue and developed workarounds in his Wimodem firmware. I'll send him a message and ask him to contribute some thoughts as well. But this problem has also been affecting our Maxiterm work around Xmodem transfers. Jay Crutti: Ham Radio Operator, K5JCJ. Computer Enthusiast. Musician. Engineer. |
||||
| JimDrew Newbie  Joined: 07/10/2020 Location: United StatesPosts: 39 |
... as you might recall, last year I posted (debated) in a thread here about implementing hardware flow control for the CMM2. Everyone was trying to tell me I was nuts and that it was not necessary. The reality is that it's not optional for high data rates. The ARM used with CMM2 has numerous spare (currently unused) UARTS with hardware flow control all built in. It's a simple matter to implement this - just 4 lines - Rx/Tx/RTS/CTS, and an interrupt handler. I have the source code to the CMM2 so I know it's not that difficult to do. I was met with some resistance so at that point I no longer paid attention to the CMM2 development. A few weeks ago I finally decided to get a RetroMax (CMM2 clone) to tinker with and I ported my WiModem firmware to the smaller pin count ESP-01S module. I let Jay know about it, and he sent me some emails about the issues he was having with X-modem transfers with what he was using and so here we are... One big issue with using ESP8266 or ESP32 modules is that the ESP libraries do not properly handle WiFi for applications like this. I spent about 2 months (literally) writing a new WiFi library for the ESP modules that allows fully asynchronous communications. The stock WiFi library is only half-duplex. It uses a timer to wait on no more data being sent to/from the modem. After a period of time without any new data being sent the data is then sent over-air. During any over-air transmission (send or receive) serial communications are halted. If you just so happen to be sending serial data to the module in the middle of an over-air transmission, any amount of data the exceeds the hardware FIFO is lost. Likewise, if the code on the module side sends serial data out during an over-air transmission that data stream will be interrupted (delayed) until the over-air is complete. This means there is a long pause in the data stream. The problem really shows up when you are expecting data within a certain time frame (like a modem transfer protocol) where timing may not fall within the protocol requirement. These protocols use a series of short, rapid, ACK/NACK packets to synchronize the data frames. Keep in mind that these protocols were developed for use with land-lines. The round trip time on a hard wired line across the globe is typically less than 25ms. Some protocols (like Z-modem) are very strict on their timing requirements. For Z-modem there are internal parameters that can be adjusted to accommodate some delay, but that is typically a 8 bit millisecond timer (0-255ms). If you do a simple wireless ping test you are going to find that the ping time varies radically depending on your router, ISP, and where your packet request is going. Having >25ms of ping time is almost always the case with outbound connections through the ISP. The further away, the longer the delay (typically). I see ping times from <1ms (locally) to hundreds of milliseconds across the U.S. My WiModem firmware allows fully asynchronous communications so that over-air data coming in or going out can occur while serial communications are also happening. Everything is handled with multiple circular buffers and DMA, and RTS/CTS is also supported. I was recently working on a new WiModem update so on a whim I bought a RetroMax and re-worked my WiModem firmware for it. Since there are only two GPIOs available on the ESP-01S module I have made one DCD (carrier detect) and the other RTS (connected to CTS on the CMM2). This would allow the CMM2 to tell the WiModem to stop sending data. I could make the DCD be CTS (RTS on the CMM2 side) instead if necessary, but since the EPS8266 can handle 3.4 megabaud input reliably I don't think that this is needed - but it's an easy option to change should it be required. My firmware can update itself from my server, so its a 20 second proposition to update firmware. I will be selling a WiModemCMM2 shortly. So... there are two issues that you are likely running into. I already know for 100% sure from testing that you can outrun the CMM2's serial input with large blocks at high speed and lose characters as a result. Flow control is necessary to prevent this. The other major issue has to do with the stock Espressif libraries not allowing asynchronous communications which results in serial data being lost when short packets with short delays are required. Edited 2021-02-22 10:32 by JimDrew |
||||
| lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3378 |
Interesting. Can you provide a test case? PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
NPHighview Senior Member Joined: 02/09/2020 Location: United StatesPosts: 203 |
If it would be of any interest, I know the guy who wrote ZModem. He's deeply interested in asynchronous communication, and has some time on his hands. Let me know via private message if you'd like to be put in touch with him. - Steve ("NPHighview") Johnson Live in the Future. It's Just Starting Now! |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 10310 |
Give it is so easy then please implement it as an option for COM1. You can use pin 16 as the extra line (COM2-TX). When you have it working I will include the code in the next beta. Things to do: Create a change to the OPEN command to select RTS/CTS use for COM1 as an option WITHOUT affecting other use of the OPEN command. Include an interlock to block COM2 use if the RTS/CTS option is chosen. Change the CLOSE command to tidy up and release the pin Implement whatever changes are needed to the interrupt routine etc. Test the new functionality and regression test existing functionality |
||||
| JimDrew Newbie Joined: 07/10/2020 Location: United StatesPosts: 39 |
@matherp, I would implement it as COM4 and add it to your existing command set. You would need to use a currently unused UART (as I described last year)... there is a UART that has Rx/Tx/RTS/CTS conveniently grouped together and super easy to route. You have to use all of the lines (Rx/Tx/RTS/CTS) associated with the exact same UART in order to have hardware flow control work, per the ST datasheet. You just can't arbitrarily take some pin and use it - at least I don't see any way in the datasheet to mux the RTS/CTS from another UART to make this work. It seems that all UART pins are hard wired, and that is pretty common really. This is why I was going to re-spin the board to add the extra dedicated COM port as a separate port connector, much like the RetroMax so it was not tied at all to the existing header. @lizby, A simple test you can do is stream a giant file from the ESP to COM1. I *think* the stock ESP firmware (the one with the funky AT commands) has the ability to stream from a URL directly. I can do that with my WiModem using HTTPGET. I am sure you can use your PC to do this as well, you just need a terminal program and a RS232<>TTL adapter cable that supports 115.2Kbps and higher. |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 10310 |
The pin I suggested is available for COM1 to use in the way you require, I do know what I'm doing! Doing this way would allow all the existing CMM2 users to wire to the existing 40-pin header. The other pins are already allocated in a future development so can't be used. If you build a board using those pins it will be incompatible with the firmware - be warned |
||||
| jirsoft Guru Joined: 18/09/2020 Location: Czech RepublicPosts: 533 |
I have the same experience with either cable serial from Mac or ESP in CMM2 on serial port. That's the reason I'm using handshaking for every 250 bytes comming from serial into CMM2, the other way is OK for Mac on cable, but for ESP is similar problem... I'm using the speed 691200 bps... Jiri Napoleon Commander and SimplEd for CMM2 (GitHub), CMM2.fun |
||||
| lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3378 |
Again interesting. I just tried 230400 on a couple of F4s: I don't have my CMM2 handy, but did this with 2 Armmite F4s (slower than CMM2) with COM2 Tx on the first connected to COM2 Rx on the second. On both: open "com2:230400,25000" as #1 On the first, create a 240-character string, "ABC...": a$="":cksum%=0:ia=asc("A")-1:for i=1 to 240:k%=(i mod 26)+ia: a$=a$+chr$(k%): cksum%=cksum%+k%: next i: ?a$: ? "block cksum=";cksum% next: a$="":cksum%=0:ia=asc("A")-1:for i=1 to 240:k%=(i mod 26)+ia: a$=a$+chr$(k%): cksum%=cksum%+k%: next i: ?a$: ? "block cksum=";cksum% This yields "1.095 cksum= 1830600"--24,000 characters sent in 1.095 milliseconds with a checksum of 1830600 (100 times the string checksum). On the second j=0:i=0:do while loc(1)>0: a$=input$(1,1): j=j+asc(a$):i=i+1: loop: ? int(i);" ck=";int(j) yields "24000 ck= 1830600" -- 24,000 characters received, checksum 1830600 (the same) If you consider that 24,000 characters is about 240,000 bits, divided by the time to send, 1.095 seconds, you get a bit rate of 219,178 with no error. That's not far off of the theoretical 230400 baud rate. Just tried again at 460800: number of bytes and checksum agreed; time .548 milliseconds, which works out to 410949 bit rate. Just tried again at 691200: number of bytes and checksum agreed; time .364 milliseconds, which works out to 659341 bit rate. Edited 2021-02-23 03:07 by lizby PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
| TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6283 |
I have succumbed to using handshaking on the ESP32. It requires one user selectable pin for the RTS. The CMM2 has no trouble keeping up with 460k baud incoming but the poor ESP needs a little help. I tried at 921k baud but file transfer speed was no different at ~23k bytes/sec. With the handshaking, transfer speed from the CMM2 is 13k bytes/sec. Not as fast as I would like but still usable. The ESP sends packets up to 5.5k long and with a sufficient size serial buffer on the CMM2, there are no obvious problems. I have now tested my FTP server with Total Commander on Windows, Total Commander on Android and gFTP on Linux. The ESP8266 with the standard AT firmware is fine for retrieving longish json data etc (I have one regular 18k file), but too many issues for FTP. Jim VK7JH MMedit |
||||
| JimDrew Newbie Joined: 07/10/2020 Location: United StatesPosts: 39 |
First of all, you really need two pins (RTS and CTS) to allow for communications with slow devices (like light pens, tablets, and anything else that can't handle a high baud rate). So, the only way to have the proper hardware flow control is by using 4 lines (Tx/Rx/RTS/CTS) of the same UART. Secondly, can you tell me where in the datasheet that shows (two pins RTS & CTS) that can be muxed from one UART to another. I looked again and the datasheet is very clear that each UART's RTS/CTS must be associated with the same UARTs Tx/Rx. I have asked my contact at ST for clarification on this. |
||||
| JimDrew Newbie Joined: 07/10/2020 Location: United StatesPosts: 39 |
That is different hardware, but that would be the ideal setup because both ports would be in sync with the CPU's clock and baud rate. Not a real-world test case for sure. |
||||
| JimDrew Newbie Joined: 07/10/2020 Location: United StatesPosts: 39 |
That is odd. I see just the opposite, even with a PC I can outrun the CMM2's serial. In your case, you need RTS on the CMM2 side to monitor the ESP. I need CTS on the ESP side to monitor the CMM2. Clearly two pins (RTS/CTS) are needed (as expected) to handle hardware flow control properly for all cases. I have many 9600 baud peripherals that can't actually handle a 9600 baud data rate, even though the bit rate is that. This is precisely why hardware flow control was invented. At 230400 baud I am seeing about 21KB per second transfers. I need to test some higher rates. The ESP chips will handle over 4mbps baud rates. Edited 2021-02-23 07:41 by JimDrew |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
So long as existing hardware works (without major hacks, preferably no minor ones and obviously this rules out a board "re-spin"), having hardware flow control sounds OK. However, I gather that cannot be done. Is that the case or not? It seems the previous debate covered how to achieve good enough I/O, if needs be using one or more pins for hardware handshaking controlled as needed by software on the CMM2. Clearly not ideal but within the above constraints. I'm all ears as to how the above can't cope with slow devices (like light pens, tablets, and ...). Anyone can, of course, design a board variant for their own use but that's no use to the rest of us. John |
||||
Turbo46 Guru Joined: 24/12/2017 Location: AustraliaPosts: 1642 |
JimDrew, Can you please explain why software control of RTS/CTS is not acceptable to you? Bill Keep safe. Live long and prosper. |
||||
| Page 1 of 3 |
|||||
 Print this page |
 |
The Back Shed's forum code is written, and hosted, in Australia. | © JAQ Software 2025 |
