Home
JAQForum Ver 20.06
Log In or Join  
Active Topics
Local Time 13:33 20 Apr 2024 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Failed attempt at a FTP server.

     Page 2 of 3    
Author Message
lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3010
Posted: 10:48pm 22 Feb 2021
Copy link to clipboard 
Print this post

I changed the buffer size to 55000 and baud rate to 921600 (open "com2:921600,55000" as #1), put 255 characters ("ABC"...) in a string variable, and wrote it 200 times--51000 characters. This took .596 seconds, a rate of 85K per second.

The receiving F4 received all characters, checksum was equal.

If the CMM2 is equivalent, and I suspect that it is, it would indicate that if the buffer size is adequate and the signal clean (which is a different issue than receiving capability), very high rates of transfer are possible.
PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8567
Posted: 10:49pm 22 Feb 2021
Copy link to clipboard 
Print this post

  Quote  First of all, you really need two pins (RTS and CTS) to allow for communications with slow devices (like light pens, tablets, and anything else that can't handle a high baud rate).  So, the only way to have the proper hardware flow control is by using 4 lines (Tx/Rx/RTS/CTS) of the same UART.  Secondly, can you tell me where in the datasheet that shows (two pins RTS & CTS) that can be muxed from one UART to another.  I looked again and the datasheet is very clear that each UART's RTS/CTS must be associated with the same UARTs Tx/Rx.  I have asked my contact at ST for clarification on this.


Try reading the datasheet (page 68, page 89), try using CubeMX and allocate the pins and let it generate the relevant initialisation code.
The other pin is the already allocated RS485 DE pin when in RTS/CTS mode.

There really is no point in me responding any more as you obviously know best. It is a miracle that the CMM2 works at all given my obvious lack of knowledge
 
frnno967
Senior Member

Joined: 02/10/2020
Location: United States
Posts: 104
Posted: 03:37am 23 Feb 2021
Copy link to clipboard 
Print this post

Respectfully, I wish these conversations wouldn't devolve into this negative back and forth. I asked Jim to come comment on this thread only because of his experience with the ESP's firmware and how the stock firmware has problems with full-duplex communications that hamper data transfers. Since that was the original topic of discussion, might we return to focusing on that instead of fighting the same flow control battle from before?

I'm sure both of you are making technically correct points, but arguing doesn't help any of us make progress on solving problems. Peter, you have obviously created a platform that works and is powerful and fun. And Jim has extensive proven knowledge of many embedded architectures and systems and has solved many issues with the ESP firmware. Can't there be some middle ground of mutual respect?
Jay Crutti: Ham Radio Operator, K5JCJ. Computer Enthusiast. Musician. Engineer.
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9052
Posted: 06:09am 23 Feb 2021
Copy link to clipboard 
Print this post

Well put, frnno967.  

On a personal note, I have never needed to use flow-control.
I'm NOT saying it is not a necessary thing in certain circumstances, but I have been using plain old UART serial on the Maximite and Micromite series since the days of the B/W Maximite and even before then, and I have never ever had an issue where I HAD to have CTS/RTS to fix any issue or to make it work.

I certainly acknowledge the need for those lines in very fast UART speeds(1Mb+ etc), but for anything up to 115k2, I never needed any kind of flow control - ever.  It always worked, and there were never any issues with bits or bytes missing provided everything was setup correctly at both ends of the link.

A simple checksum is normally about as far as I ever go.  If the checksum is not equal, then the receiving end can request the transmitter resend the packet, and you don't need flow-control for that, just good background buffering - which MMBASIC is stellar at.

My 2c only.  The point being - as far as I am concerned - flow control is not a requirement in MOST chip-to-chip applications where the UART speed is very modest.

Look at I2C or SPI - they don't have flow-control lines at all, and they work fine....
Smoke makes things work. When the smoke gets out, it stops!
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8567
Posted: 08:01am 23 Feb 2021
Copy link to clipboard 
Print this post

  Quote  Can't there be some middle ground of mutual respect?


When he extends the courtesy of accepting that others may know more than him about a platform and learns how to RTFM before pontificating perhaps.

I provided a roadmap as to how to implement RTS/CTS in a way that would allow all CMM2 users to benefit without H/W changes if he wishes to contribute to the codebase - let's see if there is more than talk.
Edited 2021-02-23 18:09 by matherp
 
JimDrew
Newbie

Joined: 07/10/2020
Location: United States
Posts: 39
Posted: 11:06am 23 Feb 2021
Copy link to clipboard 
Print this post

  Turbo46 said  imDrew,

Can you please explain why software control of RTS/CTS is not acceptable to you?

Bill


Well, because it doesn't work very well (or at all), especially at higher baud rates - and it is *extremely* slow.

I don't get it - there is a 480MHz ARM with more UARTS than you can ever hope to use, all with hardware level flow control and DMA capability (making everything seamless with basically no CPU intervention required) and yet people want to use a software flow control method?  Kind of a waste of hardware, don't you think?  What I proposed would lead to absolutely no compatibility issues due to adding an extra COM port.  I can't imagine some kind of future hardware revision that is going to take up all of the UARTS pins.  Set aside the 4 pins that the all right next to each other (and could be routed on a new board design in literally 1 minute) and call it a day. That future proofs anything you can possibly throw at it, and transfers can then be in the megabaud range and at full speed or 75 baud, without any errors.
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8567
Posted: 01:52pm 23 Feb 2021
Copy link to clipboard 
Print this post

  Quote  What I proposed would lead to absolutely no compatibility issues due to adding an extra COM port.


Please stop banging on about unused com ports - they are not going to be implemented. The compatibility issue is that over 2,500 existing users wouldn't be able to use the functionality whereas my proposal lets all users have access. I take it you are not going to roll your sleeves up so it won't happen any way so the discussion is moot.
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 3649
Posted: 02:51pm 23 Feb 2021
Copy link to clipboard 
Print this post

  JimDrew said  Set aside the 4 pins that the all right next to each other (and could be routed on a new board design in literally 1 minute) and call it a day. That future proofs anything you can possibly throw at it, and transfers can then be in the megabaud range and at full speed or 75 baud, without any errors.

How does that help people like me (one of the 2500)?

I've used flow control in the past but I haven't found a need for flow control on a CMM2 (yet) but if I do I'm NOT going to buy a new board.

If flow control can be done via software (whether via CubeMX or along the lines of the original discussion), great.

Otherwise no use to me.

John
 
lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3010
Posted: 06:35pm 23 Feb 2021
Copy link to clipboard 
Print this post

Is this the ultimate test of a pure CMM2 serial receive capability? Armmite F4 sending, CMM2 receiving on COM2.

Baud rate: 1843200; buffer: 1048576 (open "com2:1843200,1048576" as #1) (buffer of 55,000 on the F4); 1,048,560 characters sent (255*4112); calculated sending checksum=80085312

Sending with (255-byte character string a$ built as before--"ABC"...):
t%=timer: for i=1 to 4112: print #1,a$;: next i: ?(timer-t%)/1000;" cksum=";cksum%*4112
1,048,560 characters sent in 6.177 seconds.

Received with:
j=0:i=0:do while loc(1)>0: a$=input$(1,1): j=j+asc(a$):i=i+1: loop: ? int(i);" ck=";int(j)

All characters received, checksum matches: 1048560 ck= 80085312

So on a clean connection (20cm dupont cable), with nothing else happening on the CMM2, a flawless transmission rate of 169,752 characters per second is achieved for over a million characters sent.

So this would seem to indicate that for a CMM2, problems relating to receiving serial transmissions at nearly 2Mbps are not related to the ability of the CMM2 to handle that bit rate (provided the buffer receiving the transmission is large enough).

This is not to say that flow control is not needed by the CMM2 or other hardware under certain conditions, but it does indicate that those conditions are dictated by the ability of the sender/receiver or buffer size or other CMM2 activity, and it remains unclear that software controlled CTS/RTS is insufficient to maintain a satisfactory high rate of transmission.
PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed
 
frnno967
Senior Member

Joined: 02/10/2020
Location: United States
Posts: 104
Posted: 08:20pm 23 Feb 2021
Copy link to clipboard 
Print this post

I don't believe there's any issue with the CMM2's abilities to transfer data, at least at reasonable speeds ~115200 or slightly beyond. The issue is the ESP firmware, since many of us are using it as the connection method to the larger world. Despite the noise in this thread, I believe TassyJim in a different thread has overcome the ESP issues by using a flow control signal with his ESP module. If we can find a way to "standardize" this and also apply it to ESP8266/ESP-01 modules then I think things will be largely resolved. It seems the only real problem is the ESP's ability to keep up with data CMM2-->ESP.
Jay Crutti: Ham Radio Operator, K5JCJ. Computer Enthusiast. Musician. Engineer.
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5883
Posted: 08:54pm 23 Feb 2021
Copy link to clipboard 
Print this post

Arguing about hardware flow control is like trying to get a Trump supporter to vote for Biden (or the other way around, I am not trying to be political here).

After 40 years of strictly hardware flow control, I understand why Jim can't look at software solutions.
That is a pity because with software, you are not restricted by the hardware limits.

If the ESP firmware developers desired, they could make the firmware much more flow control friendly. When not in passthrough mode, there is a controlled flow of data in both directions so provided the receive buffers are large enough, traditional flow control is never needed. The same (controlled flow) goes for XMODEM etc.

With FTP, two streams are required so passthrough is not an option anyway.

The data rates I have achieved even with my third rate amateur programming is not too bad.

Any limits are with my programming skills, not MMBasic.
You don't NEED hardware flow control.

Jim
VK7JH
MMedit   MMBasic Help
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 5705
Posted: 09:08pm 23 Feb 2021
Copy link to clipboard 
Print this post

Isn't there a clash here between what the CMM2 is designed to do and what some people are trying to make it do? The computers that it's designed to "feel like" wouldn't know what to do with 115200 baud, never mind anything faster. It's already streets ahead of them. Yes it's *nice* to get higher serial speeds, but it's not actually within the CMM2's remit. It was inspired by the old 8-bit machines, with specialised hardware in those being replaced by high-speed software. If it was having problems with 1200/75 baud I'd be concerned. :)
Mick

Zilog Inside! nascom.info for Nascom & Gemini
Preliminary MMBasic docs & my PCB designs
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9052
Posted: 11:24pm 23 Feb 2021
Copy link to clipboard 
Print this post

  Mixtel90 said  Isn't there a clash here between what the CMM2 is designed to do and what some people are trying to make it do? The computers that it's designed to "feel like" wouldn't know what to do with 115200 baud, never mind anything faster.


Precisely.

Back in the 80's when I was a teenager and had an Atari 800XL, it's SIO(Serial I/O) port ran at 38k4 I am pretty sure it was.  Same speed as the standard MM console, BTW.  

From what I remember, the SIO port DID have CTS/RTS, but probably only cos the hardware of the time needed it to keep up forty years ago, as the hardware buffers would have been very small and very easy to overflow.  With processors being so fast now with tons of buffer memory, they can keep up with the serial without needing any kind of flow-control.

Anyway, I don't want to comment anymore about this, so I will shut up now.  
Smoke makes things work. When the smoke gets out, it stops!
 
frnno967
Senior Member

Joined: 02/10/2020
Location: United States
Posts: 104
Posted: 11:29pm 23 Feb 2021
Copy link to clipboard 
Print this post

  Mixtel90 said  If it was having problems with 1200/75 baud I'd be concerned. :)


That's actually the rub for me. I find 57600 or 115200 to be PLENTY fast for all the CMM2 use cases I have in mind, BBS/Terminals and file transfers with Maxiterm. I tried doing Xmodem transfers using the firmware routines at all baud rates (1200 on up) using the COM 1 & 2 ports over an ESP and it never worked reliably, either due to escaping or buffering/data loss issues. It's gotten so bad that a collaborator has been working with me these last few months writing an Xmodem routine in MMBASIC so that we would program/work around the problems with the ESP firmware. There are some firmwares out there that either are easy to use (Zimodem)  but don't work correctly with file transfers at any speed, and other firmware that works somewhat (esp8266_modem)  but drives you crazy with the command syntax or other usability problems. We're just trying to find a solution like an ESP firmware that is the best of both worlds, both speed and accuracy. If that solution is found then everyone's lives will be easier.
Jay Crutti: Ham Radio Operator, K5JCJ. Computer Enthusiast. Musician. Engineer.
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 3649
Posted: 11:31pm 23 Feb 2021
Copy link to clipboard 
Print this post

I'm happy to have flow control if needed, whether using a uart in the chip for the control lines or software toggling/checking those lines.

Just not happy if I would need a new board (or the kind of soldering I can't do).

I haven't so far needed flow control, but it could happen.

John
 
JimDrew
Newbie

Joined: 07/10/2020
Location: United States
Posts: 39
Posted: 01:09am 24 Feb 2021
Copy link to clipboard 
Print this post

  matherp said  Try reading the datasheet (page 68, page 89), try using CubeMX and allocate the pins and let it generate the relevant initialisation code.
The other pin is the already allocated RS485 DE pin when in RTS/CTS mode.


Yes, of course I looked at that info which is why I asked you the question (and sent it to ST as well) about why CubeMX will not reassign (mux) RTS or CTS from a UART to other pins.  It seems according to the datasheet (look at the fan pinout) that the UART pin functions are hard wired for each UART.

If you guys don't care about fast speed for transfers or slowing down the BASIC interpreter trying to process packets manually with software flow control, then you don't need to change anything.  I was under the assumption that you wanted this to work in all cases with both retro and modern hardware at full speed.  The only way to make that possible is by using the hardware flow control that is already there, just not currently being used.  I don't care what you guys do in regards to flow control, but I was asked about a solution to the existing issues and that is why I am here (again).
 
lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3010
Posted: 01:38am 24 Feb 2021
Copy link to clipboard 
Print this post

  JimDrew said  If you guys don't care about fast speed for transfers or slowing down the BASIC interpreter trying to process packets manually with software flow control, then you don't need to change anything.


Are we talking about only wifi transfers? Because if wired serial, 169kBps is possible, and you can create a more-than 5 megabyte buffer on the CMM2 (open "com2:1843200,5000000" as #1) so unless you have really huge files, no handshaking on receive is needed.

If your file is larger than that, you can do software RTS/CTS, receive, say, a megabyte, de-assert CTS, write what you received, re-assert CTS, and repeat for as many megabytes as needed. If it is slow, it is only as slow as the CMM2 is in writing to the SD card.

If wifi is the issue, the biggest problem seems to be with the ESP-01 or ESP32 or whatever is used to make the wifi connection. If there are bottlenecks, I suspect they will be there rather than on the CMM2 side.

If you have firmware that improves performance on the ESP-01 or ESP32, I'd be happy to test it.
PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8567
Posted: 08:03am 24 Feb 2021
Copy link to clipboard 
Print this post

  Quote  Yes, of course I looked at that info which is why I asked you the question (and sent it to ST as well) about why CubeMX will not reassign (mux) RTS or CTS from a UART to other pins.  


It does. It defaults to a particular pin. Just block that pin by pre-allocating another function and then CubeMX will pick another valid pin for the the function. Alternatively just use the dropdown on the pin to pre-select before defining the UART. Like all tools it needs learning.
Are you getting the pin numbers confused. Pin16 is the pin number on the I/O header. This is PIN-40 on the chip PA0
Edited 2021-02-24 18:12 by matherp
 
lizby
Guru

Joined: 17/05/2016
Location: United States
Posts: 3010
Posted: 01:14am 25 Feb 2021
Copy link to clipboard 
Print this post

Supposing you want to use a wired connection to send a file from a PC to the CMM2 which is too big to fit in a CMM2 serial buffer--say 10 megabytes--so you need flow control.

You don't have hardware RTS/CTS, you don't want to set up software-controlled RTS/CTS . . .  well, there's always XON/XOFF. I found a nifty utility, plink.exe which is installed along with puTTY. It can make a serial connection for you and send a file with something like this:

"\Program Files\putty\plink" -serial "com5" -sercfg "1843200,8,n,1,X" < \dl2\ftp\tst.txt

tst.txt is 10,567,840 bytes long and its checksum is 801798880.

I wrote a program for the CMM2 to request the file and send XON, send XOFF when there were more than a million bytes in the serial buffer, write the buffer and send XON, and repeat until done.


Option explicit
Dim string xon=Chr$(17),xoff=Chr$(19),ack=Chr$(6),nak=Chr$(21),a$,b$,c$,d$,fn$
Dim integer i%,j%,k%,l%,m%,n%,cksum%,oldloc%,baud%,nChar%
dim float t!,ttx!,tmr!

baud%=1843200 ' 115200 230400 460800 1843200
Open "com1:"+Str$(baud)+",2000000" As #1
fn$="tst.txt"
print "Trying to get "+fn$
open "\"+fn$ for output as #2
print #1,fn$+xon;
oldloc%=0
cksum%=0
nChar%=0
timer=0
do while loc(#1) = 0 and timer < 10000: loop
if loc(#1) = 0 then
 print "File Not Received: "+fn+"--exiting"
 print #1,chr$(27); ' EOT
 pause 1000
 end
endif
timer=0
t!=0
ttx!=0
tmr!=timer
do
 l%=loc(#1)
 if l%=0 then ' transmission done
   if cksum%>0 then ' something received
     print #1,str$(nChar%)+","+str$(cksum%)+xoff
     print " Bytes, checksum: "+str$(nChar%)+","+str$(cksum%)+xoff
     exit do
   else
     pause 2000 ' allow more time for start-up
     if loc(#1) = 0 then ' nothing received
       print "File Not Received--exiting"
'        exit do
       print #1,xon;
     endif
   endif
 elseif l%>1000000 or oldloc%=l% then ' this block is done; xoff and write to SD card
   print #1,xoff;
   ttx!=ttx!+(timer-tmr!):
   print "*";
   do while loc(#1) > 0 ' clear all characters from buffer
     b$=input$(255,#1)
     j%=Len(b$)
     For i%=1 To j%
       c$=Mid$(b$,i%,1)
       cksum%=cksum%+Asc(c$)
       nChar%=nChar%+1
     Next i%
     print #2,b$;
   loop
   print #1,xon;
   tmr!=timer
 endif
 oldloc%=l%
 pause 1000 ' wait a second
loop
t!=timer
print #1,chr$(27); ' EOT
on error skip 1
close #1
on error skip 1
close #2
print nChar%;" Characters; ";t!/1000;" seconds; ";ttx!/1000;" seconds transmitting; checksum: ";cksum%
print int(nChar%/(t!/1000));" bytes per second total; ";int(nChar%/(ttx!/1000));" bytes per second (to transmit)"


Program output:
Trying to get tst.txt
********** Bytes, checksum: 10567840,801798880
10567840 Characters;  348.832611 seconds;  70.995907 seconds transmitting; checksum:  801798880
30294 bytes per second total;  148851 bytes per second (to transmit)

So byte total and checksum match. This is over 30cm of Dupont cable--Rx,Tx,0V to a usb/serial module on the PC.

Total time to run, 349 seconds, of which only about 20% was occupied by the transfer--the remaining 80% was writing to the SD card (which is nothing special). Total transfer rate is 30K per second; actual tranmission rate, 148K per second.

Of course, the quicker thing to do would be to remove the SD card, put it in the PC, and copy the file. And I'm not sure why you would want a 10MB file on the CMM2.

But that's a benchmark for transmission of a large file to the CMM2, using flow control with no additional CMM2 hardware or firmware.

I understand that XON/XOFF is not going to be available on every device which might send serial to the CMM2, but if a transmission is likely to exceed 5 megabytes, it will probably be on a device which can support it.
PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed
 
Turbo46

Guru

Joined: 24/12/2017
Location: Australia
Posts: 1591
Posted: 06:22am 25 Feb 2021
Copy link to clipboard 
Print this post

That's a neat solution. With a companion program to write a file to the PC written so that both could run in MMBasic for DOS and the CMM2 it could be the basis for transferring files to and from the CMM2. Say to transfer graphics files et al and backup the SD card without having to remove it.

Bill
Keep safe. Live long and prosper.
 
     Page 2 of 3    
Print this page
© JAQ Software 2024