Home
JAQForum Ver 20.06
Log In or Join  
Active Topics
Local Time 18:11 28 Apr 2024 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Web Interrupts and WatchDogs

Author Message
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 06:17am 05 Mar 2024
Copy link to clipboard 
Print this post

G'day all  
I've had a niggling problem for a while when using a WebMite connected to my local WiFi network (and, WWW in general). I think I've fixed the problem now, but I'm trying to get my wee-brain around what the cause has been... Someone more in the know with Web traffic than me could have the answer.
All versions of my code have been written with a simple WebInterrupt Sub based on Geoff's Retic.bas code
In general, everything seemed to work ok, but I was getting a lot of Watchdog timeouts, plus even more of the:
 "Internal error in tcp_server_recv - Attempting recovery" errors...
which was mostly recoverable...

Process of elimination, I pinned the faults down to this sort of traffic coming down the network (with the gibberish ASCII characters that made no sense to me)
(This is just a sample of the web traffic I've logged, there's many more similar)



The ONLY thing that made sense to me is that the 'Mite receives web traffic, processes it (in this case, would send out a 404 code), but the originator of the request is no longer on line (A bot maybe?) and so a jamb-up occurs and fires the watchdog.

Not being great with web 'hand-shaking', I'm only guessing what it could be...

As a solution, I rewrote the Interrupt code to this (Sorry, lots of comments) -

WEB TCP INTERRUPT WebInterrupt 'Catch the web traffic and deal with it
settick 1000, ShowTicks, 1 'Do something every 1 second

do 'Main code loop here
 timer = 0 'Reset the timer for cycle times
 Watchdog 20000 'WatchDog time-out value
loop 'Keep looping

sub WebInterrupt 'Web data has arrived
 local integer ConnNum 'Var for connection number
 local integer ConnData(512) 'Var for web data
 ? timer 'Display the timer val for time stamp
 for ConnNum = 1 to mm.info(max connections) 'Loop through all the connections
   longstring clear ConnData() 'Clear any previous data
   web tcp read ConnNum, ConnData() 'Get the web traffic
   if llen(ConnData()) > 0 then 'If the data has anything in it, do this
     ? "Web Interrupt ---------" 'Display the page break
     ?
     if linstr(ConnData(),MM.Info(ip address)) > 0 then 'Is the WebMite address included in the data?
       ? "Format OK" 'Display that things are good
       LongString print ConnData() 'Display the data that came in
       web transmit code ConnNum, 404 'Move along, there's nothing to see here
     Else 'No WebMite address?
       ? "*** Error in Format ***" 'Display that we're not interested
       LongString print ConnData() 'Display the data that came in
       web tcp close ConnNum 'Close the connection to avoid a possible WatchDog error
     end if
     ? "-----------------------" 'Page break
   end if
 next ConnNum 'Do the next connection
 ? timer 'Display the timer val for time stamp
 ?
end Sub

Now, ONLY web traffic with the WebMite address is accepted and processed - anything else closes the connection.
So far, so good... It's been running for 36 hours without any hiccups, even with the massive amounts of dribble that comes through and gets processed.

First question: is this a normal amount and normal string format that comes through a WWW connection?
Second: Is my coding bad in that I filter the web request that only have the 'Mite IP Address in it and take action on that?

John
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5905
Posted: 06:50am 05 Mar 2024
Copy link to clipboard 
Print this post

My guess is it's a broadcast packet of some kind.
I know there a plenty of them on any network but I have not tried to work out what type is causing the problem.

I have had similar issues here.

Your cure looks like a good one and I will look forward to a report in a few days.

Jim
VK7JH
MMedit   MMBasic Help
 
Ossi
Newbie

Joined: 22/06/2023
Location: Germany
Posts: 6
Posted: 02:29pm 05 Mar 2024
Copy link to clipboard 
Print this post

Hello John,
in a "normal home environment" behind a router without open ports (e.g. FritzBox) there should be no such wild telegrams. Basically an html request should consist of readable ASCII characters. I have tried whether the WEB SERVER reacts to a ping. The ping response from the WEB Mite comes back, but my WEB SERVER interrupt gets nothing from it. Try using something like Wireshark to determine the IP address of the request.
Show your OPTION LIST. What does your WEB CONNECT look like.
Good luck with the further analysis.

Ossi
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 8582
Posted: 07:52am 06 Mar 2024
Copy link to clipboard 
Print this post

I've implemented this fix into Geoff's watering controller code and so far at 1 full day with no glitches  
If this proves to be the issue all along I'll see if I can integrate the fix into the firmware
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 05:07am 07 Mar 2024
Copy link to clipboard 
Print this post

G'day guys, thanks for the feedback on my curiosity...

Up until this point, it has been running for around 4 days with no unexplained timeouts. There's been massive amounts of ASCII gibberish come through, and all of it has been rejected without firing the WD timer.  

Of course, it's a little limiting in usability being only able to accept traffic directly targeting the Mite, so I twiddled the code to this:


sub WebInterrupt 'Web data has arrived
 local integer ConnNum 'Var for connection number
 local integer ConnData(512) 'Var for web data
 local String Host1, Host2 'Vars for hosts
 Host1 = MM.Info(ip address) 'The local IP Address
 Host2 = "210.50." 'The primary numbers for my ISP address
 for ConnNum = 1 to mm.info(max connections) 'Loop through all the connections
   longstring clear ConnData() 'Clear any previous data
   web tcp read ConnNum, ConnData() 'Get the web traffic
   if llen(ConnData()) > 0 then 'If the data has anything in it, do this
     ? "Web Interrupt -", Time$, date$ 'Display the page break, so we know web traffic has occured
     ? "Start time stamp:", timer 'Display the timer val for time stamp
     if linstr(ConnData(), "Host:") > 0 then 'Is the traffic from a legitimate host?
       ? "Host string found." 'Yes, we've found a 'host: ' string, all good so far...
       if (linstr(ConnData(), Host1) > 0) or (linstr(ConnData(),Host2) > 0) then 'Address of local WebMite, or ISP?
         ? "Valid address found." 'We have a match!
         ?
         LongString print ConnData() 'Display the data that came in
         if linstr(ConnData(), "GET /test.html HTTP") then 'Is the pointless web page requested?
           ? "Sending test.html"
           WEB TRANSMIT PAGE ConnNum, "/test.html" 'Send the pointless page
         else if linstr(ConnData(), "GET /lizard.jpg") then 'Picture requested?
           ? "Sending lizard.jpg"
           web transmit file ConnNum, "lizard.jpg", "image/jpeg" 'Yeah, send it too...
         else
           ? "Sending 404 Code"
           web transmit code ConnNum, 404 'So far, so bad... move along, there's nothing to see here
         end if
       else
         ? "Error in Address." 'The address is a dud, deal with it from here
         ?
         LongString print ConnData() 'Display the data that came in
         ? "Closing connection"
         web tcp close ConnNum 'Close the connection to avoid a possible WatchDog error
       end if
     Else 'No WebMite address?
       ? "*** Error in Format ***" 'Display that we're not interested
       LongString print ConnData() 'Display the data that came in
       ? "Closing connection"
       web tcp close ConnNum 'Close the connection to avoid a possible WatchDog error
     end if
     
   end if
 next ConnNum 'Do the next connection
 ? "End time stamp:", timer 'Display the timer val for time stamp
 ? "-----------------------" 'Page break
 ?
end Sub


I also added a useless web page to test the functionality on that side of it.
The new code now accepts ONLY the correct Mite address, and/or anything that comes from my ISP.

So far, seems to be working good, but I'll get into work next week and try to access the web page from the CNC computer and give it a solid test.
I'll keep you posted  

(Apologies for the cluttering-mess of my comments... it helps keep my train of thought on track!)

John
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 05:13am 07 Mar 2024
Copy link to clipboard 
Print this post

Oh, and just for a giggle, this is my favorite gibberish traffic I've logged...



Really?
What does that even mean!!?  

John
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5905
Posted: 06:31am 07 Mar 2024
Copy link to clipboard 
Print this post

The only traffic that should be able to get to you from the WWW are packets that have been port forwarded.

Broadcasts packets shouldn't be forwarded. That is why I suspect that they might be locally generated.

What ports have you forwarded to the webmite?

If you can save the suss packets to file instead of just printing them, it would be easier for me (and others) to examine them and possibly work out the source.
It is not easy to retype the screendumps as images.

Jim
VK7JH
MMedit   MMBasic Help
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 07:33pm 07 Mar 2024
Copy link to clipboard 
Print this post

G'day Jim,
It's been a cut/paste from the MMCC screen to Programmers Notepad for a week or so, but I had all the data saved to a local file.

Attached file is any relevant info for this, including the basic source plus the web-page and graphics - plus of course the text file of the errors (Edited with a lot of the good HTML requests taken out for a smaller file size)


Web Test 2.zip

The last entry in the error file was from 3:00 this morning... the first untrapped fault in more than a few days of running

John
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5905
Posted: 09:41pm 07 Mar 2024
Copy link to clipboard 
Print this post

Thanks for that.
I was wrong, the packets are not local but coming from the WWW and are probably some sort of port scan hitting your port 80 (or whichever external port you have forwarded).
This is normal and one of the curses we have to deal with.
My code closely follows Peters original example and looks for "GET /" rather than "HOST:" in the data.
Either would work as a basic filter.

I can see some repetition in the packets but nothing I recognise.

I will set up a system to capture external hits on my system but it might be a few days.

Jim
VK7JH
MMedit   MMBasic Help
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 12:23am 08 Mar 2024
Copy link to clipboard 
Print this post

Thanks Jim, appreciate you taking a look and glad to hear I'm not quite going crazy yet...
It seems like a lot of 'noise' for something that's normal, but as I say, I've no real idea on WWW protocols, so there were a few guesses on my part.

There's probably a few filter words that could be used ("GET /", as you say) but I selected "HOST" because I also Tx/Rx from a VB program for changing code parameters, so I wanted to keep GET, POST (etc, etc) as reserved words and not muddy the waters with HTML code within VB.
It was quick and easy to append the TxData string
Private Sub Winsock1_Connect()
 Dim tmp As Variant
 tmp = "Host: " + NodeData(NodeNum).Address + vbCrLf
 Winsock1.SendData tmp + TxData
End Sub

In the VB code through WinSock

It would be interesting to see what you can come up during your tests; I look forward to your results  

John
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 10:39pm 16 Mar 2024
Copy link to clipboard 
Print this post

'Morning all  

I've been looking a bit more into my problem because every now and then I still get a watchdog timeout after a web hit (There seems to be no pattern to the timing, nor patterns in the web hit 'language', even seemingly good web requests)
I've tried to monitor with WireShark, but as a first time user of it, I can't seem to set it up properly.

My basic networking is:
 Local IP range is 192.168.0.XXX
 Router is 192.168.0.1
 WebMite is 192.168.0.114 (Forwarded from the router)
 My desktop PC is 192.168.0.107
 My public IP is 210.50.XX.XX

So, WireShark picks up all traffic from my local PC no matter if I enter WebMite address, or access via the public IP, and I get the test webpage back as expected.
If I try to get the webpage from my phone, yes, the webpage loads onto the phone, but there's no capture in WireShark.

Setting a display filter to HTTP, or tcp/udp:80 port, it's always my desktop PC address that is listed, so I can't find what traffic is happening with the WebMite unless it comes from my desktop PC... Very frustrating!

Any WireShark experts have a clue where I'm going wrong?

John
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5905
Posted: 01:43am 17 Mar 2024
Copy link to clipboard 
Print this post

Apart from broadcast traffic, packets go to the destination device and other devices (your PC) can't see them.

Your router 'might' be able to capture data which is either saved to the router memory for later retrieval or forwarded to another device such as your PC.
Some have the ability but hide it from casual users.

One other alternative is (yet to be tested).
If your PC is wired rather than wireless, set up the PC as a mobile hotspot and connect the WEBmite to the hotspot.
That way the traffic to and from the WEBmite is going via your Ethernet port which WireShark can capture.

I also have a managed switch which can echo traffic on on port to another port. If I am desperate that is the way I go but it needs an access point for the WEBmite so the managed switch is between the router and the access point.

I have been monitoring the WEBmite with your sub, hoping for an interesting packet to appear but no luck so far. My Internet must be cleaner than yours!

Jim
Edited 2024-03-17 11:45 by TassyJim
VK7JH
MMedit   MMBasic Help
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 04:53am 17 Mar 2024
Copy link to clipboard 
Print this post

Thanks Jim, much appreciated  

  Quote  Your router 'might' be able to capture data which is either saved to the router memory for later retrieval or forwarded to another device such as your PC.
Some have the ability but hide it from casual users.


I looked at that a few weeks ago, but didn't really find anything useful there. A bit limited in knowledge on networking, I could only see that network traffic was happening, but couldn't get any real info from it.

  Quote  If your PC is wired rather than wireless, set up the PC as a mobile hotspot and connect the WEBmite to the hotspot.
That way the traffic to and from the WEBmite is going via your Ethernet port which WireShark can capture.


A good suggestion that I have running now, so I'll see what comes of that.
Unfortunately, it's given a completely new IP range with the server IP as 192.168.137.1 so it's off the usual network range (I'll have a play and see what I can figure out on that part of it)
I'll run it as-is for a few days and see what comes of it  

  Quote  I have been monitoring the WEBmite with your sub, hoping for an interesting packet to appear but no luck so far. My Internet must be cleaner than yours!


Hmmmm... that's a bit frustrating for my side of it. You might be right and I might just have a crappy ISP, so I'm chasing my tail here.
(Then again, maybe it's some state-sponsored hacking agency trying to get all the top-secret files from my computer! [KGB, XYZ, CIA, FBI, ABC, PRC... they're all after me!] )

John
 
TassyJim

Guru

Joined: 07/08/2011
Location: Australia
Posts: 5905
Posted: 06:28am 17 Mar 2024
Copy link to clipboard 
Print this post

first, you forward from the Internet to your PC port 80
then you forward that to the WEBmite with another port forward, this time on your PC

You will have to configure port forwarding to the hotspot.
You will also have to open the firewall to allow port 80 in.
This is a good step by step:
https://redfishiaven.medium.com/port-forwarding-in-windows-and-ways-to-set-it-up-c337e171086f

The only thing that needs changing is the line that does the forwarding
listenport=80 listenaddress=yourPCaddress
connectport=80 connectaddress=192.168.137.xxx (the WEBmite)

Networking is fun!

Jim

Edit. This will let you capture any odd connects but the WEBmite will still have hissy fits occasionally and you do need the hardware watchdog set up to restart.
Edited 2024-03-17 16:30 by TassyJim
VK7JH
MMedit   MMBasic Help
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 04:51am 18 Mar 2024
Copy link to clipboard 
Print this post

  Quote  first, you forward from the Internet to your PC port 80
then you forward that to the WEBmite with another port forward, this time on your PC


Thanks Jim, I think I've done it correct... I'm getting to log the web hits that show in MMCC with WireShark, but I'll leave it run for a few days and see what comes out of it.
It was a little bit of head-scratching on my part, but here's fingers crossed!

  Quote  Networking is fun!

As they say in the classics... 'No Comment'  

Ok, I'll admit - I'm learning!
Can't ask for much more than that I guess (My comms experience goes back to 422 & 485 many (many!) moons ago, then only a few moons ago with ASi Bus)

I'll keep you posted on the results  

John
 
Malibu
Senior Member

Joined: 07/07/2018
Location: Australia
Posts: 149
Posted: 11:49pm 30 Mar 2024
Copy link to clipboard 
Print this post

Morning all  
Here's the latest of my investigations for my problem (It's been a while but I've been a bit bust lately)

Here's the only part of the WireShark logs that I can guarantee is the 'fault', because I was watching the WS screen scrolling at the exact time - so I know I got a good capture.
This is a screen grab from an Excel file of the log, because I could highlight various parts of it to (hopefully) make a little more sense to me...



Down to line 10, it was a good conversation.
On line 11, a SYN was sent, but not answered - so it seems the handshaking was never completed (SYN : SYN,ACK : ACK), with 4 more tried attempts before I manually removed power and restarted.

Here's the console print to MMCC at that moment


Interestingly, the WMite watchdog didn't fire and the heartbeat continued to flash. For some reason, that seems to be the usual case running from a Hot Spot (that's what I've noticed, anyway)

I'm still at a loss to why it's happening - it 'seems' the WebMite is missing the interrupt and I thought this morning that maybe the 1-second timer tick is being served when a web interrupt arrives. Is that possible?

John
 
Print this page


To reply to this topic, you need to log in.

© JAQ Software 2024