 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : ESP8266/Micromite/sending a log
Page 1 of 2   |
|||||
| Author | Message | ||||
skyline_stu Newbie  Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Hi, I'm looking for some assistance using a Micromite MkI and a ESP8266 wifi module as a web accessed data logger. Here's a quick run down of what I'm trying to achieve. I need to monitor some inputs and log these with a time/date stamp and some other status data. I've managed mostly to get the hardware/software sorted out - I'm using an external DS3232 RTCC chip[I've considered using the spare SRAM in the RTC for logging, BTW], ATM I'm logging my data to a serial eeprom and I want to use a ESP8266 WiFi module to allow the remote downloading of my logged data via my local network. I can view and clear from the serial console just fine. There's a heck of a lot of other things that the unit does, but that's not important here. I have the Wifi server presenting the required info as a web page on my local network, but I don't know how to integrate into the web servers' html code a feature to allow 'remote' downloading/clearing of the stored data from the EEPROM. I'd like the data as a CSV type file. Here's where I'm stuck. I don't know how to: 1. Generate a 'file' of the logged data, so that it can be downloaded. I.E. The filename is the date of logged data. It can use the date from the 1st stored date. It won't be downloaded every day. 2. The required HTML code to generate a selection box for 'Download/Clear Log'. 3. The required HTML code to allow a clearing/downloading of the logged data. Quite possibly, the WiFi Module may not support the 'passing back' of the selected option to the Micromite for downloading/clearing My serial EEPROM is pretty big - 64Kbit (it was lying around) Any help or suggestions are appreciated. I've considered upgrading to a MkII and fumbling my way through a Cfunction to generate the log file if that will help. Stu Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
| akashh Senior Member  Joined: 19/01/2014 Location: IndiaPosts: 115 |
Although I have never worked with that particular model, I can help you with the logic. You can use the headers of the page to specify parameters such as the file type (text/csv) and specify that the file should be downloaded rather than viewed. By reading the GET header from the serial stream, you can identify a file name and parameters, it would be something like GET /index.htm?param1=val1¶m2=val2 HTTP/1.0
You can the either parse the file name to keep it simple, I.e a page called download.htm to download the csv and another called clear.htm to trigger a clear of the eeprom, or you could parse the parameters and have something like index.htm?action=download, etc. I hope that helps. Akash |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Cheers Akashh, I will give it a try. I still need to generate the CSV file though, I do wonder if I can somehow do that in the spare flash or ram and then have that file exported. I think I will realy need a dedicated Cfunction though, to actually 'generate' the file. Think I'm going to be busy learning and cursing today... I'm not so fluent at C or MIPS assembler. Yet. Might need to dig through the maximite source code and find Geoff's routines for some help.. Another idea is for someone more skilled than I to build a CFunction to use a MMC Card as the micromite doesn't currently support this. Anyone interested ?? Mind you, I've only got the Mk I's ATM, so I've ordered a couple of the MX170's to upgrade to MkII and will need to get the hot air station fired up. Stu Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Another thought is to temporarily 'decommission' my color maximite and use that.. It's already got the file support built in. 
Decisions, decisions.. And after a think and a read of the 'books', possibly not. Comport speeds on the CMM are limited - i need 2 ports at high speed for the ESP and the bluetooth module  ... ARGHHHH. Might need to build a duinomite.
Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
Oldbitcollector Senior Member Joined: 16/05/2014 Location: United StatesPosts: 172 |
How about a direct write to USB flash drive solution? I've been playing with those USB HOST chips from Hobby Electronics and the Micromite this week. They've got a very nice little module that supports the USB flashdrive (as well as those USB to MicroSD adapters). My Propeller/Micromite mini-computer project. |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Cheers, That board looks very interesting. I've got some PIC24FJ256GB110's in stock that support USB OTG.. Wonder if I can hack up some sample code to work as needed... More things to try. Yawn. It's been a long day or 2.. Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
TassyJim Guru  Joined: 07/08/2011 Location: AustraliaPosts: 6266 |
I don't see how a Cfunction will help much. You can read the data from your eeprom byte at a time and add a comma and <CR> as required. All this can be done 'on the fly' and the resulting data sent to the serial port without having to save it first. The speed of the comms side of things will be slower than the speed of retrieving data so the web browser at the other end will not know the difference. Jim VK7JH MMedit |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Hi Jim, Thanks for the input. My suggestion for the Cfunction is to generate a physical file which can be downloaded, archived and also viewed remotely at a later date. Rather than having a screen full of data that may be overlooked, this system needs a redundancy method to ensure someone else has a physical record of events in another place. In other words, the logging system's data is, at a later date, correlated to other logged information. My reasoning is that if I have a link on the web server to download a file and optionally clear the logs, this ensures some accountability. I currently have the web server showing the logged events on screen as needed. How do you suggest that I 'record' the data at the other end over WiFi? I'm open to any suggestion ! Stu Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
Grogster Admin Group  Joined: 31/12/2012 Location: New ZealandPosts: 9586 |
How about OpenLog? EDIT: Re-reading your post, I see you want to be able to dowload from a webpage kind of thing. OpenLog may still be useful as the logging device, but not sure if this is a good idea now that webpages have come into the mix. Smoke makes things work. When the smoke gets out, it stops! |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Hi Grogster, That OpenLog device looks great, but I've already implemented that type of system within the main micromite server's code. Unfortunatly, the micromite is using both the serial ports. One for the Wifi module, the other for a bluetooth master communicating to another micromite [with a BT slave module] which waits for specific commands to arrive and it performs other functions based on the sent commands. Controlling both serial devices eats up the two com ports. That not an issue though, I could multiplex the com ports easily. So, in reailty, I don't need more than one com port. I believe there are some issues with modified ESP8266 firmware which allows slower baud rates, so I'm steering away from using my color Maximite[which had a MMC/SD card interface] and native file support. Trouble is that the micromite doesn't have a file generation routine to read the raw data from the logger's EEPROM, create a file such as 05-01-15a.csv, store it on a SD card or whatever [spare EEPROM space], which is then passed into the EPS8266 web server and accessed from a remote terminal. The filename is based on the 1st date extracted from the 1st stored entry in the datalog. In Geoff's example garden server, he's using the line <a href="http://geoffg.net/Downloads/ServerSrc.zip">here< /a> He's obviously grabbing that file, ServerSrc.zip from his local network. In this case, the logger IS the local network. Hence, I need a Cfunction to generate the actual file [05-01-15a.csv] on the 'logger/server' micromite, which is then passed through to the wifi module and onto the remote machine when requested. Most of all, the system is operated over long distances. Field testing has shown the ESP8266 wifi's operate over enormous distances with a really good antenna. I'm sure I could go to another brand board, but I don't want to re-invent the wheel, learn a new programming language and most of all, I love my maxi & micromites ! Stu Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9586 |
Understood, and I agree with that logic. Smoke makes things work. When the smoke gets out, it stops! |
||||
| TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6266 |
Stu In the example you gave from Geoff's website <a href="http://geoffg.net/Downloads/ServerSrc.zip">here< /a> simple send a request for the file 'ServerSrc.zip' to the web server. All you have to do is process the incoming request and return the requested file. The name of the file could be anything. The person sitting at the computer can right click and save with any name he/she wants to. You could re-create the main page each day with a different default name for the file. The file can be an existing one or it can be one you create when the request is received. You do not need to use an existing file. Take a look at this web page: http://www.c-com.com.au/VK7RTV.htm The main pages and the images are on my website. the lower frame that has the realtime data is on a device similar to a micromite, located in my shed. The page with the data is created on the fly when requested. This way I have limited the amount of work the micro needs to do while still having the real time data available. Nowhere do I create a file first. In this case the file is formatted as HTML but it could be plain text or CSV etc. Jim VK7JH MMedit |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Thanks for replying again Jim :) Please tell me if I'm wrong here. Local is the Micromite, Remote is the end user on a PC, say 500m away. A. What your saying, if i understand it, is that the user, on the remote web browser hits the 'button' to download the logged data. B. The local micromite passes a request to my program, saying "hey, he hit the button to download the log data" using the returned '/GET' parameter as mentioned by Akashh. C. Then the local Micromite sends back a 'packet' saying, the file you want is called XYZ123.CSV D. The remote computer acknowledges this packet and generates a temp file buffer called XYZ123.CSV.tmp E. The local micromite simply streams data packets (??), like ROM #0 - comma - ROM #1 - comma - ROM #2 - comma - ROM#3 - <CR> - ROM #4 - ..... ....ROM#7 - <CR> - ROM #42............................................ROM45 - <CR> - <EOF> F. Remote machine saves streamed data after the EOF marker as the earlier identified file name. G. Stu is happy file is sent and goes home for beer. Which I really need after this afternoons MOST EXCELLENT experience of my mothers car almost catching fire whilst I was driving her around.. Electrical faults... Gotta love em!.. Had some time to think about this while stiting in a doctors office. As they say, couldn't see the forest for the trees. Like I said in 1st message, new to this HTML and ESP8266 gizzmo. Give me HARDWARE to design and let me programming in ASSEMBLER. No problems. [well, less anyway IMO] **** terminolgy may not be 100% right, I've tried to dumb it down so we're both on the same plane [umm, my plane !!]. I assume that the data packets still contain ASCII control characters. What got me muddled up is I'm used to the idea that there's a lot of handshaking and exchanging of data before a PHYSICAL file is transfered, rather than a simple data stream. What happens if I need to send a EOF character as part of my file, or am I asking for trouble suggesting this. I had assumed the remote computer required the overall filesize, name, rank, serial number BEFORE transferring data packets. A rather tired and toasted Stu - no pun intended. Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
MicroBlocks Guru Joined: 12/05/2012 Location: ThailandPosts: 2209 |
You first need to present an HTML page to the user with a link. [code] <!DOCTYPE html> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>uMite Data logger</title> <style> A { color: black; font-size:1em; } <style> </head> <body> <a href="/LOG">Get current log</a> </body> </html> [/code] If the user clicks the link the uMite will receive something like: [code] GET /LOG HTTP/1.1 Host: localhost [/code] Browsers will send a lot of other 'headers' that can be safely ignored. Each line ends with a <CR><LF> combination and the headers end with an empty line that has only the <CR><LF> combination and no spaces or other whitespace or characters. You need to get the first line as that is what you need to work with. The first line always has three parts. You can get those by splitting the line on the spaces. The first part is the verb: GET The second is the resource: /LOG The third is protocol and version: HTTP/1.1 You can now start to build a response You need to tell the browser what type of content you are going to send and with which character set. [code] Content-Type: text/CSV;charset=UTF-8 [/code] If you want the browser to present the user with a 'Save as' dialog you can add: [code] Content-Disposition: attachment;filename=myfilename.csv [/code] Then the browser needs to know how long the response will be. You will need to calculate the total amount of bytes that you are going to send next. This might be the trickiest bit. You might need to do a fake build of the file to get the length. [code] Content-Length:1200 [/code] The you send an empty line followed by the CSV formatted text. Edit: Here is a description of all the possible headers: http://en.wikipedia.org/wiki/List_of_HTTP_header_fields Microblocks. Build with logic. |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Thanks TZAdvantage,  Your explanation tells me everything I need to know to get this logger all finished. Greatly appreciate all the help.
And a big thanks to everyone who has provided some info! I also found this site, this person deserves a mention as his tutorials are top notch. http://www.allcreativedesigns.com.au/pages/tutorials.html Stu Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
| matherp Guru Joined: 11/12/2012 Location: United KingdomPosts: 10195 |
Stu It would be great if you could post some example code of the file transfer once working Thanks Peter |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Yup, I will do that. I'm getting there slowly. Stu Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
| skyline_stu Newbie Joined: 04/01/2015 Location: AustraliaPosts: 12 |
Hi everyone, New chips have arrived, been fitted and code conversion to MKII is being done. Will post some code up soon. Stu Fast, Cheap, Reliable. Pick any two. AND The satisfaction of a cheap job is long forgotten whilst the bitterness of poor quality remains ! |
||||
centrex Guru Joined: 13/11/2011 Location: AustraliaPosts: 320 |
Perhaps someone can help me with this... I have a ESP8266 which I can talk to and get the correct replys using TerraTerm. The 8266 runs at 9600baud but I had to tell TerraTerm that I wanted carrige return and line feed sent and added to the return. I get all the correct responses including my WiFi access point. But I cannot get any response using a uMite and MMEdit , using the small terminal program by GeoffG in the SC mag. The little blue led on the 8266 board flashes so it appears to be getting something. By the way it wanted 9600 out of the box. Any ideas. Regards Cliff Cliff |
||||
| TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6266 |
Do you mean the VT100 terminal used as a USB to TTL bridge? If so, MMEdit is unable to talk to the device. I don't know why and I have spent a lot of time trying to work it out. If you need to use CRLF as a line termination, in MMEdit, there isn't an option to change the linendings but you could set up a macro button. Jim VK7JH MMedit |
||||
| Page 1 of 2 |
|||||
 Print this page |
 |
The Back Shed's forum code is written, and hosted, in Australia. | © JAQ Software 2025 |
