 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Electronics : piclog with SER-USB converters
Page 1 of 2   |
|||||
| Author | Message | ||||
GWatPE Senior Member  Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
The logging of data from a windmill using a piclog and modern laptop computer with only USB ports presents some problems. I had posted some problems I had seen in other threads, however I do not seem to be alone with the problems. Most seem to stem from the use of the SER-USB converters that are needed with a modern laptop. I will not venture into desktop PC loggers, even though these may still have a SER comm port, as probably more power will be consumed by the PC than the logged windmill would produce. I have tried many SER-USB converters with a piclog. There is nothing wrong with the picaxe hardware or the serial data transmitted. I have modified a test unit to only transmit a constant value for voltage, and a cyclic incrementing set of values for current, RPM and windspeed. I have found that some problems that appear to come up with the piclog program are zero values, and loss of digits. The program also gives the microsoft sorry for crashing screen, if I select saving to Excel, or HTML in addition to the normal .txt log file. To overcome the software crashing problem I only save to the .txt file. I have made a dedicated file converter to create a .csv file from any Piclog, logged .txt file. This allows the data to be filtered with the favourite spreadsheet to allow removal of corrupted data. I have been unable to successfully filter the corrupt data before logging. I think the only way is to include a checksum with the data stream. This would mean a simple SUM of the 4 variables and transmitting this value to the start or end of the data stream. The piclog program would then check the validity once received, and if bytes were lost, the data would be rejected. A byproduct of this will be a slight underestimate of the Watthours, depending on the % lost data. The Piclog program is a good tool, thanks to Gizmo. I have been out of programming school for a while, so I am still sorting out the VBsyntax. I am still trying to make DIRlistBoxes and FILElistBoxes work together to make selections easier. I know sparweb and others are using the piclog. Are these other readers having any issues with the program? Gordon. become more energy aware |
||||
Gizmo Admin Group  Joined: 05/06/2004 Location: AustraliaPosts: 5019 |
I wish I could dive in and fix these problems with the VB code, but at the moment all my computer time is taken up writting code and web sites just to pay the bills, its been several months since I spent a day away from the PC. I do wonder if the serial to USB converters are having problems with the serial signal levels from the Piclog. The Piclog doesn't generate true RS232 signal levels, but it seams to work well enough for a PC serial interface. Its possible ( probable ) the USB converters dont have a very tolerant UART chip. Maybe a RS232 line conditioning IC with the PicLog will work? From memory, I wrote the PicLog code to send the datastream from the PicAxe in a crude XML format, so each chunk of data has a leading and trailing tag. The VB code should be looking for this leading and trailing tag before it considers the data valid. It can get confused if the data is very currupt, where it tries to use a trailing tag from a completly separate datastream. I did look into USB programming with VB6, but its a bit of a nightmare. The Excel and HTML file output are just text files, but with a different extension so Excel / web browser will read them as a valid file. I'll give you a call in a couple of days Gordon, got relatives with me for the next day. Glenn The best time to plant a tree was twenty years ago, the second best time is right now. JAQ |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
I will spend some more time on it. I have spent some time revising the picaxe output and changing the piclog code to look for the new serial data format. The SER-USB converters I have are all 5V logic, so the logic levels match the picaxe. I have made a test piclog program with some additional display windows that show the received discrete data bytes. There seems to be no predictable pattern as to which data is corrupted. the XML format seems to work if there are not problems with comms, but the original VBcode makes the variable 0 if there is a transmission error. If the data is say for Amps with an Alegro sensor, this introduces a large negative value in the measured variable and subsequent calculated values of Watts and Whr. I have modified the picaxe output to just a single identifier character for each type of data, and an end identifier, as well as the brackets that Gizmo used at each end of the data string. The chance of corrupt data seems to be related to the number of bytes sent, so I removed the < > and / characters. I have included a checksum, that is the sum of the 4 data variables. This system is easily decoded, as the data has a standard unchanging format from the picaxe. As long as the checksum matches the sum of the 4 data variables, the data is not corrupt and can be displayed and logged if required. My revised picaxe code and VBcode is not compatible with the original piclog. My latest VBcode only updates the data values if valid data is recieved. The output window retains the last valid data. The log file records the previous recorded data if corrupt data occurs during the logging cycle. This system is working very well so far. I may add the code for creating the .csv file to the graphing window, as that way you can preview the days data, and if needed the file can then just be converted without opening the file again. A sort of try before you buy. There are so many options. It would be good to discuss with Gizmo before posting any code. If anyone have requests for mods, now would be a good time to voice them. Gordon. become more energy aware |
||||
| Janne Senior Member Joined: 20/06/2008 Location: FinlandPosts: 121 |
Hi, It might be a good idea to invest to the rev-ed's usb programming cable if you're having trouble transmitting serial data.. I've never had any serial communication problems with it. For data storage I've used Eltima software's RS232 data logger, which simply stores the received data in a text file. Checksums.. who needs those :P If at first you don't succeed, try again. My projects |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
There does not seem to be much interest in mods to the piclog. I will refine things as I get time. Will consult with Gizmo. Elimination of corrupt data produces a better log, so I will continue with data identifiers and checksums. I will also add the .csv converter function to the graphing window as the data is already available in the piclog at this time, so the data can be checked visually before a .csv file is made. I have not determined what exactly causes the piclog to crash when the Excel button is checked. Gordon. become more energy aware |
||||
vasi Guru  Joined: 23/03/2007 Location: RomaniaPosts: 1697 |
This is why I'm starting to look over AVR's. You need dedicated Microchip microcontrollers to be able to communicate directly via USB. Almost all AVR's can do it, without special hardware in a HID maner. http://www.obdev.at/products/vusb/index.html Because here is an AVR version of Oshonsoft Basic I think is not a problem for PICAXE users to use them. Hobbit name: Togo Toadfoot of Frogmorton Elvish name: Mablung Miriel Beyound Arduino Lang |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
The way I see it re USB, is that the user front end is more the problem. the 2 stage software interface. VisualBasic will require the USB drivers. I had played with velleman USB interfaces in the past. These had the USB libraries provided that worked with VBasic. There were no problems with the comms, but the ADC was only 8bit. There are many USB components out there. Each requires a specific driver. The serial comms use a std VB interface format. As long as the SER-USB converter has drivers, the implimenting into VBapplication is not a real problem. I do not intend scratch building a USB device, but the small changes needed to improve the SER data capture at the PC end are doable. I think that if vasi has the resources to make a general purpose USB interface that will work with a picaxe, then this will be similar to the multitude of interfaces already out there. There will still be the problem of the libraries needed, so a software application can get the data. I suspect I don't have the time to go this way. Gordon. BTW the .csv converter was easily added as a button to the graph window of the piclog, that is activated only after the data is displayed. The button is deactivated once the .csv file is written. Much simpler than a dedicated program. Allows making a .csv from old data files where the Excel checkbox was not checked. Allows the data to be previewed, so only interesting graphs can be made a .csv as needed. Gizmo will probably be releasing a piclog update, when time permits. become more energy aware |
||||
| vasi Guru Joined: 23/03/2007 Location: RomaniaPosts: 1697 |
Unfortunately, rev-ed don't use 18f pics with usb support (why not? timing problems?) and implementing a software driven one as in avr's is beyond me at this time. But can be used a raw pic 18f with usb support and Oshonsof Basic (no big differences between this basic version and picaxe basic). The programming can be done with a PicKit II programmer or a clone. I don't recommend other programmers anymore. My JDM programmer covered only few pics. And this page is good for starting (with some doc here). It require also to be familiarized with Visual Basic and HidTerm ActiveX control. Such a system can be an interface between a PICAXE system and PC USB connection but too much trouble and resources... is cheaper to make the switch to PIC. Hobbit name: Togo Toadfoot of Frogmorton Elvish name: Mablung Miriel Beyound Arduino Lang |
||||
| davef Guru Joined: 14/05/2006 Location: New ZealandPosts: 499 |
Have you looked at the LUFA USB library on www.avrfreaks.net? |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
The PIC approach may be doable with a pic18 chip. I suspect that much of the processor time will be involved with just the comm aspect. I like the approach of the micro performing the required task with priority, and then just sending a string of data to the PC to log. The PC would have much more free CPU time, would probably be doing nothing when the string arrived. It would be different if the PC was controlling the process, with the USB device interfacing to the real world. I would consider the programming effort justified if the PC was in control, but reliability dictates a dedicated micro if control is required. PC's tend to crash at inoportune times. There are certainly options with PIC. I will be waiting for progress reports from those readers persuing this. Maybe a prototype USB system with example interface code in say VBasic will change my mind and give up with old style SER comms. I have found that HID type USB devices are sometimes interpreted by Windows as mouse type devices. I have found that as data is captured , unexpected pointer activity occurs. This usually happens following a sleep, or powerup, with the USB still plugged. This means that for best operation that USB activity is controlled from the PC end. This requires the PIC to wait for a PC signal before data is sent. This may require interupt sequenced data transfers. The simplicity of SER comm with VBasic is such that once the MScomm1 is set up, port number, baud rate, etc, that the line of VB code to read the ser comm port is just "logdatastring=MScomm1.input" once the string is read, then the VB application code works out what is means, including checking the data is valid. The readers experienced in USB coding could give some comparisons of expected data input code, and port configuration code that VBasic may have. I have some more ideas for the piclog program, including an automated port selection system. At the moment, the prog.cfg file port parameter is required to match a valid SER comm port before the piclog can run. I need a year of 30hour days to get time for this. Gordon. become more energy aware |
||||
| vasi Guru Joined: 23/03/2007 Location: RomaniaPosts: 1697 |
Hi davef, You have a direct link? I must be member to see that? Hobbit name: Togo Toadfoot of Frogmorton Elvish name: Mablung Miriel Beyound Arduino Lang |
||||
| vasi Guru Joined: 23/03/2007 Location: RomaniaPosts: 1697 |
Hi Gordon, Here is about a project named Pinguino and is like Arduino project but using 18F2550 and 18F4550 chips. The programing language is the same as for AVR's with a re-translation for pics, of course. I'm watching it with interest. Also, here was started a discussion about an Arduino like board (Jaluino, which can be shielded by daughter boards), with an 18F4550 chip, and with a USB lib translated in JAL. USB will be used for comms (of course, you still can use serial if you want, depending on your application). And here was started a series of tutorials about an USB board. davef, I found LUFA, thank you. Hobbit name: Togo Toadfoot of Frogmorton Elvish name: Mablung Miriel Beyound Arduino Lang |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
Hi vasi, I think these projects above highlight the difficulties I am avoiding by sticking with SER comms. These USB projects are major works in themselves. The user interface like the piclog application still would need to be made. I think the USB is more than I could tackle at the moment. I will persevere with some more ideas like graph scroll buttons, and zoom buttons on the graph window. These would be like hourly increments that would still allow zooming with the mouse for closer investigation. I would also like to be able to zoom in on an already zoomed in section without having to manually adjust the times. I am trying to keep in the spirit of the piclog, rather than requiring all new hardware. I would also like to have backward compatability with previously recorded data. Future developments may still sway me towards a USB system. Gordon. become more energy aware |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
Spent a bit of time on the piclog last night. zooming in on a zoomed graph section is progressing. Have the .csv converter button working as well. Still trying to get the automatic port detecting to work. may need a fresh look and rewrite some of the VBcode. Zooming out is also on my wish list. Gordon. become more energy aware |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
I have spent some more time with the VBasic code and have worked out how to use the dir list boxes, and file list boxes, to get a point and click method of selecting files to view in the graphing window. I use a laptop for data collecting, but tend to use the modern computer for looking at the logged data and for more detailed analysis with a spreadsheet. I now have a piclog type VBasic application that can be used independently, on a computer that does not have any comms. This application has a point and click selection of the file to view, from any directory on the current drive. I may add a drive selection window as well yet. I also have buttons for rapid selection of 4x6hour windows, and a full day selection. I have allowed for unlimited zooming within a portion of the displayed graph, without having to manually key in the times. These mods are working on a modified data set, and not Gizmos original data. Once I have mastered all these changes, I will be incorporating these into a version that will work with any existing piclog data files. I now have a rapid way of looking at the graphed data and producing .csv files that are easily imported into a spreadsheet. I do not have to enter any new file names. The original date format, with extra time info is used as the filenames for the .csv files. This allows many views of the data. I will be adding a second option to allow rapid viewing of the .csv files as well. I will be running my mods past Gizmo. I am not sure if there will be a way of just having on application that will do everything. I may incorporate an opening window that allows acccess to graphing and .csv saving even if logging cannot be performed. I have not used the raw data list, so this area of the graphing window I have used for the point and click selections. I have a lot more ideas with the logging aspect, and this VBasic application with a graphing window has many more possibilities. I have a 9channel logging at 5sec intervals on the cards. Gordon. become more energy aware |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
I have compiled a demo version of my modified file viewer, that should read voltage and times from existing piclog files. This program also works on win98 and winXP and Vista32, and Vista64 machines. I have yet to add the data filters for the RPM, Windspeed, current and Watts. Here is a link for the file. 2009-07-23_133631_picview.zip The program looks for .txt files and only allows viewing of valid piclog data. The application has to be in the folder above the location of the log folder to work at the moment. Will look into a more generic version later. Gordon. PS I was able to mod the viewer to read all the fields in the piclog data file. Here is the application and some data. 2009-07-25_092730_picview2.zip the data should be placed in a folder called logs, as a subfolder within the folder where the application is placed. become more energy aware |
||||
| Gizmo Admin Group Joined: 05/06/2004 Location: AustraliaPosts: 5019 |
Hi Gordon I got an "Path not found" error when I tried to open a piclog txt file. The file I tried to open was an old one. Could you post a log file that you know works with your viewer? Do you have my email address, you could send the source so I can have a look to see whats happening. Glenn The best time to plant a tree was twenty years ago, the second best time is right now. JAQ |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
Hi Glenn, The piclog data viewer application needs to be located in the same folder as the piclog application, or if on another computer, the piclog data files need to be located in a sub folder folder called "logs" that is in the same directory as the piclog data viewer application is installed in. Gordon. PS: The logs folder would be double clicked and then the file to view would be selected, by clicking on it in the lower file display window, before clicking the display button. become more energy aware |
||||
| Gizmo Admin Group Joined: 05/06/2004 Location: AustraliaPosts: 5019 |
Thanks Gordon, its working a treat now. Glenn The best time to plant a tree was twenty years ago, the second best time is right now. JAQ |
||||
| GWatPE Senior Member Joined: 01/09/2006 Location: AustraliaPosts: 2127 |
Hi Glenn, I did some mods to the code, and the program now has a default filename, test.txt [blank] that the program can do something with, without crashing. As long as there is a logs subfolder the data to be viewed can be placed. Any .csv files created will be placed in the logs subfolder as well. The program defaults to look in the logs folder at startup. There is an offset problem with the windspeed, still to fix. Gordon. become more energy aware |
||||
| Page 1 of 2 |
|||||
 Print this page |
