Home
JAQForum Ver 24.01
Log In or Join  
Active Topics
Local Time 19:12 05 Jul 2025 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Dual SDcard logger on wifi network

Author Message
MikeO
Senior Member

Joined: 11/09/2011
Location: Australia
Posts: 275
Posted: 09:36pm 06 May 2016
Copy link to clipboard 
Print this post

Here is my latest project, very promising results so far. I have started using ESP8266 and my wifi network for collecting data from my home sensors, weather, greenhouse etc.
I was wanting to log all my sensor data but preferably in one place and with the ability to recover selected data without stopping/interrupting the data logging.
This device is a dual SDcard logger which stores all the sensor data as it arrives via the wifi network, it writes the data to daily files but to both SDcards. This allows you to take remove one the cards and access it in a computer to take off files etc, then when you re-insert into the logger it updates the card with missing data.
I also plan to allow the attached network sensors to save there configuration data and read back as required.

I am using a CG microboardII (Micromite Plus), MMbasic 5.1 , Esp8266 with ESPbasic ver 20, network data is via UDP network protocol. The top SDcard is removable and the 2nd one sits underneath. The SDcards are selected via the 74HC157 multiplexer.



Codenquilts
 
centrex

Guru

Joined: 13/11/2011
Location: Australia
Posts: 320
Posted: 11:33am 07 May 2016
Copy link to clipboard 
Print this post

Very interesting Mike how about a little more detail.
There seems to be be a very large spread of the cost of these devices on the net can you advise where you purchased yours from.

ThanksEdited by centrex 2016-05-08
Cliff
 
MikeO
Senior Member

Joined: 11/09/2011
Location: Australia
Posts: 275
Posted: 01:36pm 07 May 2016
Copy link to clipboard 
Print this post

Hi Centrex, sorry in my enthusiasm to make a post it probably wasn't clear, this is not a commercial product it is home designed using a Micromite plus and ESP8266.

I have only just got the prototype working and haven't done any documentation as yet , was just wondering if there was any interest , ideas etc.

I can will post schematic and code later.
Codenquilts
 
centrex

Guru

Joined: 13/11/2011
Location: Australia
Posts: 320
Posted: 02:59pm 07 May 2016
Copy link to clipboard 
Print this post

Thanks Mike
The device I was referring to is the ESP8266 seems to range in price from $2 upwards.

In an earlier post you mentioned using esp basic ver 20 a little more info on loading this to the ESP8266 would be appreciated.Edited by centrex 2016-05-09
Cliff
 
MikeO
Senior Member

Joined: 11/09/2011
Location: Australia
Posts: 275
Posted: 04:42pm 07 May 2016
Copy link to clipboard 
Print this post

@ Centrex, Ok regards the ESP8266 , there is a Site and forum for this device here.
There is a very active forum and ESP basic has advanced in leaps just recently.

The device is very easy to program and all the flashing information and documentation is available on the site. The devices are available from china of course but you can get them locally for a few dollars see this thread. .

I wrote some software to run on the ESP8266 to interface with the micromite for data collection see here also my site .

The SDcard logger also uses this software to connect and save data from the network devices.

Cheers Mike.
Codenquilts
 
lew247

Guru

Joined: 23/12/2015
Location: United Kingdom
Posts: 1702
Posted: 12:00am 12 May 2016
Copy link to clipboard 
Print this post

This seems a brilliant idea, I'm surprised its not incorporated in a lot of projects already.
It looks a thing of beauty also
 
MikeO
Senior Member

Joined: 11/09/2011
Location: Australia
Posts: 275
Posted: 11:38pm 16 May 2016
Copy link to clipboard 
Print this post




A little while ago I said I would post more information on this project including schematic and code, this has now been completed and hope it may be of interest.

Just to recap this is a dual SD card data logger which can also save and read back named files for storing, say, configuration files from network connected devices. The dual SD cards work a “little like” a mirroring file server in that one of the cards can be removed (for viewing, etc, of existing files) whilst the remaining card will keep logging data, then the removed card may be returned and it will be updated with the missing data.

I had developed the project to log data from several sensors systems I have , weather, solar, irrigation, also I now use ESP8266 devices via my wifi network but it could just as well use other serial RF modules instead or indeed just as part of a stand alone system.

As it stands the ESP8266 are running ESPBasic which has the provision for UDP network transmission, so commands and data are sent between the devices on the network using this protocol, to keep it simple a multicast mode is used, then each of the devices decide if they will use the broadcasted data by a the use of a device or node name which is included in the data string, below are typical data strings, the UDP portion of the string is removed in the ESP8266.

Example data strings
Data string from weather sensor (including UDP info)

To base station, from weather, weather data fields...............
ESP_Base:,ESP_Weather,275,5,5,60,8.1026,13.6,77.8,1016.1
(This data is being sent from weather station to base station but would also be intercepted by Net SDcard as it will store any data with “ESP_” in the from field)

File Write command string
To SDcard, from device,command,filename,data fields...........
ESP_SDcard:,ESP_weather,write,weather.cfg,config1,config2,config3

File Read string
ESP_SDcard:,ESP_irrigation,read,irrig.cfg

Resulting in this file data being sent, line by line until EOF (recovered by intended device)
udp.send:ESP_irrigation:irrig.cfg,config1,config2,config3

Process to remove card, press “card load/unload” button till green led blinks and turns on fully. Safe now to remove card1. To return card, press button again till Green led blinks, now return card, card is detected and updates from other card.

Schematic


Micromite code
'http://www.codenquilts.com.au
'Michael Ogden May 2016
'network SDcard
'

Option Explicit
Option Default Integer
Option autorun On
option error continue


'Constants
const Ver$="0.52"
const GrnLed=1
const YelLed=64
const RedLed=27
const Card.1Sel=2
const Card.2Sel=3
const Card.1Pres=62
'const Card.2Pres=
const Card.ChangePulse=63
const card.button=42


'Define I/O
setpin GrnLed,dout
setpin Card.1Sel,dout
setpin Card.2Sel,dout
setpin YelLed,dout
setpin RedLed,dout
setpin Card.ChangePulse,dout
setpin Card.1Pres,din,pullup
setpin Card.button,din,pullup

'Initialise I/O
pin(GrnLed)=0
pin(YelLed)=0
pin(Card.1Sel)=1
pin(Card.2Sel)=1
pin(Card.ChangePulse)=0


'Global variables
dim secs,r,tflag,Card1Flag,card2Flag
dim mo$,yr$,day$,mon$

dim SDbuff$(100) length 100
'intialise modules

'Initialise Arrays

'Initialise Var


'init ESP8266 communications
initxbcomm '(non interupt version)
'interupts
Settick 1000 ,T1,1 'establish seconds "Tick Timer"
tflag=1 'get NTP time
if pin(card.1pres)=0 then 'check if card1 is present
card1Flag=1
else
card1Flag=0
pin(Grnled)=1 'card now missing
end if
' now draw the display
'CLS

'*** start main program loop
Do
'StartTime = Timer
Watchdog 8000
if Tflag=0 then Pulse RedLed, 50
if secs mod 5 = 0 then 'try every 5 secs till the time is obtained
'if tflag=1 then r=espQueue("time:") 'try to get NTP time data
if tflag=1 then xbsend "time:" 'try to get NTP time data
'print date$+" "time$
end if
if pin(card.button)=0 then
pulse grnled,200
pause 500
checkreleased:
if pin(card.button)=0 then goto checkreleased
CardButton
goto skip
end if
line input #5,newline$
if newline$<>"" then userprocess
skip:
loop
'***** End main program loop
End



'**************** sub routines *********************

'T1 - 1second Tick interrupt
'set up some time flags
Sub T1
secs=secs+1 'update seconds timer
if secs => 86400 then '24 hrs reached
secs = 0 'reset counter
endif
if secs mod 3600 = 0 Then 'every hour
'r=espQueue("time:") 'try to get NTP time data
xbsend "time:" 'try to get NTP time data
end if
End Sub

sub CardButton
pulse grnled,200
pause 500
print "CardButton pressed"
if card1Flag=1 then
pin(Grnled)=1 'card now missing
r=cardset(2) 'set to SDCard2
appendfile(logfilename$(),"Removed",1)
pulse yelled,50
r=cardset(1) 'set to SDCard2
appendfile(logfilename$(),"Removed",1)
pulse grnled,50
card1flag=0
else
'wait for card to be inserted
waitforcard:
pulse grnled,200
pause 500
Watchdog 8000
if pin(card.1pres)=1 then goto waitforcard
pause 100 'settle delay
pin(Grnled)=0 'card now present
card1flag=1
clearbuffer
copyfile "temp.log" 'copy temp.log to buffer
writebuffer logfilename$() 'append buffer to file on card1
deletetemplog 'delete temp file
end if

end sub

'UserProcess routine (programmed by User) is called from main loop
'to check for commands.
Sub UserProcess 'User Subroutine to process incomming communications data
local temp$,cfg$,from$
print "ESP:";newline$ 'debug
if instr(newline$,"ntp:")>0 then
tflag=0
time$=parse$(newline$,4," ") 'set the clock from NTP
mo$=parse$(newline$,2," ") 'get month
yr$=parse$(newline$,5," ") 'get Year
day$=parse$(newline$,3," ") 'get day
mon$=moy$(mo$) 'get month number
date$=day$+"-"+mon$+"-"+yr$
exit sub
endif
if instr(newline$,"ESP_Base:Call")>0 then exit sub 'Device call home so ignore
if instr(newline$,"udp.rec:")>0 then
'2nd field contains command for SDcard device
from$=parse$(newline$,2)
cfg$=parse$(newline$,4)
'print cfg$;" ";temp$
select case parse$(newline$,3)
case "write"
temp$=mid$(newline$,instr(newline$,parse$(newline$,5)))
r=cardset(2) 'set to SDCard2
appendfile(cfg$,temp$,0)
pulse yelled,50
if card1flag then
r=cardset(1) 'set to SDCard1
appendfile(cfg$,temp$,0)
pulse grnled,50
endif
case "read"
readfile cfg$,from$
case else
print "Other"
end select
exit sub
end if
if instr(newline$,"ESP_Base:")>0 then 'log data to SDcards
newline$=mid$(newline$,11)
r=cardset(2) 'set to SDCard2
appendfile(logfilename$(),newline$,1)
pulse yelled,50
'check if card 1 is pressent if so write to card1
'if not write to temp file on card 2
if card1flag then
r=cardset(1) 'set to SDCard1
appendfile(logfilename$(),newline$,1)
pulse grnled,50
else
appendfile("temp.log",newline$,1)
pulse yelled,50
endif
exit sub
endif

End Sub

'Function Cardset
'selects or toggles SD card, sets Leds
'1=SDcard1 2=SDcard2 3=Toggle SDcard
function CardSet(sd)
if sd=3 then 'toggle selected
if pin(card.1sel)= 1 then
sd=2
else
sd=1
Endif
endif
if sd=1 then 'select sdcard1
pin(card.1sel)=0
pin(card.2sel)=1
pulse Card.ChangePulse,100
cardset=1
pause 100
exit function
endif
if sd=2 then 'select sdcard2
pin(card.1sel)=1
pin(card.2sel)=0
pulse Card.ChangePulse,100
cardset=2
pause 100
exit function
endif
'error condition
pin(Grnled)=1
pin(yelled)=1
cardset=0
end function

sub writeBuffer(fname$)
local i
local a$
'set card1
r=cardset(1)
OPEN fname$ FOR append AS #1
for i=1 to 100
a$=sdbuff$(i)
if a$="" then exit for
print #1,a$
pulse grnled,10
next
CLOSE #1
end sub

sub readfile(fname$,Dest$)
local i
local a$
'set card1
r=cardset(2)
OPEN fname$ FOR input AS #1
do while not eof(#1)
line input #1,a$
if a$="" then exit do
xbsend "udp.send:"+dest$+":"+fname$+","+a$ 'add ":" so that 8266 parser can reorder the output
pulse yelled,50
i=i+1
if i>100 then exit do
loop
CLOSE #1
end sub

sub clearBuffer
local i
for i=1 to 100
sdbuff$(i)=""
next i
end sub

sub copyfile(fname$)
local i
local a$
i=1
'set card2
r=cardset(2)
OPEN fname$ FOR input AS #1
do while not eof(#1)
line INPUT #1,a$
sdbuff$(i)=a$
pulse yelled,10
'print a$
i=i+1
if i>100 then exit do
loop
CLOSE #1
end sub

sub deleteTempLog
r=cardset(2)
kill "temp.log"
end sub

function LogFileName$()
Logfilename$=MID$(DATE$,1,2) + MID$(DATE$,4,2) + MID$(DATE$,9,2) + ".csv"
end function

sub appendfile(fname$,ap$,dt)
local i,b$
OPEN fname$ FOR append AS #1
if dt=1 then
b$=date$+","+time$+","+ap$
else
b$=ap$
end if
print #1,b$
CLOSE #1
end sub

sub writefile(fname$)
local i
local a$
i=1
OPEN fname$ FOR output AS #1
for i=1 to 500
a$=sdbuff$(i)
print #1,a$
i=i+1
next i
CLOSE #1
end sub

sub Listfiles
local f$
f$ = DIR$("*.*", FILE)
DO WHILE f$ <> ""
PRINT f$
f$ = DIR$()
LOOP
end sub

'Get Month of Year
function moy$(m$)
m$=ucase$(m$)
moy$="00"
if m$="JAN" then moy$="01"
if m$="FEB" then moy$="02"
if m$="MAR" then moy$="03"
if m$="APR" then moy$="04"
if m$="MAY" then moy$="05"
if m$="JUN" then moy$="06"
if m$="JLY" then moy$="07"
if m$="AUG" then moy$="08"
if m$="SEP" then moy$="09"
if m$="OCT" then moy$="10"
if m$="NOV" then moy$="11"
if m$="DEC" then moy$="12"
end function






'********************* Library Code **************************

Function Parse$(s$,FieldNumber,d$)
Local String stringArg$
Local Integer intOldY,intY,intX
if d$="" then
d$=","
endif
StringArg$ = S$ + d$
intOldY = 1:intX=0:intY=0
Parse$ = ""
do While intY < Len(StringArg$) And intX < FieldNumber
intY = Instr(intOldY, StringArg$, d$)
intX = intX + 1
If intX = FieldNumber Then
parse$ = Mid$(StringArg$, intOldY, intY - intOldY)
Endif
intOldY = intY + 1
loop
'print s$; intx; inty; intoldy
End Function


Function Ltrim(sString As String)As String
ltrim=Str$(Val(sString))
End Function

'Comms for ESP8266 (non Interrupt ver)
'initialise routines

Sub iniTxbcomm
Dim Status=2 'set status to 2 minutes for call home
dim newline$
'Open "com1:9600,512,Xbreceive" As #5 'initialise commS
Open "com1:38400,2048" As #5 'initialise commS Non Interrupt version
End Sub



Sub xbsend (msgframe$)
Print #5,msgframe$
End Sub



Sub Callhome
xbsend (serv$+":Call,"+node$+","+str$(status)) 'Call HoMe wIth counter value (status), ie 2 mins in dataLow
End Sub


'End of ESP8266 Comms


ESP8266 code
memclear
serialprintln "Esp8266-Mcu V2.xx Initialising......"
node = "ESP_Test" 'change to reflect device node
ver = "ESP8266-Mcu-V2.92"
port = 5001
AP node
prompt = "getdata:"
serialflush
serialtimeout 100
baudrate 38400
timesetup(10,0)
'setupemail "mail.smtp2go.com" myID "myEmail" "myPassword"
'owmparam = "&units=metric&mode=json&appid=myID"
owmtown = "Nilma,AU" & owmparam
'wu.get = "weatherstation.wunderground.com/weatherstation/updateweatherstation.php?ID=myID&PASSWORD=myPassword"
timer 100, [update]
udpbegin port
udpbranch [udp.receive]
print ver
print "ID=" & node
button "Exit" [quit]
wait

[udp.receive]
strin = udpread()
'serialprintln strin
if left(strin, 12) <> "Who are you?" then goto [next_test]
udpreply "ID=" & node
return
[next_test]
if instr(strin,node) = 0 then goto [next_test1]
gosub [parse]
serialprintln "udp.rec:" & strin
return
[next_test1]
if instr(strin,"ESP_ALL") = 0 then goto [next_test2]
gosub [parse]
serialprintln "udp.bcast:" & strin
[next_test2]
if left(strin,4) <> "ESP_" then goto [exit_test]
serialprintln strin
return
[exit_test]
return


[update]
input prompt strin
if len(strin) = 0 then goto [exitupdate]
gosub [parse]
if field == "time" then gosub [gettime]
if field == "email" then gosub [sendemail]
if field == "info" then gosub [getinfo]
if field == "quit" then goto [quit]
if field == "reboot" then reboot
if field == "ts" then gosub [thingspeak]
if field == "owm" then gosub [weather]
if field == "town" then gosub [changetown]
if field == "udp.send" then gosub [sendudpdata]
if field == "wu.send" then gosub [sendweather]
[exitupdate]
wait

[sendweather]
gosub [parse]
wubla = wu.get & field
serialprint wget(wubla)
return

[thingspeak]
gosub [parse]
sstatus = field
gosub [parse]
ts.key = field
gosub [parse]
ts.fieldnum = field
gosub [parse]
ts.fielddata = field
sendts(ts.key,ts.fieldnum,ts.fielddata)
return

[sendemail]
gosub [parse]
sstatus = field
gosub [parse]
address = field
gosub [parse]
reply = field
gosub [parse]
subject = field
gosub [parse]
body = field
email address reply subject body
return

[getinfo]
serialprintln ""
serialprint "ver:"
serialprintln ver
serialprint "ip:"
serialprintln ip()
read "WIFIname" blaWIFIssid
serialprint "ssid:"
serialprintln blaWIFIssid
serialprint "flash:"
serialprintln flashfree()
serialprint "ram:"
serialprintln ramfree()
serialprintln "myid:" & node
serialprintln "port:" & port
serialprintln
return

[gettime]
y = time(year)
if val(y) < 2016 then return
serialprintln "ntp:" & time()
serialprintln "date:" & time(year) & "," & time(month) & "," & time(date)
return

[parse]
field = ""
delim = instr(strin,":")
field = delim - 1
rest = len(strin) - delim
field = left(strin,field)
strin = right(strin,rest)
return

[changetown]
gosub [parse]
owmtown = field & owmparam
return

[sendudpdata]
gosub [parse]
udp.name = field
udpwrite "192.168.0.255",port,udp.name & ":," & node & "," & strin & chr(13)
return

[weather]
owmbla = "api.openweathermap.org/data/2.5/weather?q=" & owmtown
serialprint wget(owmbla)
return

[quit]
wprint "<a href='/'>Menu</a>"
end

Codenquilts
 
lew247

Guru

Joined: 23/12/2015
Location: United Kingdom
Posts: 1702
Posted: 09:36am 17 May 2016
Copy link to clipboard 
Print this post

Thanks Mike, It's bit over my head at the moment, but I'll spend some time digesting it over the next few weeks
It's a very interesting project and I would think extremely handy to have.
Thanks for the excellent descriptions with the code, it's appreciated.
 
Print this page


To reply to this topic, you need to log in.

The Back Shed's forum code is written, and hosted, in Australia.
© JAQ Software 2025