uM2(+): data logging - cheap fast & easy


Author Message
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 11980
Posted: 02:21am 27 Mar 2016      

This one I'm pleased with

Since playing with the ESP8266 I've been interested in exploring how Serial Flash memory chips could be used for secure data logging on the Micromite as an alternative to SDcards or an external device like an Openlog.

These chips are ridiculously cheap so there is no reason not to include one on every PCB. Winbond branded chips work perfectly with the program and are recommended.





The example shown above is the one I used to develop the code and will hold 4Mbytes. Compatible chips are available from 512kbyte to 16Mbytes and the code should automatically cater for any of them.

Connections to the 8-pin chips are simple:

pin 1 (chip select) - connect to any Micromite output capable pin
pin 2 (Data out) - connect to the SPI-IN pin on the Micromite
pin 3 (write protect) - 10K resistor to 3.3V
pin 4 (GND) - connect to ground
pin 5 (Data in) - connect to the SPI-OUT pin on the Micromite
pin 6 (Clock) - connect to the SPI-CLK pin on the Micromite
pin 7 (Hold) - 10K resistor to 3.3V
pin 8 (VCC) - connect to 3.3V

The chips are a wide version of SOIC and can be easily soldered to a DIP adapter. This could then be socketed on the PCB allowing it to be swapped for reading separately in the same way as an SD card.

The intention of the coding was to hide all the workings of the chip from the user and make the user interface as simple as possible so there are just three user routines:


formatlog(chip_select_pin_number)

This routine completely erases the chip and tests it for correct erasure. In the event that the erase fails the routine will print a message and the program will end. The routine prints the ID of the chip and the size in Mbytes. It also writes to the chip an index of used pages that the logging routine uses to determine where a new record should be appended.

writelog(string-to-write, chip_select_pin_number, timestamp)

This routine appends the supplied string to the log. If the variable "timestamp" is non-zero it will automatically prepend the date and time comma-separated to the front of the string. If timestamp is 0 or not specified the string will be written as supplied. Logging strings can be constructed in the normal way using STR$ to format numbers and the "+" operator to concatenate multiple sub-strings.

readlog(chip_select_pin_number)

This routine reads the complete log and prints the contents to the console. To upload to a PC just set Teraterm (or equivalent) to record the terminal session and call readlog. The routine does not change the log data in any way so logging can be restarted after reading without any impact.

NB all of the routines are completely self-contained and can be called from the command line with no requirement for anything to have been run or set up previously - there are no global variables or constants - everything required to run is stored on the chip itself. They also obey the requirements for sharing the SPI bus with a TFT screen on the Micromite.

This means calls to the formatlog and readlog routines need never be included in the running program as they can just be called from the command line as and when required.

It really can't get any easier to include logging in any Micromite program.

The test program attached demonstrates just how easy it is to including logging commands in a program. I've tested the code using 1000, 10000, and 100000 log records. It takes 183 seconds to construct and log 10000 records.

The C-source is attached, there is nothing complicated and it was all written first in Basic and then just converted to C for improved performance.


option explicit
option default none
dim integer i
const chipselectpin=1 'chip select pin
'
' Example program to demonstrate logging to a serial flash chip
'
testdata() 'sets up the testdata generator
'
formatlog(chipselectpin) 'format the chip
'
timer=0
for i=1 to 1000 'log 1000 strings with timestamps
writelog(int2Text(i),chipselectpin,1)
next i
print "1,000 records constructed and logged in ",timer\1000," seconds"
pause 3000
'
readlog(chipselectpin) ' read back the log
'
end
'
'****************************************
'
'
sub formatlog(cspin as integer) 'erase the chip and set up the page index
local integer i,bytecount,r(2)
local s$ length 10
spi open 20000000,3,8
bytecount = getpagecount(s$,cspin)*256
print "JEDEC ID = ",s$
print "Memory size is ",bytecount\131072," Mbits"
if not (erasechip(bytecount,cspin)) then
print "erase failed"
end
endif
for i=0 to bytecount\524288
setpagewritten(i,cspin) 'mark the index pages and first real page as used
next i
spi close
print "Format complete"
end sub
'
sub writelog(s$,cspin as integer,timestamp as integer) 'write a string to the next location on the chip with optional timestamp
local integer i,x,buff(63),f,pagecount
local d$ length 10
spi open 20000000,3,8
pagecount=getpagecount(d$,cspin)
x=getnextfreepage(pagecount,cspin)
if x<>pagecount-1 then
if x<>pagecount\2048 then x=x-1 'point to the previous page to see if it has space
readpage(x,buff(0),cspin) 'read in first page
readpage(x+1,buff(32),cspin) 'read in the second page
f=getfirstfreebyte(buff())
if timestamp then
i=loadstring(date$+","+time$+","+s$,buff(),f)
else
i= loadstring(s$,buff(),f)
endif
if i>=256 then 'write the bit on the next page and set the page marker as used
writepage(x+1,buff(32),cspin)
setpagewritten(x+1,cspin)
endif
writepage(x,buff(0),cspin)
else
print "Error: Chip full"
endif
spi close
end sub
'
sub readlog(cspin as integer) 'read and print the log
local integer x,buff(63),f=0, n,i
local s$
spi open 20000000,3,8
x=getpagecount(s$,cspin)\2048 'number of indexpages
readpage(x,buff(0),cspin) 'read in first two pages
readpage(x+1,buff(32),cspin)
if buff(0) =-1 then
spi close
exit sub 'no data
endif
n=getstring(s$, buff(0),f)
print s$
do while n<>&HFF 'repeat until no more data
if f>=256 then
x=x+1
readpage(x,buff(0),cspin) 'read in next two pages
readpage(x+1,buff(32),cspin)
f=f-256
endif
n=getstring(s$, buff(0),f)
print s$
loop
spi close
end sub
'
function getnextfreepage(pagecount as integer, cspin as integer) as integer 'gets the page number of the next completely unused page
local integer i,j,r(31),p=0,found=0,indexsize=pagecount\2048
for i=0 to indexsize-1 'potentially look at all pages
readpage(i,r(),cspin)
for j=0 to 31
if r(j) then
p=j 'there must be at least one spare page if not zero
found=1
exit for
endif
next j
if found then exit for
next i
for j=0 to 63
if (r(p)>>j) and 1 then exit for
next j
getnextfreepage=i*2048+p*64+j
end function
'
sub setpagewritten(pageno as integer, cspin as integer) 'set a page as partially or completely written
local integer buff(31),i
local integer mappage=pageno\2048 'we have 2048 bits per page
local integer wordno=(pageno-mappage*2048)\64 'locate the word in the page
local integer bitno= 1<<(pageno mod 64) 'locate the bit in the word in the page
readpage(mappage,buff(),cspin)
buff(wordno) =buff(wordno) XOR bitno
writepage(mappage,buff(),cspin)
end sub
'
CFunction getpagecount 'returns the chip size in 256-byte pages and the chip ID
00000000
27BDFFD0 AFB20020 AFBF002C AFB40028 AFB30024 AFB1001C AFB00018 8CB00000
3C029D00 8C420088 00101880 00621021 8C420000 24030008 10430007 00809021
3C029D00 8C420010 02002021 24050008 0040F809 00003021 3C029D00 8C42001C
02002021 0040F809 24050005 2403009F 3C02BF80 AC435820 3C03BF80 8C625810
30420080 1040FFFD 3C02BF80 8C435820 3C03BF80 AC405820 8C625810 30420080
1040FFFD 3C02BF80 8C535820 3C03BF80 AC405820 8C625810 30420080 1040FFFD
3C02BF80 8C545820 3C03BF80 AC405820 8C625810 30420080 1040FFFD 3C119D00
8E22001C 3C03BF80 02002021 24050006 8C705820 0040F809 00000000 00133A00
24020010 AFA20010 00F43821 8E220030 00073A00 00F03821 26440001 00E03021
0040F809 00073FC3 24040006 A2440000 8FBF002C 2610FFF8 24020001 02021004
00021FC3 8FB40028 8FB30024 8FB20020 8FB1001C 8FB00018 03E00008 27BD0030
End CFunction
'
Cfunction erasechip 'size of chip in bytes, chip select pin number
00000000
27BDFFD8 AFB40020 AFBF0024 AFB3001C AFB20018 AFB10014 AFB00010 8CB30000
3C029D00 8C42001C 0080A021 24050005 0040F809 02602021 24030006 3C02BF80
AC435820 3C03BF80 8C625810 30420080 1040FFFD 3C109D00 8E02001C 3C11BF80
02602021 24050006 8E235820 0040F809 00000000 8E02001C 02602021 0040F809
24050005 240200C7 AE225820 3C03BF80 8C625810 30420080 1040FFFD 3C029D00
8C42001C 02602021 24050006 3C03BF80 8C635820 0040F809 3C119D00 3C10BF80
8E220004 0040F809 3404C350 8E22001C 02602021 0040F809 24050005 24020005
AE025820 8E025810 30420080 1040FFFD 00000000 8E025820 AE125820 8E025810
30420080 1040FFFD 00000000 8E125820 8E22001C 02602021 7C129420 0040F809
24050006 32420001 5440FFE6 8E220004 3C029D00 8C42001C 02602021 0040F809
24050005 24030003 3C02BF80 AC435820 3C03BF80 8C625810 30420080 1040FFFD
3C02BF80 8C435820 3C03BF80 AC405820 8C625810 30420080 1040FFFD 3C02BF80
8C435820 3C03BF80 AC405820 8C625810 30420080 1040FFFD 3C02BF80 8C435820
3C03BF80 AC405820 8C625810 30420080 1040FFFD 00000000 8E820004 3C03BF80
8C635820 5C400006 3C02BF80 14400014 3C029D00 8E820000 10400010 3C02BF80
AC525820 3C03BF80 8C625810 30420080 1040FFFD 3C029D00 8C42001C 02602021
24050006 3C03BF80 8C635820 0040F809 00000000 10000007 00001021 3C029D00
8C42001C 02602021 0040F809 24050006 24020001 8FBF0024 8FB40020 8FB3001C
8FB20018 8FB10014 8FB00010 03E00008 27BD0028

End Cfunction
'
CSub writepage 'page number, 256 byte data buffer, chip select pin number
00000000
27BDFFD8 AFB3001C AFB00010 AFBF0024 AFB40020 AFB20018 AFB10014 8CD30000
3C029D00 8C42001C 8C940000 00A08021 02602021 0040F809 24050005 24030006
3C02BF80 AC435820 0014A200 3C03BF80 8C625810 30420080 1040FFFD 3C119D00
8E22001C 3C12BF80 02602021 24050006 8E435820 0040F809 00000000 8E22001C
02602021 0040F809 24050005 24020002 AE425820 3C03BF80 8C625810 30420080
1040FFFD 3C02BF80 7E843C00 8C435820 3C03BF80 AC445820 8C625810 30420080
1040FFFD 3C02BF80 7E943A00 8C435820 3C03BF80 AC545820 8C625810 30420080
1040FFFD 3C02BF80 8C435820 3C03BF80 AC405820 8C625810 30420080 1040FFFD
3C02BF80 8C425820 00002021 3C03BF80 240500FF 02041021 80420000 AC625820
8C625810 30420080 1040FFFD 00000000 8C625820 10850003 3C029D00 1000FFF5
24840001 8C42001C 02602021 0040F809 24050006 24120100 3C119D00 3C10BF80
8E220004 0040F809 240400FA 8E22001C 02602021 0040F809 24050005 24020005
AE025820 8E025810 30420080 1040FFFD 00000000 8E025820 AE125820 8E025810
30420080 1040FFFD 00000000 8E125820 8E22001C 02602021 0040F809 24050006
32420001 5440FFE7 8E220004 8FBF0024 8FB40020 8FB3001C 8FB20018 8FB10014
8FB00010 03E00008 27BD0028
End CSub
'
CSub readpage 'page number, 256 byte data buffer, chip select pin number
00000000
27BDFFD8 AFB2001C AFB10018 AFBF0024 AFB30020 AFB00014 8CD20000 3C029D00
8C42001C 8C930000 00A08821 02402021 0040F809 24050005 24030003 3C02BF80
AC435820 00139A00 3C03BF80 8C625810 30420080 1040FFFD 3C02BF80 7E643C00
8C435820 3C03BF80 AC445820 8C625810 30420080 1040FFFD 3C02BF80 7E733A00
8C435820 3C03BF80 AC535820 8C625810 30420080 1040FFFD 3C02BF80 8C435820
3C03BF80 AC405820 8C625810 30420080 1040FFFD 3C02BF80 8C425820 00002021
3C03BF80 24050100 AC705820 8C625810 30420080 1040FFFD 00000000 8C705820
02241021 24840001 7C108420 1485FFF6 A0500000 3C029D00 8C42001C 02402021
0040F809 24050006 8FBF0024 8FB30020 8FB2001C 8FB10018 8FB00014 03E00008
27BD0028
End CSub
'
CFunction getfirstfreebyte
00000000
90830000 240200FF 00003021 10620014 00003821 90830001 1062000A 24020001
24020002 240500FF 24080100 00821821 90630000 54650006 24420001 10000002
00403021 00403021 10000005 00023FC3 5448FFF7 00821821 24060100 00003821
00C01021 03E00008 00E01821
End CFunction
'
CFunction loadstring 'copies a string into the write buffer
00000000
8CC30000 90820000 00621021 0043302A 14C0000A 24660001 00A31821 90870000
24C60001 24C5FFFF 0045282A A0670000 24840001 10A0FFF9 24630001 03E00008
00021FC3
End CFunction
'
CFunction getstring ' gets a string ferom the read buffer
00000000
8CC20000 00A21821 90690000 01224821 0122182A 5460000C 25220001 24430001
00A21021 90480000 24630001 2467FFFF 0127382A A0880000 24420001 10E0FFF9
24840001 25220001 00021FC3 ACC30004 ACC20000 00A92821 90A20001 03E00008
00001821
End CFunction
'
'****************************
' test data generation
'
'small
sub testdata
DATA "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten"
DATA "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", "nineteen"
'tens
DATA "twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety"
'big
DATA "thousand", "million", "billion"
dim small(19) AS STRING length 10, tens(7) AS STRING length 10, big(2) AS STRING length 10
FOR i = 1 TO 19
READ small(i)
NEXT
FOR i = 0 TO 7
READ tens(i)
NEXT
FOR i = 0 TO 2
READ big(i)
NEXT
'
end sub
.
FUNCTION int2Text(number AS integer) as string
local num AS integer, outP AS STRING length 60, unit AS INTEGER
local tmpLng1 AS integer

IF 0 = number THEN
int2Text = "zero"
EXIT FUNCTION
END IF

num = ABS(number)

DO
tmpLng1 = num MOD 100
SELECT CASE tmpLng1
CASE 1 TO 19
outP = small(tmpLng1) + " " + outP
CASE 20 TO 99
SELECT CASE tmpLng1 MOD 10
CASE 0
outP = tens((tmpLng1 \ 10) - 2) + " " + outP
CASE ELSE
outP = tens((tmpLng1 \ 10) - 2) + "-" + small(tmpLng1 MOD 10) + " " + outP
END SELECT
END SELECT

tmpLng1 = (num MOD 1000) \ 100
IF tmpLng1 THEN
outP = small(tmpLng1) + " hundred " + outP
END IF

num = num \ 1000
IF num < 1 THEN EXIT DO

tmpLng1 = num MOD 1000
IF tmpLng1 THEN outP = big(unit) + " " + outP

unit = unit + 1
LOOP

IF number < 0 THEN outP = "negative " + outP
Do WHILE ASC(RIGHT$(outp,1))<=32
outp = LEFT$(outp,len(outp)-1)
loop
int2Text = outP
END FUNCTION
'*********************************


#define writeenable 0x06
#define pageprogram 0x02
#define readstatus1 0x05
#define readdata 0x03
#define eraseall 0xC7
#define JEDEC 0x9F
#define SPIsend(a) {int j;SPIBUF=a; while((SPISTAT & 0x80)==0); j=SPIBUF;}
#define SPIread(a) {SPIBUF=a; while((SPISTAT & 0x80)==0); a=SPIBUF;}
#define SPISTAT *(volatile unsigned int *)(0xbf805810)
#define SPIBUF *(volatile unsigned int *)(0xbf805820)

long long pagecount(char id[],long long *cspin){
int pin=*cspin,i,j,k;
if(ExtCurrentConfig[pin]!=EXT_DIG_OUT)ExtCfg(pin,EXT_DIG_OUT,0);
PinSetBit(pin,LATCLR);
SPIsend(JEDEC);
SPIread(i);
SPIread(j);
SPIread(k);
PinSetBit(pin,LATSET);
IntToStr(&id[1],i*65536+j*256+k,16);
id[0]=6;
return (1<<(k-8));
}
int erasechip(long long *size, long long *cspin){
int p,pin=*cspin;
char m;
PinSetBit(pin,LATCLR);
SPIsend(writeenable);
PinSetBit(pin,LATSET);
PinSetBit(pin,LATCLR);
SPIsend(eraseall);
PinSetBit(pin,LATSET);
do {
uSec(50000);
PinSetBit(pin,LATCLR);
SPIsend(readstatus1)
SPIread(m);
PinSetBit(pin,LATSET);
} while(m & 1);
PinSetBit(pin,LATCLR);
SPIsend(readdata);
SPIsend(0);
SPIsend(0);
SPIsend(0); //start address is zero
for(p=0;p<*size;p++){
SPIread(m);
if(m!=0xFF){
PinSetBit(pin,LATSET);
return 0;
}
}
PinSetBit(pin,LATSET);
return 1;
}

void writepage(long long *address, char d[], long long *cspin){
int k=0,add,m,pin=*cspin;
add=*address<<8; //convert page number to byte number
PinSetBit(pin,LATCLR);
SPIsend(writeenable);
PinSetBit(pin,LATSET);
PinSetBit(pin,LATCLR);
SPIsend(pageprogram);
SPIsend((add>>16) & 0xFF);
SPIsend((add>>8) & 0xFF);
SPIsend(add & 0xFF);
for(m=0;m<256;m++) {
SPIsend(d[k]);
k++;
}
PinSetBit(pin,LATSET);

do {
uSec(250);
PinSetBit(pin,LATCLR);
SPIsend(readstatus1)
SPIread(m);
PinSetBit(pin,LATSET);
} while(m & 1);

}
void readpage(long long *address, char d[], long long *cspin){
int k,add,pin=*cspin;
char m;
add=*address<<8; //convert page number to byte number
PinSetBit(pin,LATCLR);
SPIsend(readdata);
SPIsend((add>>16) & 0xFF);
SPIsend((add>>8) & 0xFF);
SPIsend(add & 0xFF);
for(k=0;k<256;k++){
SPIread(m);
d[k]=m;
}
PinSetBit(pin,LATSET);
}
long long getfirstfreebyte(unsigned char d[]){
int j=0;

if(d[0]==0xFF) return 0;
for(j=1;j<256;j++){
if(d[j]==0xFF) return j;
}
return 256;
}
long long loadstring(unsigned char s[],unsigned char b1[],long long *pos){
int k,j=0,len=s[0],start=*pos;
for(k=start;k<=start+len;k++){
b1[k]=s[j];
j++;
}
return start+len;
}
long long getstring(unsigned char s[],unsigned char b1[],long long *pos){
int k,j=0,start=*pos,len=b1[start];
for(k=start;k<=start+len;k++){
s[j]=b1[k];
j++;
}
*pos=start+len+1; //start of next string
return b1[start+len+1];
}


Edited by matherp 2016-03-28