Home
JAQForum Ver 24.01
Log In or Join  
Active Topics
Local Time 15:51 05 Nov 2024 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : Maximum File Size on SD Card and File Handling

Author Message
stef123
Newbie

Joined: 25/09/2024
Location: United Kingdom
Posts: 17
Posted: 07:45am 30 Oct 2024
Copy link to clipboard 
Print this post

Hi,

i´ve implemented for my Webmite-program a subroutine, which is called each two seconds by a timer, in order to write some Log Data to a File on SD Card, which is periodically opened in "append" mode.

Some Data is written (approx 20 Bytes), after that the File I/O Buffer is flushed in order to be sure that no data gets lost and the file is finally closed.

Now, after the log File has reached a Size of 8339454 Bytes, i cannot write into this file any longer.

Is this a general PicoMite File Size limit, or could it be that the FAT-Routines now consume too much time in order to find the end of the File and append some data? Is the PicoMite / are the FAT Routines in append mode in need to crawl through the entire file before some Data can be added - by following the clusters and read in the Data for each Cluster- or do the Fat routines seek for the last Cluster of the File in question and read it in?


The next problem is reading the Log File from SD card over TFTP after the Pico has been restarted. Issuing a "tftp -i xx.xx.xx.xx get b:/log.txt" gives an empty file, although the Pico says "TFTP request to read binary file : b:/log.txt" and ends with "TFTP transmit complete" - but with no Data transmitted at all.

The file can be only read entirely when the drive is changed from A to B by issuing "B:" on the command line. Is there any workaround for this behaviour?


Thanks & Best regards
Stef
Edited 2024-10-30 17:46 by stef123
 
matherp
Guru

Joined: 11/12/2012
Location: United Kingdom
Posts: 9048
Posted: 09:09am 30 Oct 2024
Copy link to clipboard 
Print this post

Can't comment on the issue with the large file but you may well be pushing FatFS beyond its limits.
The TFTP issue is a bug which I'll fix - thanks for the report
 
stef123
Newbie

Joined: 25/09/2024
Location: United Kingdom
Posts: 17
Posted: 09:28am 30 Oct 2024
Copy link to clipboard 
Print this post

Hi Peter,

thanks for you reply. So i assume it would be better to write logs individually on a daily basis, instead of pushing it all into a single file. It would have been a bit easier for the log analyzing program i´ve wrote on the PC, but that can be solved.
 
Michal
Senior Member

Joined: 02/02/2022
Location: Poland
Posts: 120
Posted: 10:00am 30 Oct 2024
Copy link to clipboard 
Print this post

32 bit overflow ?
Overflow
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 3774
Posted: 10:59am 30 Oct 2024
Copy link to clipboard 
Print this post

8339454 is a bit under 8MB (8*1024*1024 = 8388608).

A bit of an odd number.

John
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 6655
Posted: 10:59am 30 Oct 2024
Copy link to clipboard 
Print this post

I'd be interested to know the actual format of the SD card - number of tracks/sectors/cluster sizes. Just saying "Fat32" or whatever makes assumptions.

I wonder if the difference in number is because some of the space on the card is reserved for bad sector mapping?
Edited 2024-10-30 21:01 by Mixtel90
Mick

Zilog Inside! nascom.info for Nascom & Gemini
Preliminary MMBasic docs & my PCB designs
 
stef123
Newbie

Joined: 25/09/2024
Location: United Kingdom
Posts: 17
Posted: 12:25pm 30 Oct 2024
Copy link to clipboard 
Print this post


fsck.fat 4.2 (2021-01-31)
Checking we can access the last sector of the filesystem
There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
 65:01/00
 Not automatically fixing this.
Boot sector contents:
System ID "        "
Media byte 0xf8 (hard disk)
      512 bytes per logical sector
    32768 bytes per cluster
     1170 reserved sectors
First FAT starts at byte 599040 (sector 1170)
        2 FATs, 32 bit entries
  3894784 bytes per FAT (= 7607 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 8388608 (sector 16384)
   973584 data clusters (31902400512 bytes)
63 sectors/track, 255 heads
     8192 hidden sectors
 62325760 sectors total


Totally normal 32 GB SD Card (formatted as FAT32).
Edited 2024-10-30 22:27 by stef123
 
stef123
Newbie

Joined: 25/09/2024
Location: United Kingdom
Posts: 17
Posted: 12:32pm 30 Oct 2024
Copy link to clipboard 
Print this post

Usually the Fat32-Limit is far beyond 32 GB. As with my own experiences with programming a FAT32 Driver (for 8051) and by that handling SD Cards on command level, i cannot imagine any reason aside from Speed issues why the writing gets stuck. I'll try to repeat the process without involving the Log program at all.
Edited 2024-10-30 22:35 by stef123
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 6655
Posted: 12:36pm 30 Oct 2024
Copy link to clipboard 
Print this post

Interesting.
8339454 / 32768
The error occurs at almost bang on 254.5 clusters (254.4999). I wonder if that's significant, like an 8-bit cluster counter?
Mick

Zilog Inside! nascom.info for Nascom & Gemini
Preliminary MMBasic docs & my PCB designs
 
stef123
Newbie

Joined: 25/09/2024
Location: United Kingdom
Posts: 17
Posted: 01:33pm 30 Oct 2024
Copy link to clipboard 
Print this post

Cannot repeat the issue for now, sorry. Writing a file with the same amount of data for each call of the subroutine and by the same append method creates a file larger than the size in question.

I also cannot see any performance drop at all. I´ll let it run over night to see if any errors occur. Maybe this happens if anything other is involved. My program has also an RX-Interrupt involved.
Edited 2024-10-30 23:36 by stef123
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 6655
Posted: 01:57pm 30 Oct 2024
Copy link to clipboard 
Print this post

Probably POM dependent. :)
Mick

Zilog Inside! nascom.info for Nascom & Gemini
Preliminary MMBasic docs & my PCB designs
 
Print this page


To reply to this topic, you need to log in.

© JAQ Software 2024