Home
JAQForum Ver 20.06
Log In or Join  
Active Topics
Local Time 09:40 29 Mar 2024 Privacy Policy
Jump to

Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me.

Forum Index : Microcontroller and PC projects : LINUX: Filesystem oddness....

     Page 1 of 2    
Author Message
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9030
Posted: 04:44am 18 Sep 2021
Copy link to clipboard 
Print this post

4TB WD Red drive, ext4.
76.9GB free.

Windows insists the drive has ZERO bytes available.
I tried deleting some old files on it, Windoze STILL says there are zero bytes available.

Linux reports 76.9GB free UNTIL I unmount, then remount, then even Linux says there is zero bytes available.

Getting suspicious at this point, at terminal: sudo fsck /dev/sdc1
Report comes back as "Clean".
I can access the files just fine in both Linux locally, or via the network in Windows, but if I try to copy anything to it, it moans(in windows) that the drive is full, which it is not.  In Linux, I can still copy files to the drive, EVEN THOUGH Linux is ALSO reporting zero space.  This is very odd.  

Does anyone know a more advanced fsck command I can run to perhaps do a more in-depth file system check?  The one I run from the terminal completes in about three seconds - much faster then the old ext2/ext3 fsck checks which took ages.

Photos:

















The next thing I will do, is move some files off this drive - it is quite full.
How many GB would ext4 reserve for FS maintenance on a 4TB drive?
Perhaps this is just a case of the drive being too full?
Smoke makes things work. When the smoke gets out, it stops!
 
Turbo46

Guru

Joined: 24/12/2017
Location: Australia
Posts: 1584
Posted: 06:16am 18 Sep 2021
Copy link to clipboard 
Print this post

Are the deleted files going to the recycle bin on that drive? That may cause confusion. I'd try emptying it.

Bill
Keep safe. Live long and prosper.
 
SimpleSafeName

Senior Member

Joined: 28/07/2019
Location: United States
Posts: 284
Posted: 06:21am 18 Sep 2021
Copy link to clipboard 
Print this post

How are you reading/writing an ext4 drive in Windows? The new Linux subsystem?

I did notice that you have an older version of fsck. Mine is 2.34

As for nodes, or whatever ext4 uses, that should be invisible to you I would think.

Is it just sdc3 giving you trouble?
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9030
Posted: 06:33am 18 Sep 2021
Copy link to clipboard 
Print this post

@ Turbo46: Recycle bin is empty.  Makes no difference.

@ SimpleSafeName: The ext4 drives are on a separate Linux-based computer, sitting on the network and sharing with Windoze machines via SAMBA2.  The drives are mapped to the Windoze filesystem tree - Windoze does not care about what format the drives are, as the host at the other end of the network connection takes care of all that.

Yes, just sdc1.  I can write to all the others, as there is plenty of space on them.
Smoke makes things work. When the smoke gets out, it stops!
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 5648
Posted: 06:40am 18 Sep 2021
Copy link to clipboard 
Print this post

Last time I played with a mix of linux and windows (a while ago, I admit), windows refused to see ext4 at all. A fully formatted ext4 drive always came back as having no space left according to windows, IIRC, and it just refused to do anything with it at all. Of course, that may have changed now.

I *think* the system space required on the drive depends on the number of nodes, not the amount of data. So if you save a lot of short files you'll run out of node space before you run out of data space. Or something like that. I'm very rusty on this stuff now. IIRC you can tell ext4 to increase the number of nodes (inodes?) available but I can't for the life of me remember how now.
Edited 2021-09-18 16:41 by Mixtel90
Mick

Zilog Inside! nascom.info for Nascom & Gemini
Preliminary MMBasic docs & my PCB designs
 
Volhout
Guru

Joined: 05/03/2018
Location: Netherlands
Posts: 3422
Posted: 03:39pm 18 Sep 2021
Copy link to clipboard 
Print this post

Ive seen something similar before, that drives simply refuse to be filled up to the last byte. I associated this with the fact that high density drives are lossy. Not only ssd, but also harddrives. And the os on the drive itself reserves space to swap blocks in case one gets bad. In the ATA config you can, with tools from the drive manufacturer, see these reservations, and even when the os decides to swap blocks.

Not saying this is the case in your situation, but it may be.

2 percent reservation may be totally valid.
Edited 2021-09-19 01:41 by Volhout
PicomiteVGA PETSCII ROBOTS
 
SimpleSafeName

Senior Member

Joined: 28/07/2019
Location: United States
Posts: 284
Posted: 03:51pm 18 Sep 2021
Copy link to clipboard 
Print this post

"@ SimpleSafeName: The ext4 drives are on a separate Linux-based computer, sitting on the network and sharing with Windoze machines via SAMBA2.  The drives are mapped to the Windoze filesystem tree - Windoze does not care about what format the drives are, as the host at the other end of the network connection takes care of all that.

Yes, just sdc1.  I can write to all the others, as there is plenty of space on them."

Ahh, for some reason I was thinking that you were checking/repairing the drive from Windows. That would be a neat/inadvisable trick...

And I'm familiar with SAMBA, not much use to me since I use a standalone machine at home and sneakernet works just fine for me. :)

For my home use I usually just hang out in Linux. Any required access to Windows files is just a click away. And if I need to run a Windows program in Linux I use VirtualBox. That's how I have one of my work laptops set up. Works great.


In any case, you can increase the number of inodes in EXT4 if you like:

https://askubuntu.com/questions/600159/how-can-i-create-an-ext4-partition-with-an-extra-large-number-of-inodes#:~:text=See%20the%20man%20page%20for%20mkfs.ext4.%20Option%20-N,fit%20your%20situation%20when%20you%20create%20the%20filesystem.
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 3641
Posted: 04:09pm 18 Sep 2021
Copy link to clipboard 
Print this post

  Grogster said  4TB WD Red drive, ext4.
76.9GB free.


Is this a drive you've used for quite a while and been OK?

I ask because I've met "drives" which are fake :(

  Grogster said  Linux reports 76.9GB free UNTIL I unmount, then remount, then even Linux says there is zero bytes available.


Odd. Very.

  Grogster said  Getting suspicious at this point, at terminal: sudo fsck /dev/sdc1
Report comes back as "Clean".


Please try
sudo e2fsck -f /dev/sdc1

I think your existing command is seeing the drive is "clean" so doing nothing useful.

John
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9030
Posted: 05:43am 19 Sep 2021
Copy link to clipboard 
Print this post

@ SimpleSafeName: You can get a downloadable Linux filesystem driver thing for Windoze, which give Windoze full access to Linux drives if you were to plop one in a Windoze machine.  I forget the name, but I know you can get a 3rd-party application that will add Linux filesystem support to Windows so Windoze can read/write to them just like any other Windoze drive.

@ John: Thanks for the expanded fsck thing - I will run that now.  Yes, I've had this drive since brand new, never gave any issues before, and definitely purchased a genuine one from a reputable OEM seller.  I've never met hard-drives that were fake of any brand, but I do know of the fake USB drives that are programmed to report the wrong capacity.

EDIT: Here are the results....





At this point, I plan to move some files off that drive to free up about 100GB or so, and then see if Windoze reports any space or not, and also see if anything changes as far as Linux is concerned.  Linux seems perfectly happy with the drive for the moment.

I could try rebuilding the Puppy Linux boot USB I guess.  I DID add one more drive to this system, which became sdd1, but that should not have upset anything I would not have thought....
Edited 2021-09-19 16:01 by Grogster
Smoke makes things work. When the smoke gets out, it stops!
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9030
Posted: 06:15am 19 Sep 2021
Copy link to clipboard 
Print this post

I've just discovered that in Linux, TWO different applications are reporting the change in capacity.  BOTH of them report the same, if you run them one after the other, but side-by-side, they show the difference.  pMount and Partview.





This now explains the difference in Linux, as two different applications seem to be calculating the free space differently.

Is there a command I can run from the console, to get a report on what the Kernel thinks the free space is rather then via a GUI application?
Smoke makes things work. When the smoke gets out, it stops!
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 5648
Posted: 06:22am 19 Sep 2021
Copy link to clipboard 
Print this post

Anything here of interest?
https://www.binarytides.com/linux-command-check-disk-partitions/
https://phoenixnap.com.br/kb/linux-check-disk-space
Mick

Zilog Inside! nascom.info for Nascom & Gemini
Preliminary MMBasic docs & my PCB designs
 
thwill

Guru

Joined: 16/09/2019
Location: United Kingdom
Posts: 3807
Posted: 07:39am 19 Sep 2021
Copy link to clipboard 
Print this post

  Grogster said  Is there a command I can run from the console, to get a report on what the Kernel thinks the free space is rather then via a GUI application?


Yup, it's called 'df':

   https://www.tecmint.com/how-to-check-disk-space-in-linux/

Best wishes,

Tom
Game*Mite, CMM2 Welcome Tape, Creaky old text adventures
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 3641
Posted: 02:15pm 19 Sep 2021
Copy link to clipboard 
Print this post

hmm, I'm sorry to say I don't know what's wrong (if anything).

df has some options (man df will explain) such as

df -h

which prints in "human readable" units (GB etc).

When I do
mount | grep ext4

I get just my mounted ext4 partitions e.g.
/dev/sda5 on / type ext4 (rw,noatime,data=ordered)
/dev/sda7 on /home type ext4 (rw,noatime,data=ordered)

I guess you get something similar.

John
 
Mixtel90

Guru

Joined: 05/10/2019
Location: United Kingdom
Posts: 5648
Posted: 02:43pm 19 Sep 2021
Copy link to clipboard 
Print this post

I've no idea how to attack this.
At one time I'd have set up a temporary box booting from a live filesystem (Mint) and adding this as the only HDD.

Possibly a case of back the lot up to another drive and prepare to wipe and reformat in case it's something dodgy with the partition table or something. If you do *anything* to the drive your data's at risk if it's something like that, either immediately or in the future.


And I've *finally* got round to putting a mouse socket on my CMM2. :)
Edited 2021-09-20 00:45 by Mixtel90
Mick

Zilog Inside! nascom.info for Nascom & Gemini
Preliminary MMBasic docs & my PCB designs
 
robert.rozee
Guru

Joined: 31/12/2012
Location: New Zealand
Posts: 2282
Posted: 04:49pm 19 Sep 2021
Copy link to clipboard 
Print this post

3640 / 76.8 = 48 (approximately)
1024 / 24 = 43 (approximately)

makes me wonder if you may have simply found a small bug where someone is using 1000 instead of 1024 when calculating file sizes.

also see:
https://en.wikipedia.org/wiki/Ext4
and
https://en.wikipedia.org/wiki/Journaling_file_system

with ext4, there are journaling structures on the disk that will be consuming a variable quantity of space.

i've often seen drives being reported by the operating system as having zero bytes free when the O/S wishes to signal that the drive is being treated as read-only for 'whatever' reason.


cheers,
rob   :-)
 
phil99

Guru

Joined: 11/02/2018
Location: Australia
Posts: 1721
Posted: 12:05am 20 Sep 2021
Copy link to clipboard 
Print this post

Some time ago the "Metric Authorities" decreed that the multipliers should only be based on 10^3 and 2^10 should not be used.
Drive manufacturers like this as their drives look bigger. Software writers still use both, causing some confusion.
I don't know how ext4 works but in some software the unusable space between the end of a file and the end of its allocation unit is shown as free. Reformatting with smaller allocation units can recover some of that space, if there are a lot of small files.
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9030
Posted: 12:11am 20 Sep 2021
Copy link to clipboard 
Print this post

Results of sudo df -h:





This is ALSO reporting zero available.
I think I will do Mick's suggestion - copy everything off of this drive for the moment, reformat it, and copy everything back again.  Pit of a PITA, but I am suspicious of a drive reporting zero available, when I know there is more space there, and Linux even lets me copy files to the drive - despite some of Linux's own tools reporting zero space.

I suppose one thing I could try first, is to boot up a USB version of Mint, and see if it also is reporting the same kind of oddness, then proceed from there.
Smoke makes things work. When the smoke gets out, it stops!
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9030
Posted: 12:57am 20 Sep 2021
Copy link to clipboard 
Print this post

Mint also reports zero - something odd going on here, so I will now proceed to remove everything from that drive, reformat it, and copy everything back.






Smoke makes things work. When the smoke gets out, it stops!
 
JohnS
Guru

Joined: 18/11/2011
Location: United Kingdom
Posts: 3641
Posted: 06:11am 20 Sep 2021
Copy link to clipboard 
Print this post

It may depend which user is doing the enquiry/copy and which fs (file system) kind (here, ext4).

If root, you can use every byte (not always wise as some file systems get slow as they get extremely full/fragmented).

If non-root, there is typically a threshold where the fs is shown as full to reduce/prevent the problem.

Rebuilding by copying everything will reduce fragmentation but if the issue is along the lines just mentioned will make no difference.

John
 
Grogster

Admin Group

Joined: 31/12/2012
Location: New Zealand
Posts: 9030
Posted: 06:25am 20 Sep 2021
Copy link to clipboard 
Print this post

Good point.  I will MOVE 100GB or so off that drive to one of the other ones, and see what happens.
Smoke makes things work. When the smoke gets out, it stops!
 
     Page 1 of 2    
Print this page
© JAQ Software 2024