External HD Format NTFS vs ext32

[IAmConfused]

Long story, short version. Just had a catastrophic early (less than 6 mos.) failure of an external 4TB hard drive used for Calibre library back-up. Have a replacement en route. It will arrive formatted ext32. It will be used exclusively on Win 10 via (USB) & is a 3.5 spinner (my experience with 2.5" drives not stellar e.g the current early failure & it on ext32) . Maybe just a gut hunch, or paranoia, but I'm very much inclined to format it to NTFS on receipt before putting it into service.

Am hopeful that the inherent journaling of NTFS may save my bacon down the road but am unsure if perfomance may suffer, or if my current bias is taking me down a road with only imagined benefit. All my internal Windows drives are NTFS.

Am soliciting & would very much welcome any expert advice from Calibre users having large libraries on which way to go. Am using FreeFileSync as my BU tool of choice.

[Quoth]

ext32? Do you mean exFAT? The older MS non-NT is FAT32 (win9x).

Anyway: NT (Windows) needs NTFS. Linux ext4, Apple which ever FS the installed MacOS is using.

exFAT and FAT32 are only useful to copy between different OSes.

Also I'd reformat any new drive to avoid supply chain malware, shovel-ware and ensure the file system matchs the OS in use if it's not for "sneaker net" use.


Yes, it should be NTFS for your OS and application.

[DNSB]

I would leave the format as exFAT (I'm assuming that is what you mean by ext32 though that might also refer to FAT32) but I would reformat it before use. Too many drives these days ship will all kinds of crapware on them. No real advantage that I've seen in formatting external hard drives as NTFS.

Your paranoia should extend to not trusting any storage medium. As one old saw puts it, the best way not to need a backup is to have one.

[IAmConfused]

My bad ... for clarity my ext32 reference should have been exFat. Apparently it has become common for manufacturers to so format external drives for cross platform compatibility out of the box.

If pretty much a toss up (other than potential cross platform compatibility), I'll probably go NTFS (as indicated, its journalizing may save my butt down the road) unless someone raises a performance concern of which I'm unaware

Appreciate the feedback from Quoth & DNSB

[BetterRed]

Originally Posted by Quoth

ext32? Do you mean exFAT? The older MS non-NT is FAT32 (win9x).

Anyway: NT (Windows) needs NTFS. Linux ext4, Apple which ever FS the installed MacOS is using.

exFAT and FAT32 are only useful to copy between different OSes.

Also I'd reformat any new drive to avoid supply chain malware, shovel-ware and ensure the file system matchs the OS in use if it's not for "sneaker net" use.


Yes, it should be NTFS for your OS and application.

↑ ↑ ↑ ✔

Calibre will use hardlinks (inode copies) & delete for mainstream operations involving folder/file moves on native filesystems… which is fast.

Because FAT devices don't support hardlinks calibre has to use file/folder copy & delete… which is slow. I don't know what calibre does on exFAT… not even sure exFAT even supports hardlinks.

I know exFAT doesn't support symbolic links… which I do make use of, including in the the context of calibre.

I would use NTFS.

BR

[DNSB]

Last time I looked hardlinks on a NTFS volume can only access files not folders and they can't reference files on a different volume. Reparse points/junctions can operate between volumes but they are a pain since they can not reference files nor can they reference a network volume. Symbolic links under NTFS were designed to be compatible with Unix style symlinks but require NTFS 3.1+. Personally, I found symbolic links to be a total PITA since the backup programs we were using either copied only the file path and not the data or copied the data every time a symbolic link was accessed during the backup.

On a FAT32/exFAt/etc. volume, moving a file between directories only moves the directory reference not the file data so not that much speed difference.

[BetterRed]

The backup program I use (GoodSync) lets me configure what to do with symlinks… it's one of the reasons I chose to use it well over a decade ago, probably closer to two.

If you think calibre should not be making use of hardlinks when moving book folders and format files you should take it up with Kovid.

BR

[DNSB]

Originally Posted by BetterRed

If you think calibre should not be making use of hardlinks when moving book folders and format files you should take it up with Kovid.

To quote my previous message:

Originally Posted by DNSB

Last time I looked hardlinks on a NTFS volume can only access files not folders and they can't reference files on a different volume.

I.e. you can not use hardlinks when moving folders since hardlinks are not able to access folders.

[kovidgoyal]

NTFS is way, way better than exfat. Always go for ntfs unless there is some overriding need to use exfat such as compatibility with some limited device or other OS or similar.

[BetterRed]

Originally Posted by DNSB

To quote my previous message:



I.e. you can not use hardlinks when moving folders since hardlinks are not able to access folders.

I'll leave it to someone who is better than me at dealing with those who resort to pedantry to explain how to make use of hardlinks when moving a folder and its contents to another location… quickly.

Maybe someone who is familiar with the inner workings of Windows Explorer.

BR

[DNSB]

Going back to Microsoft's documentation:

Hard links

A hard link is the file-system representation of a file by which more than one path references a single file in the same volume. To create a hard link, use the CreateHardLinkA function.

Any changes made to a hard-linked file are instantly visible to applications that access it through the links that reference it. The attributes on the file are reflected in every hard link to that file, and changes to that file's attributes propagate to all the hard links. However, the directory entry size and attribute information of the file are visibly updated only at the link through which the change was made. For example, if you clear the read-only attribute flag on a particular hard link so you can delete that hard link, and there are multiple hard links to the file, the other hard links display that the read-only attribute is still set, which isn't true. To change the file back to the read-only state, you must set the read-only flag on the file from one of its remaining hard links.

For example, in a system where C: and D: are local drives and Z: is a network drive mapped to \\fred\share, the following references are permitted as a hard link:

C:\dira\ethel.txt linked to C:\dirb\dirc\lucy.txt
D:\dir1\tinker.txt linked to D:\dir2\dirx\bell.txt
C:\diry\bob.bak linked to C:\dir2\mina.txt

This is because all the links are files on the same volume. Hard links can't reference directories, only files, and they can't reference files on different volumes.

The following references aren't permitted:

C:\dira linked to C:\dirb
C:\dira\ethel.txt linked to D:\dirb\lucy.txt
C:\dira\ethel.txt linked to Z:\dirb\lucy.txt

To delete a hard link, use the DeleteFileA function. You can delete hard links in any order regardless of the order in which they're created.

It seems rather clear that hardlinks can not be used with a directory/folder. Symlinks can but we are going round the rose with hardlinks.

[BetterRed]

Read what I wrote:

…make use of hardlinks when moving a folder and its contents to another location…

…uti nexus duro cum movendo folder et contenta in alium locum…

… utiliser des liens physiques lors du déplacement d'un dossier et de son contenu vers un autre emplacement…

…hacer uso de enlaces físicos al mover una carpeta y su contenido a otra ubicación…

Thanks Google

BR

[Quoth]

However NTFS is far better than FAT12, FAT32, exFAT. One reason is journalling. Only the DOS loaded windows didn't use it. By default NT 3.1, NT3.x, NT 4.0, NT 5.x (Win2K, XP, 2003, Vista, Win7), Win8, win1x) use NTFS.

If using a USB drive as a portable drive for different OS, then use FAT32 or exFAT.

If using a USB drive for extra storage and use directly with any application, then you use the native filesystem, which was always NTFS for all real Windows OS. WinME was the last Windows branded desktop OS that didn't use NTFS.

On XP I only used drive letters for the CD/DVD drive, boot drive and removable drives. I mounted extra internal drives into an empty directory (folder) like you'd do on Unix/Linux. I'd have done that if using a dedicated USB drive, if that works.

[ratinox]

Originally Posted by Quoth

If using a USB drive as a portable drive for different OS, then use FAT32 or exFAT.

With the caveat that contemporary Linux desktops have mechanisms to mount NTFS volumes directly, and some like OpenSUSE Tumbleweed include BitLocker-compatible drivers. If you are using just Windows and Linux desktops then you can probably use NTFS everywhere you need it. I have a small USB SSD with my emergency virtual machine for work needs. It's NTFS w/ BitLocker, VirtualBox VM images and config, and it works seamlessly on my Tumbleweed machines.

But for portability with my iPad and Windows and Tumblweed? It has to be exFAT. No choice here.

[Quoth]

Oh, I've been able to use NTFS on Linux since maybe 1999 without doing anything special.

Bitlocker is more likely to stop a user repairing than be a really needed security feature. The Browser malicious script blocking and avoiding malicious email content is far more important. Also a separate firewall, not part of Windows, so I don't believe in built-in mobile/cellular on windows, but a 3rd party external battery powered hot spot with WiFi and ethernet. I've had one since 2007.

Edit:
I also had Ext2 support on WinXP nearly 20 years ago, but 3rd party driver.

Apple is tricky. Ten years ago a 3rd level college using Mac on a Media course expected students to use an Apple format on their USB drives/sticks.