MTP Android. Scanning entire SD card

[Quoth]

Previously I set up which folder/directories should be scanned by Calibre. But now the "connect" job seems to be scanning everything. A 256 G card with 83 G byte free.

Unfortunately you can't stop that job (communicating with device), nor set up which folder/directories on which storage should be scanned until it has scanned.

Separate issue:
I also set separate default locations for epubs and pdfs, but when I "send to device" I have to explictly pick the SD card or else those are created on internal storage.

Setting default of SD Card / Card A in "Send to Device" changes that for all devices, which isn't going to work for most Kindles, most Kobos, my Sony PSR-350 or my P47L (none have extra cards).

[Quoth]

It's likely to take a long time and timed out on the first attempt. Now at 10 minutes.

I know I could power off the Android tablet and use the SD card on a USB adaptor with "Send to Disk", but not a great solution.

[Quoth]

Sorry, Calibre 7.10 on Linux Mint 21.3 with Mint Desktop. The "gvs-mtp-volume-monitor" is stopped before connecting any MTP device so that the device isn't mounted. (The "don't mount" GUI option works for Cinnamon, but not Mate).

Now at 18 minutes and still "finding objects" outside of the PDF book and ebook trees.

[Quoth]

calibre, version 7.10.0
ERROR: Error: Error communicating with device

Failed to upload file named: driveinfo.calibre to ('SD card',): 2:PTP Layer error 2002: send_file_object_info(): Could not send object info.
2:Error 2002: PTP General Error

Traceback (most recent call last):
  File "calibre/gui2/device.py", line 111, in run
  File "calibre/gui2/device.py", line 565, in _books
  File "calibre/devices/mtp/driver.py", line 230, in books
  File "calibre/devices/mtp/driver.py", line 203, in get_driveinfo
  File "calibre/devices/mtp/driver.py", line 193, in _update_drive_info
  File "calibre/devices/mtp/driver.py", line 163, in put_calibre_file
  File "calibre/devices/mtp/base.py", line 24, in synchronizer
  File "calibre/devices/mtp/unix/driver.py", line 402, in put_file
calibre.devices.errors.DeviceError: Failed to upload file named: driveinfo.calibre to ('SD card',): 2:PTP Layer error 2002: send_file_object_info(): Could not send object info.
2:Error 2002: PTP General Error

Failed again.

[Quoth]

You can't change scanned folder/directories (or even see what they are) unless the initial connection completes without error.

Time to look at the SD card directly.

[Quoth]

Perhaps the flaw is that if a new folder/directory is added, by default it is scanned.

That seems to be the issue.

so backing up extra ones and deleting content so I can untick them.

[Quoth]

The extra folders had about 60 Gbyte

[Quoth]

calibre, version 7.10.0
ERROR: Error: Error communicating with device

Failed to create folder named PDF in ('Internal shared storage',) with error: 2:PTP Layer error 2009: LIBMTP_Create_Folder: Could not send object info.
2:Error 2009: PTP Invalid Object Handle

Traceback (most recent call last):
File "calibre/gui2/device.py", line 111, in run
File "calibre/gui2/device.py", line 645, in _upload_books
File "calibre/devices/mtp/driver.py", line 439, in upload_books
File "calibre/devices/mtp/driver.py", line 415, in ensure_parent
File "calibre/devices/mtp/base.py", line 24, in synchronizer
File "calibre/devices/mtp/unix/driver.py", line 379, in create_folder
calibre.devices.errors.DeviceError: Failed to create folder named PDF in ('Internal shared storage',) with error: 2:PTP Layer error 2009: LIBMTP_Create_Folder: Could not send object info.
2:Error 2009: PTP Invalid Object Handle

Dratted PDF folder already exists.

calibre, version 7.10.0
ERROR: Error: Error communicating with device

Failed to create folder named ebooks in ('Internal shared storage',) with error: 2:PTP Layer error 2009: LIBMTP_Create_Folder: Could not send object info.
2:Error 2009: PTP Invalid Object Handle

Traceback (most recent call last):
File "calibre/gui2/device.py", line 111, in run
File "calibre/gui2/device.py", line 645, in _upload_books
File "calibre/devices/mtp/driver.py", line 439, in upload_books
File "calibre/devices/mtp/driver.py", line 415, in ensure_parent
File "calibre/devices/mtp/base.py", line 24, in synchronizer
File "calibre/devices/mtp/unix/driver.py", line 379, in create_folder
calibre.devices.errors.DeviceError: Failed to create folder named ebooks in ('Internal shared storage',) with error: 2:PTP Layer error 2009: LIBMTP_Create_Folder: Could not send object info.
2:Error 2009: PTP Invalid Object Handle

So does ebooks.

This used to work

[Quoth]

Other issues
You can't untick internal storage and SDcard folders at the same time. You have to OK main Device dialog and restart Calibre.
I mounted a main filesystem folder and connected to it with the Android tablet disconnected. I put PDFs in PDF and epubs in ebooks in that folder, exited Calibre, "Continued" the gvfs-mtp-monitor and connected the tablet. Copied files to SD card. Double checked nothing on Internal Storage. Ejected and connected to Calibre. Calibre detected the new PDFs and ebooks but shows them in "Main".
Sending to "Main" actually sends to SDCard. Shows as sent to Main. Sending to Card A fails.

The MTP for actual transfer is about x2 speed of the SD Card mounted in a USB reader, no doubt because reader is a cheap USB 2.0 and tablet is maybe using USB 3.0 (USB-C cable to a PC USB-C USB 3.0 port).

[Quoth]

Originally Posted by Quoth

This used to work

So mysteriously I used to select Card A for the SD Card, but now the SD Card appears as and is used as "Main" on Calibre main interface and Send to Device, EXCEPT in the Device Config where you select/deselect folders and you see Internal and SD Card separately, and you can only change one or the other without OK and Restart Calibre. Odd.

[kovidgoyal]

MTP is *extremely* flaky I wouldnt even try to make it work with more than a few hundred books. I suggest you temporarily move the extra folder off and then create and empy one and exclude it and then move it back, then it wont be scanned.

[Quoth]

Originally Posted by kovidgoyal

MTP is *extremely* flaky I wouldnt even try to make it work with more than a few hundred books. I suggest you temporarily move the extra folder off and then create and empy one and exclude it and then move it back, then it wont be scanned.

That's exactly what I did.

1. So I wondered could the default be that no new folder/directory is scanned unless manually added?

2. Why does the SD Card now appear as main on "Device" Column and Send to Device? Selecting Card A fails.

3. Is it a bug you can only change scanned folders on the SD Card OR Internal Storage, but not both at the same time? Was it always like that?

Spoiler:

My experience of MTP is on Win98, Win2k, XP, Win7, Win 10, Redhat/CentOS, Rasparian, Debian, DSL, Suse, Mint and Ubuntu. Gadgets including Android 4.x to 13 plus others. Maybe before Zune and Android existed.

MTP is horrible

Actually, about 80G was files on all my computers and server and just a copy, so I deleted the contents. SD card in USB reader is 5 Mbps and SD card via Tablet MTP on Caja is 8 to 11 Mbps. Too slow for 80 Gbyte! I'll set it up copying back when I go to bed.

I'm glad my actual ereaders use USB Mass Storage.

[kovidgoyal]

Originally Posted by Quoth

That's exactly what I did.

1. So I wondered could the default be that no new folder/directory is scanned unless manually added?

No not really, that would be pretty confusing. One does not expect to have to manually whitelist folders after creating them on the device.

2. Why does the SD Card now appear as main on "Device" Column and Send to Device? Selecting Card A fails.

No idea, honestly this part of the code has probably bit rotted. Devices with multiple storages are rare nowadays and I personally dont possess a single such device to test things with.
[/quote]

3. Is it a bug you can only change scanned folders on the SD Card OR Internal Storage, but not both at the same time? Was it always like that?

Probably is a bug, but see above about the bitrot. This needs someone with such hardware to step up and maintain the code paths.

[Quoth]

Originally Posted by kovidgoyal

Probably is a bug, but see above about the bitrot. This needs someone with such hardware to step up and maintain the code paths.

Unfortunately I've given up programming and the postage Ireland -> India is probably crazy.
I'll probably use the Pocketbook app via Wifi on the tablet, Calibre Content server and "opds", as unlike MTP, it's perfect.
Did you see my Kobo Libra 2 Stress test? 6200+ files in one go! No disconnects, crashes or DB corruptions (connect 2nd time for extra metadata).

Give a shout if you or a friend is transiting Shannon Airport. I've a spare Likebook Mars 7.8″ Android eink (maybe 32G max SD card) and a spare Lenovo approx 10″ LCD tablet kind of thing, which works with at least 128G, so proably up to 1G.

All my newer stuff takes SD cards to at least 1G (Oppo phone and TCL nxtpaper 11 and Raspberry Pi4b) All my smart phones (Android 4.x to 8.x) and tablets I had since 2007 take SD card, except the Apple 4S someone gave me. I even had a Win10 32 bit tablet for a while with SD card. Gave it away, though it was slightly better with 64 bit Mint.

[Renate]

Why not explicitly sync directories?
You could put the commands in a script/batch/makefile.
I just say, "make poke5" or "make motog".

adbsync /acsu /s C:\Books sdcard/Books
adbsync /acsu /s C:\Ref sdcard/Ref
...
adbtime

(adbtime is yet another utility for syncing date/time on devices for which I don't use WiFi.)