Error after import of a lot of books

[LienRag]

Hi Mobile Readers.
I recently learned how to create and manage a Calibre Library on an external drive, so I did it and imported the full folder of books I have.
It took more than a day, and then once it finished I got the following errors :

calibre, version 6.13.0
AVERTISSEMENT : Erreurs pendant l'ajout: Il y a eu des problèmes lors de l'ajout de certains fichiers, cliquez "Afficher les détails" pour plus d'information


----------------------------------------------------------------------
Échec de la lecture des métadonnées depuis le fichier :
	/media/lienrag/Elements/Livres/Source/Stephen King/La ballade de la balle elastique & L'h  omme qui refusait de serrer la main (8837)/La ballade de la balle elastique & L'homme - Stephen King.epub
Avec l'erreur :
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/utils/ipc/pool.py", line 371, in worker_main
    result = func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/ebooks/metadata/worker.py", line 81, in read_metadata
    mi, opf, has_cover = serialize_metadata_for(paths, tdir, group_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/ebooks/metadata/worker.py", line 27, in serialize_metadata_for
    f.write(cdata)
OSError: [Errno 28] Aucun espace disponible sur le périphérique

There's 700 000 lines of error messages if I'm not mistaken, similar to the one above or to the one below :

----------------------------------------------------------------------
Échec à ajouter le livremetadata
	/media/lienrag/Elements/Livres/Bibliothèque calibre/John Schember/Guide de demarrage rapide (1)/metadata.opf
Avec l'erreur :
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/add.py", line 458, in add_book
    book_id = self.dbref().create_book_entry(mi, cover=cdata)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/db/legacy.py", line 341, in create_book_entry
    ret = self.new_api.create_book_entry(mi, cover=cover, add_duplicates=add_duplicates, force_id=force_id)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/db/cache.py", line 77, in call_func_with_lock
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/db/cache.py", line 1965, in create_book_entry
    self.backend.execute('INSERT INTO books(title, series_index, author_sort) VALUES (?, ?, ?)',
  File "/usr/lib/calibre/calibre/db/backend.py", line 1022, in execute
    return self.conn.cursor().execute(sql, bindings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/cursor.c", line 235, in resetcursor
apsw.CantOpenError: CantOpenError: unable to open database file

And now I have 4 000 books (on 36 000) without metadata, even without title or author.

What caused that ? And how can I solve it ?

There clearly was sufficient space on the external drive (more than 100 Gigabytes).
Is there necessary to have some also on my /home ? And/or / ? How much ?

[JSWolf]

According to the first code block, you've run out of disk space. I don't know if that's the drive that holds the library or the drive that hold the temp files. But you need to free up some space someplace.

[kovidgoyal]

You need to have space in /tmp. Some brain dead linux systems put /tmp in RAM. Either fix that or use the calibre env var to change its tmp directory to somewhere you actually have space.

https://manual.calibre-ebook.com/cus...ment-variables

[LienRag]

Thanks.
Any idea on how much space I need ?
And, is the current situation fixable or do I need to delete the library and re-import it ?

[kovidgoyal]

You can use the bulk metadata edit dialog to set metadata from ebook files in all books. But again if you are going to be operating on huge numbers of books dont put /tmp in RAM, preferably use the same disk as your library is on to host /tmp for best performance and robustness.

[LienRag]

Originally Posted by kovidgoyal

You can use the bulk metadata edit dialog to set metadata from ebook files in all books.

Again, thanks.
But how ?
Manually, for 4000 books of which I don't know the metadata, that's not realistic.
Or is there a way to have Calibre automatically recognize the metadata from the files that are on the disk, considering that I don't know which of the 36 k books are the 4 000 ones without metadata ?

[LienRag]

Also, how much space do I need ?
I have 12 Gb of RAM (though I'm not sure that all of it was free when I launched the import - maybe 6 GiB was) and I don't have that free space on may current / nor my /home.
Is it really a good idea to put the Calibre tmp directory on the external drive ? The external drive has no free space problem, that at least would be solved.

[LienRag]

(weird, apparently my first reply is waiting for moderation, but the second passed fine...)
(not a real problem since they addressed different questions, it just makes the "also" of the second reply seems a bit weird, ah too bad)

[JSWolf]

Originally Posted by LienRag

Also, how much space do I need ?
I have 12 Gb of RAM (though I'm not sure that all of it was free when I launched the import - maybe 6 GiB was) and I don't have that free space on may current / nor my /home.
Is it really a good idea to put the Calibre tmp directory on the external drive ? The external drive has no free space problem, that at least would be solved.

How much space you need is unknown. That's why a ram drive for your temp directory is a very bad idea. You need to have more space then you might need so you need to have your temp directory on a drive with plenty of space.

[theducks]

Do them in small batches. Groups of <20 allows an easy recover when things go south. If you don't, and have a fail, you lose the whole fetch job.

And WHY do they ALL need to get external metadata? Most books have the basics, and Calibre should have used that when it added (unless you set Add to use the Filename instead)

Be more selective.
Quality Check Plugin. It has many-many checks for missing 'this data value' Simply select the results and do your Get Metadata job. (Spin - repeat)

Now for the downside of BATCH get metadata: Some of it is WRONG Some of it is inconsistent (sometimes the Author or Publisher just changes the series name) Some sites are User submitted and those are all over the place book to book. Some have summaries others have Reviews. Some have a huge list of the Authors OTHER work. IMHO (Being a Joe Friday type) They all need TLC sip and not the fire hose application)
FWIW it took me almost a year to pound the metadata (Library data) into shape for 900+ paper books I track with Calibre

[theducks]

Originally Posted by LienRag

(weird, apparently my first reply is waiting for moderation, but the second passed fine...)
(not a real problem since they addressed different questions, it just makes the "also" of the second reply seems a bit weird, ah too bad)

Rapid fire posts by new users sometimes trip the anti-spam filters.
Recently, this has also happened to old users that had not posted recent (for whatever the definition of Recent was)

[kovidgoyal]

Do them all, and yes put /tmp on the same disk as your library. How much space you need will depend on how many books their sizes, etc.

Or redo the import this time with /tmp on the same drive as your library.

[DNSB]

If you are enthused about bulk fetching metadata, you are liable to find your IP address blocked. When I last played with this and , I found that Amazon allowed me to do 3-5 metadata fetches and then I started getting timeouts. After 15 or so minutes, I could fetch another block.

[LienRag]

Now Calibre won't start..
Even withe calibre --verbose, I only get back to the bash prompt, no information is given.

calibre-debug -g gives :

calibre-debug -g
calibre 6.13  embedded-python: False
Linux-6.1.0-23-amd64-x86_64-with-glibc2.36 Linux ('64bit', 'ELF')
('Linux', '6.1.0-23-amd64', '#1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15)')
Python 3.11.2
Interface language: fr
QPA platform: xcb
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 112.28608414239481 x 112.1103448275862
Using calibre Qt style: True

And then I'm back to the prompt.

[Quoth]

Mint 21.3 with Mate Desktop

calibre-debug -g
calibre 7.16  embedded-python: True
Linux-5.15.0-117-generic-x86_64-with-glibc2.35 Linux ('64bit', 'ELF')
('Linux', '5.15.0-117-generic', '#127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024')
Python 3.11.5
Interface language: None
EXE path: /opt/calibre/bin/calibre-debug
Successfully initialized third party plugins: DeACSM (0, 0, 16) && DeDRM (10, 0, 3) && Amazon.com Multiple Countries (1, 0, 0) && Annotations (1, 17, 13) && Author Book Count (2, 2, 3) && Barnes & Noble (1, 5, 6) && Comments Cleaner (1, 14, 0) && Copy Cover To Device (2, 0, 4) && Diaps Editing Toolbag (0, 4, 0) && EpubCheck (0, 2, 5) && EpubMerge (3, 1, 0) && EpubSplit (3, 8, 0) && Find Duplicates (1, 10, 9) && Generate Cover (2, 3, 4) && Goodreads (1, 8, 2) && Hyphenate This! (0, 1, 3) && KePub Input (3, 6, 6) && KePub Metadata Reader (3, 4, 3) && KePub Metadata Writer (3, 4, 3) && KePub Output (3, 6, 9) && KindleUnpack - The Plugin (0, 83, 1) && Kobo Books (1, 9, 2) && Kobo Utilities (2, 17, 1) && KoboTouchExtended (3, 6, 9) && Library Splitter (1, 0, 7) && Manage Series (1, 5, 2) && Modify ePub (1, 8, 4) && Obok DeDRM (10, 0, 3) && Open With (1, 8, 3) && Quality Check (1, 13, 14) && Quick Preferences (1, 7, 1) && Reading List (1, 15, 4) && Resize Cover (1, 2, 1) && ScrambleEbook (0, 6, 0) && Skoob Books (1, 5, 5) && Smashwords Metadata (1, 0, 2)
QPA platform: xcb
devicePixelRatio: 1.0
logicalDpi: 145.0 x 145.0
physicalDpi: 162.56 x 161.36470588235292
ApplicationPaletteChange event ignored
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Initializing db...
[1.38] db initialized
[1.38] Constructing main UI...
DEBUG:    0.0 Annotations plugin: Error loading libiMobileDevice. This hasn't worked for a while, and is blacklisted in calibre v3.
DEBUG:    0.0 Annotations plugin: Error is:  No module named 'calibre.devices.idevice'
DEBUG:    0.0 Annotations::reader_app_support.py - loading translations
DEBUG:    0.0 Annotations::common_utils.py - loading translations
DEBUG:    0.0 Annotations::appearance.py - loading translations
DEBUG:    0.0 Annotations::config.py - loading translations
DEBUG:    0.0 Annotations::annotated_books.py - loading translations
DEBUG:    0.1 Annotations::action.py - loading translations
DEBUG:    0.0 Kobo Utilities Configuration - CreateNewCustomColumn is supported
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 KoboUtilites::dialogs.py - loading translations
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 obok::utilities.py - loading translations
DEBUG:    0.0 obok::dialogs.py - loading translations
DEBUG:    0.0 obok::config.py - loading translations
DEBUG:    0.0 obok::action_err.py - loading translations
qt.webengine.QWebEngineView OK
qt.gui.imageio: libpng warning: iCCP: known incorrect sRGB profile
qt.gui.imageio: libpng warning: iCCP: cHRM chunk does not match sRGB
qt.gui.imageio: libpng warning: iCCP: known incorrect sRGB profile
[3.87] GUI main window shown
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
EpubMerge: DEBUG: 2024-08-08 17:41:05,318: calibre_plugins.epubmerge.epubmerge_plugin(156): macmenuhack file_path:/home/mikew/.config/calibre/plugins/fanficfare_macmenuhack.txt
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
DEBUG:    1.9 No Kobo Touch, Glo or Mini appears to be connected
DEBUG:    1.9 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
DEBUG:    1.9 KoboUtilities:set_toolbar_button_tooltip - start: text='None'
DEBUG:    1.9 KoboUtilities:set_toolbar_button_tooltip - setting to text='Utilities to use with Kobo ereaders

Driver: KoboTouch'
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
[3.97] main UI initialized...
[3.97] Started up in 3.97 seconds with 7924 books

This is interesting, as I don't use Fetch News

Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring