Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 08-06-2024, 08:49 PM   #1
LienRag
Member
LienRag began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jul 2024
Device: many
Error after import of a lot of books

Hi Mobile Readers.
I recently learned how to create and manage a Calibre Library on an external drive, so I did it and imported the full folder of books I have.
It took more than a day, and then once it finished I got the following errors :

Code:
calibre, version 6.13.0
AVERTISSEMENT : Erreurs pendant l'ajout: Il y a eu des problèmes lors de l'ajout de certains fichiers, cliquez "Afficher les détails" pour plus d'information


----------------------------------------------------------------------
Échec de la lecture des métadonnées depuis le fichier :
	/media/lienrag/Elements/Livres/Source/Stephen King/La ballade de la balle elastique & L'h  omme qui refusait de serrer la main (8837)/La ballade de la balle elastique & L'homme - Stephen King.epub
Avec l'erreur :
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/utils/ipc/pool.py", line 371, in worker_main
    result = func(*job.args, **job.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/ebooks/metadata/worker.py", line 81, in read_metadata
    mi, opf, has_cover = serialize_metadata_for(paths, tdir, group_id)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/ebooks/metadata/worker.py", line 27, in serialize_metadata_for
    f.write(cdata)
OSError: [Errno 28] Aucun espace disponible sur le périphérique
There's 700 000 lines of error messages if I'm not mistaken, similar to the one above or to the one below :

Code:
----------------------------------------------------------------------
Échec à ajouter le livremetadata
	/media/lienrag/Elements/Livres/Bibliothèque calibre/John Schember/Guide de demarrage rapide (1)/metadata.opf
Avec l'erreur :
Traceback (most recent call last):
  File "/usr/lib/calibre/calibre/gui2/add.py", line 458, in add_book
    book_id = self.dbref().create_book_entry(mi, cover=cdata)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/db/legacy.py", line 341, in create_book_entry
    ret = self.new_api.create_book_entry(mi, cover=cover, add_duplicates=add_duplicates, force_id=force_id)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/db/cache.py", line 77, in call_func_with_lock
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/calibre/calibre/db/cache.py", line 1965, in create_book_entry
    self.backend.execute('INSERT INTO books(title, series_index, author_sort) VALUES (?, ?, ?)',
  File "/usr/lib/calibre/calibre/db/backend.py", line 1022, in execute
    return self.conn.cursor().execute(sql, bindings)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/cursor.c", line 235, in resetcursor
apsw.CantOpenError: CantOpenError: unable to open database file
And now I have 4 000 books (on 36 000) without metadata, even without title or author.

What caused that ? And how can I solve it ?

There clearly was sufficient space on the external drive (more than 100 Gigabytes).
Is there necessary to have some also on my /home ? And/or / ? How much ?
LienRag is offline   Reply With Quote
Old 08-06-2024, 09:35 PM   #2
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 76,951
Karma: 138256890
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
According to the first code block, you've run out of disk space. I don't know if that's the drive that holds the library or the drive that hold the temp files. But you need to free up some space someplace.
JSWolf is offline   Reply With Quote
Advert
Old 08-07-2024, 12:05 AM   #3
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 44,650
Karma: 24495948
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You need to have space in /tmp. Some brain dead linux systems put /tmp in RAM. Either fix that or use the calibre env var to change its tmp directory to somewhere you actually have space.

https://manual.calibre-ebook.com/cus...ment-variables
kovidgoyal is offline   Reply With Quote
Old 08-07-2024, 10:58 AM   #4
LienRag
Member
LienRag began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jul 2024
Device: many
Thanks.
Any idea on how much space I need ?
And, is the current situation fixable or do I need to delete the library and re-import it ?
LienRag is offline   Reply With Quote
Old 08-07-2024, 11:11 AM   #5
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 44,650
Karma: 24495948
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
You can use the bulk metadata edit dialog to set metadata from ebook files in all books. But again if you are going to be operating on huge numbers of books dont put /tmp in RAM, preferably use the same disk as your library is on to host /tmp for best performance and robustness.
kovidgoyal is offline   Reply With Quote
Advert
Old 08-07-2024, 11:55 AM   #6
LienRag
Member
LienRag began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jul 2024
Device: many
Quote:
Originally Posted by kovidgoyal View Post
You can use the bulk metadata edit dialog to set metadata from ebook files in all books.
Again, thanks.
But how ?
Manually, for 4000 books of which I don't know the metadata, that's not realistic.
Or is there a way to have Calibre automatically recognize the metadata from the files that are on the disk, considering that I don't know which of the 36 k books are the 4 000 ones without metadata ?
LienRag is offline   Reply With Quote
Old 08-07-2024, 11:59 AM   #7
LienRag
Member
LienRag began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jul 2024
Device: many
Also, how much space do I need ?
I have 12 Gb of RAM (though I'm not sure that all of it was free when I launched the import - maybe 6 GiB was) and I don't have that free space on may current / nor my /home.
Is it really a good idea to put the Calibre tmp directory on the external drive ? The external drive has no free space problem, that at least would be solved.
LienRag is offline   Reply With Quote
Old 08-07-2024, 12:01 PM   #8
LienRag
Member
LienRag began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jul 2024
Device: many
(weird, apparently my first reply is waiting for moderation, but the second passed fine...)
(not a real problem since they addressed different questions, it just makes the "also" of the second reply seems a bit weird, ah too bad)
LienRag is offline   Reply With Quote
Old 08-07-2024, 12:05 PM   #9
JSWolf
Resident Curmudgeon
JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.JSWolf ought to be getting tired of karma fortunes by now.
 
JSWolf's Avatar
 
Posts: 76,951
Karma: 138256890
Join Date: Nov 2006
Location: Roslindale, Massachusetts
Device: Kobo Libra 2, Kobo Aura H2O, PRS-650, PRS-T1, nook STR, PW3
Quote:
Originally Posted by LienRag View Post
Also, how much space do I need ?
I have 12 Gb of RAM (though I'm not sure that all of it was free when I launched the import - maybe 6 GiB was) and I don't have that free space on may current / nor my /home.
Is it really a good idea to put the Calibre tmp directory on the external drive ? The external drive has no free space problem, that at least would be solved.
How much space you need is unknown. That's why a ram drive for your temp directory is a very bad idea. You need to have more space then you might need so you need to have your temp directory on a drive with plenty of space.
JSWolf is offline   Reply With Quote
Old 08-07-2024, 12:29 PM   #10
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 30,506
Karma: 58055868
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Do them in small batches. Groups of <20 allows an easy recover when things go south. If you don't, and have a fail, you lose the whole fetch job.

And WHY do they ALL need to get external metadata? Most books have the basics, and Calibre should have used that when it added (unless you set Add to use the Filename instead)

Be more selective.
Quality Check Plugin. It has many-many checks for missing 'this data value' Simply select the results and do your Get Metadata job. (Spin - repeat)

Now for the downside of BATCH get metadata: Some of it is WRONG Some of it is inconsistent (sometimes the Author or Publisher just changes the series name) Some sites are User submitted and those are all over the place book to book. Some have summaries others have Reviews. Some have a huge list of the Authors OTHER work. IMHO (Being a Joe Friday type) They all need TLC sip and not the fire hose application)
FWIW it took me almost a year to pound the metadata (Library data) into shape for 900+ paper books I track with Calibre
theducks is online now   Reply With Quote
Old 08-07-2024, 12:34 PM   #11
theducks
Well trained by Cats
theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.theducks ought to be getting tired of karma fortunes by now.
 
theducks's Avatar
 
Posts: 30,506
Karma: 58055868
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
Quote:
Originally Posted by LienRag View Post
(weird, apparently my first reply is waiting for moderation, but the second passed fine...)
(not a real problem since they addressed different questions, it just makes the "also" of the second reply seems a bit weird, ah too bad)
Rapid fire posts by new users sometimes trip the anti-spam filters.
Recently, this has also happened to old users that had not posted recent (for whatever the definition of Recent was)
theducks is online now   Reply With Quote
Old 08-07-2024, 12:46 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 44,650
Karma: 24495948
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
Do them all, and yes put /tmp on the same disk as your library. How much space you need will depend on how many books their sizes, etc.

Or redo the import this time with /tmp on the same drive as your library.
kovidgoyal is offline   Reply With Quote
Old 08-07-2024, 07:08 PM   #13
DNSB
Bibliophagist
DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.DNSB ought to be getting tired of karma fortunes by now.
 
DNSB's Avatar
 
Posts: 41,272
Karma: 158182188
Join Date: Jul 2010
Location: Vancouver
Device: Kobo Sage, Libra Colour, Lenovo M8 FHD, Paperwhite 4, Tolino epos
If you are enthused about bulk fetching metadata, you are liable to find your IP address blocked. When I last played with this and , I found that Amazon allowed me to do 3-5 metadata fetches and then I started getting timeouts. After 15 or so minutes, I could fetch another block.
DNSB is offline   Reply With Quote
Old 08-08-2024, 11:16 AM   #14
LienRag
Member
LienRag began at the beginning.
 
Posts: 11
Karma: 10
Join Date: Jul 2024
Device: many
Now Calibre won't start..
Even withe calibre --verbose, I only get back to the bash prompt, no information is given.

calibre-debug -g gives :

Code:
calibre-debug -g
calibre 6.13  embedded-python: False
Linux-6.1.0-23-amd64-x86_64-with-glibc2.36 Linux ('64bit', 'ELF')
('Linux', '6.1.0-23-amd64', '#1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15)')
Python 3.11.2
Interface language: fr
QPA platform: xcb
devicePixelRatio: 1.0
logicalDpi: 96.0 x 96.0
physicalDpi: 112.28608414239481 x 112.1103448275862
Using calibre Qt style: True
And then I'm back to the prompt.
LienRag is offline   Reply With Quote
Old 08-08-2024, 01:45 PM   #15
Quoth
Reading till the spring
Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.Quoth ought to be getting tired of karma fortunes by now.
 
Quoth's Avatar
 
Posts: 12,518
Karma: 94058919
Join Date: Jun 2017
Location: Ireland
Device: All 4 Kinds: epub eink, Kindle, android eink, NxtPaper
Mint 21.3 with Mate Desktop
Code:
calibre-debug -g
calibre 7.16  embedded-python: True
Linux-5.15.0-117-generic-x86_64-with-glibc2.35 Linux ('64bit', 'ELF')
('Linux', '5.15.0-117-generic', '#127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024')
Python 3.11.5
Interface language: None
EXE path: /opt/calibre/bin/calibre-debug
Successfully initialized third party plugins: DeACSM (0, 0, 16) && DeDRM (10, 0, 3) && Amazon.com Multiple Countries (1, 0, 0) && Annotations (1, 17, 13) && Author Book Count (2, 2, 3) && Barnes & Noble (1, 5, 6) && Comments Cleaner (1, 14, 0) && Copy Cover To Device (2, 0, 4) && Diaps Editing Toolbag (0, 4, 0) && EpubCheck (0, 2, 5) && EpubMerge (3, 1, 0) && EpubSplit (3, 8, 0) && Find Duplicates (1, 10, 9) && Generate Cover (2, 3, 4) && Goodreads (1, 8, 2) && Hyphenate This! (0, 1, 3) && KePub Input (3, 6, 6) && KePub Metadata Reader (3, 4, 3) && KePub Metadata Writer (3, 4, 3) && KePub Output (3, 6, 9) && KindleUnpack - The Plugin (0, 83, 1) && Kobo Books (1, 9, 2) && Kobo Utilities (2, 17, 1) && KoboTouchExtended (3, 6, 9) && Library Splitter (1, 0, 7) && Manage Series (1, 5, 2) && Modify ePub (1, 8, 4) && Obok DeDRM (10, 0, 3) && Open With (1, 8, 3) && Quality Check (1, 13, 14) && Quick Preferences (1, 7, 1) && Reading List (1, 15, 4) && Resize Cover (1, 2, 1) && ScrambleEbook (0, 6, 0) && Skoob Books (1, 5, 5) && Smashwords Metadata (1, 0, 2)
QPA platform: xcb
devicePixelRatio: 1.0
logicalDpi: 145.0 x 145.0
physicalDpi: 162.56 x 161.36470588235292
ApplicationPaletteChange event ignored
Using calibre Qt style: True
[0.00] Starting up...
[0.00] Initializing db...
[1.38] db initialized
[1.38] Constructing main UI...
DEBUG:    0.0 Annotations plugin: Error loading libiMobileDevice. This hasn't worked for a while, and is blacklisted in calibre v3.
DEBUG:    0.0 Annotations plugin: Error is:  No module named 'calibre.devices.idevice'
DEBUG:    0.0 Annotations::reader_app_support.py - loading translations
DEBUG:    0.0 Annotations::common_utils.py - loading translations
DEBUG:    0.0 Annotations::appearance.py - loading translations
DEBUG:    0.0 Annotations::config.py - loading translations
DEBUG:    0.0 Annotations::annotated_books.py - loading translations
DEBUG:    0.1 Annotations::action.py - loading translations
DEBUG:    0.0 Kobo Utilities Configuration - CreateNewCustomColumn is supported
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 KoboUtilites::dialogs.py - loading translations
DEBUG:    0.0 KoboUtilites::action.py - loading translations
DEBUG:    0.0 obok::utilities.py - loading translations
DEBUG:    0.0 obok::dialogs.py - loading translations
DEBUG:    0.0 obok::config.py - loading translations
DEBUG:    0.0 obok::action_err.py - loading translations
qt.webengine.QWebEngineView OK
qt.gui.imageio: libpng warning: iCCP: known incorrect sRGB profile
qt.gui.imageio: libpng warning: iCCP: cHRM chunk does not match sRGB
qt.gui.imageio: libpng warning: iCCP: known incorrect sRGB profile
[3.87] GUI main window shown
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
EpubMerge: DEBUG: 2024-08-08 17:41:05,318: calibre_plugins.epubmerge.epubmerge_plugin(156): macmenuhack file_path:/home/mikew/.config/calibre/plugins/fanficfare_macmenuhack.txt
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
DEBUG:    1.9 No Kobo Touch, Glo or Mini appears to be connected
DEBUG:    1.9 rebuild_menus - self.supports_ratings=None, self.supports_tiles=None
DEBUG:    1.9 KoboUtilities:set_toolbar_button_tooltip - start: text='None'
DEBUG:    1.9 KoboUtilities:set_toolbar_button_tooltip - setting to text='Utilities to use with Kobo ereaders

Driver: KoboTouch'
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
[3.97] main UI initialized...
[3.97] Started up in 3.97 seconds with 7924 books
This is interesting, as I don't use Fetch News
Code:
Key 'Ctrl+Shift+N' for shortcut Browse notes is already used by Fetch news, ignoring
Quoth is offline   Reply With Quote
Reply


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
That's a lot of books! wodin Lounge 0 03-29-2017 08:19 PM
Export/Import all Calibre data - import error IanKBrown Calibre 5 02-15-2017 09:32 PM
Import on Filename changes mid import due to book error Vortex Calibre 3 12-05-2014 12:49 PM
Best way to automatically add a lot of books foxbooks Library Management 3 05-31-2012 11:41 AM
Import failed Error:404 when attempting to import from Calibre to Stanza dvond Apple Devices 0 05-13-2011 04:00 PM


All times are GMT -4. The time now is 09:44 PM.


MobileRead.com is a privately owned, operated and funded community.