Mass Import Explosion

hobolibrary · 06-17-2010, 10:57 PM

I think I've managed to blow up a nice piece of software. My fiction library is in the neighborhood of 46k files and calibre managed to get as far as H/_Misc/ before hanging on the first pass (before actually importing).

The obvious work around would seem to be adding files in slightly smaller increments.

We won't even talk about the non-fiction portion of the library. I don't have a clue how many files are in that can of worms.

Anyhow, just thought I'd drop a note. I don't know if this is a known issue or just my own problem. OS is linux 2.6.32-22 under ubuntu 10.04. The behavior is a hang - no crash, no loop. Processes drop into sleep mode and hang out indefinitely. The import of Fiction/A seems to be going OK so far.

If you need/want any more info, just ask.

Oh, calibre version is 0.6.42. Maybe this is already fixed!

DoctorOhh · 06-17-2010, 11:21 PM

Quote:

Originally Posted by hobolibrary

The behavior is a hang - no crash, no loop. Processes drop into sleep mode and hang out indefinitely. The import of Fiction/A seems to be going OK so far.

In the past this has been associated with grabbing the metadata from PDF files. The solution in those cases was to turn off the PDF metadata reader plugin and possibly use the Read metadata only from filename option.

Quote:

Originally Posted by hobolibrary

Oh, calibre version is 0.6.42. Maybe this is already fixed!

You're right 0.6.42 is a old version to be asking for help on.

speakingtohe · 06-18-2010, 02:03 AM

I had a similiar problem with about 6 gig on Vista
I turned on the merge books with similiar filenames option and it worked (suggestion from Starson). Took about 36 hours on my poor dual core spare laptop, but it did it

DoctorOhh · 06-18-2010, 02:28 AM

Quote:

Originally Posted by speakingtohe

I had a similiar problem with about 6 gig on Vista
I turned on the merge books with similiar filenames option and it worked (suggestion from Starson). Took about 36 hours on my poor dual core spare laptop, but it did it

This is another reason the OP should consider upgrading as soon as 0.7.3 is out because the feature you describe wasn't implemented until 0.6.45 and if you're going to add that many books then you really need the more advanced merge features that are not in 0.6.42.

hobolibrary · 06-18-2010, 11:30 PM

Thanks for the feedback. I'm in the process of upgrading currently. I don't really have enough free time to import one section per night when I get home from work.

Any tips on the next hurdle I'm going to face? There are a lot of books that have multiple files per directory and a lot that are multiple books in the same directory. Is there an import mode that is smart enough to deal with all of these cases (or at least close enough to get the bulk of them sorted), or am I going to be spending free time for the foreseeable future trying to straighten out my library?

Thanks for any tips you can give me.

Update: 0.7.3 displays the same problem. Turning on merging may or may not have helped. The search pass gets a bit farther into the alphabet before exploding, but not much.

speakingtohe · 06-19-2010, 05:45 AM

You might try dividing them by format, ie search for *.epub, then copy them into a temporary folder, then *.pdf etc. This would give you several large groups
Helen

06-17-2010, 10:57 PM	#1
hobolibrary Junior Member Posts: 2 Karma: 10 Join Date: Jun 2010 Device: laptop	Mass Import Explosion I think I've managed to blow up a nice piece of software. My fiction library is in the neighborhood of 46k files and calibre managed to get as far as H/_Misc/ before hanging on the first pass (before actually importing). The obvious work around would seem to be adding files in slightly smaller increments. We won't even talk about the non-fiction portion of the library. I don't have a clue how many files are in that can of worms. Anyhow, just thought I'd drop a note. I don't know if this is a known issue or just my own problem. OS is linux 2.6.32-22 under ubuntu 10.04. The behavior is a hang - no crash, no loop. Processes drop into sleep mode and hang out indefinitely. The import of Fiction/A seems to be going OK so far. If you need/want any more info, just ask. Oh, calibre version is 0.6.42. Maybe this is already fixed! Last edited by hobolibrary; 06-17-2010 at 11:00 PM. Reason: more info

06-18-2010, 11:30 PM	#5
hobolibrary Junior Member Posts: 2 Karma: 10 Join Date: Jun 2010 Device: laptop	Thanks for the feedback. I'm in the process of upgrading currently. I don't really have enough free time to import one section per night when I get home from work. Any tips on the next hurdle I'm going to face? There are a lot of books that have multiple files per directory and a lot that are multiple books in the same directory. Is there an import mode that is smart enough to deal with all of these cases (or at least close enough to get the bulk of them sorted), or am I going to be spending free time for the foreseeable future trying to straighten out my library? Thanks for any tips you can give me. Update: 0.7.3 displays the same problem. Turning on merging may or may not have helped. The search pass gets a bit farther into the alphabet before exploding, but not much. Last edited by hobolibrary; 06-19-2010 at 12:25 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Will the iPad lead to an ebook explosion?	iantyndall	Apple Devices	13	10-12-2010 03:08 PM
Mass import of books metadata from an ASCII file: HELP NEEDED	LARdT	Calibre	4	07-08-2010 05:05 PM
Hello from Western Mass	Oldvjp1	Introduce Yourself	5	03-12-2009 01:46 PM
Mobipocket convert in mass?	sirdouglas	Reading and Management	18	10-04-2008 08:49 PM
Handheld Explosion To Come From Businesses Replacing Laptops	Bob Russell	Lounge	4	08-12-2005 09:56 PM

06-18-2010, 02:03 AM	#3
speakingtohe Wizard Posts: 4,812 Karma: 26912940 Join Date: Apr 2010 Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet	I had a similiar problem with about 6 gig on Vista I turned on the merge books with similiar filenames option and it worked (suggestion from Starson). Took about 36 hours on my poor dual core spare laptop, but it did it

06-19-2010, 05:45 AM	#6
speakingtohe Wizard Posts: 4,812 Karma: 26912940 Join Date: Apr 2010 Device: sony PRS-T1 and T3, Kobo Mini and Aura HD, Tablet	You might try dividing them by format, ie search for .epub, then copy them into a temporary folder, then .pdf etc. This would give you several large groups Helen

Advert

Advert