Corrupted OPF Files in large library

unboggling · 05-29-2011, 02:48 PM

My database is corrupted. When I save books (writing metadata to opf) then import into a different library with the same structure, most of the data in user-defined fields doesn't come with it. Restore didn't help, probably made it worse.

This was my fault. I did things I shouldn't do. Changed fields in some libraries and not others. Copied books between libraries. Messed with permissions. Accessed the same library from 2 different computers thru my network while using same username. Moved library from ext HD to ext HD, used it from external HD from 2 computers. Migrated the library from computer1 to computer2 to computer3 without checking everything in between. Lost backups to stupidity. So a concatenation of things probably caused the problem(s).

I have the library on 1 computer now, changed all the permissions to "everyone rw" with one username as owner. But that didn't help. I'm presently transferring data from the user-defined fields to the Tags field, which seems to work for preserving the data in Tags thru the save and import. This will keep me busy at least several days and I'm still not sure all the records will save/import properly. It's a large library. When that's done I plan to import into a new library using just the calibre-supplied fields and work it from there.

Meanwhile, anyone have advice? Is there a relatively simple way to delete all the opf files in a copy of the library then regenerate them from metadata db? Though I'm not sure if/how corrupted metadata db is. I'm getting desperate. The idea of importing into a new database by file structure makes me shudder in dread at redoing so much work.

Thanks for any help.

-alan

theducks · 05-29-2011, 02:55 PM

Quote:

Originally Posted by unboggling

Meanwhile, anyone have advice? Is there a relatively simple way to delete all the opf files in a copy of the library then regenerate them from metadata db? Though I'm not sure if/how corrupted metadata db is. I'm getting desperate. The idea of importing into a new database by file structure makes me shudder in dread at redoing so much work.

Thanks for any help.

-alan

Switch Library to the Source Library

Select books
Right-click: Copy Library into Main (Merge) Library

is an easy way. Al Main needs is a super-set of the fields of the libraries to be added.

kovidgoyal · 05-29-2011, 02:57 PM

You can cause calibre to regenerate all OPF files from the db via the Library Maintenance menu (Start backup of metadata)

You can also restore the db from the opf files via that menu (Restore database)

unboggling · 05-29-2011, 03:12 PM

Thanks. I'll try that. If I understand correctly, I create a new library, build the fields manually and then copy books from old library to new library. Won't that just copy over the same bad opf files?

unboggling · 05-29-2011, 03:15 PM

kovid, thanks. The other day I did a restore, hoping it would help, but it seemed to make things worse. if the opfs were bad, then that made the db bad too.

unboggling · 05-30-2011, 12:27 AM

I moved all metadata from custom fields to the Tag field then deleted the custom fields. That seems to have fixed the problem of custom field data missing after saving/adding since there's no custom field data anymore. BTW ducks, previously the copy-to-other-library command apparently just copied over the same opf files because that didn't fix the problem, but thanks for the suggestion anyway. What apparently helped was deleting the custom fields, or more likely changing the metadata in all of the records.

Not sure if it's related or not but I have a continuing problem with losing books during saving/adding, approx a tenth of a one percent of total. Automerge is off and I always import title dupes. The library contains only epubs and no other formats. I tracked down two of the missing books (not easy) and only one was a title dupe with each record with its own epub format and differing metadata (eg, one tagged "acquire' and the other tagged "goodFormat." Those and the other missing book had no distinct problems and good file permissions. I've no idea what's causing the missing books.

I created a new library today and noticed that its metadata.db file has permissions for user rw, wheel r, everyone r. The metadata.db file in the older library has user rw and everyone rw -- after presumably wheel getting blitzed by my doing a global permission reset the other day on all my library files. I've seen wheel in the past and assumed it had to do with custom application privileges, but have no idea of its significance.

Thanks for any clues regarding missing books and/or wheel.

05-29-2011, 02:48 PM	#1
unboggling Wizard Posts: 1,065 Karma: 858115 Join Date: Jan 2011 Device: Kobo Clara, Kindle Paperwhite 10	Corrupted OPF Files in large library My database is corrupted. When I save books (writing metadata to opf) then import into a different library with the same structure, most of the data in user-defined fields doesn't come with it. Restore didn't help, probably made it worse. This was my fault. I did things I shouldn't do. Changed fields in some libraries and not others. Copied books between libraries. Messed with permissions. Accessed the same library from 2 different computers thru my network while using same username. Moved library from ext HD to ext HD, used it from external HD from 2 computers. Migrated the library from computer1 to computer2 to computer3 without checking everything in between. Lost backups to stupidity. So a concatenation of things probably caused the problem(s). I have the library on 1 computer now, changed all the permissions to "everyone rw" with one username as owner. But that didn't help. I'm presently transferring data from the user-defined fields to the Tags field, which seems to work for preserving the data in Tags thru the save and import. This will keep me busy at least several days and I'm still not sure all the records will save/import properly. It's a large library. When that's done I plan to import into a new library using just the calibre-supplied fields and work it from there. Meanwhile, anyone have advice? Is there a relatively simple way to delete all the opf files in a copy of the library then regenerate them from metadata db? Though I'm not sure if/how corrupted metadata db is. I'm getting desperate. The idea of importing into a new database by file structure makes me shudder in dread at redoing so much work. Thanks for any help. -alan

05-30-2011, 12:27 AM	#6
unboggling Wizard Posts: 1,065 Karma: 858115 Join Date: Jan 2011 Device: Kobo Clara, Kindle Paperwhite 10	I moved all metadata from custom fields to the Tag field then deleted the custom fields. That seems to have fixed the problem of custom field data missing after saving/adding since there's no custom field data anymore. BTW ducks, previously the copy-to-other-library command apparently just copied over the same opf files because that didn't fix the problem, but thanks for the suggestion anyway. What apparently helped was deleting the custom fields, or more likely changing the metadata in all of the records. Not sure if it's related or not but I have a continuing problem with losing books during saving/adding, approx a tenth of a one percent of total. Automerge is off and I always import title dupes. The library contains only epubs and no other formats. I tracked down two of the missing books (not easy) and only one was a title dupe with each record with its own epub format and differing metadata (eg, one tagged "acquire' and the other tagged "goodFormat." Those and the other missing book had no distinct problems and good file permissions. I've no idea what's causing the missing books. I created a new library today and noticed that its metadata.db file has permissions for user rw, wheel r, everyone r. The metadata.db file in the older library has user rw and everyone rw -- after presumably wheel getting blitzed by my doing a global permission reset the other day on all my library files. I've seen wheel in the past and assumed it had to do with custom application privileges, but have no idea of its significance. Thanks for any clues regarding missing books and/or wheel. Last edited by unboggling; 05-30-2011 at 12:36 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
903 Corrupted library	Raisan	PocketBook	1	04-09-2011 09:20 AM
Multiple "Copy to Library" not copying covers/opf files over?	Trickery	Calibre	9	10-08-2010 01:18 PM
Corrupted files	speakingtohe	Calibre	2	08-20-2010 04:31 PM
Has anyone else had problems with corrupted files?	Purple Lady	Onyx Boox	4	03-27-2010 03:46 AM
Library corrupted	rozie123	Sony Reader	1	05-26-2009 07:02 AM

05-29-2011, 02:57 PM	#3
kovidgoyal creator of calibre Posts: 44,151 Karma: 22670164 Join Date: Oct 2006 Location: Mumbai, India Device: Various	You can cause calibre to regenerate all OPF files from the db via the Library Maintenance menu (Start backup of metadata) You can also restore the db from the opf files via that menu (Restore database)

05-29-2011, 03:12 PM	#4
unboggling Wizard Posts: 1,065 Karma: 858115 Join Date: Jan 2011 Device: Kobo Clara, Kindle Paperwhite 10	Thanks. I'll try that. If I understand correctly, I create a new library, build the fields manually and then copy books from old library to new library. Won't that just copy over the same bad opf files?

05-29-2011, 03:15 PM	#5
unboggling Wizard Posts: 1,065 Karma: 858115 Join Date: Jan 2011 Device: Kobo Clara, Kindle Paperwhite 10	kovid, thanks. The other day I did a restore, hoping it would help, but it seemed to make things worse. if the opfs were bad, then that made the db bad too.