Not enough space to export/backup library. How to do partial backup?

[ilovejedd]

Originally Posted by droopy

Sirtel,
I created a new library and moved my CBZ and PDFs into it. It's about 11GB in size.
You advise zipping the new library folder up. How is that better than getting Calibre to "Export library" into the 1GB .data chunks?

Export library will backup the entire library. In my experience, it's also slower than a straight-up zip. I don't know if Calibre uses a proprietary format for archiving either.

If you only need to backup the library and not configuration data, I find it a lot simpler to just use regular tools to handle the library backup. You can still split library files anyway.

Personally, here's what I would do in your case (apart from buying another drive which is the simplest solution):

1. tag the books with the backup library name
2. for each backup set:
   1. create the backup library
   2. copy the books to the backup library
   3. create a catalog
   4. zip/tar the backup library
   5. move the zip and catalog to Google Drive
   6. delete the backup library
   7. delete only the formats for the books you just backed up from the original library but keep the metadata entries
   8. rinse and repeat

[droopy]

ILoveJedd, welcome to this thread.

Originally Posted by ilovejedd

Export library will backup the entire library.

Not sure if you had the chance to read the entire thread for context....SirTel's advice involves more than one library. For that reason, I'm afraid I'm not sure what you mean by "the entire library". Which library? Both the old and new libraries?

Originally Posted by ilovejedd

If you only need to backup the library and not configuration data, I find it a lot simpler to just use regular tools to handle the library backup. You can still split library files anyway.

Well, the plan is to do a partial backup. (If you could kindly read the rest of thread, it may help you understand what I mean.) So, though I may not need the configuration data backed up in the secondary library (the library that's gonna be backed up to the cloud), I do wish to back up the config data with the library that will remain on my laptop.

Originally Posted by ilovejedd

Personally, here's what I would do in your case (apart from buying another drive which is the simplest solution):

1. tag the books with the backup library name
2. for each backup set:
   1. create the backup library
   2. copy the books to the backup library
   3. create a catalog
   4. zip/tar the backup library
   5. move the zip and catalog to Google Drive
   6. delete the backup library
   7. delete only the formats for the books you just backed up from the original library but keep the metadata entries
   8. rinse and repeat

I noted that step g ("delete only the formats for the books, but keep the metadata entries") is rather different from SirTel's advice.
If I'm understanding you correctly, the advantage behind your plan is that I get to trim my Calibre library's size on my laptop while still knowing what books are in my entire collection. Is this correct? If so, I like this idea.
My worry/concern though with this idea is that, later on (whenever I'll reunite with my desktop computer with a larger storage capacity), it will be tedious to "merge" the "formats for the backed up books" back with the "format-less" "metadata entries". Correct me if I'm wrong.... merging the "formats" (the book files) with their corresponding metadata entries means opening each metadata entry up and drag and dropping the PDF or EPUB or CBZ or AZW into the upper-right "format" box. When I imagine having to do this for 6,000 titles, I have to say, "Sorry, but that's a whole lot of time". Am I misunderstanding the workflow for merging?

[ilovejedd]

Originally Posted by droopy

Not sure if you had the chance to read the entire thread for context....SirTel's advice involves more than one library. For that reason, I'm afraid I'm not sure what you mean by "the entire library". Which library? Both the old and new libraries?

You can choose the libraries you want to export.

Well, the plan is to do a partial backup. (If you could kindly read the rest of thread, it may help you understand what I mean.) So, though I may not need the configuration data backed up in the secondary library (the library that's gonna be backed up to the cloud), I do wish to back up the config data with the library that will remain on my laptop.

I'm aware. You'll note the steps I posted are for partial backups. I meant to type "You can still split zip files anyway" in case you prefer to upload smaller 1GB chunks for backup or something.

I noted that step g ("delete only the formats for the books, but kee the metadata entries") is rather different from SirTel's advice.
If I'm understanding you correctly, the advantage behind your plan is that I get to trim my Calibre library's size on my laptop while still knowing what books are in my entire collection. Is this correct? If so, I like this idea.
My worry/concern though with this idea is that, later on (whenever I'll reunite with my desktop computer with a larger storage capacity), it will be tedious to "merge" the "formats for the backed up books" back with the " format-less" "metadata entries". Correct me if I'm wrong.... merging the "formats" (the book files) with their corresponding metadata entries means opening each metadata entry up and drag and dropping the PDF or EPUB or CBZ or AZW into the upper-right "format" box. When I imagine having to do this for 6,000 titles, I have to say, "Sorry, but that's a whole lot of time". Am I misunderstanding the workflow for merging?

Yes, being able to know and search for offloaded books is the advantage with this method.

Iirc, there's an option in Calibre to merge similar/identical entries when copying books between libraries. No need to handle this manually.

Alternately, you could always filter the empty entries in the original by the backup tag, delete them, and then do the copy from the backup library.

[droopy]

Originally Posted by ilovejedd

Personally, here's what I would do in your case:

1. tag the books with the backup library name
2. for each backup set:
   1. create the backup library
   2. copy the books to the backup library
   3. create a catalog
   4. zip/tar the backup library
   5. move the zip and catalog to Google Drive
   6. delete the backup library
   7. delete only the formats for the books you just backed up from the original library but keep the metadata entries
   8. rinse and repeat

About Step G. What's the quickest/best way to bulk-delete the "formats" for a group of books at once?

Originally Posted by ilovejedd

I'm aware. You'll note the steps I posted are for partial backups. I meant to type "You can still split zip files anyway" in case you prefer to upload smaller 1GB chunks for backup or something.

Ok, thank you for clarifying and for taking the time to read the thread.

Originally Posted by ilovejedd

Yes, being able to know and search for offloaded books is the advantage with this method.

I love this. I was gonna create a CSV for each separate library. But your method is much nicer.

Originally Posted by ilovejedd

Iirc, there's an option in Calibre to merge similar/identical entries when copying books between libraries. No need to handle this manually.

Hopefully someone can say how to do this.

Originally Posted by ilovejedd

Alternately, you could always filter the empty entries in the original by the backup tag, delete them, and then do the copy from the backup library.

Ok. Good idea.

[Sirtel]

Originally Posted by droopy

What happens if I create a new column in one of the libraries (old or new) and merge the two libraries later on?

If there are more columns in the library you're copying TO, nothing will happen. Those column entries will just remain empty for the copied books.

If there are more columns in the library you're copying FROM, you'll get asked if you want to create the same columns in the other library. You can choose to create them or not, as you wish. The books get copied over in either case.

[theducks]

Originally Posted by droopy

What happens if I create a new column in one of the libraries (old or new) and merge the two libraries later on?

It asks ONE TIME (first time it discovers it (target) needs another field. I believe it will NOT copy any preferences that make use of that field)

[droopy]

Originally Posted by theducks

It asks ONE TIME (first time it discovers it (target) needs another field. I believe it will NOT copy any preferences that make use of that field)

Hi Ducks,
When I hear the word "preferences", I think of the "Settings" that every program has. But I think you don't mean that. What do you mean by preferences when it comes to columns?

[ilovejedd]

Originally Posted by droopy

About Step G. What's the quickest/best way to bulk-delete the "formats" for a group of books at once?

Right Click -> Remove books -> Remove all formats from selected books

I love this. I was gonna create a CSV for each separate library. But your method is much nicer.

It's still a good idea to create separate CSV files for each backup/archived library to keep on Google Drive. This just gives you more options for searching/browsing your archived books.

Hopefully someone can say how to do this.

Assuming menu options haven't changed significantly from 3.38 (on Windows 7)

Preferences -> Adding books -> Adding actions -> Auto-merge added books if they already exist ->
a. Ignore duplicate incoming formats
b. Overwrite existing duplicate formats

If you're copying from backup library to original for merging, option a. keeps the copy from original library (if it exists) and option b. keeps the copy from the backup library.


With that said, I don't know how metadata gets handled when merging. Tags, etc. are likely fine but long text such as comments might get appended. It might be better to just delete the empty entries from the original library first before copying from the backup library.

[JSWolf]

Originally Posted by droopy

Does this look good? Pls see attached screenshot

If the new library is to be synced to a cloud server, then don't do it. That can cause problems with your library.

[Sirtel]

Originally Posted by JSWolf

If the new library is to be synced to a cloud server, then don't do it. That can cause problems with your library.

The OP will zip the library folder and upload it as a zipped file.

[ownedbycats]

Originally Posted by JSWolf

If the new library is to be synced to a cloud server, then don't do it. That can cause problems with your library.

If you had read the thread, you would realize that the OP plans to archive the library folder into a compressed file and put that into the cloud.

[PeterT]

I can't help but think that attempting to zip things is pointless. Maybe a simple tar of the library makes more sense

[droopy]

Originally Posted by PeterT

I can't help but think that attempting to zip things is pointless. Maybe a simple tar of the library makes more sense

Pointless because the library's files is already quite compacted?

[PeterT]

Originally Posted by droopy

Pointless because the library's files is already quite compacted?

Correct

[ownedbycats]

Compressibility really depends on the format, doesn't it? Plain text/HTML files tend to compress pretty well. ePubs are technically just renamed zips, and trying to compress an already compressed file just tends to produce overhead. And PDF depends on how well-optimized it is in the first place.