[Old Thread] Folder Structure

[gplauche]

I know the FAQ explains why Calibre dictates folder structure so rigidly, but I still think it would be nice if Calibre was a little more flexible here.

I'd like to be able to categorize my document files like so: Fiction, Nonfiction, Comics, etc., then subcategorize within this structure by author.

Two reasons why this is useful:

1) It makes it easier to share batches of files with others. If I want to transfer all of my fiction, but nothing else, to someone's memory stick/computer/whatever then all I have to do is move the Fiction folder. I don't have to hunt through my Calibre Library folder trying to sort out which Author folder is fiction and which isn't, to say nothing of the problem of an author who writes both fiction and nonfiction.

2) I use other programs like Comix and Mendeley Desktop to manage (or view) my document library and well. It's easier to add comics to Comix when they are all in one folder. For my use of Mendeley Desktop, a reference/citation program with iTunes-style library functionality, I'd prefer to restrict it to my nonfiction documents because I don't often have need to cite science fiction, fantasy, and comics in my academic writings. Having all of my documents lumped together in Mendeley, which is what I currently have to do if I don't want to maintain two copies of all my nonfiction document files, clutters up Mendeley's library unnecessarily for my purposes.

Also, Mendeley gives me a variety of choices for how to organize my folders and name my files while also providing the tag, search, and sort functions Kovid touts as the advantage of Calibre's current approach. The point here is that it's not a case of either-or. A program like Calibre can continue to offer its tag, search, and sort interface while also providing Mendeley/iTunes/Songbird-style automatic folder organization according to options chosen by the user.

If Calibre could just watch and manage files in multiple folders (instead of one, as now) it would have most of the functionality and compatibility with other programs like Mendeley that I'm looking for. I could set Calibre to watch and manage all of my document folders while setting Mendeley to watch and manage only my nonfiction document folder (or, to go a subfolder down, only my academic nonfiction folder). Flexible options for folder structure and file name would also be nice but not as important.

[itimpi]

There are plenty of threads explaining why this is just not going to happen with Calibre.

[DoctorOhh]

Originally Posted by gplauche

1) It makes it easier to share batches of files with others. If I want to transfer all of my fiction, but nothing else, to someone's memory stick/computer/whatever then all I have to do is move the Fiction folder. I don't have to hunt through my Calibre Library folder trying to sort out which Author folder is fiction and which isn't, to say nothing of the problem of an author who writes both fiction and nonfiction.

It is easier to do what you are asking with Calibre then to do it with the best laid out file structure. Read the manual or other threads for info.

Originally Posted by itimpi

There are plenty of threads explaining why this is just not going to happen with Calibre.

[tulsa]

gplauche, I thought the same thing when I first started using Calibre, but as I've used it now for quite a while, I wouldn't have it any other way than the way it is.

I make extensive use of the tagging system. All my science fiction books are tagged science fiction, all my fantasy books are fantasy, etc. I can search/filter by tags and now you can even save a complex search.

and yes... ditto.

[gplauche]

Originally Posted by tulsa

gplauche, I thought the same thing when I first started using Calibre, but as I've used it now for quite a while, I wouldn't have it any other way than the way it is.

I make extensive use of the tagging system. All my science fiction books are tagged science fiction, all my fantasy books are fantasy, etc. I can search/filter by tags and now you can even save a complex search.

and yes... ditto.

That's not my issue. I like that aspect of Calibre very much. It's just that the way Calibre watches, manages and organizes folders is incompatible with my use of Mendeley Desktop as a research, bibliographic, and citation tool. I want to use Calibre for all of my pdfs and ebooks, because it does important things Mendeley does not (e.g., send to device, conversion to epub, etc.). But I want to use Mendeley just for my nonfiction, or more specifically academic-related nonfiction, documents; it offers full bibliographic information for multiple academic types of publications, citations in multiple formats, a web interface, syncing with an online library/database, and social networking.

To use both Calibre and Mendeley my only options currently are: 1) have them watch and manage the same general documents folder, which has the negative consequence of all of my pdfs and ebooks, not just (academic) nonfiction), cluttering up Mendeley's library and limiting my use of Mendeley's more flexible folder structure options, or 2) maintaining two copies of all my (academic) nonfiction document files and having Calibre and Mendeley watch and manage completely separate folders. Neither option is optimum. I don't have a big budget for file storage space.

If Calibre could just watch and manage multiple folders, that would give me most of the functionality I want. The other stuff is less important. I'm not sure the Save to Disk feature is as convenient as simply CTRL+C/CTRL+V or Drag & Drop of folders from one tab/window to another but, like I said, this aspect of folder structure is less important to me.

Having glanced through other threads on the topic, let me say I'm not trying to criticize or insult Kovid or anyone else. I'm just making a feature request with what I think is at least one novel reason for it: side-by-side compatibility with other programs that do similar but different and necessary things.

[Knipfty]

Why not as the makers of Mendeley to work with Calibre's file structure? In essence, you are asking the maker of Calibre to work with theirs.

If Calibre could just watch and manage multiple folders, that would give me most of the functionality I want.

You can point calibre to different libraries in the preferences. But you still get calibre's file structure.

[tulsa]

I've thought about a "watch folders" feature before.
It doesn't work with Calibre because Calibre generates its own file/folder structure.
When I download a new ebook, I put it into a temporary folder. Then I drag it into Calibre for importing. Then I delete the book from the temporary folder.
Then I add my tags and metadata.

IMO it is much easier to just let Calibre take care of the underlying file/folder organizing. Once a week, I zip the entire Calibre library folder and back it up to CD.
That way, in case Calibre goes crazy and does something weird or my computer crashes, I have a recent full backup.

[kovidgoyal]

1) Save to disk lets you do this as easily as copying a folder

2) The reason for the folder structure is not to support tagging/search/sort. It is for the following two reasons:

a) calibre supports multiple copies of the same logical book in different formats. That means each logical book needs its own folder.

b) Robustness.

- 1) In my experience when programs allow you to store files in arbitrary locations, those locations inevitably change over time and the links between the database and the locations change. I'm not saying you, personally will do this, I'm saying for a program with as large a user base as calibre, a significant fraction of calibre's users will, and this will create a support nightmare for me.

- 2) Putting files into folders is a way of storing metadata using a filesystem. This is less than optimal for various reasons I have detailed elsewhere and I want to discourage the practice.

- 3) By having the cataloguing program manage the folder structure, your folders will always match the updated metadata for the books, making it easy to browse the folder structure.

These are the advantages of the calibre approach. The disadvantages are:

1) Performance impact on windows. Windows, especially older version can't handle having too many nodes in a single folder. This will be addressed by giving users of large libraries the option of having a folder structure optimized for large collections, in the future.

2) Interoperability with other programs. You really shouldn't be using two catalog programs to manage the same set of books. And calibre comes with a comprehensive set of command line tools to allows users to use just the conversion/news download features if they want to stick with their existing catalog scheme.

In my opinion, the advantages outweigh the disadvantages.

[gplauche]

Originally Posted by Knipfty

Why not as the makers of Mendeley to work with Calibre's file structure? In essence, you are asking the maker of Calibre to work with theirs.

How? Mendeley has the more flexible folder/file structure. It can watch multiple folders as well as manage one general folder and offers flexibility in organizing the subfolders within it.

Calibre copies files from their existing folder structure and creates its own: author/title. Mendeley can organize its folder structure by author/title as well. This isn't an issue; although more flexibility in Calibre would be nice, it's not as important to me.

But how can the Mendeley developers take Calibre's author/title folder structure and pick out only the (academic) nonfiction files from it to include in its library? This is why the ability to watch AND manage/organize multiple folders is important. I'm going to ask the Mendeley developers to add the ability to manage more than one folder to Mendeley Desktop as well. But given the natures of Calibre and Mendeley, the former needs this functionality more because it is a more general-purpose program.

Keeping duplicate files might be my best option right now, but it is not optimum.

Originally Posted by Knipfty

You can point calibre to different libraries in the preferences. But you still get calibre's file structure.

Unless I'm mistaken, Calibre can only watch/manage one folder at a time.

[gplauche]

Thanks for replying to yet another folder structure thread.

Originally Posted by kovidgoyal

1) Save to disk lets you do this as easily as copying a folder

I haven't tried it yet, admittedly. It lets you save multiple books/files at once?

Originally Posted by kovidgoyal

2) The reason for the folder structure is not to support tagging/search/sort. It is for the following two reasons:

a) calibre supports multiple copies of the same logical book in different formats. That means each logical book needs its own folder.

I realize this.

Originally Posted by kovidgoyal

b) Robustness.

- 1) In my experience when programs allow you to store files in arbitrary locations, those locations inevitably change over time and the links between the database and the locations change. I'm not saying you, personally will do this, I'm saying for a program with as large a user base as calibre, a significant fraction of calibre's users will, and this will create a support nightmare for me.

I'm not a programmer, so maybe that's why I'm not seeing why it would be a problem if Calibre was able to watch and manage/organize multiple folders while keeping its author/title/logicalbook subfolder structure within these.(especially in Linux).

Originally Posted by kovidgoyal

- 2) Putting files into folders is a way of storing metadata using a filesystem. This is less than optimal for various reasons I have detailed elsewhere and I want to discourage the practice.

I'm not arguing with this. I can see how tag, search, and sort is generally superior/optimal for finding the ebook you're looking for.

Originally Posted by kovidgoyal

- 3) By having the cataloguing program manage the folder structure, your folders will always match the updated metadata for the books, making it easy to browse the folder structure.

This I don't agree with. It can be much harder to search for something in a folder in which all the files are organized by author then title. Having your files organized in folders by categories and subcategories like fiction or nonfiction, academic disciplines, fiction genre, etc., can things down considerably and make searching through author names much quicker. But this is not a major issue. One of the main reasons I want to use Calibre is its tag, search and sort features. This is generally superior to searching through folders.

Originally Posted by kovidgoyal

The disadvantages are:

2) Interoperability with other programs. You really shouldn't be using two catalog programs to manage the same set of books. And calibre comes with a comprehensive set of command line tools to allows users to use just the conversion/news download features if they want to stick with their existing catalog scheme.

In my opinion, the advantages outweigh the disadvantages.

I don't consider maintaining duplicate files in separate file structures to be optimal. If nothing else, it wastes storage space. Both Calibre and Mendeley perform some very useful functions that the other does not. I need them both. Mendeley does not handle sending files to external devices like e-readers and smartphones at all. It doesn't do format conversion. Calibre isn't a dedicated research, bibliography, and citation tool with syncing to an online library/database and social networking. That's fine. But I don't see why two or more catalog programs can't manage the same set of files so long as they play nice with each other's file structure and can watch/manage/organize multiple folders.

Ultimately your opinion is the one that counts here, of course. All I can do is try to persuade you. If I haven't persuaded you with this message, I'm probably not going to. Thanks again. Calibre is great at what it does.

[animedude01]

I think part of the problem is that Calibre should be looked at as a black box. You should not be manually fiddling with the folder structure, but using the interface. If you move/rename any books in the folders manually the database and fodlers will be out of sync. Calibre doesn't actually monitor the folder for changes, it assumes the database is correct.

It appears that Mendeley and Calibre have different user bases and were designed to do different things, it's unlike that Calibre will change significantly enough to do what you want. Of course, I'm not a developer so this is just my opinion

It seems that you will either need to keep copies of the books in both systems or use Mendeley for your academic docs and Calibre for everything else.

Of course, the brain pool might be able to come up with another way to handle it if you ask nicely and offer up the right bribes (beer generally works well)

[gplauche]

I didn't realize Calibre doesn't want any folders. I thought it watched its own.

And I thought I was being nice.

[animedude01]

Originally Posted by gplauche

I didn't realize Calibre doesn't want any folders. I thought it watched its own.

It doesn't watch it in the sense that it monitors the folder for changes. When a book is added to Calibre, the database is updated, a directory entry is created and then populated. After that, Calibre only changes the folder or books if you make a request through the GUI. If you manually change the name of a folder or move books into or out of the directories, the database isn't updated and Calibre isn't made aware of them automagically (of course this assumes I understand this all correctly, which hasn't always been the case)

Originally Posted by gplauche

And I thought I was being nice.

You were nice, the important part was the recommendation of bribes

[Worldwalker]

There seems to be a certain perception that calibre isn't "real" software because it doesn't come in a shrink-wrapped box and take a multi-hundred-dollar hit out of your credit card. Therefore, people seem to think that the developer should be not only willing but eager to throw out the entire design philosophy in order to comply with their personal preference -- which is, generally, to store metadata in the filesystem. I'm curious, by the way: have you gone to the Mendeley developers and demanded that they rewrite their software to play nice with calibre? And if not, why not?

Calibre is not meant to be a cross-linking and citation system for research papers. It's a program for managing and organizing your ebooks and reading devices, for converting between formats, for collecting news for offline reading ... it's a lot of things. But no one program can do everything, and the fact that Mendeley exists -- an entire package devoted to managing research papers -- is something of a hint that it's a very large and complex task on its own, and not something likely to be added into (or even, really, considered during the design of) a general purpose ebook manager.

Like most of the people who start these threads, you're starting out with misconceptions about how calibre works, about how it is supposed to work, and about the tools provided. For example, you wrote:

1) It makes it easier to share batches of files with others. If I want to transfer all of my fiction, but nothing else, to someone's memory stick/computer/whatever then all I have to do is move the Fiction folder. I don't have to hunt through my Calibre Library folder trying to sort out which Author folder is fiction and which isn't, to say nothing of the problem of an author who writes both fiction and nonfiction.

That's using the filesystem as metadata. What we calibre users do, in that case, isn't search through any directory structure at all. If I want to give someone all my fiction, I tell calibre to export all the books tagged "fiction" onto the memory stick. No searching, no folders, no authors ... just click the "fiction" tag, control-A to select all the books it brings up, then export.

And if I want, I can tell it to export all my alternate-history science fiction that isn't military-themed, or all David Weber's science fiction but not his fantasy, or all the books that Eric Flint wrote unless they were co-written with David Drake ... you're starting to get the picture.

You're never supposed to hunt through your calibre library folder for anything, actually. The user interface is so much more powerful a tool for finding any given book than any file browser could possibly be. We had a fellow a while back who ranted about how bad calibre was because he wanted to be able to store his books by subject: science -> chemistry -> inorganic chemistry -> intermetallic phases, I believe the example was. Which might have been fine as far as it went, but what if he owned a book that covered several different subjects in inorganic chemistry? Where would he store that? How could he find it? In calibre, he'd simply give the book a tag for each subject it covered -- "intermetallic phases" and "making things go boom" for example -- and be able to search for it under either tag.

We worry about wasted storage space, or most of us do, because we remember the days when, say, a 1.2gb drive cost me $365 at a computer show, and I was delighted to have haggled for such a good deal. Nowadays, drives are under $100 a terabyte. We really do not have to worry about saving storage space anymore. At least, not the amount that ebooks take up -- even if we duplicate our libraries for benefit of stubborn research paper organizing programs that insist on using the filesystem for metadata.

Just out of curiosity, take a look at how much space your whole library is taking up. Would it all fit on one DVD? Based on the price of a 1tb external drive in the buy.com sale flyer that arrived as I was typing this, that's 45 cents worth of HD space. I'm pretty sure a complete redesign of calibre would be more than 45 cents worth of work.

So manage your research papers in Mendeley. When you add new ones, just import those into calibre.

Y'know, all those books ought to be shoved into one huge indexed file, just to put an end to these threads.

[Starson17]

Originally Posted by kovidgoyal

The disadvantages [of the calibre approach] are:

1) Performance impact on windows. Windows, especially older version can't handle having too many nodes in a single folder. This will be addressed by giving users of large libraries the option of having a folder structure optimized for large collections, in the future.

I'm kind of surprised that no one has mentioned noticing the recent performance improvements for large libraries. The SQL coding has been greatly improved. My collection had become painfully slow to update, and is now a pleasure to work on. I know the node limits are still a concern, and performance improvements are always welcome, but I'm really pleased with the recent changes.