Possible storage plugin?

Krystian Galaj · 06-20-2010, 08:29 AM

Nearly weekly there's a new thread about storage of calibre's books in the filesystem, and asking why is it done such way.

I wonder if it would be possible to abstract the functionality of storing of calibre's files on disk in a separate plugin. There could be an abstraction layer between this plugin and the rest of calibre, defining operations calibre presently needs, or is likely to need in the future to create/delete/rename/move/setCurrent folders, read/write/create/delete/rename/copy/move files in them. The default module would simply forward those calls to system calls, and thus calibre's library would stay as it is, a subfolder tree on filesystem - but other storage plugins could also be made then. One such plugin could store all the information in a single opaque file and maintain the calibre-visible filesystem inside it. That might also allow to store Unicode filenames on old or/and crazy filesystems that don't support them, and would drive home the point that calibre's filesystem data is private to calibre. Another version the plugin would remap file/folder naming/positioning to allow tinkerers to keep the calibre file/folder structure just the way they like it, while still aloowing calibre to get all the neccessary info through known calls.

Of course, I don't know enough of calibre's workings to know all the details (it probably would make more sense to define more database-like storage structure, to represent database connections more naturally), but I think with this level of interest, removal of this functionality to a plugin might lead to creation/modification of versions of this plugin by interested people, creation of versions of the plugin customized for speedier operations on specific storage media, and less topics with people complaining that calibre isn't for them because they can't adopt it to fit their needs perfectly.

Starson17 · 06-20-2010, 08:39 AM

Quote:

Originally Posted by Krystian Galaj

I wonder if it would be possible to abstract the functionality of storing of calibre's files on disk in a separate plugin.

"Possible" - yes - but I strongly doubt it will happen. It's very deeply entwined with the other modules. Plus, most of those who are concerned with the current storage system want to directly access the library, and that can interfere with Calibre's use of its database.

If Kovid makes it easier to access the library directly, there are then going to be complaints when Calibre suddenly, and without warning, changes the filename of an ebook or the directory name in what the user will think of as "the user's library."

I suspect it's more likely that Kovid will make it harder to access Calibre's library directly instead of easier. Originally, the books in Calibre's library were entirely inaccessible - they were inside the database with the metadata - and I imagine Kovid occasionally regrets making access easier.

Krystian Galaj · 06-20-2010, 10:00 AM

Quote:

Originally Posted by Starson17

"Possible" - yes - but I strongly doubt it will happen. It's very deeply entwined with the other modules. Plus, most of those who are concerned with the current storage system want to directly access the library, and that can interfere with Calibre's use of its database.

In my own past programming experience, reworks of making abstraction layer between clearly defined resource such as boks storage and the rest of the application always made the application easier to maintain and more robust, so I'd tend to see a view of such interactions being too intertwined to change them as a surrender of the program to the mess. It's probably only my own approach to programming...

Quote:

Originally Posted by Starson17

If Kovid makes it easier to access the library directly, there are then going to be complaints when Calibre suddenly, and without warning, changes the filename of an ebook or the directory name in what the user will think of as "the user's library."

Not if the default plugin would be the one with the same functionality the current calibre has, or the plugin hiding all books inside private filesystem of SQLite database. To change that functionality, users would need to switch the plugin somewhere deep in Preferences, the way paved by texts of "make sure you know what's you're doing" and "dangerous" variety.

Quote:

Originally Posted by Starson17

I suspect it's more likely that Kovid will make it harder to access Calibre's library directly instead of easier. Originally, the books in Calibre's library were entirely inaccessible - they were inside the database with the metadata - and I imagine Kovid occasionally regrets making access easier.

I feel that by making storage means a choice calibre would benefit in the end. Surely, initially there would be more bugs detected, some code reworks done, but in the end the codebase would be more stable as a result.

That's assumming Kovid still likes to develop and manage calibre - if he's getting weary of it, and thinks often of the time when he could finish all the goals for calibre he set for himself, and switch to something more interesting, that would be one more unneccessary complication, and my arguments don't apply. I have some projects like that...

Worldwalker · 06-20-2010, 10:56 AM

The problem has nothing to do with whether Kovid is "getting weary of it" -- if he's weary of anything, it's of people who don't understand that calibre manages their books, and want to reach inside it and mess with calibre's files. There would be nothing to be gained by making it easier for users to muck around with the file database's private innards, and a great deal to be lost, not only in initial coding effort, but in subsequent support requests from people who screwed it up and corrupted the database, bricked calibre, or whatever.

Personally, I'd vote for using numerical folder names, rot-13 shifting the file names, and giving them cryptic extensions, so that users would keep their grubby hands out. It's the fact that the names are human-readable that makes humans want to mess with them. People, or at least ordinary users, don't try to fool around with the insides of the SQLite databases that calibre uses -- they understand those belong to calibre. But the files look like what they're accustomed to thinking of as "books", so they want to access them directly.

Calibre is ready and willing to export any books you happen to want externally as files of your preferred format. That's how those files should be accessed -- not by peeking up calibre's skirt. It'll slap you.

Why is it, by the way, that it is people who want changes to this one aspect of calibre, and only the people who want those specific changes, think that insulting Kovid (and by extension the other calibre developers) is the way to go about it? We've had a bit of a break lately, but at one point we were getting post like that every week or two, ranging from direct, in-the-face insults to insinuations like yours that the only reason Kovid doesn't change calibre to work the way you want it to is that he's lazy or just bored with it. I expect that from 10-karma newbies, not from someone who's been here longer than most of us (and way, way longer than me).

DoctorOhh · 06-20-2010, 11:37 AM

Quote:

Originally Posted by Worldwalker

The problem has nothing to do with whether Kovid is "getting weary of it" -- if he's weary of anything, it's of people who don't understand that calibre manages their books, and want to reach inside it and mess with calibre's files.

The OP understands this. That is why he created the thread.

Quote:

Originally Posted by Worldwalker

Personally, I'd vote for using numerical folder names, rot-13 shifting the file names, and giving them cryptic extensions, so that users would keep their grubby hands out.

As I read his words:

Quote:

One such plugin could store all the information in a single opaque file and maintain the calibre-visible filesystem inside it.

They match up with yours as well.

Quote:

Originally Posted by Worldwalker

Calibre is ready and willing to export any books you happen to want externally as files of your preferred format.

He understands this, your normal well thought out points are getting lost in the fact this is not one of those discussions. It is more a development philosophy question. I bet if built from scratch today, Kovid might well create an abstraction layer plugin system. It would make doing your idea of rot-13 shifting the file names easily doable and provide a great deal of flexibility.

Quote:

Originally Posted by Worldwalker

Why is it, by the way, that it is people who want changes to this one aspect of calibre, and only the people who want those specific changes, think that insulting Kovid (and by extension the other calibre developers) is the way to go about it? We've had a bit of a break lately, but at one point we were getting post like that every week or two, ranging from direct, in-the-face insults to insinuations like yours that the only reason Kovid doesn't change calibre to work the way you want it to is that he's lazy or just bored with it.

The OP was relating as a programmer and not implying any insults. There have been no insults in this thread.

Quote:

Originally Posted by Worldwalker

I expect that from 10-karma newbies, not from someone who's been here longer than most of us (and way, way longer than me).

Until now.

Starson17 · 06-20-2010, 11:46 AM

I thought your suggestion was driven by the number of users who complained here about the library structure. Those are exactly the users who wouldn't find your hidden "know-what-you-are-doing" plugin, and wouldn't understand why their library was being changed by Calibre. If the abstraction options were pointed out to them, and they changed the abstraction to a "better" naming format, they'd be upset by normal Calibre operation. As I said, the only reason they want a "better" library structure and a different naming format is to make it easier to do things that might interfere with Calibre's library management.

Quote:

I feel that by making storage means a choice calibre would benefit in the end. Surely, initially there would be more bugs detected, some code reworks done, but in the end the codebase would be more stable as a result.

A worthwhile goal if stability was an issue now, but it's not. I don't see any stability issues that abstraction would help with.

Moreover, it's been my experience that the more sophisticated the user, the less they feel a need to control the library structure. You're building a fix that only sophisticated users can safely use, and almost none of them seem to need or want it.

Quote:

That's assumming Kovid still likes to develop and manage calibre - if he's getting weary of it, and thinks often of the time when he could finish all the goals for calibre he set for himself, and switch to something more interesting, that would be one more unneccessary complication, and my arguments don't apply. I have some projects like that...

This shows you are definitely very new here. Calibre is actively being developed and major changes are made frequently. There are whole sections of the interface being reserved for future enhancements that are on the development road map.

Starson17 · 06-20-2010, 11:59 AM

Quote:

Originally Posted by dwanthny

I bet if built from scratch today, Kovid might well create an abstraction layer plugin system. It would make doing your idea of rot-13 shifting the file names easily doable and provide a great deal of flexibility.

Yes, it would, but why would anyone use rot-13 shifted names? They only make sense if you are trying to prevent users from direct access, and providing an option to make direct access easy is contrary to that.

Quote:

The OP was relating as a programmer and not implying any insults.

I agree - it's a programming point of view, but I'd like to see the effort for such a major rewrite justified. So far I've seen two justifications:
1) stability - something that does not seem to be a problem, (and I don't agree with), and
2) reducing complaints here from new users who want a different library structure - but even the OP says the abstraction needs to be buried and hidden from the newbie.

It's not enough. IMO, there are too many reasons not to do this - certainly not now.

chaley · 06-20-2010, 12:39 PM

I am in active discussion with Kovid about replacing calibre's back end with a plugin architecture. Kovid brought the up when I asked him about refactoring the device interface. He is not driven to give access to book files, but instead to support diverse storage possibilities. The ones he mentioned are cloud storage, non-relational DBs, and pure filesystem storage. I add virtual file systems, devices as libraries, and client-server setups. We have talked about it supporting multiple open libraries simultaneously. Discussions have halted while 0.7 goes through its growing pains, but I expect them to restart in the not far future.

There are some downsides:

1) It will be an incredible amount of work, almost a complete rewrite of the non-gui code and a substantial rewrite of the gui. By incredible, I mean thousands of hours. Fortunately, with V7 we have (almost completely) moved to a dictionary-based metadata structure, which will make the transition easier. (No, this wasn't an accident. What remains is to migrate the devices to the more general structure.)
2) It will almost certainly be behavior-inconsistent with the current calibre. It is also likely that some features will disappear. We will know only when we have arrived.
3) Such an architecture will give users enough rope to hang themselves. The support challenges are formidable now, and having random plugins floating around running on the some variant of the Glasnost 87 slate will make it near impossible. It is likely that only calibre-supplied back ends in specific configurations will be supported.

Current thoughts are to begin with migrating the device interface, with the goal of looking for the gotchas. Kovid will take the lead on this, outlining what he wants to see. Once that is semi-stable, then moving the library will be considered.

Staging has not been discussed. Kovid will work that out as time passes. That said, I will be surprised if anything appears within 6 months, and a year is more likely.

Starson17 · 06-20-2010, 01:31 PM

Quote:

Originally Posted by chaley

then moving the library will be considered.

Do you think this will change the answer to "Why doesn’t calibre let me store books in my own directory structure?" in the FAQ? I.e., will there be a question like: "How do I change the calibre directory structure?" with an answer? I totally understand the desire to support cloud storage, non-relational DBs, pure filesystem storage, virtual file systems, devices as libraries, client-server setups and multiple open libraries.

But ... is revised naming and/or better direct access to book files in the Library even on the radar? Is "Why does Calibre have to make a copy of all my ebooks?" on that radar? Those are the two issues that drive the constant beginner user queries about file structure.

Dopedangel · 06-20-2010, 01:42 PM

Quote:

Originally Posted by Starson17

Yes, it would, but why would anyone use rot-13 shifted names? They only make sense if you are trying to prevent users from direct access, and providing an option to make direct access easy is contrary to that.

I agree - it's a programming point of view, but I'd like to see the effort for such a major rewrite justified. So far I've seen two justifications:
1) stability - something that does not seem to be a problem, (and I don't agree with), and
2) reducing complaints here from new users who want a different library structure - but even the OP says the abstraction needs to be buried and hidden from the newbie.

It's not enough. IMO, there are too many reasons not to do this - certainly not now.

I am not a programmer and I really don't care about
how calibre keeps by ebooks as long as I can easily access them.
I am putting this out as its an idea I had maybe my knowledge of computers is not very good and I am completely wrong

But to me making the back-end of calibre plugin based would allow for other optimizations like keeping your library in the cloud(I am using dropbox but I mean a complete cloud based solution).
Or allowing for multiple calibre instances editing the same library from what I gather the current database system cannot support.
Even people with network storage could get optimized plugins for read write.

I guess what I am talking about would require a complete rewrite actually
you would be better of starting from scratch. Now I am wondering if I should post this or will I sound like an idiot
What the hell I will post it anyway

Starson17 · 06-20-2010, 01:48 PM

Quote:

Originally Posted by Dopedangel

Now I am wondering if I should post this or will I sound like an idiot

Not an idiot - very smart in fact. You shouldn't care how Calibre stores things, but having the ability to store in the cloud, or on other currently unsupported storage devices would be great. Client server would also be great. Dropbox semi-simulates that, as long as only one client at a time is using it

Worldwalker · 06-20-2010, 01:54 PM

Quote:

Originally Posted by Dopedangel

I am not a programmer and I really don't care about how calibre keeps by ebooks as long as I can easily access them.

The problem that has spawned this thread and many like it is the people who don't understand that the goal is to access their ebooks (via calibre) rather than the files those ebooks are in (via the filesystem) and are confused about what the purpose of calibre actually is.

And no, you don't sound like an idiot. It's an intelligent question that shows both an understanding of the issue and respect for the people involved. It's the ones who complain that they don't like how calibre organizes its private files because they want to access them without going through calibre (which makes me wonder what they're doing using calibre in the first place) who need to be concerned with how their posts make them look.

chaley · 06-20-2010, 01:58 PM

Quote:

Originally Posted by Starson17

Do you think this will change the answer to "Why doesn’t calibre let me store books in my own directory structure?" in the FAQ? I.e., will there be a question like: "How do I change the calibre directory structure?" with an answer?

It isn't a goal, at least for me. It might happen by accident. Or the answer might be 'Write a plugin that supports your personal storage scheme.'

Quote:

But ... is revised naming and/or better direct access to book files in the Library even on the radar? Is "Why does Calibre have to make a copy of all my ebooks?" on that radar? Those are the two issues that drive the constant beginner user queries about file structure.

Disclaimer: personal opinions follow:

The fundamental issues will not change. Calibre will need to know where the books are, to have some assurance that they will still be there tomorrow, and to be in control of their modification. Nobody forces people to use calibre. If how books are stored is a deal-breaker, so be it. They can get the code and change it however they want. Add to that the fact that I am not in the least interested in dealing with inevitable support questions like 'I sent 'War and Peace' to my device, but some hentai comic arrived instead. I accidentally showed it to my boss, and she fired me. How could calibre be so evil?'

So no, it isn't on my radar. But as I said, I am speaking entirely for myself. Kovid may have different ideas and goals, and his win. Other current developers might have a different radar, or new developers might come along. Que sera, sera.

Worldwalker · 06-20-2010, 02:20 PM

<-- would greatly prefer reading hentai to reading War & Peace.

theducks · 06-20-2010, 03:26 PM

I would like to see "Client-server" since I have many computers in different locations.
This would also mean that Family members could use and modify the contents (per user DB rights enforced

).
I have no issues on what/how Calibre files them, as log as I can get them when I want, from anywhere in my permitted (LAN, or WAN if made accessible) network.

06-20-2010, 08:29 AM	#1
Krystian Galaj Guru Posts: 820 Karma: 11012 Join Date: Nov 2007 Location: Warsaw, Poland Device: Bookeen Cybook	Possible storage plugin? Nearly weekly there's a new thread about storage of calibre's books in the filesystem, and asking why is it done such way. I wonder if it would be possible to abstract the functionality of storing of calibre's files on disk in a separate plugin. There could be an abstraction layer between this plugin and the rest of calibre, defining operations calibre presently needs, or is likely to need in the future to create/delete/rename/move/setCurrent folders, read/write/create/delete/rename/copy/move files in them. The default module would simply forward those calls to system calls, and thus calibre's library would stay as it is, a subfolder tree on filesystem - but other storage plugins could also be made then. One such plugin could store all the information in a single opaque file and maintain the calibre-visible filesystem inside it. That might also allow to store Unicode filenames on old or/and crazy filesystems that don't support them, and would drive home the point that calibre's filesystem data is private to calibre. Another version the plugin would remap file/folder naming/positioning to allow tinkerers to keep the calibre file/folder structure just the way they like it, while still aloowing calibre to get all the neccessary info through known calls. Of course, I don't know enough of calibre's workings to know all the details (it probably would make more sense to define more database-like storage structure, to represent database connections more naturally), but I think with this level of interest, removal of this functionality to a plugin might lead to creation/modification of versions of this plugin by interested people, creation of versions of the plugin customized for speedier operations on specific storage media, and less topics with people complaining that calibre isn't for them because they can't adopt it to fit their needs perfectly.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
New Plugin Type Idea: Library Plugin	cgranade	Plugins	3	09-15-2010 01:11 PM
How much iPad storage have you used?	Maggie Leung	Apple Devices	38	08-15-2010 03:51 AM
iPhone Two new iPhone Storage Solutions: Zoomit (SD Card Reader) & Boxnet (Cloud Storage)	kjk	Apple Devices	0	02-09-2010 07:20 PM
What Storage do you use?	Adam B.	iRex	2	04-01-2009 12:31 PM
1.4 GB Storage... HA!	TallMomof2	Amazon Kindle	22	02-28-2009 10:41 PM

06-20-2010, 10:56 AM	#4
Worldwalker Curmudgeon Posts: 3,085 Karma: 722357 Join Date: Feb 2010 Device: PRS-505	The problem has nothing to do with whether Kovid is "getting weary of it" -- if he's weary of anything, it's of people who don't understand that calibre manages their books, and want to reach inside it and mess with calibre's files. There would be nothing to be gained by making it easier for users to muck around with the file database's private innards, and a great deal to be lost, not only in initial coding effort, but in subsequent support requests from people who screwed it up and corrupted the database, bricked calibre, or whatever. Personally, I'd vote for using numerical folder names, rot-13 shifting the file names, and giving them cryptic extensions, so that users would keep their grubby hands out. It's the fact that the names are human-readable that makes humans want to mess with them. People, or at least ordinary users, don't try to fool around with the insides of the SQLite databases that calibre uses -- they understand those belong to calibre. But the files look like what they're accustomed to thinking of as "books", so they want to access them directly. Calibre is ready and willing to export any books you happen to want externally as files of your preferred format. That's how those files should be accessed -- not by peeking up calibre's skirt. It'll slap you. Why is it, by the way, that it is people who want changes to this one aspect of calibre, and only the people who want those specific changes, think that insulting Kovid (and by extension the other calibre developers) is the way to go about it? We've had a bit of a break lately, but at one point we were getting post like that every week or two, ranging from direct, in-the-face insults to insinuations like yours that the only reason Kovid doesn't change calibre to work the way you want it to is that he's lazy or just bored with it. I expect that from 10-karma newbies, not from someone who's been here longer than most of us (and way, way longer than me).

06-20-2010, 12:39 PM	#8
chaley Grand Sorcerer Posts: 12,032 Karma: 7257323 Join Date: Jan 2010 Location: Notts, England Device: Kobo Libra 2	I am in active discussion with Kovid about replacing calibre's back end with a plugin architecture. Kovid brought the up when I asked him about refactoring the device interface. He is not driven to give access to book files, but instead to support diverse storage possibilities. The ones he mentioned are cloud storage, non-relational DBs, and pure filesystem storage. I add virtual file systems, devices as libraries, and client-server setups. We have talked about it supporting multiple open libraries simultaneously. Discussions have halted while 0.7 goes through its growing pains, but I expect them to restart in the not far future. There are some downsides: 1) It will be an incredible amount of work, almost a complete rewrite of the non-gui code and a substantial rewrite of the gui. By incredible, I mean thousands of hours. Fortunately, with V7 we have (almost completely) moved to a dictionary-based metadata structure, which will make the transition easier. (No, this wasn't an accident. What remains is to migrate the devices to the more general structure.) 2) It will almost certainly be behavior-inconsistent with the current calibre. It is also likely that some features will disappear. We will know only when we have arrived. 3) Such an architecture will give users enough rope to hang themselves. The support challenges are formidable now, and having random plugins floating around running on the some variant of the Glasnost 87 slate will make it near impossible. It is likely that only calibre-supplied back ends in specific configurations will be supported. Current thoughts are to begin with migrating the device interface, with the goal of looking for the gotchas. Kovid will take the lead on this, outlining what he wants to see. Once that is semi-stable, then moving the library will be considered. Staging has not been discussed. Kovid will work that out as time passes. That said, I will be surprised if anything appears within 6 months, and a year is more likely.

06-20-2010, 02:20 PM	#14
Worldwalker Curmudgeon Posts: 3,085 Karma: 722357 Join Date: Feb 2010 Device: PRS-505	<-- would greatly prefer reading hentai to reading War & Peace.

06-20-2010, 03:26 PM	#15
theducks Well trained by Cats Posts: 30,447 Karma: 58055868 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	I would like to see "Client-server" since I have many computers in different locations. This would also mean that Family members could use and modify the contents (per user DB rights enforced ). I have no issues on what/how Calibre files them, as log as I can get them when I want, from anywhere in my permitted (LAN, or WAN if made accessible) network.

Advert

Advert