![]() |
#1 |
Guru
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 820
Karma: 11012
Join Date: Nov 2007
Location: Warsaw, Poland
Device: Bookeen Cybook
|
Possible storage plugin?
Nearly weekly there's a new thread about storage of calibre's books in the filesystem, and asking why is it done such way.
I wonder if it would be possible to abstract the functionality of storing of calibre's files on disk in a separate plugin. There could be an abstraction layer between this plugin and the rest of calibre, defining operations calibre presently needs, or is likely to need in the future to create/delete/rename/move/setCurrent folders, read/write/create/delete/rename/copy/move files in them. The default module would simply forward those calls to system calls, and thus calibre's library would stay as it is, a subfolder tree on filesystem - but other storage plugins could also be made then. One such plugin could store all the information in a single opaque file and maintain the calibre-visible filesystem inside it. That might also allow to store Unicode filenames on old or/and crazy filesystems that don't support them, and would drive home the point that calibre's filesystem data is private to calibre. Another version the plugin would remap file/folder naming/positioning to allow tinkerers to keep the calibre file/folder structure just the way they like it, while still aloowing calibre to get all the neccessary info through known calls. Of course, I don't know enough of calibre's workings to know all the details (it probably would make more sense to define more database-like storage structure, to represent database connections more naturally), but I think with this level of interest, removal of this functionality to a plugin might lead to creation/modification of versions of this plugin by interested people, creation of versions of the plugin customized for speedier operations on specific storage media, and less topics with people complaining that calibre isn't for them because they can't adopt it to fit their needs perfectly. |
![]() |
![]() |
![]() |
#2 | |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
If Kovid makes it easier to access the library directly, there are then going to be complaints when Calibre suddenly, and without warning, changes the filename of an ebook or the directory name in what the user will think of as "the user's library." I suspect it's more likely that Kovid will make it harder to access Calibre's library directly instead of easier. Originally, the books in Calibre's library were entirely inaccessible - they were inside the database with the metadata - and I imagine Kovid occasionally regrets making access easier. Last edited by Starson17; 06-20-2010 at 07:41 AM. |
|
![]() |
![]() |
Advert | |
|
![]() |
#3 | |||
Guru
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 820
Karma: 11012
Join Date: Nov 2007
Location: Warsaw, Poland
Device: Bookeen Cybook
|
Quote:
Quote:
Quote:
That's assumming Kovid still likes to develop and manage calibre - if he's getting weary of it, and thinks often of the time when he could finish all the goals for calibre he set for himself, and switch to something more interesting, that would be one more unneccessary complication, and my arguments don't apply. I have some projects like that... |
|||
![]() |
![]() |
![]() |
#4 |
Curmudgeon
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,085
Karma: 722357
Join Date: Feb 2010
Device: PRS-505
|
The problem has nothing to do with whether Kovid is "getting weary of it" -- if he's weary of anything, it's of people who don't understand that calibre manages their books, and want to reach inside it and mess with calibre's files. There would be nothing to be gained by making it easier for users to muck around with the file database's private innards, and a great deal to be lost, not only in initial coding effort, but in subsequent support requests from people who screwed it up and corrupted the database, bricked calibre, or whatever.
Personally, I'd vote for using numerical folder names, rot-13 shifting the file names, and giving them cryptic extensions, so that users would keep their grubby hands out. It's the fact that the names are human-readable that makes humans want to mess with them. People, or at least ordinary users, don't try to fool around with the insides of the SQLite databases that calibre uses -- they understand those belong to calibre. But the files look like what they're accustomed to thinking of as "books", so they want to access them directly. Calibre is ready and willing to export any books you happen to want externally as files of your preferred format. That's how those files should be accessed -- not by peeking up calibre's skirt. It'll slap you. Why is it, by the way, that it is people who want changes to this one aspect of calibre, and only the people who want those specific changes, think that insulting Kovid (and by extension the other calibre developers) is the way to go about it? We've had a bit of a break lately, but at one point we were getting post like that every week or two, ranging from direct, in-the-face insults to insinuations like yours that the only reason Kovid doesn't change calibre to work the way you want it to is that he's lazy or just bored with it. I expect that from 10-karma newbies, not from someone who's been here longer than most of us (and way, way longer than me). |
![]() |
![]() |
![]() |
#5 | ||||||
US Navy, Retired
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
![]() |
||||||
![]() |
![]() |
Advert | |
|
![]() |
#6 | ||
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
I thought your suggestion was driven by the number of users who complained here about the library structure. Those are exactly the users who wouldn't find your hidden "know-what-you-are-doing" plugin, and wouldn't understand why their library was being changed by Calibre. If the abstraction options were pointed out to them, and they changed the abstraction to a "better" naming format, they'd be upset by normal Calibre operation. As I said, the only reason they want a "better" library structure and a different naming format is to make it easier to do things that might interfere with Calibre's library management.
Quote:
Moreover, it's been my experience that the more sophisticated the user, the less they feel a need to control the library structure. You're building a fix that only sophisticated users can safely use, and almost none of them seem to need or want it. Quote:
|
||
![]() |
![]() |
![]() |
#7 | ||
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
Quote:
1) stability - something that does not seem to be a problem, (and I don't agree with), and 2) reducing complaints here from new users who want a different library structure - but even the OP says the abstraction needs to be buried and hidden from the newbie. It's not enough. IMO, there are too many reasons not to do this - certainly not now. |
||
![]() |
![]() |
![]() |
#8 |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 11,911
Karma: 7076137
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
I am in active discussion with Kovid about replacing calibre's back end with a plugin architecture. Kovid brought the up when I asked him about refactoring the device interface. He is not driven to give access to book files, but instead to support diverse storage possibilities. The ones he mentioned are cloud storage, non-relational DBs, and pure filesystem storage. I add virtual file systems, devices as libraries, and client-server setups. We have talked about it supporting multiple open libraries simultaneously. Discussions have halted while 0.7 goes through its growing pains, but I expect them to restart in the not far future.
There are some downsides: 1) It will be an incredible amount of work, almost a complete rewrite of the non-gui code and a substantial rewrite of the gui. By incredible, I mean thousands of hours. Fortunately, with V7 we have (almost completely) moved to a dictionary-based metadata structure, which will make the transition easier. (No, this wasn't an accident. What remains is to migrate the devices to the more general structure.) 2) It will almost certainly be behavior-inconsistent with the current calibre. It is also likely that some features will disappear. We will know only when we have arrived. 3) Such an architecture will give users enough rope to hang themselves. The support challenges are formidable now, and having random plugins floating around running on the some variant of the Glasnost 87 slate will make it near impossible. It is likely that only calibre-supplied back ends in specific configurations will be supported. Current thoughts are to begin with migrating the device interface, with the goal of looking for the gotchas. Kovid will take the lead on this, outlining what he wants to see. Once that is semi-stable, then moving the library will be considered. Staging has not been discussed. Kovid will work that out as time passes. That said, I will be surprised if anything appears within 6 months, and a year is more likely. |
![]() |
![]() |
![]() |
#9 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Do you think this will change the answer to "Why doesn’t calibre let me store books in my own directory structure?" in the FAQ? I.e., will there be a question like: "How do I change the calibre directory structure?" with an answer? I totally understand the desire to support cloud storage, non-relational DBs, pure filesystem storage, virtual file systems, devices as libraries, client-server setups and multiple open libraries.
But ... is revised naming and/or better direct access to book files in the Library even on the radar? Is "Why does Calibre have to make a copy of all my ebooks?" on that radar? Those are the two issues that drive the constant beginner user queries about file structure. |
![]() |
![]() |
![]() |
#10 | |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,763
Karma: 30063305
Join Date: Dec 2006
Location: Singapore
Device: Boyue
|
Quote:
how calibre keeps by ebooks as long as I can easily access them. I am putting this out as its an idea I had maybe my knowledge of computers is not very good and I am completely wrong But to me making the back-end of calibre plugin based would allow for other optimizations like keeping your library in the cloud(I am using dropbox but I mean a complete cloud based solution). Or allowing for multiple calibre instances editing the same library from what I gather the current database system cannot support. Even people with network storage could get optimized plugins for read write. I guess what I am talking about would require a complete rewrite actually you would be better of starting from scratch. Now I am wondering if I should post this or will I sound like an idiot What the hell I will post it anyway Last edited by Dopedangel; 06-20-2010 at 12:46 PM. Reason: Should have read the whole thread you guys were talking about similar later |
|
![]() |
![]() |
![]() |
#11 | |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 4,004
Karma: 177841
Join Date: Dec 2009
Device: WinMo: IPAQ; Android: HTC HD2, Archos 7o; Java:Gravity T
|
Quote:
![]() |
|
![]() |
![]() |
![]() |
#12 | |
Curmudgeon
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,085
Karma: 722357
Join Date: Feb 2010
Device: PRS-505
|
Quote:
And no, you don't sound like an idiot. It's an intelligent question that shows both an understanding of the issue and respect for the people involved. It's the ones who complain that they don't like how calibre organizes its private files because they want to access them without going through calibre (which makes me wonder what they're doing using calibre in the first place) who need to be concerned with how their posts make them look. |
|
![]() |
![]() |
![]() |
#13 | ||
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 11,911
Karma: 7076137
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
Quote:
The fundamental issues will not change. Calibre will need to know where the books are, to have some assurance that they will still be there tomorrow, and to be in control of their modification. Nobody forces people to use calibre. If how books are stored is a deal-breaker, so be it. They can get the code and change it however they want. Add to that the fact that I am not in the least interested in dealing with inevitable support questions like 'I sent 'War and Peace' to my device, but some hentai comic arrived instead. I accidentally showed it to my boss, and she fired me. How could calibre be so evil?' So no, it isn't on my radar. But as I said, I am speaking entirely for myself. Kovid may have different ideas and goals, and his win. Other current developers might have a different radar, or new developers might come along. Que sera, sera. |
||
![]() |
![]() |
![]() |
#14 |
Curmudgeon
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,085
Karma: 722357
Join Date: Feb 2010
Device: PRS-505
|
<-- would greatly prefer reading hentai to reading War & Peace.
|
![]() |
![]() |
![]() |
#15 |
Well trained by Cats
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 30,288
Karma: 57979022
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
I would like to see "Client-server" since I have many computers in different locations.
This would also mean that Family members could use and modify the contents (per user DB rights enforced ![]() I have no issues on what/how Calibre files them, as log as I can get them when I want, from anywhere in my permitted (LAN, or WAN if made accessible) network. |
![]() |
![]() |
![]() |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
New Plugin Type Idea: Library Plugin | cgranade | Plugins | 3 | 09-15-2010 12:11 PM |
How much iPad storage have you used? | Maggie Leung | Apple Devices | 38 | 08-15-2010 02:51 AM |
iPhone Two new iPhone Storage Solutions: Zoomit (SD Card Reader) & Boxnet (Cloud Storage) | kjk | Apple Devices | 0 | 02-09-2010 06:20 PM |
What Storage do you use? | Adam B. | iRex | 2 | 04-01-2009 11:31 AM |
1.4 GB Storage... HA! | TallMomof2 | Amazon Kindle | 22 | 02-28-2009 09:41 PM |