07-20-2013, 07:10 PM | #1 |
Member
Posts: 23
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
|
Calibre Network Request
Good to know you are working on a true multi user support. That is a big step! I would already be happy with a much smaller step.
My family really love calibre. Whith all those devices in the house, PC's, laptops, tablets, smartphones, ereaders and what more I decided to store all our ebook data on our main W7 server computer. And let all family members help with maintaining the database, store their books, sync their devices. And have an automated backup each night for safety. But bummer... calibre does not work that well in this scenario. Even if you mount the library via a network drive letter between two Windows 7 computers running the NTFS filesystems it seems calibre has problems and will give you strange windows errors (no access, cannot write..). Note this is not a true multi user configuration - it is not working for a single user using a mapped network drive! I have never seen problems like this with other software. Should file operations like open/read/write/rename not be just transparant over the local network? Are there bugs or limitations in the code? Users requested this scenario to work since 2010/2011. Before doing the full fledged multiuser implementation, would it be possible to first fix the one user/multiple computers/networked database scenario? I did not look at the code yet, but how much work do you think would this take? I know file locking over network drives can be tedious. But you could use other mechanisms like creating temp .lock files to make sure there is only one instance running. Like e.g. the sync program freefilesyn does. Or forget about the lock at all and let the user decide how to run it. Just make sure the open/read/write/rename works on a network drive. What do you think? Oh and ofcourse thanks for this excellent software ! |
07-20-2013, 11:07 PM | #2 |
creator of calibre
Posts: 44,161
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
|
Advert | |
|
07-20-2013, 11:10 PM | #3 |
US Navy, Retired
Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Welocome to Mobileread.
Moderator Notice Moved post to its own thread. Please read the "PLEASE READ BEFORE POSTING - The purpose of this subforum" sticky post for guidelines for posting in the development forum. Here is the current FAQ entry on setting up calibre to run from a network. |
07-24-2013, 07:21 AM | #4 | |
Member
Posts: 23
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
|
Quote:
So could you please be so kind to explain your statement "Most network filesystems lack various filesystem features that calibre uses" so we can fix the sourcecode accordingly. Thank you, just trying to help. |
|
07-24-2013, 07:35 AM | #5 | |
Member
Posts: 23
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
|
Quote:
The FAQ does not make sense because the answer to the question "on setting up calibre to run from a network" is smiply "Do not put your calibre library on a networked drive". That is not an answer ! The content server solution is not an option. The goal is to have ONE user at the time working with the full calibre software using the DB over the network. So multiple user but NO concurrency - only one user at the time. But from different locations. Solution 1: All modern network filesystems have a file locking option - if you open a Word file over the network and another user opens the same file you get a dialog box with "file is already open". This could be a simpe check in calibre? http://en.wikipedia.org/wiki/File_locking Solution 2: Alternatively you could just create a temp filename "metadata.db.locked" at startup and check for the presence of this file at startup and open it for read/write. If you succeed, you can continue. If not, it is already opened in another instance, so inform the user and quit. The normal program exit would remove this .lock file. But in case of a crash it still works because the file will be closed after some time by the OS. |
|
Advert | |
|
07-24-2013, 08:11 AM | #6 | |
Well trained by Cats
Posts: 30,133
Karma: 57500000
Join Date: Aug 2009
Location: The Central Coast of California
Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A
|
Quote:
Calibre also does a 'Transaction' with the file system (the contents of the Book Folders). You change an Author or Title, The Author or Title folders get Renamed/Moved. You change the cover, and the Cover file gets updated You change any Metadata, and Metadata.OPF (the backup) gets updated It is not as Simple as 'Let the OS locking do it'. Caching created delays alone will wreak havoc if any file is delayed and another Calibre instance makes a request before the 'transaction' has reached stability. Timing is everything. Kovid is doing a re-write to accomplish Multi User. Again, Timing. It will only be released when it is working perfectly Right |
|
07-24-2013, 08:35 AM | #7 |
creator of calibre
Posts: 44,161
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Again, do not put your calibre library on a network drive. File locking is just one problem, others include data integrity across operating systems and most importantly, the fact that various filesystem APIs are BROKEN on network file systems, and when I say BROKEN I mean BROKEN, as in they LIE about what they do.
That FAQ entry gives you THREE options for how to make your library multi-user. Just in case you missed them: 1) Use the content server 2) Use a remote desktop 3) Use a file sync tool, being careful not to run it at the same time as calibre |
07-24-2013, 11:28 AM | #8 | ||
Member
Posts: 23
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
|
calibre, version 0.9.40
Traceback (most recent call last): File "site-packages\calibre\gui2\metadata\basic_widgets.py", line 102, in commit File "site-packages\calibre\library\database2.py", line 2554, in set_title File "site-packages\calibre\library\database2.py", line 653, in set_path File "site-packages\calibre\library\database2.py", line 1529, in add_format File "site-packages\calibre\library\database2.py", line 1375, in copy_format_to File "site-packages\calibre\utils\filenames.py", line 348, in copy_path_to File "site-packages\calibre\startup.py", line 156, in local_open OSError: [Errno 13] Permission denied Arggg this is driving me nuts. All persmissions are full control for everyone on the network! Quote:
Quote:
|
||
07-24-2013, 11:30 AM | #9 |
creator of calibre
Posts: 44,161
Karma: 22670164
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
Sigh, you just wont listen, will you. All the best in your endeavors, I'm done wasting my time.
|
07-24-2013, 11:48 AM | #10 |
Member
Posts: 23
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
|
Sorry - I do not want to offend you in any way. Its just - the three options you suggest do not solve my problem. That is why I asked the question in the first place.
You do not want my feedback or my help and you do not consider any of my suggestions seriously. Ok fair enough its your project. I won't bother you again. To show you I am just a caring supporter of Calibre, which I love, I will donate USD 30 with paypal when Calibre works over a network drive (single or multi user). Thanks again and good luck with the next version. ps: you just could have directed me to github: https://github.com/kovidgoyal/calibre/issues/26 Last edited by erwin1234; 07-24-2013 at 12:12 PM. |
07-24-2013, 04:10 PM | #11 | |
Grand Sorcerer
Posts: 12,433
Karma: 74317824
Join Date: Nov 2007
Location: Toronto
Device: Libra H2O, Libra Colour
|
Quote:
|
|
07-24-2013, 06:44 PM | #12 |
Member
Posts: 23
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
|
Peter, that is a good idea, user skipcube used sysinternals to trace what was happening in the github reported issue. After that you can inspect the source code which modifies and copies the directories&filenames. But I expect no easy solution.
That issue 26 was closed by kovidgoyal without solving it. Reading that thread, it seems there is something mysterious going on in the way calibre (or the python IO libraries) work with the remote NTFS/SMB filesystem. I googled a bit and this issue does not seem that simple to solve - probably that is why kovidgoyal closed it. He is working on a totally new multi-user version anyway, I suspect this will use a true transactions based database model for all operations (SQL?, maybe client/server?). Even Microsoft has problems with their own transactions based NTFS file operations. They recommend not to use their API but use SQL filestreams instead... go figure.. I cloned the source code to see what I can track down. When I have time.. But I think kovidgoyal will come up with a multiuser version before I can find a cure for this issue |
07-25-2013, 10:53 AM | #13 |
Enthusiast
Posts: 28
Karma: 10
Join Date: Jul 2013
Device: Kobo Glo
|
Perhaps a bit missing the topic, but I don't have any problems with this combo:
OS X 10.8 Client, Openindiana (Solaris) ZFS storage server using netatalk 3. I make sure that the share is mounted before starting Calibre, nothing else. With 10.9 I'll make the move over to SMB 2.0, crossing my fingers... -FencepostErr1r |
07-26-2013, 07:23 AM | #14 | |
Member
Posts: 23
Karma: 244
Join Date: Jul 2012
Device: Sony PRS-T1
|
Quote:
The test case would be to change metadata for book titles and authors, so operations that change filenames and directory names, and see if you get consistent results. So no error messages during the changes. And, after playing with the metadata, if you check the library from whithin calibre no errors should be listed (SQL should be ok, no missing or extra titels, authors etc). The ZFS/SMB2.0 server + OSX client results would be interesting too. Anyone running other combo's that work? Like linux EFS/ZFS + SMB on the server and windows on the client? If that works I want to try running a stripped down linux instance on my windows7 server using the free VMWARE player. Last edited by erwin1234; 07-26-2013 at 07:32 AM. |
|
07-26-2013, 07:59 AM | #15 |
Enthusiast
Posts: 28
Karma: 10
Join Date: Jul 2013
Device: Kobo Glo
|
I'll provide my findings when I get around to upgrade my server in time to test it with OS X 10.9 and SMB2.0, but I can't help you with the Windows side.
But if you want to test it yourself, you may want to set up a Win 8 or Server2012 VM, because those have the latest SMB implementation available. -FencepostErr1r |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Which port will Calibre connect to a network share? | awj | Devices | 3 | 03-02-2013 10:14 AM |
Calibre with library on mapped network drive | litiki777 | Related Tools | 1 | 10-30-2012 05:31 AM |
Kobo & Calibre library over network | Issachar | Which one should I buy? | 7 | 12-20-2011 11:19 PM |
Calibre with OS X 10.6 Network User | fiveandfriends | Devices | 0 | 03-03-2011 12:15 PM |
Calibre performance over the network | mrwindex | Calibre | 11 | 07-30-2009 12:45 AM |