Calibre metadata.calibre not allowing updates

[Chuckels550]

I have been uploading ebooks onto my Kobo - it just came from RMA and I have some 400+ files to load onto it. I deleted some folders on the Kobo using Windows Explorer on Win7 64 bit. I am running into the following error when I try to send ebooks to my Kobo. Is there anyway to unlock the metadata.calibre file?

(13, 'Permission denied')

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 68, in run
File "site-packages\calibre\gui2\device.py", line 303, in _sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 304, in sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 301, in write_prefix
IOError: [Errno 13] Permission denied: u'K:\\metadata.calibre'

I think that Calibre has an issue with Sleep mode on the Kobo. If the device is in sleep mode when I set up for USB connection then the metadata.calibre file is not available for updates. If the Kobo device is powered off mode, then there doesn't seem to be an issue.

[theducks]

Originally Posted by Chuckels550

I have been uploading ebooks onto my Kobo - it just came from RMA and I have some 400+ files to load onto it. I deleted some folders on the Kobo using Windows Explorer on Win7 64 bit. I am running into the following error when I try to send ebooks to my Kobo. Is there anyway to unlock the metadata.calibre file?

(13, 'Permission denied')

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 68, in run
File "site-packages\calibre\gui2\device.py", line 303, in _sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 304, in sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 301, in write_prefix
IOError: [Errno 13] Permission denied: u'K:\\metadata.calibre'

I think that Calibre has an issue with Sleep mode on the Kobo. If the device is in sleep mode when I set up for USB connection then the metadata.calibre file is not available for updates. If the Kobo device is powered off mode, then there doesn't seem to be an issue.

That is the library scan that is used for the green (on device) check marks (not Metadata.db).
It exists on your device and can be deleted, it will just take longer to rebuild the first time you attach.
I suggest that you not transfer books until the task wheel stops spinning.

[DoctorOhh]

Originally Posted by Chuckels550

I think that Calibre has an issue with Sleep mode on the Kobo. If the device is in sleep mode when I set up for USB connection then the metadata.calibre file is not available for updates. If the Kobo device is powered off mode, then there doesn't seem to be an issue.

It's like the old Doctor! Doctor! joke.

Patient: Doctor! Doctor! it hurts when I do this.....
Doctor: Then don't do that!

Bottom line though is you have solved your own problem. You have a Kobo problem, if it doesn't work in sleep mode don't use it in sleep mode. The Jetbook Lite is the same way, prior to connecting I disable sleep mode or it may go to sleep prior to finishing what I want done.

[kgn]

Originally Posted by Chuckels550

(13, 'Permission denied')

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 68, in run
File "site-packages\calibre\gui2\device.py", line 303, in _sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 304, in sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 301, in write_prefix
IOError: [Errno 13] Permission denied: u'K:\\metadata.calibre'

I get exactly this problem when connecting to my Pocketbook 302. The thing is, this is at the very end of a connection, so it can take 15 minutes or so before you get this error. The current version I am using seems a lot better - ie 0.7.13. So far it has been connecting ok (no errors). I never get the error when using 0.7.6 either. All the versions in between these two this error comes up 99% of the time.

I am still trying to work out what the problem is - Calibre, my PB302, the SD card, etc. Seeing changing Calibre versions seems to fix the problem, I am tending towards that.

[itimpi]

The ability to 'sync' your library with the Sony has been introduced in that timescale, and since the metadata.calibre file is used as part of that process it seems that that has been you issue.

There has been mention in the release notes that the first time you try to connect with sync active (which it is by default) the connection can be tied for a while if you have a lot of books while calibre populated the metadata.calibre file. After that the delays on connection should be negligible.

[DoctorOhh]

Originally Posted by kgn

I get exactly this problem when connecting to my Pocketbook 302. The thing is, this is at the very end of a connection, so it can take 15 minutes

So you're telling us that you get the

IOError: [Errno 13] Permission denied: u'K:\\metadata.calibre'

Error too?

Does your error also happen because, like the Kobo, the Pocket302 stops communicating after it enters sleep mode, say after about 15 minutes?

I have to try this with my Jetbook Lite tomorrow night and see if I can replicate this error.

[kgn]

Originally Posted by dwanthny

So you're telling us that you get the
Error too?

Does your error also happen because, like the Kobo, the Pocket302 stops communicating after it enters sleep mode, say after about 15 minutes?

I have to try this with my Jetbook Lite tomorrow night and see if I can replicate this error.

Yep - I get that error 99% of the time. My exact report is

ERROR: Error: Error communicating with device
(13, 'Permission denied')
Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 68, in run
File "site-packages\calibre\gui2\device.py", line 282, in _books
File "site-packages\calibre\devices\usbms\driver.py", line 164, in books
File "site-packages\calibre\devices\usbms\driver.py", line 303, in sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 301, in write_prefix
IOError: [Errno 13] Permission denied: u'F:\\metadata.calibre'

The Pocketbook 302 does not have a sleep mode as far as I know. Certainly not when you are communicating with it. When the USB cable is plugged in the PB302 has the USB Active icon flashing the whole time. So no - no sleep mode.

This report was from v0.7.13. I can connect to my PB302 from v0.7.6, and it works perfectly. In fact, I just did it a few seconds ago to check. This is pointing to exactly the same calibre library paths in both versions. ie I have 0.7.6 in a VMware system pointing to a path on the host. On the host system I have 0.7.13 pointing to the self same path.

In fact, 0.7.6 works very well. It is what I always use. No version since then works with my PB302 - because of this problem.

[chaley]

Originally Posted by kgn

In fact, 0.7.6 works very well. It is what I always use. No version since then works with my PB302 - because of this problem.

Hmmm...

0.7.7 changed the way that the cache file was written. In 0.7.6 and before, the file was written line by line. On some installations/devices, this was horribly slow (3 to 4 minutes to write the cache). The change was to write the cache file in one huge write.

One possibility is that the PB302 firmware has trouble with huge writes. It could be a caching issue, where calibre closes the file but the device hasn't in fact finished writing the data. In this case, the device could refuse to permit the file to be reopened until the cache is empty, reported these back to windows as permission errors.

I am not sure what to do about this possibility, if anything. One idea is to slow down the writes, but as this would affect *everyone* it seems like a bad idea.

Another idea would be to force a filesystem 'sync' after the write. If the device behaves correctly, the sync should not return until all the data is committed to the disk.

@kgn: perhaps you are willing to try an alternate driver file. I have implemented the second option and attached the .py file (zipped). You put this file into your 0.7.13 calibre installation directory (probably "C:\Program Files\Calibre2\Lib\site-packages\calibre\devices\usbms\driver.py"), then run calibre-debug -g from a command line. When you send books, you should see lines like

DEBUG:    8.1 USBMS: starting sync_booklists
DEBUG:    8.2 USBMS: finished os.fsync()
DEBUG:    8.2 USBMS: finished sync_booklists

The second line (fsync) indicates that you are running the modified driver.

Let me know what you do and whether anything changes.

[kgn]

Originally Posted by chaley

Hmmm...
Let me know what you do and whether anything changes.

I cannot send books - the device will not connect. This is what happens when I plug in the PB302.

Basically the same as the old driver.....


Job: 1 Get device information finished
No details available.
DEBUG: 0.0 USBMS: Fetching list of books from device. oncard= None
DEBUG: 0.0 USBMS: dirs are: F:\
DEBUG: 5.4 USBMS: count found in cache: 533, count of files in metadata: 533,
need_sync: False
DEBUG: 5.4 USBMS: Finished fetching list of books from device. oncard= None
DEBUG: 5.4 USBMS: Fetching list of books from device. oncard= carda
DEBUG: 5.4 USBMS: dirs are: I:\
DEBUG: 13.8 USBMS: count found in cache: 1935, count of files in metadata: 193
5, need_sync: False
DEBUG: 13.8 USBMS: Finished fetching list of books from device. oncard= carda
DEBUG: 13.8 USBMS: Fetching list of books from device. oncard= cardb
Job: 2 Get list of books on device finished
No details available.
Setting metadata in: Dilbert at: c:\users\kerry\appdata\local\temp\calibre_0.7.1
3_v_pnhq.fb2
Setting metadata in: Daily Telegraph at: c:\users\kerry\appdata\local\temp\calib
re_0.7.13_jocl9p.fb2
Setting metadata in: Comics.com at: c:\users\kerry\appdata\local\temp\calibre_0.
7.13_vzdbjg.fb2
DEBUG: 15.9 USBMS: uploading 3 books
DEBUG: 23.6 USBMS: finished uploading 3 books
Job: 3 Upload 3 books to deviceilbert, Daily Telegraph, Comics.com finished
No details available.
DEBUG: 24.2 USBMS: adding metadata for 3 books
DEBUG: 24.2 USBMS: finished adding metadata
DEBUG: 25.6 USBMS: starting sync_booklists
Job: 3 Upload 3 books to deviceilbert, Daily Telegraph, Comics.com finished
No details available.
Job: 4 Send metadata to device finished
(13, 'Permission denied')

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 68, in run
File "site-packages\calibre\gui2\device.py", line 303, in _sync_bookli
sts
File "C:\Program Files (x86)\Calibre2\Lib\site-packages\calibre\device
s\usbms\driver.py", line 306, in sync_booklists
File "C:\Program Files (x86)\Calibre2\Lib\site-packages\calibre\device
s\usbms\driver.py", line 301, in write_prefix
IOError: [Errno 13] Permission denied: u'F:\\metadata.calibre'

(13, 'Permission denied')

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 68, in run
File "site-packages\calibre\gui2\device.py", line 303, in _sync_booklists
File "C:\Program Files (x86)\Calibre2\Lib\site-packages\calibre\devices\usbms\
driver.py", line 306, in sync_booklists
File "C:\Program Files (x86)\Calibre2\Lib\site-packages\calibre\devices\usbms\
driver.py", line 301, in write_prefix
IOError: [Errno 13] Permission denied: u'F:\\metadata.calibre'

[chaley]

Originally Posted by kgn

I cannot send books - the device will not connect. This is what happens when I plug in the PB302.

Basically the same as the old driver.....

I don't think my theory is correct, because there is no evidence in the log that the metadata.calibre file was opened for write before the permission denied error happened. It was successfully opened for read.

The exception is being thrown when the file is opened for write, not at the write itself. That open hasn't changed since 0.7.0, which points at something wrong with either the metadata.calibre file, the memory itself, or some mysterious sequence of operations. Six experiments to try come to mind. The first 4 are benign. The last two, and especially the last one, are more dangerous.

In the experiments below, in all cases start calibre with calibre-debug -g, so we can get the debug output. If you wish, you can delete the experimental driver.

1) I note that you have both memory and a card. Try the experiment with the card not in the device.

2) Start calibre, then plug in the device. Using a text editor that supports Unix-format line endings (\n), open the file f:\calibre.metadata, then write it back. The goal of the experiment is to find if you get a permission denied error on the save, with calibre running and having opened the file.

3) Ensure calibre is not running. Plug in the device. Copy (not move) the metadata.calibre off the device, then delete the file. Unplug the device, then plug it back in. Copy the file back. Start calibre and try to add a book.

4). Ensure calibre is not running. Add a book to the device main memory by hand, by copying the book to the device. Start calibre. You should see a line saying 'need sync = true' in the debug output, and calibre will attempt to write the file. If the exception is not raised, then the problem is being introduced later.

5) (more radical) Delete the metadata.calibre file. Start calibre. Wait some amount of time while metadata.calibre is being rebuilt. The problem with this approach is that the cached metadata is lost. When calibre rebuilds it from information in the files, the metadata could be wrong, making the OnDevice indications incorrect.

6) (most radical) Format the memory and resend all the books.

[carrie]

Originally Posted by Chuckels550

(13, 'Permission denied')

Traceback (most recent call last):
File "site-packages\calibre\gui2\device.py", line 68, in run
File "site-packages\calibre\gui2\device.py", line 303, in _sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 304, in sync_booklists
File "site-packages\calibre\devices\usbms\driver.py", line 301, in write_prefix
IOError: [Errno 13] Permission denied: u'K:\\metadata.calibre'

I've been having the exact same problem. I tend to only use PDF & LIT formats and I noticed that it only seemed to be the LIT files that were suffering from this error.

Basically the files were uploaded to my reader but the meta data wasn't transferred with it so Calibre doesn't recognize it correctly to note if it's been transferred to the reader or not.

I had to go back and delete the files from my reader and re-send to device, that seemed to solve the problem.