Beta Beta Test of Major New Features

[Starson17]

This is a non-sticky discussion thread for the unofficial beta beta development available from lp:~cbhaley/calibre/charles.custcol

Charles,
Instead of doing this by email or PM, this is easier for me, so I'm moving the discussion here.

Our last round is quoted below:
1) I'll check the attempt to delete and get back to you.
2) I suspected you might not see the oddity of LN, FN sorting
3) Yes, there are often unsuspected benefits of new features.
4) New bug:
a) connect to empty folder, add books
b) disconnect
c) reconnect to same folder. The author is now listed as title and title as author for each book.

Originally Posted by chaley

I never see this. Is there any chance that something has the file open, preventing its deletion?

It would help if you checked that calibre actually attempts to delete the file. A print statement in devices.usbms.driver at line 205 would do it. Best would be to print the value that unlink returns.

Actually, I didn't see it because I keep names in LN, FN order.

Is this new behavior? Did it do something different before? Ahhh, you wouldn't know.

I can correct the ondevice scanner and metadata correcter to recompute author_sort for those books that don't have it, then send the metadata back to the 'device'. Once I do that, I will use author_sort.

Interesting use of the feature. My guess is that more like this will show up once people more creative than me get their hands on it.

[chaley]

Originally Posted by Starson17

2) I suspected you might not see the oddity of LN, FN sorting

I have checked with Kovid, and it is broken. Oddly enough, it has been broken since forever. I will fix it soonly .

4) New bug:
a) connect to empty folder, add books
b) disconnect
c) reconnect to same folder. The author is now listed as title and title as author for each book.

I don't see this one either. Books I add show the right info.

As an experiment, disconnect from the folder, go to that folder and delete the file .metadata.calibre, and then reconnect to the folder. The device driver will rebuild the metadata from what is found in the books. Are the results different?

[chaley]

Ahhh -- you were adding a new book, not copying from the library?

I just tried doing that, and it worked for me.

This operation will use the preference settings for adding books. Any chance you have ticked 'read metadata only from file name' and that your regexp is getting the two reversed?

My settings are vanilla: no boxes checked on the add preferences tab, with the standard regexp. I added an epub that had metadata in it, and the metadata was right. I added a text file and got the text file name as title with author unknown.

[Starson17]

[QUOTE=Starson17;913208]This happens with .txt format files. In one pdf it mangled, in another, it looks correct. Some .lits are OK, others not. ...
Still testing, this is a prelim report.
The data looks correct in the GUI during the initial import. It's wrong during the reconnect. The errors can be seen in the .metadata.calibre cache file.

[Starson17]

Originally Posted by chaley

Ahhh -- you were adding a new book, not copying from the library?

No, in each case I was transferring from the library. See below.

I just tried doing that, and it worked for me.

That's why it's beta

This operation will use the preference settings for adding books. Any chance you have ticked 'read metadata only from file name' and that your regexp is getting the two reversed?

They were coming from the main library. No options checked on Adding books.

My settings are vanilla: no boxes checked on the add preferences tab, with the standard regexp. I added an epub that had metadata in it, and the metadata was right. I added a text file and got the text file name as title with author unknown.

What happened with a Send to Device - Main Memory from main library of a txt file?

[chaley]

Originally Posted by Starson17

What happened with a Send to Device - Main Memory from main library of a txt file?

Works for me.

I wonder if the delete problem is not related to this? Clearly something is stopping calibre from certain file manipulations. If the metadata cache is not being written, then it tries to determine the metadata from the file. For text files this is clearly impossible, so it uses the filename pattern stored in preferences.

    pat = re.compile(prefs.get('filename_pattern'))
    name = name.replace('_', ' ')
    match = pat.search(name)

In other words, it will use the last value you put into the regexp box for adding books. If this regexp does not match the filename pattern used for sending to device (also in preferences), then nothing good will happen.

I will look at the path names for local drives. I can easily imagine that the paths are not being normalized to windows (you run vista?), which might cause trouble.

[Starson17]

Somewhere I lost a reply to you. The author/title munge is visible in the .metadata.cache file. I'm multitasking some jobs for the wife: welding oxygen and goldfish - don't ask.

[Starson17]

Originally Posted by Starson17

The author/title munge is visible in the .metadata.cache file.

This is for the SD card mounted in the SD reader. Initial transfer of ebooks looks perfect in the GUI, on the SD card and in the .metadata.cache (AFAICT). Disconnect the folder and reconnect to it - nothing else. Then open the .metadata.cache file and I see all kinds of munge in author/title. Authors become Unknown or title. Titles messed up, etc.

I suggest we focus on this, then see if the delete is still an issue.

[Starson17]

Originally Posted by chaley

If the metadata cache is not being written, then it tries to determine the metadata from the file.

AFAICT, it is correctly writing the metadata.calibre file (I assume that is the cache) on the SD card (and on the HD for that matter - yes I'm on Vista). Then it munges it on a reconnect.

I probably do have a mismatch between the regex used to Add Books and the pattern used to save the book out. Normally I have the option to read metadata only from filename on, and I vary the Add Books import regex. Does that regex come into play during the reconnect to the folder (or during the initial transfer from main library to Device)?

In other words, it will use the last value you put into the regexp box for adding books. If this regexp does not match the filename pattern used for sending to device (also in preferences), then nothing good will happen.

Does this happen even with a simple transfer from the main GUI library?

[Starson17]

Originally Posted by Starson17

Does this happen even with a simple transfer from the main GUI library?

It looks like the reconnect is attempting to determine the ebook metadata from the ebooks a second time. It reads the metadata internally from the file, if it can, and if not, uses the regex on the filename. Then it revises the correct metadata in the cache file and ends up with a mess. I can fix this by controlling my settings, but it will be problematical for others.

I'll test to verify. (I think this is what you were saying, but it just sank in.)

[chaley]

Yes, that is what I was saying, and yes, it will be problematic. It shouldn't recompute the metadata.

I am now looking at when that happens and why. I have seriously broken the code, unfortunately, so it will be a little while before I get back to you.

[Starson17]

Originally Posted by chaley

Yes, that is what I was saying, and yes, it will be problematic. It shouldn't recompute the metadata.

I am now looking at when that happens and why. I have seriously broken the code, unfortunately, so it will be a little while before I get back to you.

Can you post the default Add Save regex and the Send to Device regex. I can recreate something similar to them, but I'd like to use the defaults, and I don't seem to have them in my notes.

[chaley]

Under some conditions that I haven't yet figured out, it uses
re.compile(r'^(?P<title>[ \S]+?)[ _]-[ _](?P<author>[ \S]+?)_+\d+'))

The default in preferences is '(?P<title>.+) - (?P<author>[^_]+)'

[Starson17]

Originally Posted by chaley

The default in preferences is

OK, this will confirm that setting regex to defaults mostly fixes things. I say "mostly" due to the oddity of text file metadata. I vaguely recall that calibre attempts to divine the metadata from inside the text file rather than always getting it from the filename. If the "metadata only from filename" option is off, it is getting the metadata of a txt file incorrectly on the reread. Apparently calibre never writes the correct metadata internally into a txt file, so with that option off, on the reread of metadata, it's always wrong. Other formats have correct metadata written internally, or if not, the metadata is picked up from the filename on the reread, so I saw errors only with txt files after the default regexes were set.

Turning on the "metadata only from filename" option fixed the txt file problem.

I'll check the delete issues now that this bug is at least identified.

[Starson17]

There's no confirm dialog on delete.