Koreader: How to get metadata to device?

[WS64]

I'm working right now on an enhancement to add a search function to Koreader that allows a user to search for any text in any of Calibre's columns and then select a book from the list of results and open the book.
The first part works already, I can search for books and it automatically opens the first book it finds, I "just" have to work on the second part, to show the search results. I will work in the next days on that.

But there is still the question, how to get the metadata on the device?
I can see a few options for that.
I could read the data from the build-in reader. I will not follow this way since it has a few disadvantages: First, different databases for Kobo and Kindle, and none for Android devices. Second, hidden directories. I have all my books in one of those to make sure Nickel does not see my books, so my Kobo database does not have any metadata.

Another way is to use Calibre.
I guess all of us use Calibre, but still there are different ways how we put our books to the device. Some will send the files to Kobo, some to Kindle, and I choose a third way by connecting my library to a folder and load all the files into this folder and copy them via robocopy to my Kobos.
One way to use Calibre is to use the file .metadata.calibre that Calibre puts into the folder with your books. But this file is pretty huge and difficult to read, that's why I right now do not use this one neither.

What I use is Calibre's export catalog to CSV function. I upload this CSV file to my reader and read all the necessary info out of that one.
In my opinion necessary are: author, title, path to book. Once I have those it should not be a problem at all to add nearly every other of Calibre's columns, actually in my solution so far you can already use any column.

But my question, is there a better way to get the metadata to the device?
Something that is easy to handle for the end user?
There are some problems involved in this, e.g. the path to the book. Right now I have created a column in Calibre that repeats the setting from "Send to device" ({author_sort[0]}/{authors}/{title}), but since the default setting is to also change the characters to English equivalents this becomes {author_sort[0]:transliterate()}/{authors:transliterate()}/{title:transliterate()}, and if you have more complicated settings the string here will also get more complicated. Another problem is that this way I will not find out which format the book has (e.g. epub or pdf). Right now I don't care, I just check if I find either an epub or a pdf, I probably have to enhance this by all other possible file types.

Any suggestions?

[Ken Maltby]

If I remember correctly FAT has the metadata in the directory, not sure if there is any actually in the file itself. Correction it appears that, that changed with Vista. They are using the term "Properties" now and what you are looking for is under the details tab. Guess you could see if some of the data that you can see looking at the properties/details can be found using a HexEditor on the file.

OR You could copy a file, of the type you are interested in, then use the "Remove Properties and Personal Information" function and make a change to the copy's properties, then you could compare the two files and it should show were the difference occurs.


Luck;
Ken

[WS64]

No, the data I am looking for is for sure not in the file or the file properties.
Of course you could modify the file itself so the metadata become part of the epub, but again that is not a solution I would prefer, it means to modify all files and it will be slow.

[hawhill]

Errm, you get the data by extracting the metadata from the files, I'd suggest. I think most of our file backends already have some metadata access items. Metadata typically _is_ part of the epub. Or the PDF. Or the DJVU file.

Of course, one could start another way. However, most existing readers use the metadata embedded into the document files. I'd say that is what users expect. Extracting is a bit I/O intensive, so typically, metadata extracted is cached in a database of some kind, where a relationship to the actual file is stored alongside.

[WS64]

Well, my idea is for sure NOT to search through the whole file system but use the data from Calibre since this is already there. And for sure the data I have in Calibre are well-kept while the data in my books are not.
My solution (CSV export from Calibre) so far works super fast, right now I'm testing if I can use the data from metadata.calibre instead since then I could avoid having to export the CSV file every time...

[embryo]

I think that the "metadata.calibre" could be much faster and easier than the 'extraction' solution.
This file (one for every storage media) have all the tags, the custom columns and the -relative to the root- path of every book.
With the full filename, you also have the extension (pdf, epub ...)

There could be a field (or a settings file) that the user could specify the path and the format ({author_sort[0]}/{authors} - {title}) of the library.

The CSV way is also very good idea, with just one more easy step for the user.
I could live with that. ;o)

Because the default setting of Calibre is to change the Unicode characters to English equivalents, I 'Save to Disk' first and then copy the files to the reader by myself.

[davidfor]

WS64: The metadata.calibreis an easy way to do this, but it means that a connection to calibre has to be made at some time. The file is JSON format. There are libraries around for different languages to read that. Hopefully there is one you can use.

[WS64]

I wrote my own function so far to get the data from metadata.calibre, and it still is pretty fast (right now I can get a search result back within 30 seconds for a (manipulated) calibre file with 19000 datasets.)
Just have to think of something to get back data that go over a few lines, but that won't be a big problem either.

One question to that file, is the name metadata.calibre or .metadata.calibre? On my Kobos the file has the dot, on my Cybook not. Of course I can simply search for both and take the one I can find (if there is any, this solution for sure is only for Calibre user!), but I wonder what the naming conventions here are...

[eschwartz]

I have no idea why anything, but on my Kindle it is "metadata.calibre", no preceding dot.

[davidfor]

Originally Posted by WS64

One question to that file, is the name metadata.calibre or .metadata.calibre? On my Kobos the file has the dot, on my Cybook not. Of course I can simply search for both and take the one I can find (if there is any, this solution for sure is only for Calibre user!), but I wonder what the naming conventions here are...

It is device/driver specific. But, only the folder device, the Hanlin BOOX qand anything that subclasses of these should have the dot. Are you connecting to the Kobo using the folder device?

[WS64]

Originally Posted by davidfor

It is device/driver specific. But, only the folder device, the Hanlin BOOX qand anything that subclasses of these should have the dot. Are you connecting to the Kobo using the folder device?

Yes. Since I have to fill three readers I thought it would be easiest to have a master on my harddrive and robocopy this one to the readers.
So I check for metadata.calibre first and if not successful for the dot version, thanks!

[Ken Maltby]

Originally Posted by WS64

Yes. Since I have to fill three readers I thought it would be easiest to have a master on my harddrive and robocopy this one to the readers.
So I check for metadata.calibre first and if not successful for the dot version, thanks!

A file or folder name with a preceding dot/period is treated by Linux programs as "Hidden", this includes the Kobo program that processes files for its database. That could be a reason for the difference.

Luck;
Ken

[embryo]

Using 485, although the search is producing results, when I click on them I get a "File does not exist" message.
My books are in "/mnt/sd/.Books" and Calibre can see them just fine.

An other thing is that when I search a term in Unicode, it is case sensitive regardless of what I set in the Options.

[WS64]

Is the LIBRARY_PATH really set to "/mnt/sd/.Books"?
Manually check the metadata.calibre file in this folder for one of the books, does the lpath point to a valid book?

No idea yet about the case (in)sensitive...

[Ken Maltby]

Originally Posted by embryo

Using 485, although the search is producing results, when I click on them I get a "File does not exist" message.
My books are in "/mnt/sd/.Books" and Calibre can see them just fine.

An other thing is that when I search a term in Unicode, it is case sensitive regardless of what I set in the Options.

I am using 485 as well and it works fine in my AuraHD. It must have been defaulting to case insensitive, with my trial, I used all lower case and found items that were both upper and lower case. Not sure if the changes to the handling of Unicode are in effect for 485.

Luck;
Ken