Can custom book data be displayed in a custom column?

[kiwidude]

I store a key called 'goodreads_id' in the plugin data store for books using db.add_custom_book_data() as recommended here and it is working great.

I am wondering if it is possible (for if a user wanted it) to display that value in a custom column. Alternatively for example have a column that indicates whether the value is populated. So users could see which books they have linked to Goodreads (and search based on this).

I had a brief look at program template functions, but I couldn't obviously see how you would have access to the custom book data.

Is it possible? I know chaley loves a challenge...

[chaley]

No. The template processor can see information only if it is in a Metadata instance. Every attribute in a Metadata instance must be serializable (able to be persistent), so it cannot contain a db handle.

If you want to provide this feature, then you will need to ask the user for a CC to use and populate it yourself.

[kiwidude]

That's a shame, thanks.

[chaley]

Thinking further about this, a few changes would make it possible. I am willing to make them if Kovid says 'go ahead' (something I am not convinced he will do for various reasons).

The idea is to include all the custom book data in the meta2 view. It would be done similarly to how the author sort map is done today, but returning a JSON-encoded string such as
{ "custom_data_name" : JSON_data_from_DB , custom_name_2, JSON_data_2, ...}. This string would be added to the values returned by get_metadata (mi.custom_data = row[nnn]). A template function would be provided that decodes that data, selects out a custom_data_name, then selects an attribute from that name.

An alternative to adding the information to the view is doing the necessary select in get_metadata. My problem with this approach is that we might end up doing the same select multiple times, because there is no guarantee that get_metadata is not called more than once for the same book.

This scheme has the advantage that almost no processing is required unless a) custom data exists, and b) someone uses the information.

The disadvantages that I can see are:
1) Even if there is no custom data, there would be a small performance penalty to put the None into the view. I think this penalty is very close to zero.
2) If there is custom data, then the JSON strings must be fetched and concatenated. This operation might add a few percent to the time to build the view, depending on the amount of data.
3) The template function would be rather slow, because it would decode the JSON data. It would also require that the custom data be a dict of (dicts/lists). The function would look something like "custom_data(data_name,item_name)". If the data under data_name is a list, then item_name must be an integer. If the data is a dict, then item_name must be a string.

With the above, one can create a composite custom column to show the custom data. For example, and making a large number of assumptions, the goodreads_id could be displayed using the template

program: custom_data('goodreads_plugin', 'goodreads_id')

It could become Yes/No using

program: test(custom_data('goodreads_plugin', 'goodreads_id'), 'Yes', 'No')

[kovidgoyal]

There's going to be an identifiers table added to the database to support non ISBN identifiers, lije DOI, lccn, google, adobe, asin, etc. so that will be the correct solution for this.

[kiwidude]

So now you have committed the new identifier stuff, would I be correct in saying that the correct way to display one of the new identifiers in a custom column is to use a program template and do something like:

name:GoodreadsId
def evaluate(self, formatter, kwargs, mi, locals, val):
    return mi.get_identifiers().get('goodreads_id','')

and then in my custom column I do this:
{'':GoodreadsId()}

Or is there a better way?

EDIT: I suppose this would improve things:

name:Identifier
def evaluate(self, formatter, kwargs, mi, locals, val):
    return mi.get_identifiers().get(val,'')

and then in my custom column I do this:
program: Identifier('isbn')

[chaley]

Originally Posted by kiwidude

So now you have committed the new identifier stuff, would I be correct in saying that the correct way to display one of the new identifiers in a custom column is to use a program template and do something like:

name:GoodreadsId
def evaluate(self, formatter, kwargs, mi, locals, val):
    return mi.get_identifiers().get('goodreads_id','')

and then in my custom column I do this:
{'':GoodreadsId()}

Or is there a better way?

The first thing you must do is add the identifier to the book using db2.set_identifier. Once that is done, then to see the identifier in a custom column, use the new 'select' function, which is a generalized version of what you proposed. Something like {identifiers:select(goodreads_id)}.

To search, use 'identifiers:goodreads_id:what-have-you.' This is a keypair search (something new), searching for key:value pairs. Matching the key and the value are independent, and can use contains, exact, and regexp matching. For example, identifiers:good:true will find all books with a key containing 'good'.

[kiwidude]

Originally Posted by chaley

Once that is done, then to see the identifier in a custom column, use the new 'select' function, which is a generalized version of what you proposed. Something like {identifiers:select(goodreads_id)}.

Ahhh a new function, no wonder I couldn't spot that on the doc page. I figured you would put something more convenient than a write your own one in...

I have been trying to test this just with the isbn value column since I haven't set my goodreads id yet. I can do queries in the search bar like this:
identifiers:isbn:9781842320136

But when I put this as a custom column it displays as blank:
{identifiers:select(isbn)}

[chaley]

Originally Posted by kiwidude

But when I put this as a custom column it displays as blank: {identifiers:select(isbn)}

I should have mentioned -- this morning during testing I found that Metadata.format_field did the wrong thing with the identifiers dict. I fixed it, but it isn't in trunk yet.

If you want to test, then add

=== modified file src/calibre/ebooks/metadata/book/base.py
--- src/calibre/ebooks/metadata/book/base.py	2011-03-01 15:05:18 +0000
+++ src/calibre/ebooks/metadata/book/base.py	2011-03-02 08:50:11 +0000
@@ -595,6 +595,8 @@
             elif key == 'series_index':
                 res = self.format_series_index(res)
             elif datatype == 'text' and fmeta['is_multiple']:
+                if isinstance(res, dict):
+                    res = [k + ':' + v for k,v in res.items()]
                 res = u', '.join(sorted(res, key=sort_key))
             elif datatype == 'series' and series_with_index:
                 res = res + ' [%s]'%self.format_series_index()

The change should appear in trunk tonight.

[kiwidude]

Cool. I was trying all sorts of variants wondering if I had screwed up, knowing chaley never steers me wrong, heh. I've got a workaround of my own custom function as per my edited post above I did while you were posting a response in the meantime anyways.

One thing I noticed when I was trying the identifiers:isbn:xxx search. It seems to be doing a starts with match (if I drop a few digits off the ISBN I get multiple results). Is that intentional, I wouldn't have thought that would be particular useful for an "identifier" field? If it is intentional, how do I force an exact match?

EDIT: Never mind, realised it was just a normal search and just use :=