Template to search notes?

[chaley]

Originally Posted by Comfy.n

Today on v7.18, it took about the same time as yesterday while running from source, on a timed attempt. ~ 12 minutes.

Well, more thought required ...

How many author notes do you have?

[Comfy.n]

11877

[chaley]

@Comfy_n: I am trying to see why this takes so long and failing. Having a large library to work with will help. Any chance you would be willing to share your metadata.db and the contents of the .cal_notes folder? The notes.db isn't enough -- I need the resources as well.

[Comfy.n]

sure I can

[chaley]

Originally Posted by Comfy.n

sure I can

Thanks.

I think I have found the bottleneck. Getting your metadata.db and the notes folder will let me be sure.

NB: I don't need the books themselves.

[chaley]

Try this template. On my machine and using your library the search completes in 4 or 5 seconds.

Note that it is suitable to be used as a stored template with 2 arguments. I tested it as "books_with_notes_containing_text" with this as the actual template search.

python:
def evaluate(book, context):
    if context.arguments is None or len(context.arguments) != 2:
        # Set these to what you want
        field_name = 'authors'
        value_in_note = 'a'
    else:
        field_name = context.arguments[0]
        value_in_note = context.arguments[1]

    db = context.db.new_api

    # check if we have already cached the notes
    note_items = context.globals.get('items_with_notes', None)
    if note_items is not None:
        # We've already fetched which items have notes.
        # Get the cached search result values
        note_search_results = context.globals['note_search_results']
        item_name_map = context.globals['item_name_map']
    else:
        # First time. Get the items with notes and initialize
        # the search value cache.
        note_items = db.get_all_items_that_have_notes(field_name)
        context.globals['items_with_notes'] = note_items
        note_search_results = {}
        context.globals['note_search_results'] = note_search_results
        # db.get_item_id() uses a linear search. Avoid this by getting
        # and caching the map
        item_name_map = db.get_item_name_map(field_name)
        context.globals['item_name_map'] = item_name_map

    # Check if this book is a match -- that the field has a note containing
    # the desired text.
    
    # We must first get the item_id for the value in the field to be checked.
    field_value = book.get(field_name)
	if not field_value:
		return ''
    # if the field is multi-valued, use the first value
    if isinstance(field_value, list):
		if len(field_value) == 0:
			return ''
        field_value = field_value[0]
    # Now get the cached internal ID of the value in field_name
    item_id = item_name_map[field_value]

    # Does the item have a note? If not, give up now.
    if item_id not in note_items:
        return ''

    # The item has a note. Have we already checked it?
    if item_id not in note_search_results:
        # Item has a note but we haven't seen it before. Do the compare
        # on the plain text version of the note.
        result = ''
        # Get the note.
        note = db.notes_data_for(field_name, item_id)
        if note:
            # Get the plain text of the note.
            note = note['searchable_text'].partition('\n')[2]
        if note:
            # use a case insensitive compare to check if the search value is in the note
            from calibre.utils.icu import primary_contains
            result = 'Yes' if primary_contains(value_in_note, note) else ''
        # Cache the result of the comparison
        note_search_results[item_id] = result
        context.globals['note_search_results'] = note_search_results
    # Return the cached value
    return note_search_results.get(item_id, '')

[Comfy.n]

yes that works