Register Guidelines E-Books Today's Posts Search

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 09-13-2024, 12:27 PM   #16
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,097
Karma: 7908993
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by Comfy.n View Post
Today on v7.18, it took about the same time as yesterday while running from source, on a timed attempt. ~ 12 minutes.
Well, more thought required ...

How many author notes do you have?
chaley is offline   Reply With Quote
Old 09-13-2024, 12:37 PM   #17
Comfy.n
want to learn what I want
Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.
 
Posts: 1,353
Karma: 6874872
Join Date: Sep 2020
Device: Calibre E-book viewer
11877
Comfy.n is offline   Reply With Quote
Old 09-13-2024, 03:14 PM   #18
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,097
Karma: 7908993
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
@Comfy_n: I am trying to see why this takes so long and failing. Having a large library to work with will help. Any chance you would be willing to share your metadata.db and the contents of the .cal_notes folder? The notes.db isn't enough -- I need the resources as well.
chaley is offline   Reply With Quote
Old 09-13-2024, 03:16 PM   #19
Comfy.n
want to learn what I want
Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.
 
Posts: 1,353
Karma: 6874872
Join Date: Sep 2020
Device: Calibre E-book viewer
sure I can
Comfy.n is offline   Reply With Quote
Old 09-13-2024, 04:33 PM   #20
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,097
Karma: 7908993
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by Comfy.n View Post
sure I can
Thanks.

I think I have found the bottleneck. Getting your metadata.db and the notes folder will let me be sure.

NB: I don't need the books themselves.

Last edited by chaley; 09-13-2024 at 05:13 PM.
chaley is offline   Reply With Quote
Old 09-14-2024, 12:41 PM   #21
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,097
Karma: 7908993
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Try this template. On my machine and using your library the search completes in 4 or 5 seconds.

Note that it is suitable to be used as a stored template with 2 arguments. I tested it as "books_with_notes_containing_text" with this as the actual template search.
Click image for larger version

Name:	Clipboard01.jpg
Views:	53
Size:	71.4 KB
ID:	210811

Code:
python:
def evaluate(book, context):
    if context.arguments is None or len(context.arguments) != 2:
        # Set these to what you want
        field_name = 'authors'
        value_in_note = 'a'
    else:
        field_name = context.arguments[0]
        value_in_note = context.arguments[1]

    db = context.db.new_api

    # check if we have already cached the notes
    note_items = context.globals.get('items_with_notes', None)
    if note_items is not None:
        # We've already fetched which items have notes.
        # Get the cached search result values
        note_search_results = context.globals['note_search_results']
        item_name_map = context.globals['item_name_map']
    else:
        # First time. Get the items with notes and initialize
        # the search value cache.
        note_items = db.get_all_items_that_have_notes(field_name)
        context.globals['items_with_notes'] = note_items
        note_search_results = {}
        context.globals['note_search_results'] = note_search_results
        # db.get_item_id() uses a linear search. Avoid this by getting
        # and caching the map
        item_name_map = db.get_item_name_map(field_name)
        context.globals['item_name_map'] = item_name_map

    # Check if this book is a match -- that the field has a note containing
    # the desired text.
    
    # We must first get the item_id for the value in the field to be checked.
    field_value = book.get(field_name)
	if not field_value:
		return ''
    # if the field is multi-valued, use the first value
    if isinstance(field_value, list):
		if len(field_value) == 0:
			return ''
        field_value = field_value[0]
    # Now get the cached internal ID of the value in field_name
    item_id = item_name_map[field_value]

    # Does the item have a note? If not, give up now.
    if item_id not in note_items:
        return ''

    # The item has a note. Have we already checked it?
    if item_id not in note_search_results:
        # Item has a note but we haven't seen it before. Do the compare
        # on the plain text version of the note.
        result = ''
        # Get the note.
        note = db.notes_data_for(field_name, item_id)
        if note:
            # Get the plain text of the note.
            note = note['searchable_text'].partition('\n')[2]
        if note:
            # use a case insensitive compare to check if the search value is in the note
            from calibre.utils.icu import primary_contains
            result = 'Yes' if primary_contains(value_in_note, note) else ''
        # Cache the result of the comparison
        note_search_results[item_id] = result
        context.globals['note_search_results'] = note_search_results
    # Return the cached value
    return note_search_results.get(item_id, '')
chaley is offline   Reply With Quote
Old 09-14-2024, 12:49 PM   #22
Comfy.n
want to learn what I want
Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.Comfy.n ought to be getting tired of karma fortunes by now.
 
Posts: 1,353
Karma: 6874872
Join Date: Sep 2020
Device: Calibre E-book viewer
yes that works
Comfy.n is offline   Reply With Quote
Old 09-21-2024, 07:16 PM   #23
Montana Harper
Zealot
Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.
 
Montana Harper's Avatar
 
Posts: 108
Karma: 2029154
Join Date: Sep 2013
Location: Pacific Northwest
Device: iPad Mini, iPhone 12, Kindle Paperwhite 3
Quote:
Originally Posted by chaley View Post
Try this template. On my machine and using your library the search completes in 4 or 5 seconds.
This is the solution I chose, as my library is currently 17000+ books and is probably going to expand relatively quickly. (It's my fanfic library, the size of which is limited only by my interest and AO3's server stability.) It works wonderfully!

One question, though: is there any way to modify that template to iterate over all the authors for a given title? That would be the absolute ideal solution for me.

Montana Harper is offline   Reply With Quote
Old 09-22-2024, 08:59 AM   #24
chaley
Grand Sorcerer
chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.chaley ought to be getting tired of karma fortunes by now.
 
Posts: 12,097
Karma: 7908993
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
Quote:
Originally Posted by Montana Harper View Post
One question, though: is there any way to modify that template to iterate over all the authors for a given title? That would be the absolute ideal solution for me.

This template checks every value for a field in a book. If the field is "authors" then it checks every author.
Code:
python:
def evaluate(book, context):
	if context.arguments is None or len(context.arguments) != 2:
		# Set these to what you want
		field_name = 'authors'
		value_in_note = 'note'
	else:
		field_name = context.arguments[0]
		value_in_note = context.arguments[1]

	db = context.db.new_api

	# check if we have already cached the notes
	note_items = context.globals.get('items_with_notes', None)
	if note_items is not None:
		# We've already fetched which items have notes.
		# Get the cached search result values
		note_search_results = context.globals['note_search_results']
		item_name_map = context.globals['item_name_map']
	else:
		# First time. Get the items with notes and initialize
		# the search value cache.
		note_items = db.get_all_items_that_have_notes(field_name)
		context.globals['items_with_notes'] = note_items
		note_search_results = {}
		context.globals['note_search_results'] = note_search_results
		# db.get_item_id() uses a linear search. Avoid this by getting
		# and caching the map
		item_name_map = db.get_item_name_map(field_name)
		context.globals['item_name_map'] = item_name_map

	# Check if this book is a match -- that the field has a note containing
	# the desired text.	
	# We must first get the item_id for the value in the field to be checked.
	field_values = book.get(field_name)
	if not field_values:
		return ''
	
	# We want to check every value in the item, so use a list.
	# If the given field is not multi-valued, turn it into a list
	if not isinstance(field_values, (tuple, list, set)):
		field_values = (field_values,)

	# Loop over the field values, checking each one. Stop on first success
	result = ''
	for field_value in field_values:
		# Get the cached internal ID of the value in field_name
		item_id = item_name_map[field_value]

		# Does the item have a note? If not, give up now.
		if item_id not in note_items:
			continue

		# The item has a note. Have we already checked it?
		result = note_search_results.get(item_id)
		if result is not None:
			# We've already checked this item.
			if result:
				# It matched. Break out of the loop
				break
			# It didn't match. Check the next item
			continue
		# Item has a note but we haven't seen it before. Do the compare
		# on the plain text version of the note.
		# Get the note.
		note = db.notes_data_for(field_name, item_id)
		if note:
			# Get the plain text of the note.
			note = note['searchable_text'].partition('\n')[2]
		if note:
			# use a case insensitive compare to check if the search value is in the note
			from calibre.utils.icu import primary_contains
			result = 'Yes' if primary_contains(value_in_note, note) else ''
		# Cache the result of the comparison
		note_search_results[item_id] = result
		if result:
			break

# Cache the updated results
	context.globals['note_search_results'] = note_search_results
	return result

Last edited by chaley; 09-29-2024 at 06:39 AM. Reason: Minor correction to the template
chaley is offline   Reply With Quote
Old 09-28-2024, 10:24 PM   #25
Montana Harper
Zealot
Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.Montana Harper ought to be getting tired of karma fortunes by now.
 
Montana Harper's Avatar
 
Posts: 108
Karma: 2029154
Join Date: Sep 2013
Location: Pacific Northwest
Device: iPad Mini, iPhone 12, Kindle Paperwhite 3
Quote:
Originally Posted by chaley View Post
This template checks every value for a field in a book. If the field is "authors" then it checks every author.
Perfect!

Montana Harper is offline   Reply With Quote
Reply


Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Template Search: Exact matching ownedbycats Library Management 3 04-03-2022 06:01 PM
Template: Converting a search & replace into a template ownedbycats Library Management 11 03-26-2021 05:32 AM
Nova Pro : Notes Template MachinaCarnis Onyx Boox 0 01-30-2020 02:57 PM
template or search feature question bulldogmo Calibre 2 08-06-2014 07:34 PM


All times are GMT -4. The time now is 03:46 AM.


MobileRead.com is a privately owned, operated and funded community.