Full Text Service (FTS) - request

Yarrow · 02-07-2024, 10:24 AM

Really love FTS BUT would welcome FTS returns to show the number of citations of the search term per book as this would show most relevant books to view rather than current position of just showing one. Or alternatively an option to choose 1, 2, 3, 4, 5 returns per book. Is this possible?

kovidgoyal · 02-07-2024, 10:32 AM

No, sadly the FTS engine only tracks a single match per document (book format). You can open the book in the viewer and see th enumber of matches.

Yarrow · 02-07-2024, 11:18 AM

Thanks Kovid. I thought that if it was possible to have more than one match shown or the number of matches per book it would help deciding which books to go to to search further. I have a large library and sometimes get 30+ books returned and it's hard to choose which ones to view. Not Grumbling as a total fan of Calibre and the FTS option. Really grateful for your work over many years on Calibre.

kovidgoyal · 02-07-2024, 11:02 PM

Yeah, I can see your use case, however it's simply not supported by the underlying engine. It could theoretically be implemented by writing a custom SQLITE FTS5 extension function, but that's rather a lot of work to implement. Maybe someday when I have the time.

ownedbycats · 02-08-2024, 02:20 AM

I would find this useful as well. (I think I might've actually requested it once.)

Yarrow · 02-08-2024, 12:03 PM

Quote:

Originally Posted by kovidgoyal

Yeah, I can see your use case, however it's simply not supported by the underlying engine. It could theoretically be implemented by writing a custom SQLITE FTS5 extension function, but that's rather a lot of work to implement. Maybe someday when I have the time.

Many thanks for thinking about doing this. I know how busy you must be.

By the way, love Mumbai. My son was working there for a few years and we went to see him a few times. He lived in Bandra West.

Yarrow · 02-08-2024, 12:06 PM

Quote:

Originally Posted by kovidgoyal

Yeah, I can see your use case, however it's simply not supported by the underlying engine. It could theoretically be implemented by writing a custom SQLITE FTS5 extension function, but that's rather a lot of work to implement. Maybe someday when I have the time.

Yarrow · 02-08-2024, 12:07 PM

:

Quote:

Originally Posted by ownedbycats

I would find this useful as well. (I think I might've actually requested it once.)

Yarrow · 02-09-2024, 02:34 PM

An alternative approach would be to use an 'AI engine' to interrogate a calibre library. Just noticed that the IOS app Evernote is offering users the option to search with AI - whatever that means in this context.

Quoth · 02-09-2024, 03:18 PM

Quote:

Originally Posted by Yarrow

An alternative approach would be to use an 'AI engine' to interrogate a calibre library. Just noticed that the IOS app Evernote is offering users the option to search with AI - whatever that means in this context.

Daft approach.

DNSB · 02-09-2024, 03:40 PM

Quote:

Originally Posted by Yarrow

An alternative approach would be to use an 'AI engine' to interrogate a calibre library. Just noticed that the IOS app Evernote is offering users the option to search with AI - whatever that means in this context.

If you have the money to spare...

BetterRed · 02-09-2024, 05:13 PM

Quote:

Originally Posted by Yarrow

An alternative approach would be to use an 'AI engine' to interrogate a calibre library. Just noticed that the IOS app Evernote is offering users the option to search with AI - whatever that means in this context.

Maybe this would be of interest, it's on my list of things to look at ==>> Open Semantic Search

For the past decade or so I've used Windows Search and X1 to content search calibre libraries. And, since its release in 2016, in conjunction with the Drop Search Results calibre plugin.

BR

kovidgoyal · 07-04-2024, 01:24 AM

Oh and I should clarify the results as shown in calibre are already ranked by the bm25 algorithm so when you get multiple results for a query you should start with the topmost one. Generally it will be more relevant than subsequent ones. What you are asking for is basically using a simpler relevance ranking by number of matches only rather than bm25. I looked into this and I dont really see how that would be helpful. That said, its not out of the question to provide such an alternate ranking, but its not worth the effort to me, patches welcome.

Here is a technical writeup on how bm25 works:
https://www.elastic.co/blog/practica...-its-variables

02-07-2024, 10:24 AM	#1
Yarrow Enthusiast Posts: 45 Karma: 10 Join Date: Aug 2007 Location: Lancashire, UK Device: Sony PRS500& PRS505; iPad; and Kindle	Full Text Service (FTS) - request Really love FTS BUT would welcome FTS returns to show the number of citations of the search term per book as this would show most relevant books to view rather than current position of just showing one. Or alternatively an option to choose 1, 2, 3, 4, 5 returns per book. Is this possible?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
FTS for UTF-8 Text Formats in Non-TXT File Extensions	DaltonST	Development	7	03-12-2023 08:40 PM
Full-text search not really full-text	secasa	Library Management	5	10-01-2022 11:11 PM
[Feature Request] Manage "Full Text" search	shuvashish76	Library Management	7	07-17-2022 12:54 AM
Feature request: full text search	Laisvunas	EPUBReader	3	04-03-2011 12:47 PM

02-07-2024, 10:32 AM	#2
kovidgoyal creator of calibre Posts: 44,523 Karma: 24495784 Join Date: Oct 2006 Location: Mumbai, India Device: Various	No, sadly the FTS engine only tracks a single match per document (book format). You can open the book in the viewer and see th enumber of matches.

02-07-2024, 11:18 AM	#3
Yarrow Enthusiast Posts: 45 Karma: 10 Join Date: Aug 2007 Location: Lancashire, UK Device: Sony PRS500& PRS505; iPad; and Kindle	Thanks Kovid. I thought that if it was possible to have more than one match shown or the number of matches per book it would help deciding which books to go to to search further. I have a large library and sometimes get 30+ books returned and it's hard to choose which ones to view. Not Grumbling as a total fan of Calibre and the FTS option. Really grateful for your work over many years on Calibre.

02-07-2024, 11:02 PM	#4
kovidgoyal creator of calibre Posts: 44,523 Karma: 24495784 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Yeah, I can see your use case, however it's simply not supported by the underlying engine. It could theoretically be implemented by writing a custom SQLITE FTS5 extension function, but that's rather a lot of work to implement. Maybe someday when I have the time.

02-08-2024, 02:20 AM	#5
ownedbycats Custom User Title Posts: 9,530 Karma: 64500175 Join Date: Oct 2018 Location: Canada Device: Kobo Libra H2O, formerly Aura HD	I would find this useful as well. (I think I might've actually requested it once.)

02-09-2024, 02:34 PM	#9
Yarrow Enthusiast Posts: 45 Karma: 10 Join Date: Aug 2007 Location: Lancashire, UK Device: Sony PRS500& PRS505; iPad; and Kindle	An alternative approach would be to use an 'AI engine' to interrogate a calibre library. Just noticed that the IOS app Evernote is offering users the option to search with AI - whatever that means in this context.

07-04-2024, 01:24 AM	#13
kovidgoyal creator of calibre Posts: 44,523 Karma: 24495784 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Oh and I should clarify the results as shown in calibre are already ranked by the bm25 algorithm so when you get multiple results for a query you should start with the topmost one. Generally it will be more relevant than subsequent ones. What you are asking for is basically using a simpler relevance ranking by number of matches only rather than bm25. I looked into this and I dont really see how that would be helpful. That said, its not out of the question to provide such an alternate ranking, but its not worth the effort to me, patches welcome. Here is a technical writeup on how bm25 works: https://www.elastic.co/blog/practica...-its-variables