Counting is hard: word breaks and XPath

[isarl]

Hello, thank you for taking the time to read my post.

I am interested in counting words in ebooks. “Simple!” you might respond. “Use the Count Pages plugin!” If this is your suggestion, then thank you – I already do! However, I am specifically interested in counting the words in only part of an ebook.

I got this idea when I started using KoboUtilities and examined the format in which the Kobo stores my “last reading position”, which looks like this:

Text/Chapter02.xhtml#kobo.27.1

In other words, it stores the location as the name of the document in the book's spine, and an XHTML tag ID (these tags and their ID values are inserted by the Kobo driver when converting to .kepub). This means that I can use the OEB container data types provided by Calibre to do something like:

from calibre.ebooks.oeb.polish.container import get_container
book = get_container("/path/to/book.kepub")
chapter = book.parsed("Text/Chapter02.xhtml")
excerpt = ''.join(chapter.xpath("//*[@id='kobo.27.1']/following::*//text()"))
from calibre.spell.break_iterator import count_words
count_words(excerpt)

The above code counts all words in the document (“chapter”/section/split) of that name after the indicated XHTML span. However, because str.join is unaware of display rules for HTML block elements, successive paragraph tags are joined without any space, and usually the first word of the next paragraph is counted along with the last word of the preceding paragraph (although some punctuation results in the correct count). Example:

>>> count_words("Hello.How are you?")
3
>>> count_words("‘Hello.’How are you?")
4

I am hesitant to roll my own word counting function as I strongly suspect that the ICU code core to count_words is much better than anything I can come up with. Is there perhaps a better XPath query I can use for this, or some other mechanism to excerpt the content I wish to count? Should I still use XPath, only be more intelligent about how I count words? (Perhaps I can drop the "//text()" suffix and be smart about iterating over the returned nodeset, e.g. counting words for each paragraph tag separately? But I'm not sure how I would do this without exhaustively enumerating every possible block-type tag name I might have to consider, and this also completely ignores that an individual book might have style rules which change one or more block elements to display inline.)

My ultimate goal with this code is to take two reading positions like Kobo stores, and count the words between them. There is extra logic involved in determining, “Are the starting and ending positions in the same document? Do the names documents exist in this book? Do the named tags exist in their documents?” which I have omitted here for the sake of brevity.

Thank you again for taking the time to read my post! Even if you can't help, I appreciate your time, and I hope you have a lovely day.

~isarl

[kovidgoyal]

See the code in polish/spell.py for many different ways of counting words some of which you should be able to adapt

[isarl]

Thank you Kovid! I appreciate the pointer and look forward to exploring the classes and methods available.