[GUI Plugin] Count Pages

[PeterT]

Because who knows how many OTHER edge cases like that exist.

[rpgmaker]

Originally Posted by PeterT

Because who knows how many OTHER edge cases like that exist.

I know but it's good to encourage people to report them. Even if they can't be fixed (or dev decides they aren't worth a fix) it's good to be aware.

[PeterT]

You probably have to open this as a calibre bug; the plugin uses

from calibre.utils.wordcount import get_wordcount_obj

    book_text = _read_epub_contents(iterator, strip_html=True)
    wordcount = get_wordcount_obj(book_text)

to get the word count

[JSWolf]

Originally Posted by PeterT

Is it that critical that the word count be precise?

In the book I just read, it makes a difference of least 112 words. Why bother to count the words if it's possibly inaccurate?

[BetterRed]

Originally Posted by rpgmaker

I know but it's good to encourage people to report them. Even if they can't be fixed (or dev decides they aren't worth a fix) it's good to be aware.

Are there any ISO endorsed standards for counting words?

If not, then it's not a bug, its merely a difference of opinion on how composite words ought to be counted.

There are many users (possibly millions) who have used the PI against their existing libraries (possibly billions of books), most (probably vast majority) of whom are only interested in consistent relative counts.

Any change would have to be an 'opt-in ' choice and it would open up discussion to the vagaries of different word separators and different languages - e.g. I would not want a non-breaking hyphen to count as a word separator.

BR

[PeterT]

The only use I've ever had is to get a rough estimate of the size of a book; I see no use in knowing the precise number of words. Besides, do you count the words in the index, or the table of contents?

[JSWolf]

[QUOTE=PeterT;3236052]The only use I've ever had is to get a rough estimate of the size of a book; I see no use in knowing the precise number of words. Besides, do you count the words in the index, or the table of contents?/QUOTE]

Well, I've reported the bug in the Calibre forum. But, as for ToC, I usually delete it as the NCX does just fine.

Anyway, thank you for finding the source of the bug. If it gets fixed, all well and good. If not, not that big a deal.

[jackie_w]

Just FYI for anyone interested ...

Based on a very quick look, the current algorithm for calculating Wordcount appears to be taken from main calibre code (calibre.utils.wordcount, author="Ryan Ginstrom") rather than being created specifically for the 'Count Pages' plugin.

I imagine getting it changed would involve either:

1. Convincing Kovid/Ryan Ginstrom to change it calibre-wide (good luck with that ) or
   
2. Replacing a couple of lines in your own personal copy of 'Count Pages' statistics.py
   
   From:
   

       wordcount = get_wordcount_obj(book_text)
       return wordcount.words

   To:
   

       words = _my_wordcount_algorithm(book_text)
       return words
   
   def _my_wordcount_algorithm(text):
       wcount = <Your personal wordcount algorithm here>
       return wcount

ETA: Damn - not quick enough!

[rpgmaker]

Originally Posted by BetterRed

Are there any ISO endorsed standards for counting words?

If not, then it's not a bug, its merely a difference of opinion on how composite words ought to be counted.

...

... I would not want a non-breaking hyphen to count as a word separator.

You're right and you bring up a very interesting point. As far as I'm concerned this settles it, let alone the fact that the word count seems to come from calibre itself. This plugin has never been "accurate" in the traditional sense of the word anyways... it's value lies in the rough idea it gives you about how long a book is, specially when compared to others in your library and its usefulness in that aspect hasn't change one bit because of this.

[theducks]

Whose rules?

IIRC in Jr High Typing (Royal Manuals), 7 (error free) characters was a word for your WPM count.
This Duck only needed 1 digit most of the time

[Divingduck]

Originally Posted by JSWolf

I have found some bugs in count pages that can cause it to count two words as one word.

except…if
except—if
except–if

All of those cases cause two words to be counted as one. I do hope someone is available to fix this. Thanks.

Is it so simple?

What will you do with words like 3D printer in German language called 3-D-Drucker as one word? Count it as 3 words is definitely wrong for that language and this kind of exceptions happen a lot more. Guess, in other languages too. To cover this you will need a dictionary for each language and I am quite sure, you will not cover all exceptions as e.g. in German language there is no rule to prevent constructions with a "-" between words. This is often used for a better reading of long word constructions.

[davidfor]

Originally Posted by Divingduck

Is it so simple?

What will you do with words like 3D printer in German language called 3-D-Drucker as one word? Count it as 3 words is definitely wrong for that language and this kind of exceptions happen a lot more. Guess, in other languages too. To cover this you will need a dictionary for each language and I am quite sure, you will not cover all exceptions as e.g. in German language there is no rule to prevent constructions with a "-" between words. This is often used for a better reading of long word constructions.

It isn't obvious from JSWolf's post, but the last one is actually an "en-dash", not a hyphen. I have no idea whether that should be considered a word delimiter or word joiner.

The method Kovid has mentioned for word counting accepts a locale. That should sort the differences out between the languages.

[davidfor]

I wasn't going to do this, but, Kovid added an extra method and the count wasn't taking into account the language of the book, so...

Attached is a beta version of the plugin that uses the ICU Word Iterator to do the count. This does the word count using the language set in the book. If this is not set, it will default to English. For older versions of calibre that do not include the appropriate methods for the ICU Word Iterator, the word count will use the older method.

But, for this beta, the count is actually being done twice. The old method is always done and printed to the job log. Then it attempts to use the new method. I have done this to get an idea of how much difference there is for the two methods. For English, the difference is small enough that it doesn't bother me. But, for other languages, it might be more. I don't have enough non-English test books to check. I would be interested to know if there is a significant difference for any language.

To view the two counts, you need to open the job list (click "Jobs" in the bottom right of the calibre window), select the count pages job and press the "Show job details".

If anyone finds a problem, please report it. If none are found, and there are no objections to changing the word count algorithm, I will arrange to release this sometime next week.

[kovidgoyal]

@davidfor: I suggest defaulting to the calibre interface language rather than English when no language is available (from calibre.utils.localization import get_lang)

[BetterRed]

Would it be possible to retain the 'old' method, at least as a configurable option.

Added: I appreciate I don't have to install the new version, but I prefer to keep all my software up to date.

Also what will it do with non-breaking hyphens in English, maybe someone can point me at the relevant ICU doco for that level of detail - looked for it, but failed to find it, but ...

BR