[GUI Plugin] WordDumb

[booklover2111]

Originally Posted by xxyzz

v2.0.2
use absolute redis path on macOS, rm useless try except

I just updated the new version. After that, your plugin doesn't work anymore. It always shows this error


calibre, version 5.6.0 (win32, embedded-python: True)
Conversion error: Failed: Generating Word Wise

Starting job: Generating Word Wise
Job: "Generating Word Wise" failed with error:
Traceback (most recent call last):
File "calibre\gui2\threaded_jobs.py", line 83, in start_work
File "calibre_plugins.worddumb.parse_job", line 26, in do_job
File "calibre_plugins.worddumb.database", line 76, in start_redis_server
File "subprocess.py", line 854, in __init__
File "subprocess.py", line 1307, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified

Called with args: (<calibre.db.cache.Cache object at 0x000000E956B02340>, [28], 'C:\\Users\\MyPC\\AppData\\Roaming\\calibre\\plugi ns\\WordDumb.zip') {'notifications': <queue.Queue object at 0x000000E956D55B80>, 'abort': <threading.Event object at 0x000000E956D55B50>, 'log': <calibre.utils.logging.GUILog object at 0x000000E959207400>}

[xxyzz]

Originally Posted by booklover2111

I just updated the new version. After that, your plugin doesn't work anymore. It always shows this error

WordDumb needs redis(https://redis.io/download) since v2.0.0. The reason for that is redis is way more efficient than python sqlite3 library. I don't know how to install redis on Windows, you can try install it on WSL(https://en.wikipedia.org/wiki/Window...stem_for_Linux). I haven't try it yet since I don't use Windows.

You also can stick with the old version if you don't care much about the performance. But I still encourage you try the new version, it's really fast.

Update: get redis running on Windows doesn't worth the effort, it just waste time. v1.8.1 uses multiple threads so it's not too slow but uses lots CPU resources.

[booklover2111]

Originally Posted by xxyzz

WordDumb needs redis(https://redis.io/download) since v2.0.0. The reason for that is redis is way more efficient than python sqlite3 library. I don't know how to install redis on Windows, you can try install it on WSL(https://en.wikipedia.org/wiki/Window...stem_for_Linux). I haven't try it yet since I don't use Windows.

You also can stick with the old version if you don't care much about the performance. But I still encourage you try the new version, it's really fast.

I have tried to install redis on my computer with no success ( I dual-boot windows and Linux Mint by the way). When I installed redis, on the terminal it shows the error like this .

I'm so confused, because I couldn't manage to install redis on the computer (I'm no expert when it comes to Linux and terminal things).

I really really like your plugin, and I think the final solution is to downgrade to its older version, but it is also a lost cause. After googling the net for an hour, I couldn't find any tutorial or instruction to help me.

[xxyzz]

Originally Posted by booklover2111

I have tried to install redis on my computer with no success ( I dual-boot windows and Linux Mint by the way). When I installed redis, on the terminal it shows the error like this

You don't need to compile redis, you can install it via package manager. Please read the README(https://github.com/xxyzz/WordDumb#how-to-use). The commands for Ubuntu should work for you.

[booklover2111]

Using the commands from your HOW-TO page, I have installed redis successfully, but when I ran the plugin, it still didn't work. Anyway, I figured out how to install older version of your plugin by using a zip file. Now it works well (it takes a while but at least it works- that is a huge achievement for me). Thank for your support. Keep up the good work.

[xxyzz]

Originally Posted by booklover2111

Using the commands from your HOW-TO page, I have installed redis successfully, but when I ran the plugin, it still didn't work.

Could you please post the error message?

Maybe you installed an old version of redis? Please try the following command and see if it will install redis 5 or 6, The last command will print the redis version.

$ sudo apt update
$ sudo apt upgrade redis
$ redis-server -v

I just found how to install redis on Windows! Download redis installer from https://github.com/tporadowski/redis/releases and select 'add redis to PATH' option in the installer.

[booklover2111]

Originally Posted by xxyzz

Could you please post the error message?

Maybe you installed an old version of redis? Please try the following command and see if it will install redis 5 or 6, The last command will print the redis version.

$ sudo apt update
$ sudo apt upgrade redis
$ redis-server -v

I just found how to install redis on Windows! Download redis installer from https://github.com/tporadowski/redis/releases and select 'add redis to PATH' option in the installer.

I couldn't update to redis 5 or 6. After I ran your commands, it shows version 4.0 I don't remember the exact number, but I have installed Window version, and it worked wonder.

[xxyzz]

v2.1.0 Download wordnet in job thread.

Plugin zip file is smaller.

[xxyzz]

v2.1.1 use python packages from zip file directly

You may want to delete some useless folders generated by worddumb previously by clicking "Preferences" -> "Miscellaneous" -> "Open calibre configuration directory" -> go to "plugins" folder. Only "worddumb-db1" and "worddumb-wordnet3.0" folders are needed, other folders that has name start with "worddumb-" can be deleted.

[xxyzz]

v3.0.0: create X-Ray file on Linux and Windows. Requires python3(using pip3 to download numpy).

X-Ray entities are created by nltk's ne_chunk().

This feature doesn't support macOS because numpy(required by nltk) has some .so files can't pass library validation unless they are signed by the same key of calibre.

[PoP]

Wow, I am impressed by the quality of the generated X-Ray. Thank you.

[xxyzz]

Originally Posted by PoP

Wow, I am impressed by the quality of the generated X-Ray. Thank you.

Machine learning is really amazing(that's how NLTK find named entities but I don't know how it works). Thanks to projects like NLTK, Wikipedia and many others this feature can be achieved more easily. I think there is a lots of room for improvement in speed and accuracy.

[xxyzz]

v3.2.0

- enable unix domain socket for redis

- count people in x-ray correctly

[xxyzz]

v3.3.0 load data from json, doesn't require redis anymore.

This version is even faster, I tested it on the book 1984 and it only took one or two seconds to create word wise file!

v3.4.0 fix text position for kfx

[sammayor]

I am getting error while running plugin :

calibre, version 5.12.0 (win32, embedded-python: True)
Conversion error: Failed: Generating Word Wise

Starting job: Generating Word Wise
Job: "Generating Word Wise" failed with error:
Traceback (most recent call last):
File "D:\calibre\src\calibre\gui2\threaded_jobs.py" , line 83, in start_work
File "calibre_plugins.worddumb.parse_job", line 43, in do_job
File "calibre_plugins.worddumb.parse_job", line 110, in find_named_entity
File "C:\Users\sam\AppData\Roaming\calibre\plugins\Word Dumb.zip\.venv\lib\python3.8\site-packages\nltk\chunk\__init__.py", line 185, in ne_chunk
File "C:\Users\sam\AppData\Roaming\calibre\plugins\Word Dumb.zip\.venv\lib\python3.8\site-packages\nltk\data.py", line 757, in load
File "C:\Users\sam\AppData\Roaming\calibre\plugins\word dumb-numpy1.20.1\numpy\__init__.py", line 143, in <module>
File "C:\Users\sam\AppData\Roaming\calibre\plugins\word dumb-numpy1.20.1\numpy\_distributor_init.py", line 26, in <module>
File "ctypes\__init__.py", line 373, in __init__
OSError: [WinError 193] %1 is not a valid Win32 application

Called with args: (<calibre.db.cache.Cache object at 0x000001FFAAB86C40>, [12444]) {'notifications': <queue.Queue object at 0x000001FFBA77ECA0>, 'abort': <threading.Event object at 0x000001FFBA77E610>, 'log': <calibre.utils.logging.GUILog object at 0x000001FFBA77E790>}