Dealing with non-Python Dependencies

[Mikowmer]

Hello!

I'm currently attempting to troubleshoot someone else's plugin for Calibre. The plugin in question is here: https://github.com/naclander/Calibre...-Driver-Plugin

At the moment, it looks like certain Rust binaries in the Cryptography module aren't importing correctly due to an 'unknown location'.

ImportError: cannot import name 'asn1' from 'cryptography.hazmat.bindings._rust' (unknown location)

The abi3.so binary files for those specific submodules are present. However, when the plugin is being loaded, those binaries are ignored.


Would it be possible to get some help on including non-Python dependencies in a Calibre plugin?

[kovidgoyal]

The appropriate folders need to be on sys.path before attempting the import.

[Mikowmer]

Originally Posted by kovidgoyal

The appropriate folders need to be on sys.path before attempting the import.

... Meaning?

Is that the appropriate folders for the module, or the non-Python code?

[kovidgoyal]

The folder that contains the python package you are importing has to be on sys.path if it is and the abi version of the compiled module is a match it will import.

[Kenivia]

I've posted a workaround on GitHub here. You were right, the path added to sys.path was the path to the zip file.

Having the library somewhere else is not the most elegant solution, is there a better way to do this? Maybe the plugin could decompress the zip file temporarily?

[kovidgoyal]

with self:
   import whatever

see the __enter__ method of the Plugin base class

[xdgc]

I just found this plugin and am having the same trouble, two years later. (It's abandonware so not taking updates.)

The plugin is a zip plugin. All the dependencies are in the zip, and Python can import python text or bytecode from a zip, but it can't import native code from a zip. (Presumably this is because python text/bytecode can be compiled or executed from memory, but the OS loader has to memory-map native DSOs.)

Kovid pointed to the Plugin base class's __enter__ method, which in principle knows how to handle this situation. It scans the zipfile for unimportable files and unpacks if needed. But this is a device plugin and device plugins are not loaded as context managers, so that's inapplicable on the surface.

After some ponderment, what I think Kovid is cryptically suggesting is that the library code's startup method, prior to importing modules that include DSOs, should use itself as a context manager. Indeed this patch allows the pycrypto code to load:

diff --git a/__init__.py b/__init__.py
index e548b30..71c41bb 100644
--- a/__init__.py
+++ b/__init__.py
@@ -20,12 +20,17 @@ class RemarkablePlugin(DevicePlugin):
     supported_platforms = ["linux", "windows", "osx"]
     version = (1, 2, 3)  # The version number of this plugin
     minimum_calibre_version = (0, 7, 53)
+    seen_device = False

     FORMATS = ["epub", "pdf"]

     MANAGES_DEVICE_PRESENCE = True

     def startup(self):
+        with self:
+            return self._startup()
+
+    def _startup(self):
         # Use the plugins directory that's included with the plugin
         sys.path.append(self.plugin_path)
         global remarkable_fs

Onward now to the next issue: RemarkablePlugin's packaging script uses your system Python, whose compiled output is very likely not to be loadable by Calibre's Python.