Plugin Development

[KevinH]

I have started this thread to help answer any plugin developer's questions, and to provide links to some common code examples to help speed development of Sigil plugins.

Attached below are the following:


libgui.zip
a python module compatible with both python 2.7 and 3.4 that hides the details
of using Tk for generating simple modal dialogs such as fileChooser(), folderChooser(),
fileSaveAs(), and selectFromList(). Also included are routines to convert ebook titles
to safe file names, and to get the path to the User's home directory.
The main routine illustrates simple examples on how to use these routines in your plugin.

sampleOutput.zip
a sample epub output plugin that supports font obfuscation and removes the Sigil specific metadata. It requires Sigil-0.9.3 to work due to bugs in earlier launcher code.

testme3_v031.zip
a sample edit plugin that provides many simple examples of how to use the plugin interface

[KevinH]

Pre-Release Versions for Plugin Developers

Because storing preferences can be important for a Plugin, DiapDealer has designed a new python launcher preferences class and integrated it right into the *container classes, so persistent preferences storage is easily available for all plugins.

These changes are new since the Sigil 0.8.2 release and they will appear in the next future version of Sigil.

To illustrate how to create and use these settings, DiapDealer has designed two simple plugins that clearly demonstrate how the interface should be used.

See the attached "PrefsExampleGroups_v0.0.2.zip" and "PrefsExampleSimple_v0.0.2.zip".

Important
In order to use this preferences interface, you will need to be using Sigil v0.8.3 or greater.

It's possible to update only the python plugin launcher files if you can't wait for Sigil 0.8.3. To facilitate that we have thrown together a launcher_updater python program that will try to automatically find your Sigil installation and properly copy in the python files. *NOTE: this is unnecessary if you have Sigil 0.8.3 or higher.* The instructions for updating the plugin launcher files are being left in the event that that future preview releases appear.

To run the launcher_updater: do the following:

1. download the zip file attached to this post, do not unzip it

2. In a terminal, cd to where this zip file is (again, do not unzip it) and run the following command: (and yes you are telling python to run a zip file!).

Note that on Linux you will probably need to run this command with "sudo" and on Windows you should open a command-prompt as Administrator

python launcher_updater_20141204.zip

Before using this updater, please verify that it matches the following md5sum:

/sbin/md5 launcher_updater_20141204.zip
MD5 (launcher_updater_20141204.zip) = 9a4a2fbc79aacc736f95d74cc4b9eea9

If you know how to manually install the new python launcher files, you can simply unzip the launcher_updater and look in the payload folder.

Note, these launcher files will only work with stock Sigil 0.8.2, NOT Sigil 0.8.1.

Note, this pre-release returns a launcher_version value of 20141204 versus the earlier version in Sigil 0.8.2

And as always the very latest version of the python launcher code used by Sigil can always be be found here:

https://github.com/Sigil-Ebook/Sigil...unchers/python

[KevinH]

Here is a mapping of Sigil version with the version information set by the plugin launcher code:

bk.launcher_version()

Sigil Version      Launcher Version
--------------      -------------------
Sigil-0.8.0         Did_not_exist
Sigil-0.8.1         Did not exist
Sigil-0.8.2         20141120
Sigil-0.8.3         20141204
Sigil-0.8.4         20141204
Sigil-0.8.5         20141204
Sigil-0.8.6         20141204
Sigil-0.8.7         20141204
Sigil-0.8.900       20150909
Sigil-0.8.901       20151001
Sigil-0.9.0         20151024
Sigil-0.9.1         20151120
Sigil-0.9.2         20151215
Sigil-0.9.3         20160130
Sigil-0.9.4         20160313
Sigil-0.9.5         20160325
Sigil-0.9.6         20160605
Sigil-0.9.7         20160909
Sigil-0.9.8         20170227
Sigil-0.9.9         20171212
Sigil-0.9.10       20180723
Sigil-0.9.11       20190214
Sigil-0.9.12       20190218

We will keep this table up to date. And we will continue to increment the launcher_version with each new release so that it can be used as a proxy for Sigil version in all versions of Sigil in the Sigil-0.9.X series.

Information as to when significant new plugin features were added or bugs fixed is provided below:

20150909 (Sigil-0.8.900 pre-release)
------------------------------------------
- lxml, sigil_bs4, PIL, regex, six, html5lib first added to Bundled Python
- spellcheck support first added
- sigil_gumbo parser support first added
- FlightCrew is now a Sigil Plugin


20151001 (Sigil-0.8.901 pre-release)
------------------------------------------
- major bug fixes for sigil_bs4 prettyprint_xhtml and serialize_xhtml
- epub3 interface features for pugins
- cssutils, cssselect, chardet first added to Bundled Python
- validation plugins now autoclose


20151024 (Sigil-0.9.0 release)
----------------------------------
- Use Bundled Python option first introduced
- fix bug in PluginRunner that try to well-form check xml files as xhtml
- fix launcher epub3 interface to convert null properties to None
- fix for pluginhunspell.py to try and find hunspell on Linux systems


20151120 (Sigil-0.9.1 release)
----------------------------------
- fix bad bug in PluginRunner that coerced xhtml to xml when new files added
- fix to prevent missing hunspell from stopping all plugins
- fix for bundled PIL and cssutils on Mac OS X
- creation of testplugin_v010.zip to allow builders to test their plugins
- made launcher success and error messages robust to non-utf8 strings


20151215 (Sigil-0.9.2 release)
----------------------------------
- Simplified the UseBundled Interpreter Logic
- revamped sigil_bs4 prettyprint_xhtml and serialzie_xhtml to be more robust
- update sigil_bs4 to use numeric entities for raw n0n-breaking spaces
- better handle void tags in sigil_bs4/prettyprint_xhtml and serialize_xhtml


20160130 (Sigil-0.9.3 release)
----------------------------------
- Improved plugin epub3 handling
- Added epub_version interface
- Many epub_utils bug fixes for font support
- sigil_gumbo_bs4_adapter bug fixes for namespaced attributes


20160313 (Sigil-0.9.4 Release)
-----------------------------------
- Fixed epub_utils IDPF font obfuscating routine to better match the spec.
- Other minor fixes and cleanups


20160325 (Sigil-0.9.5 Release)
-----------------------------------
- Changed sigil-bs4 to treat both ruby and rt tags as inline when prettyprinting xhtml


20160605 (Sigil-0.9.6 Release)
-----------------------------------
- added support for epub3 bindings element in opf


20160909 (Sigil-0.9.7 Release)
-----------------------------------
- extend validation plugin interface with add_extended_result() method to allow better cursor positioning
- remove support for python2.7 only plugins


20170227 (Sigil-0.9.8 Release)
-----------------------------------
- make Sigil's UI and spellchecker language settings available to plugins (bk.sigil_ui_lang, and bk.sigil_spellcheck_lang)
- fix bug in prettyprint_xhtml in sigil_bs4 (put back inadvertently dropped is_void_tag routine)
- harden plugin interface code to properly unquote/quote hrefs


Please note, currently the only supported versions of Sigil are the very latest release version (at this time Sigil-0.9.8), and Sigil master. If users are using any earlier versions of Sigil, they are strongly recommended to update to the current Sigil Release.

Hope this helps,

KevinH

[KevinH]

Hi,

Please note there currently exists 4 types of Sigil plugins:

1. edit - used to create modify and delete files in the ebook currently being edited by Sigil

2. input - used to load a newly created epub into Sigil

3. output - used to read the ebook currently in Sigil and convert or generate a new ebook file you store externally. No changes to the current ebook inside Sigil are made.

4. validation - used to return error messages from epub validators run on the current ebook in Sigil. It inherits from the output plugin.


For specific information on interface calls available for each type of plugin, check out:

bookcontainer.py for the interface for an "edit" type plugin

outputcontainer.py for the interface for an "output" type plugin

inputcontainer.py for the interface for an "input" container

validationcontainer.py for the interface for a "validation" type plugin

The current Sigil Plugin Framework Documentation is up-to-date with the upcoming Sigil 0.9.991 (pre Sigil-1.0) release. Here it is as an epub for reference by Sigil Plugin Developers.

Hope this helps,

KevinH

[KevinH]

Hi,

I just tested this and the metadata in the Sigil ebook was in fact properly copied over by that function in all of my tests.

So please post a sample epub that displays this problem as I can not recreate this at all.
I will track it down and get it fixed.

Thanks!

Kevin

Originally Posted by Toxaris

Perhaps add a function in epub_utils.py that to also have the OPF including metadata is copied? In the current bk.copy_book_contents_to function the OPF is not including the metadata (is there a good reason for that?).

[Toxaris]

Originally Posted by KevinH

Hi,

I just tested this and the metadata in the Sigil ebook was in fact properly copied over by that function in all of my tests.

So please post a sample epub that displays this problem as I can not recreate this at all.
I will track it down and get it fixed.

Thanks!

Kevin

Will try again tomorrow and do some additional testing first. From the looks of it, the opf is recreated. The manifest has a different sequence it seems.

[Toxaris]

Ok, I found the culprit. The difference is in the OPF, specifically the metadata tag. In the my test-book I have:

<opf:metadata xmlns:opf="http://www.idpf.org/2007/opf" xmlns:dc="http://purl.org/dc/elements/1.1/">
...
</opf:metadata>

If I remove the red part, it does work. The tag is valid though as far as I know.

So, it appears that the OPF is not copied, but recreated.

[DiapDealer]

Originally Posted by Toxaris

Ok, I found the culprit. The difference is in the OPF, specifically the metadata tag. In the my test-book I have:

<opf:metadata xmlns:opf="http://www.idpf.org/2007/opf" xmlns:dc="http://purl.org/dc/elements/1.1/">
...
</opf:metadata>

If I remove the red part, it does work. The tag is valid though as far as I know.

So, it appears that the OPF is not copied, but recreated.

I noticed something similar to this when using the epub3 output plugin, but hadn't tracked down the culprit before now. Having the opf prefix in the metadata tag (which seems to be valid according to Flightcrew) causes the resulting epub3 to be created without a metadata section in the opf. Might need to accommodate unnecessary, but apparently not invalid, prefixes in the opf.

[KevinH]

Hi DiapDealer and Toxaris,

Thanks for figuring that out. The actual content.opf text itself is never used in the launcher. It literally parses the content.opf from inside Sigil and then provides an interface to change its pieces and to rebuild it on the fly.

The code to parse the Sigil content.opf simply did not properly handle the case of using explicit opf: prefixes on the tags. The content.opf that is rebuilt on the fly then was missing the metadata. The rebuilt-on-the-fly version does not use explicit opf: prefixing (as it is redundant give how it creates the metadata tag), the new version of the opf_parser.py simply removes any redundant opf: prefixes from tag names.

This in turn allows all of the metadata to be properly parsed and captured, and therefore the rebuilt on the fly version to now have the metadata.

The upcoming new release is expected shortly and it will have that bug fixed.

I have attached an updated opf_parser.py, that can be used until then.

Thanks so much for your bug report!

Kevin

[Toxaris]

Kevin, this version does handle the metadata perfect.

[Toxaris]

Another question, in the bookcontainer class (that is for edit plugins, right?), there is a function to replace files in the ePUB after adaptation, writefile.
I need the id from the OPF and I need to input data. For text based files that would be a string I pressume. However, what if I want to replace (or add for that matter) a non-text based file like a font, audio or video? Must I use some kind of binary stream?
Can you shed some light on that?

[KevinH]

Hi Toxaris,

Originally Posted by Toxaris

Another question, in the bookcontainer class (that is for edit plugins, right?)

Yes that is the container class for "edit" plugins - it was not named "editcontainer.py" because it existed before we thought of the separate plugin types, and as such it kept its original name "bookcontainer.py"

there is a function to replace files in the ePUB after adaptation, writefile.
I need the id from the OPF and I need to input data. For text based files that would be a string I pressume. However, what if I want to replace (or add for that matter) a non-text based file like a font, audio or video? Must I use some kind of binary stream?

Can you shed some light on that?

If you are editing or changing an already existing file then "writefile" is the way to go (assuming the file is part of the manifest). The media-type field from the manifiest is used to determine what should be done with the data you want to write.

Based on the media-type if it is text (css or xhtml/xml) the data is tested to see if it is unicode or a bytestring. If a bytestring it is assumed to be utf-8, if unicode, it is converted to utf-8 and written to the output file.

Based on the media-type,if it is not text, the data is assumed to be a string of bytes and it is written (as binary) unchanged to the output file.


Now if you want to create an entirely new file that should be part of the manifest, you would use the "addfile". Here you can explicitly pass in a media-type or a media-type can be deduced from the filename extension. And again, the media-type determines how the data will be treated.

Hope this helps,

KevinH

[Doitsu]

@KevinH: Does bk.addfile() support bytearrays?

I did a quick test and came up with the following proof-of-concept code, which will add a blackletter ttf font located in the plugin's directory to the epub:

#!/usr/bin/env python
import os, inspect, uuid

def run(bk):
    basename = 'WallauRundgotisch-Heavy.ttf'
    binary_path = os.path.join(os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))), basename)
    file_handle = open(binary_path, 'rb')
    uniqueid = 'id' + str(uuid.uuid4())[24:82]
    data = bytearray(file_handle.read())
    file_handle.close()
    mime = 'application/x-font-ttf'
    bk.addfile(uniqueid, basename, data, mime)
    
    return 0

def main():
    print('I reached main when I should not have\n')
    return -1

if __name__ == "__main__":
    sys.exit(main())

This code appears to be working, but I'm wondering if there is a better and/or more desirable method to add binary files?

[KevinH]

Hi Doitsu,

Originally Posted by Doitsu

@KevinH: Does bk.addfile() support bytearrays?

Technically it is either a "bytestring" ie. a standard non-unicode string in python2.7 or it has type "bytes" in python3.4. But yes, as long as you specify the media-type properly, the binary data will not be touched and will be written properly.

A "bytearray" is a closely related but different type under both python2.7 and python3.4

I did a quick test and came up with the following proof-of-concept code, which will add a blackletter ttf font located in the plugin's directory to the epub:

#!/usr/bin/env python
import os, inspect, uuid

def run(bk):
    binary_file = 'WallauRundgotisch-Heavy.ttf'
    binary_path = os.path.join(os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe()))), binary_file)
    file_handle = open(binary_path, 'rb')
    uniqueid = 'id' + str(uuid.uuid4())[24:82]
    basename = 'Fonts/' + binary_file
    data = bytearray(file_handle.read())
    mime = 'application/x-font-ttf'
    bk.addfile(uniqueid, basename, data, mime)
    
    return 0

I would simplify this only slightly and not use the bytearray class at all. Also the basename should not have the directory at all, the correct place inside of Sigil should again be determined by the mime-type/file extension automatically.

Using DiapDealer's KindleImport as a model:

SCRIPT_DIR = os.path.normpath(os.path.dirname(os.path.dirname(os.path.abspath(inspect.getfile(inspect.currentframe())))))

def run(bk):
    binary_file = 'WallauRundgotisch-Heavy.ttf'
    binary_path = os.path.join(SCRIPT_DIR, binary_file)
    data = ''
    with open(binary_path, 'rb') as f:
        data = f.read()
    uniqueid = 'id' + str(uuid.uuid4())[24:82]
    bk.addfile(uniqueid, binary_file, data, 'application/x-font-ttf')
    return 0

FWIW: I personally do not like creating uuid based manifest ids. I have my own unique id creator routine I invoke. Something along the lines of:

id = create_unique_id("fnt", 0)

def create_unique_id(base, cnt):
    id = base + "%d" % cnt
    while(bk.id_to_mime(id,None) is not None):
        cnt +=1
        id = base + "%d" % cnt
    return id

Others will simply pick a name and preface it with "x" until unique. There is of course no one "right" way.


Hope something here helps.

Kevin

[Doitsu]

Hi Kevin,

Originally Posted by KevinH

Also the basename should not have the directory at all, the correct place inside of Sigil should again be determined by the mime-type/file extension automatically.

I must have mixed it up with other functions that require an href.

Originally Posted by KevinH

FWIW: I personally do not like creating uuid based manifest ids. I have my own unique id creator routine I invoke.

Thanks that is very helpful. I merely used uuid(), because it's so easy to use.

Thanks!
D.