[FileType Plugin] YVES Bible Plugin

[ClashTheBunny]

YouVersion allows you to download Bibles offline. They are just simple html snippets of each chapter of the Bible encoded in a strange fashion. Using this plugin, you can "add book" for the "manifest.yves" for each Bible on your Android's SD Card. It can then be converted to any format that Calibre knows about. If you have a newer version of YouVersion, rename the _version_ file to manifest.yves and it should work with 0.0.8.

For example, in /sdcard/.youversion/.persistence/bible_v3/versions_1/ is _version_ for the King James version of the Bible. Just copy the whole 'version_1' directory over to your computer somewhere, rename _version_ to manifest.yves and "add" the manifest.yves. Or, if your android phone shows up like a drive on your computer, just open the //sdcard/.youversion/.persistence/bible_v3/versions_1/ directory, COPY _version_ to manifest.yves and then import that file straight from there. It will import a htmlz formatted book into your library that you can then convert to epub or mobi.

For old versions, in /sdcard/.youversion/bibles/1/ is manifest.yves for the King James version of the Bible. Just copy the whole '1' directory over to your computer somewhere and "add" the manifest.yves from it. Or, if your android phone shows up like a drive on your computer, just open the /sdcard/.youversion/bibles/1/manifest.yves file straight from there. It will import a htmlz formatted book into your library that you can then convert to epub or mobi.

It imports everything, so for example if the Bible has introductions or something, it will be added to the book that it is a part of, even if it's not "canonical".

It is a very simple concatenation of all the files with a simple chapter header put between each chapter. Don't expect anything beautiful. It's just easier on the eyes than an Android phone. It's also searchable offline. There are so many times that the Bible is just useless to me if I can't find a specific phrase.

Either way, it's here on github: https://github.com/ClashTheBunny/calibre-yves-input

This is clearly fair use of the files on SD Card because they are not encrypted and there is no reason you would distribute any of these, you already are able to download them for free. Don't ask me for any of the data that this is built from. That would be VERY wrong. Just get you own Bible app and download for offline usage.

Changelog:

Version 0.0.8:
Update to allow it to read new bible_v3 bibles now. Just rename the _version_ file to manifest.yves and you should be off to the races.

Version 0.0.7:
This version is compatible back to 1.25: the version that comes on Ubuntu 14.04. Since it's LTS, there may be many people who will use that version for the years to come, so it's helpful to have it supported. It goes back on sub-classing HTML2ZIP and instead just instantiates an instance of it and calls it. There is a bug of some sort in that version that doesn't overwrite the .name property on a plugin that has been imported at the top of the file. This should not change other functionality.

Version 0.0.6:
Make the yves file read in with binary, so that it is compatible with Windows. Windows has different ways of treating binary and text files, which can remove information.

Version 0.0.5 (unreleased):
Refactored thing out of __init__.py so that I could test them easier from the command line.

Version 0.0.4:
Cleans up some general things and adds support for Right-to-Left Languages and fixes languages that don't have a two letter code.

Todo:

Footnote parsing
Better chapter and book separation
Option to exclude extraneous text, like introductions and such
Add more documentation on converting to EPub or Mobi, like book and chapter xpath statements.

[eschwartz]

Usually, they do what you did.

Plugins are distributed through their MobileRead thread, and the canonical location is via the attachment you put here. the calibre website indexes the plugin threads and caches the attached plugin for the plugin updater.

You could also just move the "src/calibre_plugins/yves_input/" folder into the root of the repo If you plan on making multiple plugins, then for organization purposes I would use a separate repo for each. Probably no big deal, though.

For quick packaging, you can use calibre-customize.exe. It allows you to update the plugin in calibre, and then you just upload the working plugin from your configuration folder.

on the plugin and Welcome to MobileRead!

[eschwartz]

Originally Posted by ClashTheBunny

The last thing that is still not working is that I don't know how to add yves to the list of files that you can "open". I just always have to choose "all files" before it will let me select yves, even thought the plugin says that yves is the right extension.

Maybe you need to extend the BOOK_EXTENSIONS?

https://github.com/kovidgoyal/calibr..._init__.py#L29

[ClashTheBunny]

Originally Posted by eschwartz

Maybe you need to extend the BOOK_EXTENSIONS?

https://github.com/kovidgoyal/calibr..._init__.py#L29

I looked into this and I don't see a place in the API documentation where this would be extendable. It seems that this is built into a plist for osx in the setup section of the source, which makes me think that maybe a humble plugin would not have much effect on the app.

I also don't see many FileTypePlugin plugins out there, maybe this is a fairly non-standard thing to try to add file types to the list. For example, the plugin that I started with, calibre-doc-input, adds support for the .doc extension, but it's already in the BOOK_EXTENSION list. It seems that even if it's not supported in calibre, it's in the BOOK_EXTENSION list to allow calibre to manage the type of document.

[ClashTheBunny]

Originally Posted by eschwartz

You could also just move the "src/calibre_plugins/yves_input/" folder into the root of the repo If you plan on making multiple plugins, then for organization purposes I would use a separate repo for each. Probably no big deal, though.

I had copied another plugin called calibre_doc_input, so I took the directory structure from there, hoping that it would be convenient. Looks like I'm going to just move it around so I can just upload the github zip. This forum allows uploading via URL, so it should make development pretty easy with the shift around in source.

Originally Posted by eschwartz

on the plugin and Welcome to MobileRead!

Thanks! It's good to be here!

[ClashTheBunny]

Originally Posted by ClashTheBunny

Looks like I'm going to just move it around so I can just upload the github zip. This forum allows uploading via URL, so it should make development pretty easy with the shift around in source.

Github creates the zip with a parent directory and calibre isn't smart enough to look in the directory. I guess I can add a to src/calibre/customize/zipplugin.py" _locate_code to look one directory deep if there is nothing but a directory in the zip.

[kovidgoyal]

BOOK_EXTENSIONS controls what filetypes calibre recognizes as being ebooks. This list is not meant to be affected by file input plugins, which influence only what files calibre can convert.

As for creating the zip file, dont use github export, use calibre-customize. If you must use github export then simply move the files into the root of the repository.

And you need to have both __init__.py and the .txt file with the plugin import name in the root of the folder, for calibre to recognize the plugin, that is not going to change.

[eschwartz]

Yep, plugin folder should be the repo, not be in the repo.

[ClashTheBunny]

Yep, tried that, and github still doesn't work with their way of trying to prevent tarbombs. They will not let you download a zip file without the name of the repo and the name of the branch as the root directory.

It seems that tarbombs are the policy though, so no releases from github are possible.

[kovidgoyal]

Dont use github's zip export then. In any case you want the plugin zip file to not be compressed for performance while loading the plugin code, calibre-customize takes care of that for you automatically.

[ichbindasauge]

The plugin doesn't seem to work for me. I have installed it and added the manifest.yves file to the library but calibre just adds the single file, it will not convert it to zip or anything else. Anything else I have to do?

[ClashTheBunny]

Thanks so much for trying this!

What platform are you on?

Did you keep the manifest.yves in the same location as all of the directories containing zips? AKA, manifest.yves is in the same directory as GEN directory, which contains 1.yves, 2.yves, etc?

If that doesn't work, could you do the "restart calibre in debug mode" and then post the text file output of the session after you try the import again? That will tell me where the script is failing.

[ichbindasauge]

Windows 8.1 64 bit

This is what I get from the debug mode:

Spoiler:

Running file type plugin YVES to ZIP failed with traceback:
Traceback (most recent call last):
File "site-packages\calibre\customize\ui.py", line 154, in _run_filetype_plugins
File "calibre_plugins.yves_input.__init__", line 33, in run
File "json\__init__.py", line 338, in loads
File "json\decoder.py", line 366, in decode
File "json\decoder.py", line 382, in raw_decode
ValueError: Unterminated string starting at: line 1 column 56320 (char 56319)

Added manifest to db in: 0.447000026703 seconds

[ClashTheBunny]

Okay I think I found the problem, could you try 0.0.6? I made a rookie mistake on not reading the yves files in binary mode, which works on Linux and OSX, but isn't too happy on Windows.

[ClashTheBunny]

Also, this was the part of the traceback that I needed. I'll copy it here so that you can delete the rest of the traceback if you worry about privacy issues, but so that I can still see it if I need it later.

Originally Posted by ichbindasauge

Running file type plugin YVES to ZIP failed with traceback:
Traceback (most recent call last):
File "site-packages\calibre\customize\ui.py", line 154, in _run_filetype_plugins
File "calibre_plugins.yves_input.__init__", line 33, in run
File "json\__init__.py", line 338, in loads
File "json\decoder.py", line 366, in decode
File "json\decoder.py", line 382, in raw_decode
ValueError: Unterminated string starting at: line 1 column 56320 (char 56319)