Python 2 to Python 3, Polyglot & Plugins

[DaltonST]

Kovid,

For future planning purposes only, by what month/year will plugin developers need to have changed their Python 2 code to use "...\src\polyglot"?


When do you expect polyglot to become stable enough that plugin developers can begin using it and hence begin completely re-testing their plugins?


I have 15 plugins, so it is not something that can be done quickly and easily.


Thanks.



DaltonST

[kovidgoyal]

polyglot is already stable, feel free to use it whenever you like.

[jhowell]

I have questions about the planned Python 3 migration.

Are you going to put out a Python 3 beta version for Windows anytime soon?

Are you planning a hard cutover from 2 to 3 are you going to release both versions together for awhile?

[kovidgoyal]

Not any time soon no. The python 3 port is nowhere near ready. If and when it is ready, there will be a long period when both python 2 and python 3 builds of calibre on all platforms will be available so people have time to port plugins and find regressions in the python 3 port.

At the moment the plan is that when the python 3 port is ready and many plugins have been ported, I will stop releasing python 2 builds and all future calibre releases will be python 3 based.

Note however that this is very early stages, these plans might change depending on ground realties. The one this that is certain is that I do intend to provide at least a few months for plugin porting.

[jhowell]

Thanks. That sounds great.

[eschwartz]

Note that you don't actually need to use polyglot at all, but if you do, it's simple enough to just import from it and bump the minimum calibre version. But they are mostly pretty simple things.

I'd suggest going forward, to reduce the cognitive burden you should rely on any __future__ imports available (especially unicode_literals), make sure to mark all relevant strings as b'' and use modern python idioms that work natively on both. The latter two are things that any relatively recent code should probably have been doing anyway.

...

It's quite easy to reimplement the bits that you need, since they are, by and large, very simple try/except blocks. You could also rely on the fact that calibre has always had dependencies (dateutil, more recently html5lib) which require python-six, and just import needed functionality from e.g. six.moves which would make things a lot simpler. I think six 1.10 has been available at a minimum since a very long time...

Kovid has decided that he does not wish to rely on python-six, but you're not required to hold the same opinions.

Note: polyglot has been available since calibre 3.32 with the following contents (many more things were only added in the latest release):

from polyglot.builtins import is_py3, reraise, zip, map, filter, iteritems, iterkeys, itervalues
from polyglot.urllib import urlopen, urlencode

[jackie_w]

Originally Posted by eschwartz

... I'd suggest going forward, to reduce the cognitive burden you should rely on any __future__ imports available (especially unicode_literals), make sure to mark all relevant strings as b'' ...

Hi eschwartz,

I've read the above many times and I still don't really understand what you're saying. I'm pretty sure the problem is at my end

1. All my plugin .py files have this as the first line:
   

   from __future__ import (unicode_literals, division, absolute_import, print_function)

   Are you saying this will still be OK in Python 3?
   
   
2. When do you need the b'' thing? Can you give an example of Python2 vs 3 where it would be needed?
   
   
3. Re: polyglot.builtins iteritems, iterkeys, itervalues.
   I have a lot of code that looks something like:
   

   for k,v in dict.iteritems():
       do something with k and v

   Is there some reason why the following minor change wouldn't work OK in both Py2 and Py3?
   

   for k,v in dict.items():
       do something with k and v

[eschwartz]

1) Yes! All __future__ imports are defined by the python language committee to always remain valid, in perpetuity. https://docs.python.org/3/library/__future__.html

Originally Posted by The Python Software Foundation

No feature description will ever be deleted from __future__. Since its introduction in Python 2.1 the following features have found their way into the language using this mechanism:

Some __future__ imports are no-ops since as early as python 2.2. They denote that the feature became mandatory in a given version, and thereafter serve as executable documentation that such and such a thing happened, once.
You're doing everything you need to, here.

However, there are other cases where you need simple modernization. For example, in several cases your ScrambleEbook plugin uses the idiom:

raise MyException, errmsg

This is considered to be "old-style" in python2, and is fully removed in python3. On python3 it returns:

  File "/home/eschwartz/git/calibre/src/calibre/customize/zipplugin.py", line 182, in load_module
    import_name), 'exec', dont_inherit=True)
  File "calibre_plugins.scrambleebook_plugin.action", line 117
    raise SelectedBookError, errmsg
                           ^
SyntaxError: invalid syntax

This will work on both, and is preferred even in python2:

raise MyException(errmsg)

3) This change will indeed work on both python2 and python3. However, it misses the point of why one would want to use iteritems on python2: it is more efficient and uses less memory to use an iterator.

# python2
>>> {"one":"two", "three": "four"}.items()
[('three', 'four'), ('one', 'two')] # this is a list
>>> {"one":"two", "three": "four"}.iteritems()
<dictionary-itemiterator object at 0x7f6874c5f940> # this is an iterator
>>> {"one":"two", "three": "four"}.viewitems()
dict_items([('three', 'four'), ('one', 'two')])  # this is an iterator
# python3
{"one":"two", "three": "four"}.items()
dict_items([('one', 'two'), ('three', 'four')])  # this is an iterator

The dictionary-itemiterator and dict_items do not create an extra list, which you then toss away. OTOH that also makes it useless in many other cases, since you cannot count len(foo) or index foo[0], if foo is not even a list. (It is curious to note, that iteritems, an iterator, does not have a len, but dict_items which is the native python3 items() and the backported python2 viewitems, and is a view object, does have a len, but is still not subscriptable as foo[0] either way.)

So you trade inefficiency for portability if you use a simple items().

2) b'' strings are for example what you get if you:

>>> with open('filename', 'wb') as f:
>>>     f.write('Hello World')
TypeError: a bytes-like object is required, not 'str'

Which matters a great deal for people working with files. Also, by default, lxml.etree processing works with byte types.

One example of an article discussing the difference and why it matters, is here: https://medium.com/@andreacolangelo/...w-27dc02ff2686

To use a practical example, after I updated your plugin to use "raise SelectedBookError(errmsg)", and updated all dict.iteritems() to use from polyglot.builtins import iteritems (requires calibre >= 3.41.0) and iteritems(dict), I still got the following message. But this came from calibre itself.

Traceback (most recent call last):
  File "calibre_plugins.scrambleebook_plugin.action", line 161, in show_dialog
  File "calibre_plugins.scrambleebook_plugin.scrambleebook", line 189, in __init__
  File "calibre_plugins.scrambleebook_plugin.scrambleebook", line 256, in initialise_new_file
  File "calibre_plugins.scrambleebook_plugin.scrambleebook", line 1094, in get_run_check_error
  File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/check/main.py", line 81, in run_checks
    errors += check_opf(container)
  File "/home/eschwartz/git/calibre/src/calibre/ebooks/oeb/polish/check/opf.py", line 373, in check_opf
    n, c = raw.index('name="'), raw.index('content="')
TypeError: argument should be integer or bytes-like object, not 'str'

Now let us take a look at that line in calibre: https://github.com/kovidgoyal/calibr...ck/opf.py#L373

raw.index is acting on

raw = etree.tostring(cover)

etree.tostring returns a bytestring, and calibre then tries to index it using a simple str() type, denoted by

'name="'

This does not work -- on python2, str and bytes are the same, but on python3, str is of type unicode, which is not comparable to bytes, and the solution is to index it using

b'name="'

This will be fixed by the following commit in calibre (which is currently in my fork, but I will PR it soon): https://github.com/eli-schwartz/cali...2d483a654b56ef
EDIT: now officially merged: https://github.com/kovidgoyal/calibr...0543e13ef72ac0

The final fix to make the ScrambleEbook plugin work flawlessly AFAICT is to replace all use of unicode() with unicode_type() as imported from polyglot.builtins once again. Although when you used 'mystring %s' % unicode(len(foo)), you can simply drop the unicode cast. You can directly format the string using an integer returned by len() without first converting it to a unicode string.

[kovidgoyal]

I should note that a good way to find many of the more common/simple things that need changing and even to change them automatically in your plugin's python files is to use the modernize tool

https://python-modernize.readthedocs...est/index.html

or the 2to3 tool

https://docs.python.org/3/library/2to3.html

This tool however will make your files incompatible with python 2 so it should be used to detect issues only, not fix them automatically.

[jackie_w]

@eschwartz,
Thank you very much for explaining things. It was good of you to take the time with such a detailed reply It looks like there are a few things I can be working on to tidy up my old-fashioned code. Most of my plugins are not publicly shared so I can experiment without upsetting anyone if/when I make mistakes.

@kovid,
Thank you for the links The homework should keep me busy for a while.

[JSWolf]

Originally Posted by jackie_w

@eschwartz,
Thank you very much for explaining things. It was good of you to take the time with such a detailed reply It looks like there are a few things I can be working on to tidy up my old-fashioned code. Most of my plugins are not publicly shared so I can experiment without upsetting anyone if/when I make mistakes.

@kovid,
Thank you for the links The homework should keep me busy for a while.

What plugins have you written for your own use? Maybe some of them might be of use to others.

[Terisa de morgan]

Originally Posted by jackie_w

@eschwartz,
Thank you very much for explaining things. It was good of you to take the time with such a detailed reply It looks like there are a few things I can be working on to tidy up my old-fashioned code. Most of my plugins are not publicly shared so I can experiment without upsetting anyone if/when I make mistakes.

Thank you for reminding me that I have plugins too (private and public)

[jackie_w]

Originally Posted by JSWolf

What plugins have you written for your own use? Maybe some of them might be of use to others.

Yes, they might, but there are a variety of reasons why sharing to the public can be problematic. Mainly:

• some of them are variations on existing plugins (Modify Epub, KoboTouchExtended, EpubCheck) so I'm not going to tread on the toes of the original creators or other noble souls who try to maintain orphaned plugins.
  
• some of them have some overlap with standard calibre (e.g. epub-to-kepub, kepub-to-epub, epub2-to-epub3). I prefer to avoid confusing anyone.
  
• a lot of my Editor Tool plugins may not work 100% for all users, for all epubs. Not a problem when I'm using them myself because I know the limitations. On the first day of public release I'd probably be inundated with complaints about the 1% of corner cases that I don't care about (you'd probably be first in the queue ). This is when programming as a fun retirement hobby turns into PITA programming as real work. I've no desire to spend large amounts of time creating error-checks for people who might do something dumb or who just prefer to edit epubs differently to me. I'll leave the all-singing, all-dancing plugins to the professionals.

Having said all that I'm always happy to share code with other plugin hobbyists as a base for creating their own stuff. But I don't want ongoing maintenance responsibility - especially now - during the transition from python 2 to 3.

[Terisa de morgan]

Originally Posted by jackie_w

a lot of my Editor Tool plugins may not work 100% for all users, for all epubs. Not a problem when I'm using them myself because I know the limitations. On the first day of public release I'd probably be inundated with complaints about the 1% of corner cases that I don't care about (you'd probably be first in the queue ). This is when programming as a fun retirement hobby turns into PITA programming as real work. I've no desire to spend large amounts of time creating error-checks for people who might do something dumb or who just prefer to edit epubs differently to me. I'll leave the all-singing, all-dancing plugins to the professionals.

I totally, totally understand you.

[Terisa de morgan]

Originally Posted by jackie_w

Yes, they might, but there are a variety of reasons why sharing to the public can be problematic. Mainly:

• some of them are variations on existing plugins (Modify Epub, KoboTouchExtended, EpubCheck) so I'm not going to tread on the toes of the original creators or other noble souls who try to maintain orphaned plugins.
  
• some of them have some overlap with standard calibre (e.g. epub-to-kepub, kepub-to-epub, epub2-to-epub3). I prefer to avoid confusing anyone.
  
• a lot of my Editor Tool plugins may not work 100% for all users, for all epubs. Not a problem when I'm using them myself because I know the limitations. On the first day of public release I'd probably be inundated with complaints about the 1% of corner cases that I don't care about (you'd probably be first in the queue ). This is when programming as a fun retirement hobby turns into PITA programming as real work. I've no desire to spend large amounts of time creating error-checks for people who might do something dumb or who just prefer to edit epubs differently to me. I'll leave the all-singing, all-dancing plugins to the professionals.

Having said all that I'm always happy to share code with other plugin hobbyists as a base for creating their own stuff. But I don't want ongoing maintenance responsibility - especially now - during the transition from python 2 to 3.

BTW... I would be interested in those editor plugins... so I can see how they work and perhaps evolve them to my own use cases.....