Beta testers needed

[kovidgoyal]

Hi all,

I've spent the last two weeks doing some major surgery to calibre's internals to remove a dependency on ImageMagick for image processing. The result is that that the calibre installation size decreases by 15MB and (hopefully) all image processing functionality remains intact. The immediate motivation for this change was some security issues recently found in ImageMagick, though it has been on my long term TODO list for a while.

However, since these changes are fairly invasive, I may well have broken bits of functionality here and there. Particularly if you use any third party plugins that do image processing. So I would appreciate it if some of you could downloads betas and try them out.

In addition to the image processing changes, I also changed the way file open/save dialogs are used in calibre on windows (they are now used via a helper process) -- this should hopefully resolve most of the crashes caused by badly designed Shell extensions on windows.

Please report any problems you encounter here.

https://download.calibre-ebook.com/betas/

Thanks.

P.S. There is an extra windows 64-bit build in that folder that has Qt 5.5.1 and was compiled with Visual Studio 2015 (calibre-64bit-2.56.0-vs2015.msi) Dont install that unless you want to test out even more beta code (calibre 3.0 will be based on it)

[MerlinMama]

Since installing the beta version, I can no longer find the Generate Cover Plug-in. I had it on my toolbar but now it's gone, and if I go to add it back, it is not available. It does, however, still show as installed in the list of plugins.

[davidfor]

Originally Posted by MerlinMama

Since installing the beta version, I can no longer find the Generate Cover Plug-in. I had it on my toolbar but now it's gone, and if I go to add it back, it is not available. It does, however, still show as installed in the list of plugins.

Unfortuntely, Generate Cover uses ImageMagick, so it won't load. I don't really use it, but will have a look at it over the weekend if no one else does.

The Marvin XD plugin also uses ImageMagick. But, as it doesn't work with recent iOS versions, I don't have any reason to look at it.

[MerlinMama]

Originally Posted by davidfor

Unfortuntely, Generate Cover uses ImageMagick, so it won't load. I don't really use it, but will have a look at it over the weekend if no one else does.

I hope it can be made to work. I use it VERY often. For now I'll go back to the 'regular' Calibre version and keep my fingers crossed...

[kovidgoyal]

I have created a portability layer for plugins, so most plugins that use ImageMagick should still work. However, the one part that could not be ported is writing text on images.

For that, the plugin would need to switch over to using QPainter (which is a better approach anyway and is what the generate covers code in calibre itself uses). Feel free to ask me if you have questions on how to port any specific functionality.

[davidfor]

Originally Posted by kovidgoyal

I have created a portability layer for plugins, so most plugins should still work. However, the one part that could not be ported is writing text on images.

For that, the plugin would need to switch over to using QPainter (which is a better approach anyway and is what the generate covers code in calibre itself uses). Feel free to ask me if you have questions on how to port any specific functionality.

OK. I just quickly looked at the code for Generate Cover and couldn't find matches for a few things. The QPainter hint will probably be enough to get started.

[kovidgoyal]

So other than the Generate Cover Plugin, no issues?

[davidfor]

Nothing I saw. But, I just realised I didn't have the cover options turned on in the KoboTouch driver. They've been working fine while I have been running from source, but I'll check with the beta tonight.

[eschwartz]

The only thing I've really done with images since you started cutting out imagemagick is generating covers using the builtin feature after FanFicFare downloads.

But FWIW that seems to be working fine.

[kovidgoyal]

The basic use cases should all be fine, I've tested them pretty extensively myself. The problem is calibre is so large and has so many parts that there is no way I can cover every combination -- which is why I'm hoping for a spot of testing from the user community before I release this code.

[Terisa de morgan]

I'm using it now. Quite basic use: edition and library management, but I use regularly a number of plugins (although not graphical) and no problem at all.

[davidfor]

I've done some testing of sending the cover images tonight. There were no problems. I can't see a difference between the old and new versions. Including greyscale covers. I thought they were being skipped in Windows.

[jackie_w]

@ Kovid,

I found a problem when using my ScrambleEbook plugin. The problem arises when trying to scramble GIF images (JPG, JPEG, PNG, SVG seem OK).

This is the error message:

calibre, version 2.56.0
ERROR: Unhandled exception: <b>TypeError</b>:file() argument 1 must be encoded string without NULL bytes, not str

calibre 2.56 [64bit]  embedded-python: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: DeDRM && Obok DeDRM && ScrambleEbook && Fantastic Fiction && Goodreads && JS Mod Epub && Diaps Editing Toolbag && DOC Input && KePub Metadata Writer && Barnes & Noble && Search The Internet && Count Pages && Wiki Reader && Quality Check && Open With && Copy Cover To Device && EpubSplit && JSTest Plugin && KindleUnpack - The Plugin && KePub Metadata Reader && Extract ISBN && KePub Input && Modify ePub && KoboTouchCSSEdit && Kobo Utilities && KePub Output && KoboBooks
Traceback (most recent call last):
  File "calibre_plugins.scrambleebook_plugin.scrambleebook", line 334, in create_scramble_book
  File "calibre_plugins.scrambleebook_plugin.scrambleebook", line 409, in __init__
  File "calibre_plugins.scrambleebook_plugin.scrambleebook", line 437, in scramble_main
  File "calibre_plugins.scrambleebook_plugin.scrambleebook", line 540, in scramble_img
  File "site-packages\calibre\utils\magick\legacy.py", line 135, in export
  File "site-packages\Pillow-2.4.0-py2.7-win-amd64.egg\PIL\Image.py", line 2093, in open
TypeError: file() argument 1 must be encoded string without NULL bytes, not str

This is the Python method where the error occurs (the red line). It processes all image formats except .SVG:

Spoiler:

def scramble_img(self, name, scramble_dgts=False):
    if self.eb.mime_map[name] in OEB_RASTER_IMAGES:
        data = self.eb.parsed(name)
        oldimg = Image()
        oldpath = self.eb.name_to_abspath(name)
        try:
            oldimg.load(data)
            wid, hgt = oldimg.size
        except:
            wid, hgt = (50, 50)
        try:
            fmt = oldimg.format
        except:
            x, x, fmt = get_nameparts(name)
        
        newimg = Image()
        newimg.load(self.dummyimg)
        newimg.size = (wid, hgt)
        
        self.eb.replace(name, newimg.export(fmt.upper()))

• The relevant import: from calibre.utils.magick import Image
• self.eb is a standard calibre container object (epub/azw)

I've attached a test epub in case you need it.

[jackie_w]

@Kovid,

I'm doing well here! I've also found a problem when using my Copy Cover plugin to send covers to Pocketbook devices which must be .BMP 4BPP grayscale format.

This is the error message:

calibre, version 2.56.0
ERROR: Unhandled exception: <b>ValueError</b>:Failed to export image as BMP3 with error: Unsupported image format

calibre 2.56 [64bit]  embedded-python: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.9
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Successfully initialized third party plugins: DeDRM && Obok DeDRM && ScrambleEbook && Fantastic Fiction && Goodreads && JS Mod Epub && Diaps Editing Toolbag && DOC Input && KePub Metadata Writer && Barnes & Noble && Search The Internet && Count Pages && Wiki Reader && Quality Check && Open With && Copy Cover To Device && EpubSplit && JSTest Plugin && KindleUnpack - The Plugin && KePub Metadata Reader && Extract ISBN && KePub Input && Modify ePub && KoboTouchCSSEdit && Kobo Utilities && KePub Output && KoboBooks
Traceback (most recent call last):
  File "calibre_plugins.copy_cover_to_device.dialogs", line 91, in do_book_action
  File "calibre_plugins.copy_cover_to_device.dialogs", line 140, in copy_to_pocketbook_16
  File "site-packages\calibre\utils\magick\legacy.py", line 92, in save
  File "site-packages\calibre\utils\magick\legacy.py", line 139, in export
  File "site-packages\calibre\utils\img.py", line 124, in image_to_data
ValueError: Failed to export image as BMP3 with error: Unsupported image format

These are the relevant plugin methods. The error relates to the red line:

Spoiler:

def copy_to_pocketbook_16(self, screen, aspect, cdata, dest):
    img, rescale = self.cover_resize(screen, aspect, cdata)
    # Resize image then convert to greyscale
    if rescale:
        img.size = rescale
    img.quantize(number_colors=16, colorspace='GRAYColorspace')
    # save_cover_data_to(img, dest)
    img.type = "PaletteType"
    img.save(dest, 'BMP3')
    return rescale

def cover_resize(self, screen, aspect, cdata):
    img = Image()
    img.load(cdata)
    rescale = None
    if aspect:
        swidth, sheight = screen
        cwidth, cheight = img.size[:2]
        # calc new image size
        # Set rescale factor to match screen height
        rescale = (int(sheight * cwidth / cheight), sheight)
        if rescale[0] > swidth:
            # if too wide for screen, set to match screen width
            rescale = (swidth, int(swidth * cheight / cwidth))
    else:
        rescale = screen
    return img, rescale

The ImageMagick-related imports are these:

from calibre.utils.magick.draw import save_cover_data_to
from calibre.utils.magick import Image, create_canvas

I tested as well as I can given I don't own most of the supported devices (Kindle, Nook) and my Pocketbook battery is dead I discovered today. I have no reason to believe there is any problem with:

• create_canvas - used for Kindles only
• save_cover_data_to - used for Sony (JPG), Nook (JPG), Kindle (PNG)
  For Pocketbooks, save_cover_data_to had to be replaced by the workaround code (provided by you in 2012 if you can remember that far back ) you see above as it was not saving the in 4BPP format when using Linux.

[kovidgoyal]

Originally Posted by davidfor

I've done some testing of sending the cover images tonight. There were no problems. I can't see a difference between the old and new versions. Including greyscale covers. I thought they were being skipped in Windows.

No, I implemented a grayscale routine that works on all Qt versions.