[Plugin] CustomCleanerPlus

[slowsmile]

Fixed cleanup issues on plugin exit(v0.1.4)

[Thasaidon]

Marvellous.

Thanks very much

Originally Posted by slowsmile

@Thasaidon...I've just updated this plugin(v0.1.3) to include a "Save selections" checkbox which, when ticked, will save your current dialog selections for all future sessions until you change it. I hope it's OK for you.

[slowsmile]

Fixed a problem with the "Save selection" chkbutton not properly saving the current chkbutton selections in the user dialog window(v0.1.5).

[slowsmile]

KevinH has confirmed that there were no problems for this plugin on a Mac. He did say that the text size in the dialog was very small. Will increase the dialog text size as advised.

[roger64]

Hi

I used this plugin to correct
- images
- suppressing divs

It works great for these two items. Some books are really cluttered with lots of useless divs. Maybe there could be room for a small improvement.
about divs
I think that suppressing divs is not very advisable when it concerns images. It replaces the tag div with a tag p, but the second often (at least for French text) introduces a text-indent (while the former never carries one with it) and so the centering of the image is defective.
about small images
For the images, I remarked the creation of some inline code like

style="txt-align:center"

where

style="text-align:center"

seems more correct.

[slowsmile]

roger64...Regarding the 'txt-align: center' error -- thanks for spotting that mispelling. That will be corrected in the next change soon.

As regards the code that handles image tags -- this can vary. If "Remove div tags" is not selected by the user then all that's added to the image(<img) styling is 'alt=""' if it is needed. This is for epub 2 html compliance so that it will not fail Epubcheck.

But if the "Remove all div tags..." is selected by the plugin user then all div lines with associated </div> tags will be removed throughout the html code. This means that div tags will also be removed from all <img> tags if they are used.

Similarly if you select "Reformat smaller images..." then only the height and width values are changed or converted from pixel values to % values as inline styling within the <img> tag. Nothing else is touched or changed in the <img> tag or in any associated div tag(unless you have also chosen to remove all div tags as well).

The gist of what I'm saying is that the plugin will not automatically remove or suppress divs. It will only remove divs if you choose that option.

Thanks again. I'll get the change for the mispelling error out as quick as I can.

[slowsmile]

roger64...I've removed some code to fix the problem when you use div tags to surround an <img> tag. 'Reformat smaller ebook images..." should now work without problems (provided that you do not also select "Remove all div tags...")

This fix is in v0.1.8(the current download)

[roger64]

Wow! Flash reaction.

Thank you.

[JSWolf]

Could a version of this plugin be written for the Calibre editor? Thanks.

[slowsmile]

@JSWolf...I'm afraid that I'm not at all experienced at writing plugins for Calibre -- never done it, though I do frequently use that app. And if I did write a cleanup plugin for Calibre it would probably take me a while to learn the Calibre way of doing things(ie the libraries). I'm currently learning PyQT5(for use with Sigil plugins), which seems much easier to use and far less cranky than Tkinter. I believe that the Calibre plugins have been using Qt for a while now. So I am interested.

But for now, I'm not saying yes or no to your request. I'll look into it first and then let you know later on this forum if I can do an equivalent cleanup plugin for Calibre. Hope this is OK.

Also, just a reminder -- my Sigil plugin can cleanup both epubs and imported html docs. So what you could perhaps do is simply convert your doc to epub with Calibre, then just open your epub in Sigil and run my CustomCleanerPlus plugin. Doing this will cleanup and remove any dross html data that isn't needed in the epub html, cleanup your stylesheets and will also change or remove html data to ensure compliance with epub 2 (this helps to reduce Epubcheck validation errors later).

[slowsmile]

Update: This plugin has been updated(v0.2.8):

* Any id attribute value found to contain an illegal digit start character will be repaired by swapping out the illegal digit character and replacing it with an alpha character('x'). This check/repair only occurs in the epub's html files only. So this additional functionality should at least go part way in helping people to resolve Epubchecks infamous "colon" id error.

* All excess spacing will be removed from all id values in html files only.

* All internal links containing bookmarks will also be appropriately repaired and updated.

[slowsmile]

Plugin Update: Changes in v0.2.9:

* Fixed a problem with the version updater.

* Fixed a formatting problem with the epub TOC file.

[AlanHK]

Feature requests:

Filenames with spaces cause epubcheck to complain, and they are potentially troublesome.
So an option to change all filename spaces to underscores?
Quite tedious to do by hand with a few dozen files.

"Mend XHTML Source Code" -- can this be made to convert Windows-1252/ANSI html files to UTF-8?
HTML with that encoding that displays perfectly in a browser comes though with garbage characters if imported to Sigil normally, even if it's correctly described in the header.

How about a warning if undefined classes are applied in the HTML? Often from typos or missing definitions.

Also regarding these automated fixes:

-- Trims the epub stylesheet(s) - removes any unneeded or redundant class properties from the css
I like to snoop through interesting new stylesheets, and I can then use the Sigil tool to strip them if I want to.

-- Removes all html tags that are empty or that contain just spaces
These are often in a file because a linespace was intended. So after spotting these I often use them to modify the html, e.g. convert the following tag to one with "margin-top: 2em", for example.

So my preference is these be reported, but fixing them be made optional.

[slowsmile]

Filenames with spaces cause epubcheck to complain, and they are potentially troublesome.
So an option to change all filename spaces to underscores?
Quite tedious to do by hand with a few dozen files.

It's certainly possible to have spaces in filenames in the Book Browser. But to cure that in plugin code would be quite a difficult task because there are strict IDPF relationships and dependencies that exist between the epub filenames, ids and hrefs(links) that exist in the content.opf and toc.ncx files in the epub. I've already looked into trying to do this and realized fairly quickly that it was not easy because of all the dependencies and strict rules that apply to that data. So, IMHO, I'm afraid that your request is not really realistic and probably not even be possible at the plugin level.

Mend XHTML Source Code" -- can this be made to convert Windows-1252/ANSI html files to UTF-8?

Sorry but I'm going to have to say no again. The CustomCleanerPlus plugin is limited to doing just two specific tasks. It's first primary task is to clean out the dross proprietary data from the html or epub. It's second main task is to ensure epub 2 code compliance. In other words the plugin is a cleaner -- it's not a fixer or a converter. Also, converting several different encodings within your file to unicode is also not such an easy task. These problems are usually related to the user using too many applications to edit or format his or her doc or code and not ensuring that these editing environments are in unicode. Unless you are careful you will have cp1252, latin-1, ANSI and over 100 other possible encodings pop up in your epub. You can try patching your file in code using just the favourites like all the UTF forms, latin-1 and cp1252 etc but you can never really be sure that you have covered all bases. I could be wrong but I think Sigil also does a basic unicode check whenever you open an epub or import an html file into Sigil.

-- Trims the epub stylesheet(s) - removes any unneeded or redundant class properties from the css
I like to snoop through interesting new stylesheets, and I can then use the Sigil tool to strip them if I want to.

My meaning of the word "redundant" used above is wrt unnecessary style property declarations in the css. Here's an example:

my-style {
font-size: 100%;
line-height: 1.2em;
}

The "font-size:100%" and the "line-height: 1.2em" are unnecessary because you should always go with the vendor defaults for the latter properties for standard epubs(and for Kindle mobi). This may or may not result in an empty style declaration. But if you don't remove these properties from your epub then this might also cause problems on some vendor conversions.

-- Removes all html tags that are empty or that contain just spaces
These are often in a file because a linespace was intended. So after spotting these I often use them to modify the html, e.g. convert the following tag to one with "margin-top: 2em", for example.

Sorry again, but I can't accommodate your wish with this one either. For two reasons. First I've found the best way to format your ebook as a Word doc or ODT doc to be converted to epub is to ensure that you have created named paragraph styles for all your headings, text and spacing in your doc. The second reason is that the blank lines or hard line breaks with the enter key were causing havoc with my plugin image formatting on test using the plugin. Epub xhtml breaks also have completely different behaviour when compared with html breaks and this was a big problem initially for me with this plugin. That's really why I decided to just remove all hard breaks from the epub on cleanup.

[AlanHK]

Originally Posted by slowsmile

It's certainly possible to have spaces in filenames in the Book Browser. But to cure that in plugin code would be quite a difficult task because there are strict IDPF relationships and dependencies

Yes, but Sigil handles all that when you use the GUI rename-- I hoped a plugin could use that. I suppose not so easy.

Also, converting several different encodings within your file to unicode is also not such an easy task. These problems are usually related to the user using too many applications to edit or format his or her doc or code and not ensuring that these editing environments are in unicode.

These aren't documents made to be part of an epub, but HTML web files I'm repurposing.

As it is, I do a bunch of S&R and convert to character entities so it all ends up as plain ASCII.

Originally Posted by slowsmile

I could be wrong but I think Sigil also does a basic unicode check whenever you open an epub or import an html file into Sigil.

No. Just gives you a bunch of garbage without any notice.

Originally Posted by slowsmile

My meaning of the word "redundant" used above is wrt unnecessary style property declarations in the css. Here's an example:

It would be clearer if you listed all the properties removed.

Originally Posted by slowsmile

Sorry again, but I can't accommodate your wish with this one either. For two reasons. First I've found the best way to format your ebook as a Word doc or ODT doc to be converted to epub is to ensure that you have created named paragraph styles for all your headings, text and spacing

If by "hard linebreaks" you mean carriage return /linebreak characters, they don't matter, my concern is with "html tags that are empty or that contain just spaces".

Again, these are not files I created, but ones I'm importing or trying to fix.

Some books have a blank para between every text para. These have no significance and I will just delete. Others though have them before a section break and I will use them to insert a proper code.
e.g.
Search: <p></p> <p>
Replace: <p class="topspace">

Or maybe <p></p><p></p> <p>
or whatever. They should not be in the final epub; but if these are just filtered out I lose information.

Blank paragraphs sometimes indicate a vertical space is required; sometimes they are useless side effects. So I would never want to just delete them before being sure they are bathwater and not baby.

I've been in publishing for over 20 years; I gave up long ago trying to get authors to follow any formatting rules. 90% of them have no clue what a style is and do not want to know. So the files I get are never to spec and require inspection to see if the code reflects the intention.