[Plugin] RemoveUnusedBookmarks - removes all unused bookmarks from epubs

[slowsmile]

@DiapDealer...You're right, I did assume that all epubs contain a toc -- my mistake with apologies to both you and KevinH. OK I'll add some code that walks through the ncx and opf looking for hrefs with ids. I think the safest way would be to gather the href ids from both the xhtml files and toc.ncx and just use that as a check list for id removal from the xhtml files.

[DiapDealer]

No problem. It's not always easy seeing the various ways that someone else's garbage can expose the flaws in the workflow of the code you wrote using lovingly cherry-picked test-cases.

[KevinH]

Also the ncx can have an actual page table of page numbers and links into the xhtml. These ids are required and are not something that can be removed without breaking the entire page table.

[KevinH]

Here is a link to adobe pagemap.xml files that can contain links to ids in xhtml pages in epub2 and to ncx page tables. Various examples can be found by a websearch.

https://www.safaribooksonline.com/bl...-ncx-pagelist/

And here are a few starting point in our own Wiki:

https://wiki.mobileread.com/wiki/Page_numbers
https://wiki.mobileread.com/wiki/Ado...tions#Page-map
https://wiki.mobileread.com/wiki/NCX


And by the way, the sample epub that was just posted by BeckyEbook correctly uses fragments in the opf guide section so parsing the opf guide will be needed as well.

[slowsmile]

@Kevin...Thanks for providing the links. I admit that I don't know much about Adobe page-maps.

I've read the links that you provided and some of the information surprised me. For instance here is an excerpt from your Safari-O'Reilly link:

"The catch? page-map is (intentionally) an extension to ePub and adding a page-map file to your ePub will make it invalid. On top of that, unless the reading systems is based on Adobe software (like Digital Editions and the Sony Reader), page-map will have no effect."

The above emphasized text seems to indicate that if you have Adobe's page-map feature in your epub then it will not pass Epubcheck. So what you are asking me to do is to accomodate and check a proprietary epub feature(page-maps) that invalidates an epub. The page-map feature also only works on ADE and Sony ereaders that obviously use their own proprietary epub standard(not the IDPF epub standard).

Could you please clarify the reasons why I should include checks on page-map links in my plugin. My own view is that I see no point in my plugin supporting Adobe's proprietary page-map feature when that feature invalidates epubs i.e. I would prefer that my plugin properly follows the IDPF standard for epub 2. I should also mention that, in my research on page-maps, Calibre dropped their page-map plugin in 2010 and it was never replaced.

KevinH said:

And by the way, the sample epub that was just posted by BeckyEbook correctly uses fragments in the opf guide section so parsing the opf guide will be needed as well.

As I've already reported to Becky, after I ran the IDErrorCheck and RemoveUnusedBookmarks plugins on her epub, it passed Epubcheck without any problems. Nevertheless, I will also check all the href ids in the opf with this plugin as you've advised, just to make sure.

[DiapDealer]

Let users worry about validating their own epubs. If it worked when it was going in and you can prevent breaking it with your plugin, you probably should. Especially something as industry standard as page maps (regardless of their spec-compliance).

[slowsmile]

@ DiapDealer...I'm afraid that I strongly object to allowing page-map checks in my plugin because it sets a bad precedent. To me there's no grey area. You either follow one standard like the IDPF standard or you don't.

Here's an example of problems that would arise if you don't follow one standard. Say a Sigil user requests an automatic page-map build feature for Sigil and the developers respond that their request does not follow the IDPF standard and his feature request is refused. And then the user responds by saying that there is a already a Sigil plugin that supports and processes page-maps. How are you going to respond to that user? If the above precedent already exists and allows a Sigil plugin to process page-maps then what possible justification can the Sigil developers have for refusing the user's new page-map build feature request?

From the above, here's another interesting question for you: Should all Sigil plugins follow the same epub standard as the Sigil app?(I would really like an answer to this question)

[DiapDealer]

Originally Posted by slowsmile

@ DiapDealer...I'm afraid that I strongly object to allowing page-map checks in my plugin because it sets a bad precedent. To me there's no grey area. You either follow one standard like the IDPF standard or you don't.

Whatever. They're your plugins.

Originally Posted by slowsmile

Here's an example of problems that would arise if you don't follow one standard. Say a Sigil user requests an automatic page-map build feature for Sigil and the developers respond that their request does not follow the IDPF standard and his feature request is refused. And then the user responds by saying that there is a already a Sigil plugin that supports and processes page-maps. How are you going to respond to that user?

I'm going to say, "so what?"

Originally Posted by slowsmile

If the above precedent already allows a Sigil plugin to process page-maps then what possible justification do the Sigil developers have for refusing the user's new page-map build feature request?

You're being a little obtuse. There's clearly a difference between adding feature requests to Sigil's core and not expecting a plugin to break a working epub. Besides ... Sigil already makes non-spec-complaint concessions to remain compatible with other popular software and popular epub creation techniques. And we certainly try to accommodate the opening of non-compliant epubs without breaking internal links wherever possible. Sigil doesn't blindly dump/drop/break things that don't meet spec.

But I'm done here. If you're comfortable with your plugins breaking people's working epubs because they contained non-spec content, then have at it. I'm more of a "first do no harm" contributor, myself. *shrug*

[slowsmile]

@DiapDealer...Thanks for your response and for your patience. Just to keep things proper, in my release notes I'll add in a statement in Caveat that states that page-maps are not supported by this plugin. Secondly, I'll add a plugin routine in my epub validation that checks if the epub ncx has page-maps. If page-maps are found then the user will be notified by dialog message and plugin run will immediately be cancelled.

[KevinH]

Sounds like a compromise. But FYI - NCX page tables are epubcheck compliant and have the same issue as well as guide sections in the opf that use fragments. Both of these are epub2 compliant and can contain ids that are used in xhtml files.

[slowsmile]

@KevinH...I'm not sure what you mean by page tables. Are they the same as Adobe page-maps? I don't even know whether the page-map info that I have read is up to date from the links in your post above.

I also don't understand why page-maps are regarded as an industrial standard when the only major ereaders that support and use Adobe page-maps are ADE and Sony which I don't really regard as a particularly significant portion of the ereader market -- so how can page-maps be regarded as an accepted industrial standard ?

Anyway, I'll get these plugin changes done as quickly as I can.

[Doitsu]

@slowsmile: ADE pagemaps are an Adobe extension and books that contain them won't pass epubcheck. They're actually not that common.

The NCX page tables that KevinH referred to are (epub 2.0.1-compliant) page number tables wrapped in <pageList> tags. They're rarely found in epub2 books because only some DAISY-compliant apps support them. (BTW, they are supported by the epub3 output plugin.)

Since they basically use the same syntax as TOC entries they also might contain fragment ids.

If you don't want to deal with them, simply check for the presence of <pageList> tags in toc.ncx and display a warning.

[slowsmile]

@Doitsu...Thanks for that useful info and for the pageList advice. To check for a page-map file in epub is fairly easy. I'm using the bk.getpagemapid() which should always return id = "map" if a page-map.xml file is present in the epub. For pageList -- as you say -- I'll probably have to look for that tag in the ncx.

[DiapDealer]

I take no issue with you not wanting to support page-maps in your plugins. I only took issue with the notion that it was OK break them because they're not spec (Adobe), or because you didn't want to allow for them (ncx).

Warning and bailing if they're present is a perfectly reasonable and conscientious compromise. Thanks.

[slowsmile]

@DiapDealer...I appreciate what you've said and I agree with you for the most part. As I've described above, I think I've put in enough plugin checks to ensure that the plugin will not break the epub if there is a page-map file present. I've also just put in another warning and bailout about not using pageLists in the ncx as Doitsu has suggested. Hopefully that should be enough.