ePub creation tools : what's missing ? wishlist / dialogue

[zelda_pinwheel]

We've been having an interesting discussion in the french forum recently about what is missing in the apps we currently have available to us for creating epub files (i'm talking about editors, not converters), and what we wish we had. In other threads, at least 2 people (that i have seen) have mentioned their intentions to write an epub editor. In still other threads, there have been references to various details that could be done better. calibre is great in many ways, but it's not a fully-featured editor. feedbooks does a brilliant job making valid code and hierarchical structure, but a lot of people want an offline app, and feedbooks doesn't handle images yet.

So, I thought I would create ONE THREAD TO RULE THEM ALL where users can list their desiderata and any interested developpers could discuss what is possible / not possible, their intentions, etc. With a little luck we might even end up creating the ONE APP TO RULE THEM ALL right here ! wouldn't that be brilliant.

Here to start are some of the things we want as users and creators of epub books, taken from the discussion in french in this thread. Please feel free to add your own, and I hope the developpers will be interested in participating / responding as well !! Valloric, llasram, kovid, wallcraft, Komenor, et al. i'm thinking of you, for example. (<-- non-exhaustive list !!)

DESIRED FEATURES

1. epub files must be valid (html tidy, epub check...) and conform to best practices.

2. the editor should be able to accept multiple html / xhtml flows and create one document with a hierarchical TOC. It should also be able to accept one html flow and semantically parse the hierarchy of the document (part, chapter, section...) according to the tags used (h1, h2, h3, etc.), creating logical divisions for a properly structured epub document and a hierarchical TOC, the way that feedbooks does.

3. It should be able to handle images and relatively advanced css markup (dropcaps, for instance).

4. Ideally, it should be accessible even to users with no knowledge of html / css code with a full wysiwyg UI, although advanced options should be available if you do know the code (direct code editing should be possible), again similar to the feedbooks interface.

I'll probably have more to add later but i've got to go for the moment, so i'll stop here and open the discussion. What do you want from an editor ? i'll be looking forward to seeing what comes out of this !

[Jellby]

Since I like writing my XHTML and CSS code by hand, I won't ask this from an editor, but I would appreciate a user-friendly way to generate and edit the opf (metadata, manifest, spine, guide) and the ncx (hierarchical table of contents) files.

[GeoffC]

5. Choice of end formats, to cater for readers that do not support ePub.

6. Pick and chose title-page image.

[brewt]

Font Embedding. The only thing that'll do that off-the-shelf now is Indesign, and well, that's the only thing it does "well".

Macros, search & replace (not just text but styles, fonts, sizes, etc), grammar checker, spell-checker, target viewing application compensation, multiple format input & output, auto paragraph re-justification & hyphenation, multiple language dictionaries, built-in browser, metadata management, multi-style select, discontinuous select, table generation/correction, cross references such as indicies and footnotes, rss feeds....

You said "killer app", didn't you Z?

-bjc

[Valloric]

Originally Posted by zelda_pinwheel

So, I thought I would create ONE THREAD TO RULE THEM ALL where users can list their desiderata and any interested developpers could discuss what is possible / not possible, their intentions, etc.

This is a very good idea.

Originally Posted by zelda_pinwheel

1. epub files must be valid (html tidy, epub check...) and conform to best practices.

I've been thinking about this and writing down ideas in my "epub editor idea book"... there are certain problems here.

For one, we have the epub standard. Naturally, all editors should output valid epub, that's a given, but the best practices... currently, the Sony Reader--more specifically, mobile DE--has limits on chapter length (content file limit is 300KB). While this is unfortunate, I can live with it. But there are other problems and flukes of mobile DE that one has to take into account. The page numbers on the side, for instance. One needs to add margins so the numbers don't overlap the text, etc.

So a standards compliant epub may not work in mobile DE, and if it does, it may not look as nice as one specifically tailored to it. And I'm not blaming DE on certain limitations (like the 300KB limit) that are the result of the platform it runs on. Other epub reading applications for the Reader (or other devices) would probably hit a similar wall.

My current working idea is this: the editor exports two types of epub files. Both are standards compliant. One follows the standard and nothing else. No size limits, no special margins etc. Pure epub, without any consideration for the reading application or the platform. Let's call this "Standards Epub".

Then there's the second output option. This epub type is also fully compliant, but has the size limitations, special margins and other necessities for an enjoyable read on something like the Sony Reader. Let's call this "Mobile Epub".

I know no one likes the idea of two different epub files. One risks the creation of a sub-format. But I don't see a way around this. A purely standards compliant epub is a must; current practical limitations may (and will) disappear with time. But one must also be realistic: there's no point in high-horsing around with compliant epubs that nobody can read on portable devices like the Sony Reader.

I am very much interested in what other people think about this.

Originally Posted by zelda_pinwheel

3. It should be able to handle images and relatively advanced css markup (dropcaps, for instance).

I've been following the dropcaps thread(s), and it's a good example of why I would want an editor to export two kinds of epub files: one that follows the standard, and one that's tailored around flukes and idiosyncrasies of certain reading software.

Originally Posted by zelda_pinwheel

4. Ideally, it should be accessible even to users with no knowledge of html / css code with a full wysiwyg UI, although advanced options should be available if you do know the code (direct code editing should be possible), again similar to the feedbooks interface.

My working idea is to have a WYSIWYG interface, but with a "view code" view, like for instance in Dreamweaver. If someone wants to mess with it directly, they should be able to.

[kovidgoyal]

1. and 2. are incompatible.

[kovidgoyal]

Also I think you need to more clearly define what the use case is for this tool. Is it for authors who want to write books, or is it for proofreaders who want to touch up and convert existing books.

If the former, far more important that the features you have outlined would be features to support actual writing, like for example, keeping notes associated with characters/places/events.

If the latter then all that's needed is a wrapper around any good html editor with a wyswyg mode.

[Valloric]

Originally Posted by kovidgoyal

If the former, far more important that the features you have outlined would be features to support actual writing, like for example, keeping notes associated with characters/places/events.

Who in their right mind would use an ebook editor to actually write a book? The editor I'm thinking of is for proofreaders and copyeditors.

[zelda_pinwheel]

Originally Posted by Valloric

This is a very good idea.

i'm glad you think so ; i had you in mind when i thought of it.

I know no one likes the idea of two different epub files. One risks the creation of a sub-format. But I don't see a way around this. A purely standards compliant epub is a must; current practical limitations may (and will) disappear with time. But one must also be realistic: there's no point in high-horsing around with compliant epubs that nobody can read on portable device like the Sony Reader.

I am very much interested in what other people think about this.

well, you're right, this isn't the optimum situation, however it seems a reasonable compromise to me. particularly since the "standards epub" can also be used for archival purposes and as a source format for conversion to other formats as needed. i'm not an expert though ; i'll be interested to see other reactions.

My working idea is to have a WYSIWYG interface, but with a "view code" view, like for instance in Dreamweaver. If someone wants to mess with it directly, they should be able to.

right, that's rather the idea i had in my head as well.

Originally Posted by kovidgoyal

1. and 2. are incompatible.

why are 1 and 2 incompatible, kovid ?

Originally Posted by kovidgoyal

Also I think you need to more clearly define what the use case is for this tool. Is it for authors who want to write books, or is it for proofreaders who want to touch up and convert existing books.

the users of this tool as i see it are very much the *latter* case you describe : they are the same kinds of people using BookDesigner or ETI's eBook Publisher or Mobipocket Creator today, but who want a good tool for making epub files. so really closer to dreamweaver than to word or any writing tools. i imagine they would *already* have the finished text, either PD texts or their own manuscripts, and just want to turn them into ebooks. so, yes, a wrapper around a good html editor with a wysiwyg mode. but i think there are some particularities specific to epub that would need to be addressed (creation of the different file types, creation of the epub container...).

anyway this discussion looks to be off to a good start ! i'm glad to see it !

[Valloric]

Originally Posted by zelda_pinwheel

particularly since the "standards epub" can also be used for archival purposes and as a source format for conversion to other formats as needed.

Exactly. There are people out there who want to use epubs purely for storage.

Originally Posted by zelda_pinwheel

the users of this tool as i see it are very much the *latter* case you describe : they are the same kinds of people using BookDesigner or ETI's eBook Publisher or Mobipocket Creator today, but who want a good tool for making epub files. so really closer to dreamweaver than to word or any writing tools. i imagine they would *already* have the finished text, either PD texts or their own manuscripts, and just want to turn them into ebooks. so, yes, a wrapper around a good html editor with a wysiwyg mode. but i think there are some particularities specific to epub that would need to be addressed (creation of the different file types, creation of the epub container...).

Again, I agree completely. This is exactly how I see an epub editor: a tool that facilitates the manual creation of an epub book from pre-existing text. BD for epub... without the suck.

[kovidgoyal]

Originally Posted by zelda_pinwheel

why are 1 and 2 incompatible, kovid ?

If you accept aribtrary HTML as input and want to output standards compliant HTML the only way to do that is to basically strip the HTML down to a basic internal markup and then re-export it. This is for example what BookDesigner does. There is no way you can accept arbitrary HTML input and losslessly convert it to standards compliant HTML output (and no htmltidy doesn't do this).

So really what the tool will have to do is:

1) Accept html input
2) parse the html input into some simple internal markup
3) Try to auto identify structural components (or ask the user to provide input to help identify them)
4) Provide an editor interface for the internal markup
5) Export the internal markup to EPUB

[mtravellerh]

That sounds really good. I would like to have a way for direct source access and editing, though, maybe a search and replace with RegEx (I use regular expressions quite a lot). Otherwise a GUI thing would be great. One free open source wysiwyg editor that springs to mind is NVU, an open source editor I use a lot is Notepad ++. As those are open source, they could be easily integrated as edit tools.

[Valloric]

Originally Posted by kovidgoyal

So really what the tool will have to do is:

1) Accept html input
2) parse the html input into some simple internal markup
3) Try to auto identify structural components (or ask the user to provide input to help identify them)
4) Provide an editor interface for the internal markup
5) Export the internal markup to EPUB

My understanding exactly. Only I'm thinking of making the "simple" internal markup not so simple. But yes, one has to parse the initial HTML and create a new one.

[mtravellerh]

Originally Posted by kovidgoyal

If you accept aribtrary HTML as input and want to output standards compliant HTML the only way to do that is to basically strip the HTML down to a basic internal markup and then re-export it. This is for example what BookDesigner does. There is no way you can accept arbitrary HTML input and losslessly convert it to standards compliant HTML output (and no htmltidy doesn't do this).

So really what the tool will have to do is:

1) Accept html input
2) parse the html input into some simple internal markup
3) Try to auto identify structural components (or ask the user to provide input to help identify them)
4) Provide an editor interface for the internal markup
5) Export the internal markup to EPUB

If you do that (5), people like Coolmicro will get up and shout again that the resulting epub is not conform to standard and that the html code is not "clean". (I really do not care about "clean or dirty" code myself, as long as it does what it has to do, like Calibre does for example). So I am all for it.

[zelda_pinwheel]

Originally Posted by Valloric

This is exactly how I see an epub editor: a tool that facilitates the manual creation of an epub book from pre-existing text. BD for epub... without the suck.

yes, please without the suck.

Originally Posted by kovidgoyal

If you accept aribtrary HTML as input and want to output standards compliant HTML the only way to do that is to basically strip the HTML down to a basic internal markup and then re-export it. This is for example what BookDesigner does. There is no way you can accept arbitrary HTML input and losslessly convert it to standards compliant HTML output (and no htmltidy doesn't do this).

So really what the tool will have to do is:

1) Accept html input
2) parse the html input into some simple internal markup
3) Try to auto identify structural components (or ask the user to provide input to help identify them)
4) Provide an editor interface for the internal markup
5) Export the internal markup to EPUB

okay, i see what you mean. you are right of course ; i was making the assumption that the input would be clean and valid code, which cannot necessarily be assumed.

out of curiosity, Valloric, have you seen the feedbooks wysiwyg editor ? i HIGHLY recommend you take a look at it and at how the book creation process is handled ; it is excellent, and it is easily accessible even to people who know nothing at all about html / css, however also gives access to the source code for more knowledgeable users. really, in my mind, the tool i would like would be very close to the feedbooks interface, with just a few modifications. notably i definitely want to be able to insert images, which feedbooks does not yet support.

Originally Posted by mtravellerh

If you do that (5), people like Coolmicro will get up and shout again that the resulting epub is not conform to standard and that the html code is not "clean". (I really do not care about "clean or dirty" code myself, as long as it does what it has to do, like Calibre does for example). So I am all for it.

why ? the exported code can easily be clean, that is one of the goals.