How to export indexes from indesign to epub

[Ryn]

Hi all,

I am involved in a project that involves exporting to epub 15-20 books of about 500 pages each.

All these books have large and essential indexes that need to make it into the epub, preferably linked.

As the indexes were created by hand, and outside of indesign, that program does not know how to make them active. It would be really nice if indesign knew how to link page numbers, which it self-creates, to a list of page numbers in the same document. Sadly, it does not.

As one of the source files was not in indesign, but in QuarkXpress, and in a version which I do not have, I have experimented with loading the PDF into Acrobat and exporting it to a Word file. This gives me the page numbers, which are carried with all the other header and footer stuff into the word file.

I then did the following:

1. Exporting the Word document to ePub with OpenOffice Writer's writer2ePub plugin. This yields an epub with for each page an xhtml document.
2. In Sigil, regexing the page numbers of the book to a self-closing <a> tag with the proper id - like pxxx - and moving them into the topmost paragraph tag of the page.
3. Then merging all the pages that are not the index into one xhtml file. Doing the same for the pages that are part of the index.
4. Iterating through the index file, using regexes to find page numbers and link them to the proper anchor inside the book document. There are ins and outs to this, that I will gloss over here.
5. Finally, splitting the book up into its logical chapters - usually one xhtml file per chapter, same with the notes etc. Thankfully, Sigil knows how to manage the links, once xhtml files are broken up.
6. Sadly, bc of the PDF export, there is a lot of cleaning still to do, as hyphenation is not understood by acrobat's PDF reading and exporting system. Also, there will be headers and/or footers, lots of unwanted hard and soft returns, whitespace, and no styles except (hopefully) italics, bold, super and subscripts. Also, the index now only links to the page, and not to the proper paragraph or sentence, which might be possible in an ePub, but is now beyond the pale for this project.

It is definitely a half-way solution, but one that can be simplified to some extent within Sigil, by using the Saved Searches functionality.

The whole index thing took me about two hours, from start to finish, for a 4k entry index. One book down, fourteen (or more) to go.

I am posting this as a hack, but in this community it is likely there are minds that can spot weaknesses in my approach. Feel free to shoot holes, and help me out for the remainder of the series.

------
As an edit: for those who come after, looking for a solution, there are valuable comments in the thread.

Outstanding ones from my perspective have been the realization that there are indesign plugins that export the page numbers into epub, here:

Originally Posted by BeckyEbook

http://epubsecrets.com/why-i-use-page-list-and-how.php
http://epubsecrets.com/page-list-all...e-doing-it.php

The link to the script is dead, so I'm listing it from web archive:
http://web.archive.org/web/201912181...orohikoscripts

You can write directly to Laura, but I have a feeling you'd better check out the "EPUB Accessibility Using InDesign" video tutorial (available from Lynda.com or Linkedin), which AFAIK includes the PageStaker and EPUBOgrify script.

Saving the best for last, there is actually a script that links "dead" indexes to their indesign page numbers, and exports them to epub. It's called LiveIndex, and can be bought here. (I have no relationship to this developer.)

https://www.id-extras.com/products/liveindex/

[DiapDealer]

Before you get impatient, Jon, and unilaterally declare this as irrelevant to Sigil... don't. I've chosen to leave it here, as there could be Sigil-specific ideas presented for improvement.

[Doitsu]

Originally Posted by Ryn

All these books have large and essential indexes that need to make it into the epub, preferably linked.

Depending on the structure and format of your book you might find the following simple plugins that I wrote helpful:

FootnoteLinker

PageList

Incremental IDs

Also check out all other Sigil plugins.

[Tex2002ans]

Originally Posted by Ryn

All these books have large and essential indexes that need to make it into the epub, preferably linked.

In ebooks, "Real Page Numbers" (RPNs) are a pain and aren't as helpful as they seem...

And Linked Indexes, to do them properly, requires a massive amount of manual intervention/cleanup.

Over the years, Me + Hitch have written an enormous amount on these two topics. See some of the latest discussion from earlier this year:

• My 2020 post in "Correct Page Numbers in Kindle?"
• Hitch's 2016 post in "Real Page Numbers for Reflowable Kindles"

(And if you want to know more about RPNs + Indexes in ebooks, read/follow all those links to all the other threads where we cover every pro/con from every angle.)

Originally Posted by Ryn

As the indexes were created by hand, and outside of indesign, that program does not know how to make them active. It would be really nice if indesign knew how to link page numbers, which it self-creates, to a list of page numbers in the same document. Sadly, it does not.

Even if you created InDesign Indexes, InDesign still doesn't export all the proper page numbers (pageList + page-list) to EPUB.

Back in 2016, David Kudler asked a similar question in "Getting InDesign to export pagelists to ePub3 (reflowable)".

I pointed to a 2015 article written by Joshua Tallent "How to Add a page-list to an EPUB" (now dead, so here's an Archive.org backup) + a 2015 article from EPUBSecrets, "Page List: All the Cool Ebook Developers Are Doing It".

To my knowledge, not much has really changed since.

Many of those methods require you to put some sort of tag/character at the end of each page, then convert that using some outside tool, then manually link all the Index links.

Just a few months ago, I wrote one such method in "Create index on epub from printed book".

Originally Posted by Ryn

[...] This gives me the page numbers, which are carried with all the other header and footer stuff into the word file.

I then did the following:

Great, great... I'll give some rolling commentary on your steps.

Originally Posted by Ryn

1. Exporting the Word document to ePub with OpenOffice Writer's writer2ePub plugin. This yields an epub with for each page an xhtml document.

Okay. A page-per-XHTML-file might be a good way to leave some sort of marker on every page.

Then you could potentially use the method I pointed out above in "Create index on epub from printed book", then use Doitsu's "Incremental IDs plugin":

Chapter01:

<h1>Chapter 1</h1>
<p>This is the beginning.</p>
<p class="pagenum">1</p> <--- Footer exported from the original document.

Convert with Incremental IDs (or Regex):

<h1>Chapter 1</h1>
<p>This is the beginning.</p>
<span epub:type="pagebreak" id="page1" title="1"/>

Move Top:

<span epub:type="pagebreak" id="page1" title="1"/>
<h1>Chapter 1</h1>
<p>This is the beginning.</p>

Note: That <span epub:type> format is EPUB3. EPUB2 you would use slightly different code.

2. In Sigil, regexing the page numbers of the book to a self-closing <a> tag with the proper id - like pxxx - and moving them into the topmost paragraph tag of the page.

That works too... If you are able to export reliable headers/footers.

Although that's usually a lot of extra cruft you usually have to clean up and sift through.

Depending on the document, it may be best to trash all the header/footers (or not export them at all), then renumber from scratch using some other tools.

3. Then merging all the pages that are not the index into one xhtml file. Doing the same for the pages that are part of the index.

Yep, this is great. Let Sigil/Calibre do the hard work for you:

1. You merge the entire book into one or two monolithic XHTML file/s.

Let's call them:

• merged.xhtml
• Index.xhtml

2. You can then add your <a>s around all your page numbers in your Index:

Index (Before):

<p>Dogs, 1</p>

Index (After Regex):

<p>Dogs, <a id="index-dog-1" href="../Text/merged.xhtml#page1">1</a></p>

3. Then use Sigil to split the XHTML files into their individual chapters, and all the anchors will be automatically updated to the files/chapters they belong:

<p>Dogs, <a id="index-dog-1" href="../Text/Chapter01.xhtml#page1">1</a></p>

4. Iterating through the index file, using regexes to find page numbers and link them to the proper anchor inside the book document. There are ins and outs to this, that I will gloss over here.

Yeah, it's a complicated mess.

Forms like "381–385" vs. "381–5" OR "385n10".

Indexes are extremely information dense and come in many variations, and usually it's not just a simple page number.

I went into some details in the "Create Index [...]" topic above. (Plus definitely in the famous "Real Page Numbers" topics.)

5. Finally, splitting the book up into its logical chapters - usually one xhtml file per chapter, same with the notes etc. Thankfully, Sigil knows how to manage the links, once xhtml files are broken up.

And once you linkify the Index, just be careful of the ~300KB soft filesize limit for EPUB. Sometimes the Indexes get so large, you have to split them into 2 or more files.

6. Sadly, bc of the PDF export, there is a lot of cleaning still to do, as hyphenation is not understood by acrobat's PDF reading and exporting system. Also, there will be headers and/or footers, lots of unwanted hard and soft returns, whitespace, and no styles except (hopefully) italics, bold, super and subscripts.

Yep, having a clean source document is the most important step.

I discussed a lot of that back in 2019, "Workflow for simultaneous EPUB and PDF production?"

Also, the index now only links to the page, and not to the proper paragraph or sentence, which might be possible in an ePub, but is now beyond the pale for this project.

... yeah, down to the sentence/paragraph-level ain't happening any time soon... Me + Hitch have also discussed that one to death.

It would require the Indexer to have access to the actual source files + the perfect mix of skills that very few are even equipped for.

Page numbers and/or Chap.Subchap are about as good as you're going to get.

Side Note: For just a piece of that discussion, look at my 2016 Post #129 from "Sick of Amazon Kindle books without Page Numbers...". I came up with this concept of "Format-Specific" and "Format-Neutral", and I still think it's a genius analysis.

[Hitch]

INDEXES!

"Niagara Falls, slowly I turn, step-by-step..."

OH, NEVER MIND. I was trying to put the Abbott & Costello Niagra Falls video in here. If you want to see it, go here: https://www.youtube.com/watch?v=8KpsUlvzbkk



Hitch, frequent Index Victim

[DNSB]

Originally Posted by Hitch

INDEXES!

"Niagara Falls, slowly I turn, step-by-step..."

OH, NEVER MIND. I was trying to put the Abbott & Costello Niagra Falls video in here. If you want to see it, go here: https://www.youtube.com/watch?v=8KpsUlvzbkk

[Hitch]

Originally Posted by DNSB

Thank you. I don't know why I seem to have a brain-blank about how to make it work in this forum.

Hitch

[JSWolf]

Originally Posted by Hitch

INDEXES!

"Niagara Falls, slowly I turn, step-by-step..."

OH, NEVER MIND. I was trying to put the Abbott & Costello Niagra Falls video in here. If you want to see it, go here: https://www.youtube.com/watch?v=8KpsUlvzbkk



Hitch, frequent Index Victim

Three Stooges Niagara Falls.

[Ryn]

Originally Posted by Tex2002ans

In ebooks, "Real Page Numbers" (RPNs) are a pain and aren't as helpful as they seem...

I hope you don't mind if I'm not quoting and responding to the entire post. I'm all in favor of being complete, but legibility is also a concern.

First of all, thanks for your extensive reply.

Second, I'm not at all interested in having parity between book page numbers and e-book page numbers - RPNs as you call them. I have tried implementing such things from time to time, but have found reader implementation spotty, and can see little added value for either reader or publisher.

You mention some alternatives to the steps I have used; I will investigate them further when I have occasion to do so - it's always good to have multiple paths home

As the rest of this project consists of indesign files, and I generally don't export those to epub page-by-page, I was considering working from the PDFs that indesign outputs. Converting them to word docs with acrobat, and then exporting those to epub using oowriter seems the way to go to easily get the page numbers. It remains a hassle to clean all the cruft out, though.

I's welcome a way to do things more easily through indesign, perhaps using the method you mention where each page gets a special character which Sigil can replace with page-break tags, and then to use the Sigil plugin for serialized ids you mention.

As I never use indesign for anything except making epub exports, I would welcome some input as to how to go about this in indesign. It's not my favorite program, although my limited experiences with quark have managed to knock the adobe product off the utmost bottom rank.

[Tex2002ans]

Originally Posted by Ryn

First of all, thanks for your extensive reply.

No problem. (I'm slightly famous around here for that. )

Originally Posted by Ryn

As the rest of this project consists of indesign files, and I generally don't export those to epub page-by-page, I was considering working from the PDFs that indesign outputs. Converting them to word docs with acrobat, and then exporting those to epub using oowriter seems the way to go to easily get the page numbers. It remains a hassle to clean all the cruft out, though.

Export to EPUB directly from InDesign, then clean up the HTML from there. This will carry over all the original markup (Headings, Italics, Smallcaps, etc.).

Remember, a book isn't just pure text, the underlying formatting is just as important.

PDF is one of the worst input formats there is, and you'll lose much of the original markup + introduce errors and other junk while converting to any other formats.

It's almost always better to always go from:

Source -> EPUB (Directly)

than to do:

Source -> PDF -> Word -> EPUB

where each step in the chain may introduce more issues.

* * *

If InDesign File Is Using Styles

Great. You're going to have an easier job.

In InDesign, there's such a thing as Style Mapping:

• Style Mapping | Learn how to map the character, paragraph and object styles in your document to fine-tune the appearance of the final EPUB file.
• "Mapping Styles and Tags | Adobe Indesign Accessibility"
• Adobe InDesign | Paragraph and character styles

If InDesign File Is NOT Using Styles

Prepare for pain...

(This is the more likely scenario, since 99%+ of people who use InDesign/Word/LibreOffice don't know or use Styles when designing documents.)

You'll have to manually clean up all the code, and every single book is going to generate wildly different cruft. And boy, oh boy, does InDesign love to generate iBooks-friendly bloat in their CSS.

Side Note: On Styles...

I've also written about Why/How Styles are so important, most recently:

• 2020, Post #50+52 in "eBook Formatting in Sigil"
• 2019, "MS Word vs Open Office Word"
  
  • Make sure to check out those 2 videos I linked to.
    

I think this is #1 the most important step there can be. Clean input helps EVERY single step down the line.

If people designed their documents with Styles+Accessibility in mind first, it would make everyone's life much easier.

(While steps between programs are different, the Styles concept is similar across all.)

Originally Posted by Ryn

As I never use indesign for anything except making epub exports, I would welcome some input as to how to go about this in indesign. It's not my favorite program, although my limited experiences with quark have managed to knock the adobe product off the utmost bottom rank.

And I also try to get everything out of InDesign ASAP.

~100% of the InDesign work I get is... directly formatted... so it's a mess. I've only met one designer who actually used InDesign with proper Styles.

Originally Posted by Ryn

Second, I'm not at all interested in having parity between book page numbers and e-book page numbers - RPNs as you call them.

But I'm scratching my head over here...

If not using RPNs, then what's the clickable links you're trying to accomplish in the Index?

Are you trying to do a:

Cats, [1], [2], [3]
Dogs, [1], [2], [3]

(Sigil Index style?)

* * *

But RPNs do serve some purpose, especially for Accessibility reasons (blind readers) + citations, book clubs, etc.

And for Linked Indexes, page #s seem to make a lot more sense.

Originally Posted by Ryn

I have tried implementing such things from time to time, but have found reader implementation spotty, and can see little added value for either reader or publisher.

On Usability of these "many-to-one"-type links...

In your favorite search engine, type:

many-to-one Hitch site:mobileread.com

That'll lead you to many threads over the years where Hitch discusses them. This issue is severe in Glossaries, Indexes, and even sometimes Footnotes.

[Ryn]

Hi Tex. Apologies for my late answer. Somehow, I didn't get notified of your reply, and as I do not visit MR every day, here we are.

I do use indesign for limited things like exporting files to epub, and am familiar with the mechanics and best practices of that route. Of course, it is often way easier to go the direct route than through PDF.

Perhaps my question was not as clear as it could have been. What I really wanted to know was: how do you export the page numbers in an indesign export to epub? It's not an option in the export dialogue box, nor is it something I can easily put together using indesign's byzantine search module.

Is there another way? It would be nice, seeing as most of the volumes in this project are in indesign. The reason I went the PDF route in the OP was bc that particular volume was created in some quark version I do not possess.

[Hitch]

Originally Posted by Ryn

Hi Tex. Apologies for my late answer. Somehow, I didn't get notified of your reply, and as I do not visit MR every day, here we are.

I do use indesign for limited things like exporting files to epub, and am familiar with the mechanics and best practices of that route. Of course, it is often way easier to go the direct route than through PDF.

Perhaps my question was not as clear as it could have been. What I really wanted to know was: how do you export the page numbers in an indesign export to epub? It's not an option in the export dialogue box, nor is it something I can easily put together using indesign's byzantine search module.

Is there another way? It would be nice, seeing as most of the volumes in this project are in indesign. The reason I went the PDF route in the OP was bc that particular volume was created in some quark version I do not possess.

I'm sorry, I literally feel an idiot asking this, but: do you mean, when you ask about the 'page numbers," the page numbers for the actual pages? From INDD to ePUB? Is that what you're asking about?

Hitch

[Ryn]

Originally Posted by Hitch

I'm sorry, I literally feel an idiot asking this, but: do you mean, when you ask about the 'page numbers," the page numbers for the actual pages? From INDD to ePUB? Is that what you're asking about?

Hitch

Yup. For linking an extensive and essential index, created outside of indesign and which indesign does not recognize.

Edit: not one index in fact but dozens, in a big project.

[BeckyEbook]

I'm not sure if this is exactly what you need, but I will post a few links that may lead you to come up with your own solution.

http://epubsecrets.com/why-i-use-page-list-and-how.php
http://epubsecrets.com/page-list-all...e-doing-it.php

The link to the script is dead, so I'm listing it from web archive:
http://web.archive.org/web/201912181...orohikoscripts

You can write directly to Laura, but I have a feeling you'd better check out the "EPUB Accessibility Using InDesign" video tutorial (available from Lynda.com or Linkedin), which AFAIK includes the PageStaker and EPUBOgrify script.
The latter is not so important anyway, because it is a simple change that can be done in Sigil.

[Hitch]

Originally Posted by Ryn

Yup. For linking an extensive and essential index, created outside of indesign and which indesign does not recognize.

Edit: not one index in fact but dozens, in a big project.

You don't. There isn't any easy or magic or automatic way to export the RPNs (Real Page Numbers). You create them manually.

You open up the ePUB; you open up the PDF. You find the first page-end. You search for that bit of text--typically, 5-10 characters will do. When you find it, you create the anchor, like P01, P02, etc.

Then, after all the anchors are done: then you write a script if you're lucky--or do it manually if you aren't--that links all the index entries that go to page 1, to P01, all the index entries that go to P02, to 2 and so forth.

That's it. Knowing Tex, he has some mad coding that will do some of this more easily than I've described, but that's the fundamental process, right there. And it's entirely possible that there are Sigil or Calibre addins that already do the 2nd part, the linking part, that I don't know about, as my band of Merry Minions use our internal, proprietary clips/programs to do that.

That's the basic procedure.

Hitch