Formatting all books in library to same format

[JSWolf]

Originally Posted by deback

I use the Modify ePub plugin and Convert to automate many fixes. Convert will fix many things, if you add enough things in the Styling and Transform sections of Look and Feel.

I too use Modify ePub and it works very well.

Don't convert to try to fix things. It will make the code more difficult to fix things. Go from the original code.

Some of the things I fix automatically in most of the books I convert are (I run Modify and then Convert):

All indents are changed to 1.2em, all line-heights are changed to 1.2, remove blank line spaces (but sometimes I have to manually edit the paragraph CSS code to fix this or manually delete the lines that add the blank line spaces), change the "body" side margins to 30pt from 5pt, add the Charis font and remove all other font style CSS codes in the .css file, remove all the CSS codes that are invalid according to Calibre (these are all in the Transform section), justify all text in the paragraphs (manually have to justify some of the sections at the beginning of the books), change the CSS code for cover images that are coded as auto and auto to 100% and 100% (I have a macro that adds the CSS code for this in the .css file, so I only change the class to calibre3a and hit ctrl-shift-5 to add the CSS code quickly), paragraph font size is changed to 1em from whatever the original code is, and many more automated things I can't think of at the moment.

Before you dive into the CSS, in the Calibre editor is an option to remove all extra CSS code. Use that to make the CSS easier to edit.

I too change the indents to 1.2em. I remove the line heights as your Kobo has a setting for line height and you can set it for what you want. It's better that way. Margins should be set to 0. Again Kobo has a setting for margins and you can only increase, not decrease the margins. So again, leaving the margins at 0 lets you choose what you want. There's no need to embed Charis SIL (or the even better ChareInk version that I made). You can side load the fonts you want and use the Kobo firmware setting to select which font you want. As for justification, remove all text-align: left and justify. There again is a setting in the firmware for justification. Also, in classes that affect the body styling, remove any font-size 1em or less. Also, any you get any sort of space of 5% or more, I change that to less. What that is depends on where the space is. For ToC headers, I use a top/bottom margin of 0.8em. But you may find something like 1 or 1.5em to be more to your liking. If there are any widows and orphans in the CSS, I remove those and put a widows: 1 and orphans: 1 in the body style (if there is not a body style, add one).

Convert removes all the unused CSS codes

So does the option in the editor in the Tools menu. And you don't end up with a conversion.

Modify removes a lot of things that shouldn't be in the files, and on and on.

Yes, it does.

I also highlight all the files and run the ToC plugin to edit any ToCs that need to be fixed (like add ToC from files when there are no ToCs, rename chapter numbers to Chapter 1, Chapter 2, etc, rename Chapter One, etc, to Chapter 1, etc). The ToC plugin will open the next file after you click on OK for each prior file.

The ToC I check to see if there is a Cover as the first entry so when you first view the ePub, you go right to the cover. Without Cover as the first entry, you do not see the cover and you start at whatever the first entry is. I don't go changing the names in the ToC as they are fine as is.

My only wish is that Convert would do the Check Book and Automatically fix errors automatically during the conversions. I have to do this manually, but there are usually very few errors found after I run the Modify and Convert plugins.

You can install the epubcheck plugin for Calibre and that allows you to use epubcheck to validate your ePub. I've never had a problem with ePub that are validated with epubcheck. I also sometimes use the Calibre validator. If both say there are no errors, there are no structural errors.

Another thing I do is remove the internal ToC (the HTML ToC) and I remove things like "About the Publisher" a list of the author's other books, any reviews, advertisements, previews for other books, newsletter links, piracy notices, and any other rubbish I don't need. You just delete those files in the text section. The NCX ToC will automatically be edited to remove those files if they are there. Also, if there are any embedded fonts I don't want, I delete them and remove the code from the CSS. If you do have any embedded fonts youdo keep, in the Tools menu is where you can find the option to subset the embedded fonts to make them smaller and remove any that are not used.

When I get done doing all of this (most of it is automated), my files are very clean looking and they all have the same format in the text for the paragraphs.

And you don't have any errors so the ePub should work no problem.

Some files are so badly coded and have tons of <spans> and other poor coding that it can take much longer than five minutes to edit the file properly. If I really want a certain book, I will spend the time to edit everything, including the spellcheck and the Report (to fix punctuation and remove improper symbols due to scanning errors, etc).

Yes, some are rather badly coded and it does take some time to fix those. But it's up to you if you want to spend the time to fix them.

But remember, DO NOT CONVERT!

[steven168]

wow seem too deep for me

[deback]

I've been very happy with my system and will keep doing it that way. I've never had a problem when converting files more than once, so I will not be making any changes and will continue to convert all files after modifying all of them. Many of my settings in the Styling and Transform sections will fix many things, without having to manually do a lot of editing.

In Common Options/Look & Feel, I have checked: Subset all embedded fonts, Expand CSS, and embed the Charis SIL font (which I love, so there's no reason to find any other font). In Styling, I have the Fonts box checked at the bottom. I have many extra CSS entries in Styling and many rules in Transform Styles. If anyone wants to see the lists of those entries, let me know. Those entries have saved me tons of time since I added them.

In Page Setup, I have 30pt for left and right margins. (This is a must, because I can't stand it when the text is almost flush with the side margins.)

In Modify, I have all boxes checked, EXCEPT for the following:

Remove OS artifact files, De-indent, Add unmanifested files to manifest, remove unmanifested files from ePub, Remove margins from Adobe .xpgt files, Remove Adobe resource DRM meta tags, Flatten TOC, Encode HTML in UTF-8, Modify @page and body margin styles, Append extra CSS, Remove inline javascript and files, Add/replace metadata jacket, Remove existing cover, insert or replace cover, and Remove non dc: metadata elements.

I just wish the Remove page-map.xml feature in Modify actually worked all the time. That's one thing I have to constantly watch for, because if there's a page-map.xml file in the ePub, there is usually a problem with the page-numbering in ADE. And, once in a while, Check Book will find codes related to .xpgt files in the ePub that I have to have automatically fixed, so that feature in Modify doesn't work all the time, either.

Another thing that happens once in a while is that my extra CSS for the class "calibre" doesn't always add text-align: justify;, and my extra CSS for the class "body" doesn't always change the left and right margins to zero, so I'll have to manually edit them when that happens:

.calibre {
margin-top: 0;
margin-bottom: 0;
line-height: 1.2;
text-align: justify;
}
body {
margin-left: 0;
margin-right: 0;
line-height: 1.2;
}
.body {
margin-left: 0;
margin-right: 0;
line-height: 1.2;
}


Everyone has their own way of doing things, and everyone has their own opinions. I spent a lot of time perfecting my system, and it's about as perfect as it can be (except for the few things I still have to manually edit, mainly due to publishers' incompetent methods of coding). But I will read through your last post a few times to see if I can find anything I want to change. Thanks.

[deback]

Originally Posted by steven168

wow seem too deep for me

It would have been too deep for me, too, when I first started using Calibre. But after using it for a few years, I finally decided to learn more about the features.

First, I learned how to edit files and use the CSS coding--to fix errors and reformat everything. Then I asked a few questions in this forum, received some very useful responses, did a lot of reading about different things, and then I came up with my current system of making ePub files be the way I want them to be.

[deback]

Originally Posted by JSWolf

Don't convert to try to fix things. It will make the code more difficult to fix things. Go from the original code.

I have to disagree with this. Converting (with all of my settings) has saved me tons of time by not having to manually edit things that I used to edit.

Before you dive into the CSS, in the Calibre editor is an option to remove all extra CSS code. Use that to make the CSS easier to edit.

By the time I'm checking the book (after running modify and convert), all the unused CSS codes have been deleted by Convert. I used to also run Polish to delete the unused CSS codes, but then I discovered that Convert automatically deleted them, so I quit using Polish.

I too change the indents to 1.2em. I remove the line heights as your Kobo has a setting for line height and you can set it for what you want. It's better that way. Margins should be set to 0. Again Kobo has a setting for margins and you can only increase, not decrease the margins. So again, leaving the margins at 0 lets you choose what you want. There's no need to embed Charis SIL (or the even better ChareInk version that I made). You can side load the fonts you want and use the Kobo firmware setting to select which font you want. As for justification, remove all text-align: left and justify. There again is a setting in the firmware for justification. Also, in classes that affect the body styling, remove any font-size 1em or less. Also, any you get any sort of space of 5% or more, I change that to less. What that is depends on where the space is. For ToC headers, I use a top/bottom margin of 0.8em. But you may find something like 1 or 1.5em to be more to your liking. If there are any widows and orphans in the CSS, I remove those and put a widows: 1 and orphans: 1 in the body style (if there is not a body style, add one).

I don't use Kobo. I read ebooks using ADE 2 on my laptop, which works the best for me for many reasons. I had to quit using two different Nooks because of the hassle involved, and with a laptop, I can switch between ADE and a browser or whatever. So, all the things you mentioned in the above paragraph are done automatically by Modify and Convert.

The ToC I check to see if there is a Cover as the first entry so when you first view the ePub, you go right to the cover. Without Cover as the first entry, you do not see the cover and you start at whatever the first entry is. I don't go changing the names in the ToC as they are fine as is.

I used to add an entry for the Cover in every file I did, but then I decided to quit doing that, because (1) it really wasn't important to me, and (2) it was too time-consuming when modifying/converting/editing hundreds of books.

You can install the epubcheck plugin for Calibre and that allows you to use epubcheck to validate your ePub. I've never had a problem with ePub that are validated with epubcheck. I also sometimes use the Calibre validator. If both say there are no errors, there are no structural errors.

The Check Book feature in the editor is good enough for me. Again, it would be too time-consuming to add more things to my system. I've never heard of Calibre validator, so I might try to find more info about that.

Another thing I do is remove the internal ToC (the HTML ToC) and I remove things like "About the Publisher" a list of the author's other books, any reviews, advertisements, previews for other books, newsletter links, piracy notices, and any other rubbish I don't need. You just delete those files in the text section. The NCX ToC will automatically be edited to remove those files if they are there. Also, if there are any embedded fonts I don't want, I delete them and remove the code from the CSS. If you do have any embedded fonts you do keep, in the Tools menu is where you can find the option to subset the embedded fonts to make them smaller and remove any that are not used.

This would be way too time-consuming to add to my system, due to the volume of books I work with. Removing rubbish pages in each book is not important to me (sometimes, I do remove certain pages, but not too often). I'm mainly concerned with making sure the paragraph text is justified and has side margins of 30pt and that the text is a nice font, like Charis. Also, my settings do remove all fonts and font codes automatically, and the Charis font that is embedded is automatically subset. It really depends on the book I'm working with. If I'm editing and reformatting an OpenLibrary file, then I don't include any junk pages, and I make sure that everything is perfect.

But remember, DO NOT CONVERT!

Again, I DISAGREE! I've had no problems with converting, and doing that has saved me hundreds of hours that I used to spend doing manual editing. Before I started using my current system, I would open each file in the editor and hit the buttons for beautify, smarten punctuation, remove unused CSS, ToC, and then check book. With my current system, I just hit check book, and once in a while, I'll have to manually edit something (usually the code for the cover image or a typo in one of the CSS codes), so the amount of time I spend now is much less than it used to be due to all the settings, extra CSS, and transform rules I've added to the conversion process.

[steven168]

this seem to be a big topic on Calibre

[deback]

JSWolf - I just tried your two versions of ChareInk. Found no difference between the NoWeight version and the Charis version, but I would not be able to read books with the regular version of ChareInk, since my first inclination would be to edit the book and get rid of the bold code. It hurt my eyes to look at it. That is one of the other things I always manually edit...when the paragraph text is coded to be bold.

[JSWolf]

Originally Posted by steven168

wow seem too deep for me

It sounds a lot more complicated then it really is.

[JSWolf]

Originally Posted by deback

JSWolf - I just tried your two versions of ChareInk. Found no difference between the NoWeight version and the Charis version, but I would not be able to read books with the regular version of ChareInk, since my first inclination would be to edit the book and get rid of the bold code. It hurt my eyes to look at it. That is one of the other things I always manually edit...when the paragraph text is coded to be bold.

The NoWeight version is different than the regular Charis SIL in that Caris SIL has no kerning and CareInk does.

[JSWolf]

Originally Posted by deback

I have to disagree with this. Converting (with all of my settings) has saved me tons of time by not having to manually edit things that I used to edit.

Once you get used to editing the CSS, it does get to be pretty easy. PLus, in NOT converting, you get easier code (sometimes) to edit.

By the time I'm checking the book (after running modify and convert), all the unused CSS codes have been deleted by Convert. I used to also run Polish to delete the unused CSS codes, but then I discovered that Convert automatically deleted them, so I quit using Polish.

Since the best way to edit is with the Editor after running Modify ePub, there won't be any converting and you can remove excess CSS from the editor.

I don't use Kobo. I read ebooks using ADE 2 on my laptop, which works the best for me for many reasons. I had to quit using two different Nooks because of the hassle involved, and with a laptop, I can switch between ADE and a browser or whatever. So, all the things you mentioned in the above paragraph are done automatically by Modify and Convert.

The OP does use a Kobo. That's why I mention some things the way I did. Your ways are not optimal for a Kobo or any ePub reading software that support line hight, margin, and justification settings.

I used to add an entry for the Cover in every file I did, but then I decided to quit doing that, because (1) it really wasn't important to me, and (2) it was too time-consuming when modifying/converting/editing hundreds of books.

It's important when using a Kobo Reader. But most NCX ToC does come with an entry for the cover. So it's not needed to be done for most ePub.

The Check Book feature in the editor is good enough for me. Again, it would be too time-consuming to add more things to my system. I've never heard of Calibre validator, so I might try to find more info about that.

I find that validating using Check Book and epubcheck produce ePub that do not have any issues with a Reader. I've read of many people having issues and that includes Kindle, Kobo, Sony, and others. The reason for most of the problems is due to eBooks with structural errors. So it is less time-consuming then having your eBook Reader not work properly.

This would be way too time-consuming to add to my system, due to the volume of books I work with. Removing rubbish pages in each book is not important to me (sometimes, I do remove certain pages, but not too often). I'm mainly concerned with making sure the paragraph text is justified and has side margins of 30pt and that the text is a nice font, like Charis. Also, my settings do remove all fonts and font codes automatically, and the Charis font that is embedded is automatically subset. It really depends on the book I'm working with. If I'm editing and reformatting an OpenLibrary file, then I don't include any junk pages, and I make sure that everything is perfect.

Removing the junk pages is not too time consuming. Most are obvious what they are by the filename. So you check the ones you are not sure of or think are junk pages based on thr filename and when you find a junk page, you just delete it. That's all. It's not hard and it's not really time consuming.

Again, I DISAGREE! I've had no problems with converting, and doing that has saved me hundreds of hours that I used to spend doing manual editing. Before I started using my current system, I would open each file in the editor and hit the buttons for beautify, smarten punctuation, removed unused CSS, ToC, and then check book. With my current system, I just hit check book, and once in a while, I'll have to manually edit something (usually the code for the cover image or a typo in one of the CSS codes), so the amount of time I spend now is much less than it used to be due to all the settings, extra CSS, and transform rules I've added to the conversion process.

I have to disagree with you. DON'T CONVERT! It can make editing more difficult. You don't need to beautify anything except the CSS. Most eBooks don't need the punctuation smartened. Sometimes the cover XHTML will need editing and so will the titlepage XHTML.

And there are some bugs in converting which means you will need to go into the code and find where the conversion went wrong and manually fix them. For example, a font size of small is converted to 9pt. That's smaller that a size of small and it too small to read.

And some of the things I say to remove will make the eBook much easier to deal with. Once you've removed all excess CSS, searching/replacing is very easy. It's a lot easier to do it my way than your way. And with my way, you learn more about the code which is a good thing vs. a blind conversion where you don't learn much of anything.

[retiredbiker]

Over the past year or so, I've had about 3,000 books open in the editor - anything from a quick fix to major surgery. If I've learned anything, it's that no matter how I try to automate or streamline some function, the next book breaks the mould and is totally screwed up by whatever I thought I had working so well. The variation in ebook coding is so vast it is just amazing. It keeps the hobby interesting!

I have come to agree about not converting most of the time. I'm so used to the editor it is just easier. And if I type in find <div.*?>, replace with nothing, and hit "replace all", I know I can backtrack if I have to.

But I keep finding old short stories from early mystery and sci-fi magazines that have all in-line styling and no CSS files--the only way I've found to deal with those is to convert, as it gets rid of the in-line stuff and creates a CSS file. The originals can be so dense with code it's migraine time to try and read it. No idea how else to do that.

This thread is amazing, it is like a history of just about everything I've bumped into and learned or tried over the past year. You guys are wonderful.

[deback]

JS - Most of everything you said in your last post does not apply to me. I know how to code with CSS and HTML. I'm a book editor, FPS--and valedictorians tend to learn how to do things easily and quickly--and achieve the best results.

I have lots of extra styling codes that fix a lot of things that I used to fix manually. I have lots of transform rules added that fix a lot of things that I used to have to spend time fixing manually. Conversion works great for me and does NOT make editing any harder and does not mess up things. Actually, conversion makes everything much easier. Once in a while, I'll have to manually edit something, but that happens far less often than when I used to do all the editing manually.

In fact, I've saved hundreds of hours (or more) by modifying and converting with all my extra settings that fix a lot of things automatically for me. I can load up 100 files, run modify, run convert, run check book, and run ToC on each of the files, and be done with all of it in a very short amount of time.

The results in all the books are fantastic, they all work fine, all the errors found by Calibre are gone (with very few manual edits needed), everything is justified, all the blank line spaces are gone, all the indents are how I want them, the text looks very nice, and I love how my files look when I'm done. They look much cleaner and are much better than when I started, that I can tell you.

I wish I could show you my system. If I could, I'm 100% sure you would quit criticizing me for everything. You might also want to save time and use my system, which would save you tons of time, with much better results.

[deback]

BTW, one of my settings in Transform is to change any "font-size: small;" and "font-size: medium;" to the size I want them to be. You see, all of my settings, extra CSS codes added, and transform rules take care of a lot of the things you are still editing manually. There are many things that can be done automatically in Extra CSS and Transform.

Punctuation should be smartened when you have edited a book and corrected many punctuation errors, added quotation marks, etc. I can't imagine why you said that most books don't need the punctuation smartened. But I have that performed automatically in my settings.

Beautify is necessary to fix the arrangement of everything, including line spaces between all the HTML codes. One advantage of hitting the beautify button is that when finding and replacing, you don't need the \s+ in the Find string. If you don't hit the beautify button, Find will not find all occurrences (when you don't use \s+), but after you hit the beautify button, Find will find all occurrences (without having to use \s+ in the Find string). This makes editing much quicker, because then you just have to highlight whatever you want to search for and paste it into the Find box. Quite often, there will be spaces involved, but it won't matter if you just hit the Beautify button first. (I'm talking about a string that contains space between the end of one paragraph and the beginning of the next paragraph.)

I definitely don't do "blind" conversions. You know nothing about me, so you have no idea that I'm a perfectionist, I know a lot about editing books, CSS codes, HTML, and much more that I won't go into here. I've spent hundreds of hours experimenting and perfecting my system and have edited many thousands of books to be the way I want them to be.

When you're dealing with thousands of books (like I have), there are some things that would be very time-consuming to do (for example, removing pages that publishers have added is not something I have any interest in doing and would be very time-consuming, because most of the HTML pages I see have names like blahblah_split_001.html, etc). And doing some of the things you say you do manually would be a complete waste of time, when I can do (and already do) many of them automatically.

[BetterRed]

Since TashPerth started this thread, he/she hasn't made any more posts - I wonder why.

I'll stick my neck out and suggest TashPerth is a she (Natasha) from the Scottish Perth or the Aussie one. She has obviously moved on whilst others - all men I'll venture - continue to wage a mansplaining battle even after she's stopped listening and left the room - yawning.

BR

[Divingduck]

@BR, absolutely right. It's a discussion.

[OT]
Only an idea from my side to the debaters: Why not making a separate new thread like a "How to do things better or easier" and make it as a sticky post if it is useful for others too. And a separate discussion thread of discussing things around it. That would have much more benefit as these kind of discussions in lots of threads.
[/OT]