![]() |
#1 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,130
Karma: 91256
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
|
Regular expressions, Calibre and you- an introduction (Archived)
Revision 8 (27.09. 2010)
The intent of this introduction is not so much to explain all finesses of regular expression usage, but rather to explain enough to handle some common tasks in Calibre and get new users started and knowledgeable enough that they can further educate themselves using the (rather technical) explanation given in the Python documentation, linked through the Calibre manual. So, let's get started. First, a word of warning and a word of courage: This is, inevitably, going to be somewhat technical- after all, regular expressions are a technical tool for doing technical stuff. I'm going to have to use some jargon and concepts that may seem complicated or convoluted. I'm going to try to explain those concepts as clearly as I can, but really can't do without using them at all. That being said, don't be discouraged by any jargon, as I've tried to explain everything new. And while regular expressions themselves may seem like an arcane, black magic (or, to be more prosaic, a random string of mumbo-jumbo letters and signs), I promise that they are not all that complicated. Even those who understand regular expressions really well have trouble reading the more complex ones, but writing them isn't as difficult- you construct the expression step by step. So, take a step and follow me into the rabbit's hole. Where in Calibre can you use regular expressions? There are a few places Calibre uses regular expressions. There's the header/footer removal in conversion options, metadata detection from filenames in the import settings and, since last version, there's the option to use regular expressions to search and replace in metadata of multiple books. What on earth is a regular expression? A regular expression is a way to describe a particular string of characters (string for short). (Technical note: I'm using string here in the sense it is used in programming languages: a string of one or more characters, characters including actual characters, numbers, punctuation and so-called whitespaces (linebreaks, tabulators etc.). Please note that generally, uppercase and lowercase characters are not considered the same, thus "a" being a different character from "A" and so forth. In Calibre, regular expressions are case insensitive in the search bar, but not in the conversion options. There's a way to make every regular expression case insensitive, but we'll discuss that later.) It gets complicated because regular expressions allow for variations in the strings it matches, so one expression can match multiple strings, which is why people bother using them at all. More on that in a bit. Care to explain? Well, that's why we're here. First, this is the most important concept in regular expressions: A string in itself is a regular expression that matches itself. That is to say, if I wanted to match the string "Hello, World!" using a regular expression, the regular expression to use would be Code:
Hello, World! That doesn't sound too bad. What's next? Next is the beginning of the really good stuff. Remember where I said that regular expressions can match multiple strings? This is were it gets a little more complicated. Say, as a somewhat more practical exercise, the ebook you wanted to convert had a nasty footer counting the pages, like "Page 5 of 423". Obviously the page number would rise from 1 to 423, thus you'd have to match 423 different strings, right? Wrong, actually: regular expressions allow you to define sets of characters that are matched: To define a set, you put all the characters you want to be in the set into square brackets. So, for example, the set Code:
[abc] Code:
[a-z] Code:
[a-zA-Z] Code:
Page [0-9] of 423 Code:
Page [0-9][0-9] of 423 Hey, neat! This is starting to make sense! I was hoping you'd say that. But brace yourself, now it gets even better! We just saw that using sets, we could match one of several characters at once. But you can even repeat a character or set, reducing the number of expressions needed to handle the above page number example to one. Yes, ONE! Excited? You should be! It works like this: Some so-called special characters, "+", "?" and "*", repeat the single element preceding them. (Element means either a single character, a character set, an escape sequence or a group (we'll learn about those last two later)- in short, any single entity in a regular expression.) These characters are called wildcards or quantifiers. To be more precise, "?" matches 0 or 1 of the preceding element, "*" matches 0 or more of the preceding element and "+" matches 1 or more of the preceding element. A few examples: The expression "a?" would match either "" (which is the empty string, not strictly useful in this case) or "a", the expression "a*" would match "", "a", "aa" or any number of a's in a row, and, finally, the expression "a+" would match "a", "aa" or any number of a's in a row (Note: it wouldn't match the empty string!). Same deal for sets: The expression Code:
[0-9]+ Code:
Page [0-9]+ of 423 A note on these quantifiers: They generally try to match as much text as possible, so be careful when using them. This is called "greedy behaviour"- I'm sure you get why. It gets problematic when you, say, try to match a tag. Consider, for example, the string "<p class="calibre2">Title here</p>" and let's say you'd want to match the opening tag (the part between the first pair of angle brackets, a little more on tags later). You'd think that the expression Code:
<p.*> Code:
.* Code:
<p.*?> There's actually another way to accomplish this: The expression Code:
<p[^>]*> Well, these special characters are very neat and all, but what if I wanted to match a dot or a question mark? You can of course do that: Just put a backslash in front of any special character and it is interpreted as the literal character, without any special meaning. This pair of a backslash followed by a single character is called an escape sequence, and the act of putting a backslash in front of a special character is called escaping that character. An escape sequence is interpreted as a single element. There are of course escape sequences that do more than just escaping special characters, for example "\t" means a tabulator. We'll get to some of the escape sequences later. Oh, and by the way, concerning those special characters: Consider any character we discuss in this introduction as having some function to be special and thus needing to be escaped if you want the literal character. So, what are the most useful sets? Knew you'd ask. Some useful sets are Code:
[0-9] Code:
[a-z] Code:
[A-Z] Code:
[a-zA-Z] Code:
[a-zA-Z0-9] Code:
\d is equivalent to [0-9] \w is equivalent to [a-zA-Z0-9_] \s is equivalent to any whitespace Code:
[^a] Code:
[^0-9] Code:
<p[^>]*> But if I had a few varying strings I wanted to match, things get complicated? Fear not, life still is good and easy. Consider this example: The book you're converting has "Title" written on every odd page and "Author" written on every even page. Looks great in print, right? But in ebooks, it's annoying. You can group whole expressions in normal parentheses, and the character "|" will let you match either the expression to its right or the one to its left. Combine those and you're done. Too fast for you? Okay, first off, we group the expressions for odd and even pages, thus getting Code:
(Title) (Author) Code:
(Title|Author) You can of course use the vertical bar without using grouping parentheses, as well. Remember when I said that quantifiers repeat the element preceding them? Well, the vertical bar works a little differently: The expression "Title|Author" will also match either the string "Title" or the string "Author", just as the above example using grouping. The vertical bar selects between the entire expression preceding and following it. So, if you wanted to match the strings "Calibre" and "calibre" and wanted to select only between the upper- and lowercase "c", you'd have to use the expression "(c|C)alibre", where the grouping ensures that only the "c" will be selected. If you were to use "c|Calibre", you'd get a match on the string "c" or on the string "Calibre", which isn't what we wanted. In short: If in doubt, use grouping together with the vertical bar. You missed... ... wait just a minute, there's one last, really neat thing you can do with groups. If you have a group that you previously matched, you can use references to that group later in the expression: Groups are numbered starting with 1, and you reference them by escaping the number of the group you want to reference, thus, the fifth group would be referenced as "\5". So, if you searched for "([^ ]+) \1" in the string "Test Test", you'd match the whole string! That's really incredibly useless... Oh, you'll see. You missed something. In the beginning, you said there was a way to make a regular expression case insensitive? Yes, I did, thanks for paying attention and reminding me. You can tell Calibre how you want certain things handled by using something called flags. You include flags in your expression by using the special construct Code:
(?flags go here) Code:
test(?i) Another useful flag lets the dot match any character at all, including the newline, the flag "s". If you want to use multiple flags in an expression, just put them in the same statement: "(?is)" would ignore case and make the dot match all. It doesn't matter which flag you state first, "(?si)" would be equivalent to the above. By the way, good places for putting flags in your expression would be either the very beginning or the very end. That way, they don't get mixed up with anything else. I think I'm beginning to understand these regular expressions now... how do I use them in Calibre? Let's begin with the conversion settings, which is really neat. In the structure detection part, you can input a regexp (short for regular expression) that describes the header or footer string that will be removed during the conversion. The neat part is the wizard (Go ahead, give Kovid some karma already, the man deserves it!): Click on the wizard staff and you get a preview of what Calibre "sees" during the conversion process. Scroll down to the header or footer you want to remove, select and copy it, paste it into the regexp field on top of the window. If there are variable parts, like page numbers or so, use sets and quantifiers to cover those, and while you're at it, rememper to escape special characters, if there are some. Hit the button labeled "Test" and Calibre highlights the parts it would remove were you to use the regexp. Once you're satisfied, hit OK and convert. Be careful if your conversion source has tags like this example: Code:
"Maybe, but the cops feel like you do, Anita. What's one more dead vampire? New laws don't change that." </p><p class="calibre4"> <b class="calibre2">Generated by ABC Amber LIT Conv<a href="http://www.processtext.com/abclit.html" class="calibre3">erter, http://www.processtext.com/abclit.html</a></b></p><p class="calibre4"> It had only been two years since Addison v. Clark. The court case gave us a revised version of what life was Code:
<b class="calibre2"> Code:
</b> Code:
<b.*?>.*?</b> Code:
<b.*?>\s*Generated\s+by\s+ABC\s+Amber\s+LIT.*?</b> Another thing you can use regular expressions for is to extract metadata from filenames. You can find this feature in the "Adding books" part of the settings. There's a special feature here: You can use field names for metadata fields, for example (?P<title>) would indicate that calibre uses this part of the string as book title. The allowed field names are listed in the windows, together with another nice test field (Remember that karma you wanted to give Kovid?). An example: Say you want to import a whole bunch of files named like "Classical Texts: The Divine Comedy by Dante Alighieri.mobi" (Obviously, this is already in your library, since we all love classical italian poetry ![]() Code:
(?P<title>.+) - (?P<author>[^_]+) Code:
[a-zA-Z]+: (?P<title>.+) by (?P<author>.+) The last part is regular expression search and replace in metadata fields. You can access this by selecting multiple books in the library and using bulk metadata edit. Be very careful when using this last feature, as it can do Very Bad Things to your library! Doublecheck that your expressions do what you want them to using the test fields, and only mark the books you really want to change! In the regular expression search mode, you can search in one field, replace the text with something and even write the result into another field. A practical example: Say your library contained the books of Frank Herbert's Dune series, named after the fashion "Dune 1 - Dune", "Dune 2 - Dune Messiah" and so on. Now you want to get "Dune" into the series field. You can do that by searching for "(.*?) \d+ - .*" in the title field and replacing it with "\1" in the series field. See what I did there? That's a reference to the first group you're replacing the series field with. Now that you have the series all set, you only need to do another search for ".*? - " in the title field and replace it with "" (an empty string), again in the title field, and your metadata is all neat and tidy. Isn't that great? By the way, instead of replacing the entire field, you can also append or prepend to the field, so, if you wanted the book title to be prepended with series info, you could do that as well. As you by now have undoubtedly noticed, there's a checkbox labeled "Case sensitive", so you won't have to use flags to select behaviour here. Well, that just about concludes the very short introduction to regular expressions. Hopefully I'll have shown you enough to at least get you started and to enable you to continue learning by yourself- a good starting point would be the Python documentation for regexpes. One last word of warning, though: Regexpes are powerful, but also really easy to get wrong. Calibre provides really great testing possibilities to see if your expressions behave as you expect them to. Use them. Try not to shoot yourself in the foot. (God, I love that expression...) But should you, despite the warning, injure your foot (or any other body parts), try to learn from it. Credits: Thanks for helping with tips, corrections and such:
Edit history:
Last edited by Manichean; 01-26-2011 at 05:37 PM. Reason: edit, see history |
![]() |
![]() |
#2 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
|
You need to be careful about deleting everything between <p> and </p> tags. In that particular example book if you did that you would delete actual book text in addition to the headers.
While it's generally a good idea to always try to remove both the opening and closing tags, the only format I think that's critical for is epub. Calibre will force the files into xhtml spec if it discovers they're out of spec. (I think for epub it assumes they're in spec, so you could really screw up epub) generally .*? is better than .*, and will usually do what users actually want it to. I'd use ? instead of *? to make something optional. You can think of brackets [] as single character groupings, but for string groupings use parentheses and | (one|two|three|four) A few other useful expressions: Matching p tags with any styles/ids: <p[^>]*> Never specify actual spaces in your regular expression. Use \s, which tells regex to look for a space. Better yet use \s+ or \s*, which match one or more spaces or zero or more spaces respectively. I make liberal use of \s* in my expressions because you never know when a stray space will hurt you. \s* also has the benefit of passing through any whitespace including tabs and carriage returns. So when you really do need to match everything between <p></p>, except your opening and closing tags are across lines, you can use \s* to get you there. Last edited by ldolse; 09-23-2010 at 03:12 PM. |
![]() |
Advert | |
|
![]() |
#3 |
creator of calibre
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 44,273
Karma: 23661992
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
|
calibre will try to fix broken html for EPUB as well.
|
![]() |
![]() |
#4 | |
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 11,911
Karma: 7076137
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
This exposes one of the problems with regular expressions. Using regexps, it is difficult to do delimited matching in the constrained case, and impossible in the general case. Doing it right usually requires a recursive state machine, which by definition cannot be described by a regular expression. For fun, try to write a regular expression that matches any palindrome. (http://en.wikipedia.org/wiki/Palindrome. Examples: abcdedcba or 'madam im adam' with spaces ignored.) You will fail. ![]() Edit: the paragraph above deals with computational theory and does not belong in a tutorial. However, it might be useful for Manichean, which is why I added it. Last edited by chaley; 09-20-2010 at 04:06 AM. |
|
![]() |
![]() |
#5 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,130
Karma: 91256
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
|
Thanks for the suggestions, all of you. I'll edit those in sometime the next few days.
I do have one question, though... I don't see that in this example. I haven't actually tried it, but wouldn't matching, in this case, start at "<p class="calibre4"><b class="calibre2">Generated by..." and finish with the closing "</p>"? In which case, no book text would be removed? |
![]() |
Advert | |
|
![]() |
#6 | |
US Navy, Retired
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
As I see it in your example above, a brief review makes it look like you are saying to start your regex with <p class="calibre4"> and end with </p> without being much more specific this will include many other paragraphs of text as well. Looking forward to the finished primer. |
|
![]() |
![]() |
#7 | |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,130
Karma: 91256
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
|
Quote:
|
|
![]() |
![]() |
#8 | ||
US Navy, Retired
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 9,865
Karma: 13806776
Join Date: Feb 2009
Location: North Carolina
Device: Icarus Illumina XL HD, Nexus 7
|
Quote:
I'm not arguing that you were trying to remove everything between <p> </p> tags. I was just pointing out the obvious way ldolse saw the connection of deleting anything between <p> </p> tags. A new user reading a primer would take away the opening and closing info from your code tags and then try to start applying expressions to everything in between. To stem off this type of confusion your top code box should be more specific. To show the opening tag by including "<p class="calibre4"><b class="calibre2">Generated by". You stop the initial part of the primer in a spot that could get and energetic user in trouble. One more thing, I could be far off base, but since folks will be seeing the below code in their book viewer, html viewer or Sigil, and the below will be word wrapped in those viewers (or not I'm not sure), wouldn't it be better to put it in quotes so users can see the entire picture? Quote:
![]() ![]() Update: In my experience if the above is in a <p class="calibre4"> tag most every paragraph in the book will be using that tag too. That's why any primer needs to emphasize ways of limiting your expression to avoid accidentally removing your entire text. Last edited by DoctorOhh; 09-20-2010 at 06:05 AM. |
||
![]() |
![]() |
#9 | |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,130
Karma: 91256
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
|
Ah, now I see what you mean. Thanks for the input, especially the part about getting energetic users in trouble, I'll have to think about that.
Quote:
|
|
![]() |
![]() |
#10 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,452
Karma: 10484861
Join Date: May 2006
Device: PocketBook 360, before it was Sony Reader, cassiopeia A-20
|
Great work.
I am big fan of Regular Expressions, and I have recently started to use Calibre for other things than for just a conversion now and then. I will keep close eye on this thread. Regular Expressions are very powerful stuff and deserve to be popularized a little bit more. ?, + and * are called quantifiers, because they quantify whatever lies before them. The very first thing that a beginner needs to know about those standard quantifiers, you can see in any RE implementation is, that they are GREEDY. Yes, there are also non-greedy quantifiers, as one of previous posters pointed out. In Python syntax those are *?, +?, ??. Yes, there are *many* different syntaxes for Regular Expressions. I won't go further, I do not want to scare our dear readers away ;-) A '*' quantifier will eat as much of the string as it can. Let's have an example. You have string 'AuthorFirstName AuthorLastName - series - title.epub' and you want to match 'AuthorFirstName AuthorLastName - '. So, you write an expression like: '.* - ' to match Author. But! '.' matches any character and '*' quantifier takes as much as possible, so instead of matching 'AuthorFirstName AuthorLastName - ' as you have intended, you will match 'AuthorFirstName AuthorLastName - series - ' You need to search for '[^-]* - ' '[^-]' means match ANY character BUT '-' If the first character in a group is '^' the rest of group is effectively a list of characters that are NOT supposed be matched. I very, *very* strongly recommend THE best^H^H^H^Hmost exhaustive (pun intended) book ever written about Regular Expressions - Mastering Regular Expressions - Book on regular expressions by Jeffrey Friedl, published by O’Reilly. Please see http://docs.python.org/library/re.html for Recomandation about which version of book to use The book is difficult, but worth its weight in gold if you want to understand Regular Expressions. |
![]() |
![]() |
#11 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 1,337
Karma: 123455
Join Date: Apr 2009
Location: Malaysia
Device: PRS-650, iPhone
|
I did mean '<p[^>]*>', thanks. I was aware of the problem with internal attributes being a problem for that expression, but I figured that was getting into a rat-hole for this level of tutorial and in practice it's pretty rare.
|
![]() |
![]() |
#12 |
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,130
Karma: 91256
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
|
Edited the tutorial to include suggestions as of now. It'd be great if the resident gurus could give it another read. Thank you, guys- you know who you are
![]() |
![]() |
![]() |
#13 | ||||||||||||||||||||||
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 11,911
Karma: 7076137
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Thank you for writing this!
Comments below. Some are pedantic, but I can't help it. ![]() Quote:
Quote:
Quote:
Quote:
Quote:
![]() Quote:
'\D'==[^0-9], \s== (set of whitespace) (this one is important), \w==[a-zA-Z0-9_] (Note: I see that you did this further down. ![]() Quote:
You might also want to introduce the word 'element', which means a character or class or (eventually) group. Quantification applies to the previous element. (The computer scientist in me wants to get into recursion, but that would be a disaster. ![]() Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
![]() Quote:
Quote:
Quote:
Quote:
![]() Last edited by chaley; 09-21-2010 at 08:00 AM. |
||||||||||||||||||||||
![]() |
![]() |
#14 | ||||
Wizard
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 3,130
Karma: 91256
Join Date: Feb 2008
Location: Germany
Device: Cybook Gen3
|
First of all, thank you again for your comments. This is actually where it starts to become a learning experience for me as well.
I'll definitely go back and edit the text again, but right after reading this, a few comments: Quote:
![]() Quote:
Quote:
Quote:
![]() By the way, concerning your comment on palindromes a while back: I think I see what you mean. I believe I've figured out how to match any palindrome of a given length not containing whitespaces (as in I couldn't match "madam im adam"), but that's about as far as I got. |
||||
![]() |
![]() |
#15 | |||
Grand Sorcerer
![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() ![]() Posts: 11,911
Karma: 7076137
Join Date: Jan 2010
Location: Notts, England
Device: Kobo Libra 2
|
Quote:
Code:
(?iLmsux) (One or more letters from the set 'i', 'L', 'm', 's', 'u', 'x'.) The group matches the empty string; the letters set the corresponding flags: re.I (ignore case), re.L (locale dependent), re.M (multi-line), re.S (dot matches all), re.U (Unicode dependent), and re.X (verbose), for the entire regular expression. (The flags are described in Module Contents.) This is useful if you wish to include the flags as part of the regular expression, instead of passing a flag argument to the re.compile() function. 1) ignore case is turned on by default, and therefore cannot be turned off. 2) (in python) the flags affect the entire expression, even if they occur later in the expression. So, to use DOTALL to match tags split across lines (<a tags are famous for this>, I would do something like '(?s)<.*?>'. Re.M is also incredibly useful, because it allows you to use anchored expressions that match in the middle of the document. For example, '(?im)^<a.*?\/a>$' would match hyperlink tags that start at the beginning of a line, end at the end of a line, but perhaps contain line endings. Quote:
Quote:
<professorial_mode> The general case cannot be solved with regular expressions because REs don't have the notion of 'stack'. Said another way, and getting a bit formal, all REs by definition can be translated into a deterministic finite state machine. The important part here is that the number of states is known from the RE, and is fixed for all utterances (text to be matched). Parsing utterances in a palindromic language requires a state for each letter up to the center point so the machine can match the right letter after the center point. Such a machine requires len(utterance)/2 states. Thus the number of states is unbounded, meaning that the grammar for the language cannot be described using an RE. Because of the above problem, compilers usually use multiple grammars. One describes the input alphabet (identifiers etc) and symbols, and can often be an RE. Another describes the order of symbols, and is almost always a non-regular context-free grammar. Sometimes there is are more grammars for certain constructs or for the optimizer. </professorial_mode> ![]() |
|||
![]() |
![]() |
Tags |
regexp calibre tutorial |
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
Problem with regular expressions | Manichean | Conversion | 10 | 02-03-2011 02:27 PM |
Custom Regular Expressions for adding book information | bigbot3 | Calibre | 1 | 12-25-2010 06:28 PM |
Help with Regular Expressions | ghostyjack | Workshop | 2 | 01-08-2010 11:04 AM |
Regular Expressions help needed | Phil_C | Workshop | 20 | 10-03-2009 12:14 AM |
BookDesigner v5 and regular expressions | ShineOn | Sony Reader | 11 | 08-25-2008 04:06 PM |