Using News Recipes: Start Here

[Starson17]

What's a "News Recipe?"
The official Calibre definition is:

A recipe is a set of instructions that teaches calibre how to convert an online news source, like a magazine or a blog into an e-book. A recipe, is essentially python code. As such, it is capable of converting arbitrarily complex news sources into e-books. At the simplest level, it is just a set of variables such as URLs that give calibre enough information to go out onto the internet and download the news.

Almost any RSS feed or HTML-based web site can be converted into an e-book. Calibre has "builtin" recipes for over one thousand popular websites, newspapers, magazines and blogs in more than 30 languages and 50 countries. Examples include: CNN, Wired Magazine and the Dilbert comic strip. A recipe can be run manually or it can be scheduled to run automatically.

Where do I find recipes?
To see the "builtin" recipes, click the button marked "Fetch News" on the top icon bar of Calibre (it has a large "N" and is red). You will see the "Schedule news download" dialog screen appear. Recipes are organized by language (country) on the left side of this screen. You can also find recipes in this forum, you can modify builtin recipes within Calibre or you can write your own recipes using the tips and tricks you find here.

How do I run a recipe to create an e-book?
Click the button marked "Fetch News" on the icon bar and select a recipe from the language (country) listing on the left side of the "Schedule news download" dialog screen. You will then see related options for that recipe on the right side of the screen. You can choose to immediately download the selected recipe (click "Download now") or schedule it for automatic download at a specified time. Recipes you write, recipes you have modified based on builtin recipes and recipes you get from others are called "custom recipes." They will appear under the heading "Custom," which will appear at the top of the language(country) list on the left side of the dialog screen after you have added your first custom recipe.

Can I change the output format of a recipe?
Yes. By default, the recipe system creates EPUBs. If you set "Preferences - Behavior - Preferred output format" to another format, the recipe will be automatically converted to your preferred output format after it runs. Some output formats, such as MOBI for Kindles, cause the recipe system to add special information to the output, which causes the reader to treat the recipe-created ebook as a periodical, not an ebook. Thus, a MOBI format ebook automatically created by the recipe system may be slightly different from a recipe created default EPUB that is later manually converted to MOBI format.

Where do I find the e-book created by a recipe?
The e-book appears in your Calibre library after a recipe successfully runs. The author will be "calibre" and the title will be the recipe title - usually followed by the download date. It will appear at the top of your list of books, if you haven't resorted them since they were created. The book date will be the date of download. By default the recipe-created e-book will have two tags: "News" and a tag that matches the recipe name. You can also assign extra tags to be added to the e-book created by a recipe by selecting the "Advanced" tab on the right side of the "Schedule news download" dialog screen. You can find your recipe created e-books by the usual searching options or in the Tag Browser under News (sorted by recipe name), under Author (look for "calibre"), or under Tags (by the tags of "News" or recipe name).

Do I have to pay for e-books created by recipes?
No. You may need to add a username and password on the right side of the "Schedule news download" dialog screen if the website requires registration. Some sites have free registration and you simply register at the site to get the username and password needed for the recipe. Some recipes come in both free and subscription versions. For recipes written to be used with paid subscription sites, you will have to buy a subscription from the website. No money is received by Calibre's author or the recipe developers. You can donate to Calibre here, and some prolific developers and recipe authors accept donations.

What recipes are other people using?
Recipe Stats lists recipes in order of their frequency of use. Web sites change frequently, and the matching recipe has to be changed accordingly. When you run a builtin recipe, Calibre automatically checks an online recipe database and uses the most recent version of that recipe. When run from the Calibre GUI, Calibre will automatically fetch and use the most recent version of a builtin recipe, even if you haven't recently updated the Calibre program. When a recipe is run from the command line (with ebook-convert) the automatic fetch only works if the builtin recipe is run by "Title.recipe" (a file with that name should not be in the local directory). This automatic check process is used to provide the recipe statistics of use seen at the link above. It's also why changing Calibre's locally stored copy of a recipe won't have any effect. The local copy is only used as a backup if the online recipe database is not accessible. (If you want to modify a built in recipe - continue reading).

How do I use a custom recipe from a *.recipe file?
Download a .recipe file from here or get one from a friend, etc. It must be a file in the *.recipe (or *.py) form. In the Calibre GUI, click the small black triangle to the right of the button marked "Fetch News" and select the "Add a custom news source" option to open the custom news dialog screen. Note that you add, create and modify recipes only from the "Add a custom news source" screen and you schedule and run recipes only from the "Schedule news download" dialog screen. On the custom news dialog screen that opens, click "Load recipe from file" and browse to your *.recipe file, then click "Open" and it will immediately appear in a list on the left side of this screen under "Available user recipes." You can now exit this screen by clicking the "Close" button on the lower right - say yes to the confirmation box that appears to confirm you want to exit. Return to the "Schedule news download" dialog screen and you will see the new custom recipe listed under the "Custom" heading where it can be selected and run or scheduled, just like a builtin recipe.

How do I use a custom recipe that was posted as text (code), not as a file?
You have two options:

1. Select all the recipe text from here, save it as a file named with the .recipe extension and follow the steps above, or
2. Select all the recipe text and go to the "Add a custom news source" dialog screen. Click the button on the upper right of the dialog screen marked "Switch to Advanced mode." (If you see "Switch to Basic mode," you are already in Advanced mode.) Delete everything in the box below the "Switch to Advanced mode" button (the text to be deleted starts with "class AdvancedUserRecipe" and ends "feeds = []"), then paste in the text/code you copied from here. The final step is to press the button marked "Add/Update recipe" on the left side of the dialog screen. Until you do that the recipe isn't saved. You will see the title of the recipe appear on the left side of the dialog screen under "Available user recipes."

For both methods, be very careful to preserve all indents and spaces from the text/code. Indentation is very important in Python, which is the language Calibre uses for recipes. If there aren't any indents in the code you have copied from here, the recipe probably won't work. Note that MobileRead's forum software preserves critical spaces and indent formatting, but does not display that formatting, so you can't just copy a recipe from here that isn't in CODE tags. For a recipe that isn't inside CODE tags, you can start a reply with a quote of the posted recipe and you will see all required formatting. Copy the recipe, then simply close the quote reply without saving it. When saving the recipe as a file, use a text editor, or save in text mode. Windows users can use Notepad.

How do I modify a builtin recipe to make it work better?
You can't, even though some builtin recipes are specifically written to be modified by the user. You can only modify a personal copy of the builtin recipe. The Arcamax, GoComics and comics.Com builtin recipes can retrieve hundreds of different comic strips, but by default they only retrieve a few. You have to modify a copy of them to select the comics you want (by removing the "#" hash/pound comment character from in front of comics you want). To make a copy of a builtin recipe (turning the copy into a custom recipe), start with the "Add a custom news source" dialog screen. Click the "Customize builtin recipe" button on the lower left. Select the desired builtin recipe to be modified from the list that appears and click "OK." This creates your personal copy of the selected builtin recipe. It appears in the "Available user recipes" custom recipe list. You then modify the copy as described below.

If you really need to modify a builtin recipe to change it for everyone, for example because you've got a fix for one that's broken, you can post your fix here and request Kovid to change it. If you think your fix or improvement is good enough to warrant changing the existing builtin, please make that clear in your post and explain why it's better than the existing one. Kovid reads this forum and will pick up improved recipes or new recipes, but we don't want to make him have to guess about what the change is.

How do I modify a custom recipe?
Whether you want to change a custom recipe you wrote, one you obtained from here, or a copy of a builtin recipe created according to the instructions above, the procedure is the same. Select a recipe from the "Available user recipes" list on the "Add a custom news source" dialog screen. The recipe code for that recipe will appear on the right side. Make the changes you need and press the button marked "Add/Update recipe" (it's found on on the lower left side of the dialog screen). Until you press that button your changes to the recipe aren't saved. You will get a warning that you are about to overwrite another custom recipe of the same name (unless you have changed the line "title = 'Recipe Name'" in the recipe). Let Calibre overwrite. Note that if you are working on a builtin recipe, it is only a copy. It will have the same name as the original builtin recipe (unless you changed the title line), but it will appear under "Custom" instead of under a language(country) on the schedule screen. You will have two recipes of that name under the Scheduling screen, so make sure you chose the correct one. Unlike the original builtin, your copy can never be updated or changed by Calibre. If the original is updated, you will need to manually update your modified version.

What if a built in recipe doesn't work?
Even a tiny change to a website or RSS feed can make a recipe stop working. A common symptom of a change to the website is that the articles will be blank, but you will still get an index/table of contents. To track it down, try another recipe. If that works, try going to the website or RSS feed for the recipe. If that works, it's likely that the site has changed and the recipe needs to be fixed. If those don't work, it's not the fault of the recipe. If the recipe is broken, post a polite notice here and the recipe author may respond. You can also post a bug report (however, Kovid, Calibre's author, does not maintain all recipes, even if it's a builtin). Your post should include a description of the problem, and the error report. You can get the error report by clicking the spinning "Jobs" icon (lower left of the Calibre GUI), selecting the failed recipe from the job list and clicking the "Show job details button." Select the entire text, and paste it into your message here. Before posting it here press the "Spoiler" button (an eye with a red "X" across it) in the reply editor here. Then paste the lengthy text between the Spoiler tags. That will cause the lengthy text to hide under a Spoiler tag so your post is easy to read. Clicking the Spoiler tag expands it and only those who need to see the error text will need to wade through it.

I want to fix a broken recipe myself. What should I do?
Recipes vary in complexity, but often they break due to a change in the source site that merely rearranges the content. When a recipe is written, the author uses tags in the site html code to identify/keep desirable content and remove what isn't wanted. If those tags are changed, the recipe throws away the content you want. There is one simple change to a recipe that may cause it to start working better. Make a custom copy of the builtin recipe, as above. Then edit it to remove or comment out (using the hash "#" symbol in front of each line) all the lines or groups that start:

• keep_only_tags
• remove_tags
• remove_tags_before
• remove_tags_after

If the recipe is not too complex, and the site change is minor, this may cause the recipe to stop throwing away the content you want. You are also likely to get lots of content you don't want. If you begin to get the content you want, you can return each removed or commented line or group, one by one, to find the problem. If these changes fail to improve things, you will need to see here and here for more info on recipe writing.

What if there isn't a recipe for my favorite site?
Start a thread and feel free to ask for someone to write it for you. Lots of recipe gurus hang out here. Remember that this is all volunteer. You are more likely to get help if you provide as much help as possible. A link to the RSS feed is essential. Posting links to several sample articles from the feed will also help so a potential author can see how the links are constructed. If the articles have links to print versions, posting those links along with matching article links would also be helpful. If free registration is needed, you can create a test registration and offer to PM the username and password.

How do I write my own recipe?
Start here for the basic tutorial. Next go here for the API. Here is the documentation for Mechanize for low level control of cookies, headers and HTML handshaking.

There is a simple basic method of asking Calibre to fetch an RSS feed: It's the Basic Recipe mode on the "Add a custom news source" dialog screen. To use the Basic Recipe Mode all you have to do is provide one or more RSS feeds. Unfortunately, the results are often not very satisfying, as the resulting ebook will contain various undesirable content, such as advertising, non-working links, etc. A Basic Mode Recipe can be converted to an Advanced Mode Recipe to improve it. The recipe authors that hang out in the recipe forum enjoy helping others learn to write recipes -feel free to ask questions.

You can look at all current recipe code for the builtin recipes here.

How about some tips on writing my own recipes?
The basic tutorial is a good start. Most recipe authors write their recipes using a separate editor and run them with ebook-convert from the command line. I keep three files open in my editor when writing and debugging. They are: 1) the .recipe file (_Test_1.recipe), 2) a batch/script file that runs the recipe with verbose and testing options and 3) a text file (_Test.txt) with error messages and verbose output from the last recipe run.

I prefer this in the batch/script file:

ebook-convert _Test_1.recipe _Test_1 --test -vv > _Test.txt

(The tutorial link above suggests using alternative debug options for ebook-convert that produce an actual ebook and several stages in conversion from the initial HTML to the ebook format specified, but I prefer to wait until the recipe is producing good HTML before spending the time to create an ebook.)

The output will be saved as html in the folder _Test_1. Clicking on the index.html file in that folder will let you examine the output in your browser. Adding print statements to the recipe at critical points will let you track what's happening. Specifically, adding a line like this:

print 'At this point in the code the value of variable_X is: ', variable_X

will print the variable_X into the _Test.txt file. Make sure you properly Python indent that line for the part of the recipe code you are working on.

For studying the source page, FireFox with the FireBug addon works very well. If you have complicated issues relating to cookies and HTTP headers, the TamperData addon is excellent.

See here for a detailed explanation of using auto_cleanup to help produce a good ebook with minimal effort. It's particularly useful for combining content from multiple sources. That link points to a specific article in the Calibre Tips and Tricks Blog, which I highly recommend, even though it's not exclusively dedicated to recipes.

When you are done with your recipe, post it in the forum for others to use.

We'd like your input, even though this thread is a sticky and is closed. If you find anything here confusing, or think of anything that should have been here, but wasn't covered, please start a thread and comment.