[GUI Plugin] Search the Internet

[kiwidude]

This plugin was designed to allow quick navigation from the selected books in the library view in Calibre to a choice of websites in your web browser.

By default websites such as FantasticFiction, Amazon, Google and Wikipedia are assigned, however around 100 more website links are included for you to select from and you can add/remove your own to this list.

I mainly use this plugin to look for better quality metadata than that available from the default Calibre sources, however it is also useful to get detailed reviews, identify missing books in a series, accurate publication dates, find similar authors etc.

Main Features:

• Launch website searches for books matching author, title, isbn or any other book metadata field
• Can assign multiple pages to open at once in a special group
• Easy to use configuration dialog accessed via right-click menu or Preferences->Plugins
• Over 100 predefined website urls for you to choose from, plus the ability to add your own.
• Supports HTTP POST searches for websites that cannot use a querystring
• Menu is fully customisable to change ordering, add submenus, keyboard shortcuts and images
• Import and export capability for backup or sharing with other users
• Help file web page available from the configuration dialog

Special Notes:

• Help Documentation
• Change History
• Requires Calibre 2.0.0 or later. Use of template language features may require a later version of calibre

Tutorial for Adding a Website Link:

Spoiler:

In this example we will add a link to a website not currently in the plugin (www.fictionwise.com)

1. The first step is to identify how the website accepts search information. Just try using the search box on the website, taking note of what appears in your browser bar when the results are displayed.
2. If the url displayed in the browser bar contains a "querystring" then this will be the simplest website to add. Take for example www.goodreads.com. When I type "hunger games", the url displayed is http://www.goodreads.com/search?q=hunger+games. So we know that a simple substitution of plugin tokens like {title} in place of "hunger+games" would give us a search url we could re-use. This is called an HTTP GET type of query.
3. If however the url displayed does not contain the text you searched, this means an HTTP POST query is taking place. This means the data we search with must be sent using code in the web page and requires a bit more work. If we search for the author "lyn gala" on fictionwise.com, the url displayed is http://www.fictionwise.com/servlet/mwsearch. So this tells us this website falls into this latter category.
4. The next step will be to work out what we need to pass to the website. If you are using HTTP GET, you can usually just experiment with the website, trying different search dropdowns it has and taking note of how the url of the results changes depending on whether you search by author, title etc. Every website handles this differently.
5. For HTTP POST websites we must use view source page from your browser. We will be looking for "<form " in the html displayed for the special code that submits the search data. In fictionwise.com we find "<form name=searchform..." near the top. We can see in the "action" that data will be sent to "/servlet/mwsearch". which matches the url we saw displayed in the browser. We can see a bunch of <input> elements including searchstring which will contain our text, a <select> element called searchtype with values like title and author. There are also some hidden <input> tags for action and template
6. We need to put together our name=value pairs of data to supply, using either fixed values specific to the website, or dynamic values using plugin tokens like {title} to pass the title of the selected book. In this case to search on fictionwise.com for an author we know we will want to set searchtype=author and searchstring={author}
7. We are now ready to start configuring this in the plugin for testing. In the search the internet plugin, choose Customize plugin and click on the green plus button to add a new row.
8. Type a value into the Title column, like "Fictionwise for Author"
9. In the Method column on the far right, we will choose POST or GET. At first glance, it seems like we should use POST in this particular case.
10. In the Url column, we need to put where we will send our search query and what data to send. We know that the url will start with "http://www.fictionwise.com" and according to the action attribute on the form we must add "/servlet/mwsearch" to this. To be able to pass our pairs of name=value data we will then use the same syntax as if we were doing an HTTP GET query of "?name=value1&name2=value2" etc.
11. So in this case the full url we will use is:
    

    http://www.fictionwise.com/servlet/mwsearch?searchstring={author}&searchtype=author&action=view&template=searchresults.htm

12. Right now we have enough to begin testing. Right-click on our new menu row in the configuration dialog, and choose "Test url" and verify the website is displayed as you would expect. If the website has no matches for that particular test data, use the "Test url using..." menu option instead.
13. As an experiment, try flipping the Method to GET instead of POST. Just because the website when you searched didn't use/display such a URL indicating a GET query, sometimes they will still go ahead and accept it. The advantage if it does is that will make your navigating to it slightly faster. In the case of fictionwise.com, we find the above url does actually work, so now we can keep the Method column set to GET.
14. Our final common step for the website is to specify an image to appear in the menu for this item. The most common source is what as known as a "favicon" and is the icon you see in your web browser next to the url when you go to the website or bookmark a page.
15. Right-click on our row in the configuration dialog and choose "Add image...". In this case fictionwise.com has a favicon so we will use the plugin to retrieve it. So in the dialog select "From web domain favicon" and in the box next to it type "www.fictionwise.com". In the "Save as filename" box, give it a unique name, like "fictionwise" and click ok.
16. In the Image column, select "fictionwise.png" in the dropdown that should now be present.
17. At this point we have a fully working menu item. Move it up and down the list as desired and specify any other options like whether to appear in a submenu, a keyboard shortcut etc.
18. Add additional variations as required for searching for a title, for a book (author and title), for ISBN etc.

The final step if desired is to make your urls available for other users on MobileRead.

1. Select your fictionwise menu items together in the plugin configuration dialog.
2. Right-click and choose Export. Type a name to save this as. This will save the configuration information and images into a zip file.
3. Attach the zip file to a forum post in this thread. Job done.

You can find the resulting zip file from this tutorial attached to this post. If you think all of the above sounds like lots of work, then you may appreciate how much effort I have spent pre-configuring over 100 URLs in this plugin already for you

Paypal Donations:

• If you find this or any of my other plugins useful please feel free to show your appreciation. I have spent many hundreds of unpaid hours in their development and support so any encouragement for me to continue is appreciated!
  

[carmenchu]

First, thanks again for the plug-ins!
Now, a small trouble/question: searching 'Manybooks for Author'
* I noticed that they seem to require {author_sort} (yes, I keep my authors unsorted)
* Tried to substitute {author} with {author_sort} in the plug-in, but the field-value was not picked up - i.e., the search was for '{author_sort}', literally.
Is there some work-around? I wish to add Project Gutemberg, and {author_sort} would be thing there as well, I think. Alternatively, I am going to try {author_spaced}.

[kiwidude]

Originally Posted by carmenchu

First, thanks again for the plug-ins!
Now, a small trouble/question: searching 'Manybooks for Author'
* I noticed that they seem to require {author_sort} (yes, I keep my authors unsorted)
* Tried to substitute {author} with {author_sort} in the plug-in, but the field-value was not picked up - i.e., the search was for '{author_sort}', literally.
Is there some work-around? I wish to add Project Gutemberg, and {author_sort} would be thing there as well, I think. Alternatively, I am going to try {author_spaced}.

The plugin does not currently use the template processor, so substituting other values than those listed in the help file for the plugin will not get you anywhere

For author, your only choices currently are {author} or {author_spaced}. They both pull from the same database field in Calibre, the only difference is the former will pass the author as "Joe+Smith" and the latter as "Joe Smith" (required for lovereading.co.uk).

Why would you not populate the author sort field in Calibre? It is populated automatically if your tweak is set correctly, it is one of those "set and forget" fields for me that I never need to subsequently edit?

[cheveguerra]

Good afternoon,

I just updated to 0.7.44 and then added this plugin, I didn't had it installed before, so I don't know if the error is just in 0.7.44, but the menu just does not appear, the plugin installed all right, I restarted Calibre, it DOES appear in "preferences/plugins/user interface action plugins", it is green and I can customize it, but the option does not appear in the menu when I right-click on a book!

I tried to go back to caliber 0.7.42, but I get a message that a newer version is installed and it just quits.

Any idea if there is something wrong with mi PC or is it possble that the new version of Caliber is he culprit.

Regards

Jose Alberto Guerra

[theducks]

Originally Posted by cheveguerra

Good afternoon,

I just updated to 0.7.44 and then added this plugin, I didn't had it installed before, so I don't know if the error is just in 0.7.44, but the menu just does not appear, the plugin installed all right, I restarted Calibre, it DOES appear in "preferences/plugins/user interface action plugins", it is green and I can customize it, but the option does not appear in the menu when I right-click on a book!

I tried to go back to caliber 0.7.42, but I get a message that a newer version is installed and it just quits.

Any idea if there is something wrong with mi PC or is it possble that the new version of Caliber is he culprit.

Regards

Jose Alberto Guerra

Did you do the "Customise Menu" step
and re-start?

[cheveguerra]

Yes!! , I don't think is necesary to customize it to work, but anyway I added some submenus, un-selected some websites, applied changes, re-started and nothing yet!

[DoctorOhh]

Originally Posted by cheveguerra

re-started and nothing yet!

You missed the step add to context menu. In this step the plugin is added to the right click menu.

Originally Posted by kiwidude

7. By default a GUI action plugin will not appear in any menus or toolbars. To add it to any of these you must go to Preferences->Customise the toolbar. In the dropdown at the top choose where you want the new "action", such as "The context menu for the books in the Calibre library". Select the new plugin within the "Available actions", click the add button and move up/down as required.

[cheveguerra]

... yes!! . . . I guess the instructions were TOO long for me to read them all!

Thanks!!

Everything is working right now!

[Maggie-Me]

Very very nice... Thanks

[Loosheesh]

Thanks for this plugin - I added it and everything is working fine. I would like to add a couple of other search sites (FictionDB and FictFact especially) and don't know how to generate the correct url that will launch a search for a particular title/author etc. Is there a way to get this for any website? Thanks in advance

[kiwidude]

Originally Posted by Loosheesh

Thanks for this plugin - I added it and everything is working fine. I would like to add a couple of other search sites (FictionDB and FictFact especially) and don't know how to generate the correct url that will launch a search for a particular title/author etc. Is there a way to get this for any website? Thanks in advance

FictFact is no problem. Download the zip attachment with this post, rename the .zip to .stip extension, and then use the "Import" right-click from the plugin configuration dialog.

FictionDB on the other hand is a problem. From my quick testing they don't seem to support querystring URLs, only those supplied via it's own webpages via HTTP POST. So for instance this query below contains all of the "right data" to do an author search for Lee Child, but being in the URL rather than hidden inside the request body (as is the case when you click the submit button on their website) is still not enough to get the desired result, as it will not be an HTTP POST submission when effectively typed into a browser location url field:
http://www.fictiondb.com/search/sear...htxt=lee+child

The closest you can get imho is to just create an entry in your menu of "FictionDB", put the url below and manually type in the author/title on the search page when you use it. Well unless someone else finds a way for this website
http://www.fictiondb.com/search/searchresults.htm

In terms of a general "how do you do it", my approach is to use the search box on the website, then look at the url displayed for the page it takes you to with a smaple search. For most websites that is sufficiently useful. You then substitute in the special token values like {author} where you see the things you searched for and you now have a reusable url you can add to your plugin configuration. Do the various combinations of author, title searches etc to see what differs and job done.

For some websites through they do not display a useful url to you - they hide that search data and display a URL based on data in their database rather than what you typed in (like a numeric id representing the book/author). In that scenario I go digging on the search page using "View source page". There it should list the various special sauce variable values you need and the url that clicking search would submit to (usually with a "POST" operation). Quite often you can then take that information and construct a url to do what you need, assigning the various variable names into a querystring and the website link will work. However there is the odd site like FictionDB which is setup in such a way as to block these types of requests. It could be for instance that they do so to prevent web page scraping. In that case I believe you are a bit scuppered unless others have suggestions, I haven't done web development myself for many years.

[kovidgoyal]

@kiwidude: You can create POST requests with the mechanize browser object.

[kiwidude]

Originally Posted by kovidgoyal

@kiwidude: You can create POST requests with the mechanize browser object.

Kovid to the rescue again - thanks for that... I think... (wanders off grumbling about neverending work to do)...

[Loosheesh]

Originally Posted by kiwidude

FictFact is no problem. Download the zip attachment with this post, rename the .zip to .stip extension, and then use the "Import" right-click from the plugin configuration dialog.

- did this and it works like a dream!

Originally Posted by kiwidude

FictionDB on the other hand is a problem. From my quick testing they don't seem to support querystring URLs, only those supplied via it's own webpages via HTTP POST. So for instance this query below contains all of the "right data" to do an author search for Lee Child, but being in the URL rather than hidden inside the request body (as is the case when you click the submit button on their website) is still not enough to get the desired result, as it will not be an HTTP POST submission when effectively typed into a browser location url field:
http://www.fictiondb.com/search/sear...htxt=lee+child
The closest you can get imho is to just create an entry in your menu of "FictionDB", put the url below and manually type in the author/title on the search page when you use it. Well unless someone else finds a way for this website
http://www.fictiondb.com/search/searchresults.htm

- I'd already tried a similar link but yours takes me to the advanced search, which is even better.

Thanks for all your help, especially that bit on how to set up the search urls - that will definitely be handy for later so I'm saving the instructions

[marvin_2]

Google can be your friend for finding books on FictionDB:

http://www.google.com/search?q=%22{author}%22+%22{title}%22+site%3Ahttp%3A%2F%2Fwww.fictiondb.com%2F&hl=en&num=10&lr=&ft=i&cr=&safe=images&tbs=