impserve: serving content and accessing the internet for the 1200/1150 - Page 2

ashkulz · 08-29-2008, 11:31 AM

Quote:

Originally Posted by nrapallo

I have tried the same ebook reader actions under two 'surf the internet' solutions: one being impserve; the other being 'Linreb+privoxy'. I only used the REB1200, so far, haven't tried the EBW1150 yet.

I attach the log results from both and you can see that the transfer times are much slower in impserve. Now to find the cause of it and if it is system related...

BTW, Ashish, notice the last log entry; it's a delete request from the online bookshelf that could be implemented in impserve. Linreb also ignores that delete request. Food for thought...

I know the problem, it's that when the log is printed the it does a reverse lookup on the IP address of your ebook, which is what causes the latency. I fixed it by adding an entry to the hosts file yesterday, but forgot to add the fix: I'm attaching 0.2.1 for now. I'll be working on a 0.3 release for tomorrow, so please let the feature requests come in -- I'd really appreciate a comprehensive list so I know what to shoot for

nrapallo · 08-29-2008, 12:18 PM

Quote:

Originally Posted by ashkulz

please let the feature requests come in -- I'd really appreciate a comprehensive list so I know what to shoot for

OK, first on my list would be, as stated before:

Cache "Content-type: text/html" transfers and before you send them onto the ebook reader, in response to the HTTP GET, text substitute, at least:

Code:

- <a href>  WITH <a UNDERLINE="YES" href> (so that you can SEE hyperlinks)
- <body> WITH <body style="margin-left:2%; margin-right:2%"> (so there's no edge-bleeding - can use 2px instead of 2% or even 0)

In addition, to simulate a back function, remember the last html "visited" and when the directory icon is tapped ( HTTP requests to directory.ebooksystem.net, I think ) then go there. It would be nice to remember ALL sites visited so that we could go back to the start page, if we liked!

However, I see where this could pose a problem; if we want a quick restart. I would recommend that a 'bookshelf' request then clear this cached list of websites visited and restore the directory icon's function. What do you think?

That's it for now...

Did I mention whispernet?

Ret · 08-29-2008, 04:54 PM

Don't forget to:
- Handle internet links within ebooks. Impserve should be able of stripping the redirection URL that ebooksystem.net uses.
- Change content-type header to application/x-softbook when downloading an IMP file from a server that doesn't know that MIME type.
- Let the reader delete ebooks from the online bookshelf.

that's all that comes to mind.... THANKS!

nrapallo · 08-29-2008, 09:34 PM

Quote:

Originally Posted by ashkulz

I know the problem, it's that when the log is printed the it does a reverse lookup on the IP address of your ebook, which is what causes the latency. I fixed it by adding an entry to the hosts file yesterday, but forgot to add the fix: I'm attaching 0.2.1 for now. I'll be working on a 0.3 release for tomorrow, so please let the feature requests come in -- I'd really appreciate a comprehensive list so I know what to shoot for

WOW, that returned and maybe improved my surfing speed. No more slow downs with WinXP and impserve!!!

Truly amazing!

I can't wait to see your version 0.3...

(wipes chin now)

p.s. Ashish, I attach a privoxy log showing three ways to get a .imp ebook from Mobileread.com; each with a different Content-type! Please see attached text log.

This may help determine the best way to download .imp ebooks directly from Mobileread!

ashkulz · 08-30-2008, 05:31 AM

Quote:

Originally Posted by nrapallo

WOW, that returned and maybe improved my surfing speed. No more slow downs with WinXP and impserve!!!

Truly amazing!

I can't wait to see your version 0.3...

(wipes chin now)

p.s. Ashish, I attach a privoxy log showing three ways to get a .imp ebook from Mobileread.com; each with a different Content-type! Please see attached text log.

This may help determine the best way to download .imp ebooks directly from Mobileread!

I've already figured out #2 and #3: look at the "Content-disposition" header, and if it provides a filename call mimetypes.guess_type on the file which will give the correct MIME type. I'm more hesitant of doing it for URLs where they don't specify Content-disposition and the URL ends with .imp and the content-type is not correct: it may be a valid URL. You can choose to override it in the plugins system I plan to develop.

nrapallo · 08-30-2008, 07:33 AM

Quote:

Originally Posted by ashkulz

I've already figured out #2 and #3: look at the "Content-disposition" header, and if it provides a filename call mimetypes.guess_type on the file which will give the correct MIME type. I'm more hesitant of doing it for URLs where they don't specify Content-disposition and the URL ends with .imp and the content-type is not correct: it may be a valid URL. You can choose to override it in the plugins system I plan to develop.

I think Example #3, in my privoxy log file above, would be the most common, as it will come about when surfing the IMP ebook upload forum threads and selecting the .imp attachment therein (doesn't end with .imp).

Example #2 is what would be expected if the "IMP Download Guide" could be prepared by Alexander in .imp format and/or .html format similar to the current 'Mobipocket Download Guide'. Again doesn't end with .imp.

Example #1 (the one you haven't nailed down yet) can rarely be found, so rare that I had trouble finding a single example to surf to, until I edited a message here and surfed to that link. So do we need to handle this situation? Maybe not!

Ashish, could you check the booklist creation code. Whenever there are more than 100 books IN TOTAL being sent (i.e. 101+ ), the booklist doesn't get processed. For 100 or below books it is fine!

My logs between the REB1200 and GEBLibrarian showed that the booklist should be sent in chunks of 100 books (each with its respective booklist length) at a time and then the INDEX=1 URL encoded tag will properly be bumped up to INDEX=101 for the next batch of 100 books and so on. Right now the INDEX gets bumped up by a very large amount and hence causes that aborted booklist attempt. This is an excerpt of my GEBLibrarain log that I was referring to:

Code:

GET http://bookshelf.ebooksystem.net/bookshelf/default.asp?INDEX=1&REQUEST=100&SHOW_HIDDEN=YES HTTP/1.0
User-Agent: Allegro HTTP Client
Host: bookshelf.ebooksystem.net
Accept-Encoding: deflate, x-des-encrypted

...

GET http://bookshelf.ebooksystem.net/bookshelf/default.asp?INDEX=101&REQUEST=100&SHOW_HIDDEN=YES HTTP/1.0
User-Agent: Allegro HTTP Client
Host: bookshelf.ebooksystem.net
Accept: text/x-booklist
Accept-Encoding: deflate, x-des-encrypted

ashkulz · 08-30-2008, 08:57 PM

NOTE: impserve has now been renamed to ebookutils, and is hosted at http://gitorious.org/projects/ebookutils using Git, which should easily allow other developers to make changes. I plan to merge impserve, impmake, 2 new python utilities which replace ebw1150+rebcomm and pdfread under this umbrella.

Changes in ebookutils-0.3:
- impserve: handle paging of booklist requests
- impserve: handle downloading of IMP books which are sent as an attachment (direct download of IMP from mobileread works)
- impserve: implement a plugin system and provide 2 default plugins to change the URL/content before being sent to the device.
- impmake: add an -f option to specify the files to be included, with one file per line
- impserve: refactored the code heavily, please run impserve/run.py from python.

ashkulz · 08-30-2008, 09:09 PM

Ok, the only functionality I can think of which remains is:

[Ret] Handle internet links within ebooks (do the redirection which ebooksystem.net)
[nrapallo] add back button functionality
=> for this I plan to add a plugin point where you can hook into all requests coming for ebooksystem.net and refactor the code to use it. Plugins can then be written with that functionality in mind, without touching the core code.

[Ret] Let the reader delete ebooks from the online bookshelf.
why do we need that, really? just remove it from your filesystem, and next time you visit the booklist it's gone (it's autorefreshed on every request).

Any other feature requests?

ashkulz · 08-31-2008, 01:54 PM

I've released ebookutils-0.4 having changes only in impserve. I consider it feature-complete for now, so pending any more feature requests I don't think it's going to change much.

Changes in ebookutils-0.4:
- impserve: redirect to the content page if bookstore/directory requested (remove the static page)
- impserve: allow books to be deleted from online bookshelf
- impserve: add support for back button with directory/store icon (going to online bookshelf erases history)
- impserve: make the proxy server single-threaded
- impserve: add support for browsing from links in ebooks, but it looks
broken like it triggers a bug in the firmware and gives a 400 error. Please try it at your end and let me know.

shighfield · 08-31-2008, 08:10 PM

This is going to replace the wBookwise Libraian I think.

nekokami · 09-01-2008, 01:13 AM

I haven't tried it yet (my daughter has my ebw1150), but does it (or could it) pipe non-DRM versions of other format books through a converter to IMP so those could be downloaded through this system as well? I'm thinking of Gutenberg books, and possibly even CBZ.

ashkulz · 09-01-2008, 02:49 AM

Quote:

Originally Posted by nekokami

I haven't tried it yet (my daughter has my ebw1150), but does it (or could it) pipe non-DRM versions of other format books through a converter to IMP so those could be downloaded through this system as well? I'm thinking of Gutenberg books, and possibly even CBZ.

Yes, that can be done. All that would need to be done is write a plugin which intercepts the URL or format, do the conversion (via calling impmake on the command line) and return the converted book. That's one of the main reasons I built the plugin system

However, eBook Publisher is still very particular about the HTML it gets and crashes on invalid HTML (which is out there a lot) so that'd have to be run through HTML Tidy + doing misc cleanup on it.

Let me know what kind of conversion you envisage, and I can (attempt) to write the plugin or you can do it yourself, if you want ...

ashkulz · 09-01-2008, 02:57 AM

Quote:

Originally Posted by shighfield

This is going to replace the wBookwise Libraian I think.

Thanks, shighfield

Looks like no one else is downloading/using it for now as the earlier versions work equally well..

Ret · 09-01-2008, 10:11 AM

Thank you again ashkulz! Great effort! In such a short time you've made a great tool!

Quote:

Originally Posted by ashkulz

[Ret] Let the reader delete ebooks from the online bookshelf.
why do we need that, really? just remove it from your filesystem, and next time you visit the booklist it's gone (it's autorefreshed on every request).

I see you included this in version 0.4 but let me describe different scenarios where it can be useful: Even though I don't use that feature myself, sometimes you want to just delete a book while browsing your bookshelf, it's easier. Besides sometimes filenames are "cryptic", not including author or bookname, while on the reader all that information is displayed so it's easier to find the book.

Cheers

ashkulz · 09-01-2008, 10:46 AM

BTW, can someone verify that browsing from ebooks works? the data is served from impserve, but it apparently triggers a bug in the ebook and it asks to reset itself ... can you verify that the same happens with you?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Confused about personal content re: eBookwise 1150	locomouse	Fictionwise eBookwise	6	11-25-2009 01:52 PM
PDFRead - reading PDFs on the 1100/1150/1200 eBook Readers	ashkulz	IMP	87	06-04-2009 03:43 PM
How do you get new content onto your eBookwise 1150 / REB 1200? (pick all that apply)	nrapallo	Fictionwise eBookwise	23	02-03-2009 05:34 PM
Hacking EBW-1150 to browse the Internet	sputnik	Fictionwise eBookwise	44	08-27-2008 09:40 AM
Comparing 1150 to 1200 or 2150	Katelyn	Fictionwise eBookwise	1	04-29-2007 12:38 PM

08-29-2008, 04:54 PM	#18
Ret Connoisseur Posts: 81 Karma: 480 Join Date: Dec 2004	Don't forget to: - Handle internet links within ebooks. Impserve should be able of stripping the redirection URL that ebooksystem.net uses. - Change content-type header to application/x-softbook when downloading an IMP file from a server that doesn't know that MIME type. - Let the reader delete ebooks from the online bookshelf. that's all that comes to mind.... THANKS!

08-30-2008, 08:57 PM	#22
ashkulz Addict Posts: 350 Karma: 705 Join Date: Dec 2006 Location: Mumbai, India Device: Kindle 1/REB 1200	NOTE: impserve has now been renamed to ebookutils, and is hosted at http://gitorious.org/projects/ebookutils using Git, which should easily allow other developers to make changes. I plan to merge impserve, impmake, 2 new python utilities which replace ebw1150+rebcomm and pdfread under this umbrella. Changes in ebookutils-0.3: - impserve: handle paging of booklist requests - impserve: handle downloading of IMP books which are sent as an attachment (direct download of IMP from mobileread works) - impserve: implement a plugin system and provide 2 default plugins to change the URL/content before being sent to the device. - impmake: add an -f option to specify the files to be included, with one file per line - impserve: refactored the code heavily, please run impserve/run.py from python.

08-30-2008, 09:09 PM	#23
ashkulz Addict Posts: 350 Karma: 705 Join Date: Dec 2006 Location: Mumbai, India Device: Kindle 1/REB 1200	Ok, the only functionality I can think of which remains is: [Ret] Handle internet links within ebooks (do the redirection which ebooksystem.net) [nrapallo] add back button functionality => for this I plan to add a plugin point where you can hook into all requests coming for ebooksystem.net and refactor the code to use it. Plugins can then be written with that functionality in mind, without touching the core code. [Ret] Let the reader delete ebooks from the online bookshelf. why do we need that, really? just remove it from your filesystem, and next time you visit the booklist it's gone (it's autorefreshed on every request). Any other feature requests?

08-31-2008, 01:54 PM	#24
ashkulz Addict Posts: 350 Karma: 705 Join Date: Dec 2006 Location: Mumbai, India Device: Kindle 1/REB 1200	I've released ebookutils-0.4 having changes only in impserve. I consider it feature-complete for now, so pending any more feature requests I don't think it's going to change much. Changes in ebookutils-0.4: - impserve: redirect to the content page if bookstore/directory requested (remove the static page) - impserve: allow books to be deleted from online bookshelf - impserve: add support for back button with directory/store icon (going to online bookshelf erases history) - impserve: make the proxy server single-threaded - impserve: add support for browsing from links in ebooks, but it looks broken like it triggers a bug in the firmware and gives a 400 error. Please try it at your end and let me know.

08-31-2008, 08:10 PM	#25
shighfield Groupie!? GROUPIE!?!? Posts: 319 Karma: 649 Join Date: Aug 2008 Location: The Dirty Shwa Device: EBW 1150, Libra Color	This is going to replace the wBookwise Libraian I think.

09-01-2008, 01:13 AM	#26
nekokami fruminous edugeek Posts: 6,745 Karma: 551260 Join Date: Oct 2006 Location: Northeast US Device: iPad, eBw 1150	I haven't tried it yet (my daughter has my ebw1150), but does it (or could it) pipe non-DRM versions of other format books through a converter to IMP so those could be downloaded through this system as well? I'm thinking of Gutenberg books, and possibly even CBZ.

09-01-2008, 10:46 AM	#30
ashkulz Addict Posts: 350 Karma: 705 Join Date: Dec 2006 Location: Mumbai, India Device: Kindle 1/REB 1200	BTW, can someone verify that browsing from ebooks works? the data is served from impserve, but it apparently triggers a bug in the ebook and it asks to reset itself ... can you verify that the same happens with you?

Advert

Advert