Hacking EBW-1150 to browse the Internet

[nrapallo]

Originally Posted by Ret

Reading those caps I can recall some things:
You see the device uses Allegro HTTP client.

Is there a reference manual for this client on the web. I found the Allegro FTP documentation, but not this HTTP one.

It sends HTTP GET requests to the server. If you use a local "Librarian" you can fetch any HTML file.

By local "Librarian" you mean the proxy server program (like Linreb or GEBLibrarian).

My proof of concept fetched a static html file containing one link to google.com. That prooved that the device could surf the net.

I've served up static html files in the 'content' directory that the reader accesses when displaying 'bookshelf.ebooksystem.net/content/' requests, but no surfing was allowed; even tried the google home page.

I also noticed that the reader's 'bookshelf page' was very weak in it's support of html pages so I would probably rule that out to serve up static html to its 'bookshelf' requests.

If you want to generate and fetch your own list of books you'll have to fetch an HTTP response containing the string "Content-Type: text/x-booklist" and then the list of books in this format:

3:B:ebook<TAB>BOOKNAME<TAB>AUTHOR<TAB>CATEGORY<TAB>SIZE<TAB>URL_TO_BOOK<TAB>1<TAB>17
Taken from:
3:B:eBook	eBook User's Guide Version 2.2	Thomson Consumer Electronics	General Interest	452630	http://bookshelf.softbook.net/bookshelf/default.asp?BOOK_ID=eBook&SOURCE_ID=3&SOURCE_TYPE=B	1	17

If the device asks for a book the server should respond with an HTTP response containg "Content-Type: text/x-book" (which I infer, since it's not on those caps)

Ya, I omitted those requests from the capture file as they were binary transfers in a text file. I do see the 'content-type: x-softbook' being responded in the Linreb program code. That C code also shows the string replied to the x-booklist request as you have indicated above. We're on the same page...

Does this help you? Hope it does Take a closer look at your caps. I think all you need to know is almost all there
Good luck

I think it narrows down where to patch into but I still can't reproduce that 'surfing' ability. Here's to more explorations...(or someone having a copy of your java code)

[Ret]

Originally Posted by nrapallo

Is there a reference manual for this client on the web. I found the Allegro FTP documentation, but not this HTTP one.By local "Librarian" you mean the proxy server program (like Linreb or GEBLibrarian).I've served up static html files in the 'content' directory that the reader accesses when displaying 'bookshelf.ebooksystem.net/content/' requests, but no surfing was allowed; even tried the google home page.

I don't know anything about Allegro HTTP client.
Yes, for Librarian I meant the proxy server + local HTTP server.

If you already served static HTML files to the reader and it rendered it ok, then you've done all I did before. If a link in that file pointing to google doesn't work from your reader, I'd say it has something to do with your proxy settings or Linux box (I've worked on XP) or some kind of network related issue (DNS, URL redirection, etc). Try using direct links with full IP addresses (that don't require DNS requests).

[nrapallo]

Originally Posted by Ret

I don't know anything about Allegro HTTP client.
Yes, for Librarian I meant the proxy server + local HTTP server.

If you already served static HTML files to the reader and it rendered it ok, then you've done all I did before. If a link in that file pointing to google doesn't work from your reader, I'd say it has something to do with your proxy settings or Linux box (I've worked on XP) or some kind of network related issue (DNS, URL redirection, etc). Try using direct links with full IP addresses (that don't require DNS requests).

I'm using a WinXP computer to communicate with the reader. I'm behind a router and working in local IP space (198.168.X.X). The Linreb program seems to work well and v0.2 has a windows executable that you can run in WinXP (check out v0.2).

I think there could be one or more of the following working against me getting this to work:
- ETI may have 'fixed/blocked' that ability to surf the net from 2005 to now with recent firmware updates
- your java code actually 'processed' the requests by manipulating the info sent whereas the Linreb program merely responds with the requested data. No .asp code is being executed by Linreb when the HTTP requests come in from the reader.
- the ability to 'surf the net' is there and I'm too 'blind' to see it.

I will continue experimenting with the Linreb code...

[Ret]

From the moment you can use Linreb with your reader, the ability to surf the net is there. You have to remember that what Linreb does is redirect all traffic to ebooktechnologies servers to the local server listening on localhost. In tech terms: it puts an entry in your HOSTS file on your windows machine that translates the ebooktechonologies.com to 127.0.0.1 and runs a web server listening on localhost.

My java code did nothing magic or on the fly, just serve a static html file. You say you can do the same, so you have pretty much everything done. Am I correct? The reader shows your custom static html files, rigth?

[nrapallo]

Originally Posted by nrapallo

I'm using a WinXP computer to communicate with the reader. I'm behind a router and working in local IP space (198.168.X.X). The Linreb program seems to work well and v0.2 has a windows executable that you can run in WinXP (check out v0.2).

I think there could be one or more of the following working against me getting this to work:
- ETI may have 'fixed/blocked' that ability to surf the net from 2005 to now with recent firmware updates
- your java code actually 'processed' the requests by manipulating the info sent whereas the Linreb program merely responds with the requested data. No .asp code is being executed by Linreb when the HTTP requests come in from the reader.
- the ability to 'surf the net' is there and I'm too 'blind' to see it.
I will continue experimenting with the Linreb code...

Originally Posted by Ret

From the moment you can use Linreb with your reader, the ability to surf the net is there. You have to remember that what Linreb does is redirect all traffic to ebooktechnologies servers to the local server listening on localhost. In tech terms: it puts an entry in your HOSTS file on your windows machine that translates the ebooktechonologies.com to 127.0.0.1 and runs a web server listening on localhost.

My java code did nothing magic or on the fly, just serve a static html file. You say you can do the same, so you have pretty much everything done. Am I correct? The reader shows your custom static html files, rigth?

Yes, basically a combination of the last two points (in bold) in my post above is taking place.

The Linreb program doesn't do anything but "serve up" static html pages in the 'content' directory. No external html pages are fetched from the internet and placed there. No HOSTS file is 'changed', but all HTTP requests from the reader are RESPONDED to by the proxy program, not the ETI servers. Again, there is no 'processing' done by the Linreb program other than 'give up' what it sees in the local directories.

Have you tried running the program? It does work with the EBW1150 conntected via the usb cable and with the HTTP Proxy server set up according to the GEBLibrarian-way. Presumably you know this already i.e. 192.168.0.X where X is your machine's IP last octal.

I think the Linreb program doesn't use the 127.0.0.1 method directly, but instead relies on the EBW1150 reader to have the HTTP proxy set in Settings as well as use the usb for proxy HTTP. Maybe this is the difference in making it work and not?

[Ret]

OK, I've downloaded and run LinReb (nice software)
I think I now know what your problem is: It's Linreb's lack of proxy feature.
This happens because the reader is set to use an HTTP proxy (which should run in a local IP). In this case, ALL HTTP requests are handled by Linreb which runs in that specific IPort. So, if the requests are made to those known ETI servers (hardcoded in the readers), Linreb responds. If they are made to different domains (e.g. google.com) Linreb doesn't do anything because it's coded in that way; and because it's also not coded to act as a proxy, it doesn't even forward those requests to the required server onn the internet.
There are two "workarounds for this"
1- Use an HTTP proxy in the reader that points to a separate proxy server that handles web requests AND forwards the ones to ETI server to LINREB.
2- Don't set use proxy settings on the reader, run Linreb in port 80, and change the HOSTS files to forward ETI domain to Linreb in local IP. This is the approach that I coded before (By the way... I've found one old version of it).
Does this help you clear all out?
Good luck!

[nrapallo]

Originally Posted by Ret

OK, I've downloaded and run LinReb (nice software)
I think I now know what your problem is: It's Linreb's lack of proxy feature.
This happens because the reader is set to use an HTTP proxy (which should run in a local IP). In this case, ALL HTTP requests are handled by Linreb which runs in that specific IPort. So, if the requests are made to those known ETI servers (hardcoded in the readers), Linreb responds. If they are made to different domains (e.g. google.com) Linreb doesn't do anything because it's coded in that way; and because it's also not coded to act as a proxy, it doesn't even forward those requests to the required server onn the internet.
There are two "workarounds for this"
1- Use an HTTP proxy in the reader that points to a separate proxy server that handles web requests AND forwards the ones to ETI server to LINREB.
2- Don't set use proxy settings on the reader, run Linreb in port 80, and change the HOSTS files to forward ETI domain to Linreb in local IP. This is the approach that I coded before (By the way... I've found one old version of it).
Does this help you clear all out?
Good luck!

I think you are right on the mark! Any chance of getting your java code (early version) to acheive that surfing ability again? Please post your results here if you get it working. I could never see where in Linreb I could "get" those webpages but I thought the reader was doing the surfing for me. I guess not!

After I got that 'surf the internet' feature "activated", I then wanted to modify Linreb to "get" .imp ebooks from our uploaded ebooks forum here directly ala the Mobipocket Download Guide.

You have given me a great headstart with your findings and hopefully this can evolve into something useful. I am going to try this new (and hopefully correct) way as soon as possible.

Thanks again; you have been most helpful!

[Ret]

I think you don't really need my code but if you want it you can get it from here
That program runs a webserver listening on port 80. When it is run, it loads the HOSTS file in memory and adds some entries. It listens for requests and serves the files (sample.html should be named index.html). When you close it, it puts the HOSTS files back to original state.

That's not the approach used by Linreb since it doesn't run on port 80 (which needs some privileges).
What you really need is to add some code in Linreb to let it handle requests for domains other than ebooksystem.net and forward the responses back to the reader. In other words, always act as a proxy except when trying to connect to ETI servers.

If you don't want to add code to Linreb (maybe for maintenance reasons) you should run a separate proxy that directs all traffic to ebooksystem.net to Linreb's port. (I tried to do that but I could find a way to do a port redirection.)

One last thing: I think It'll be a lot of work parsing all the Uploaded book forum section and presenting them to the reader.

[Nate the great]

Originally Posted by Ret

One last thing: I think It'll be a lot of work parsing all the Uploaded book forum section and presenting them to the reader.

That's something that can be automated. We already have a download guide in Mobipocket that is generated automatically.

But it's not worth Alex's time to make one for the Ebw-1150 until we know that it will work. Also, he will need two things. What is the largest file size that the 1150 can handle comfortably? Is there a Linux command line tool for generating IMP ebooks? (got a link?)

[nrapallo]

Originally Posted by Ret

I think you don't really need my code but if you want it you can get it from here

Are you kidding; I think I have found the Holy Grail with that java code. Very interesting; perhaps I can merge some segments into Linreb. But since I didn't write Linreb (I only found it), my progress will be all uphill. That's OK though, I like a challenge.

BTW, quick (stupid) question, did you get it to work on the EBW1150 as you remembered it?

That program runs a webserver listening on port 80. When it is run, it loads the HOSTS file in memory and adds some entries. It listens for requests and serves the files (sample.html should be named index.html). When you close it, it puts the HOSTS files back to original state.

I tried this already with the EBW1150 (as you previously suggested earlier), but none of the HTTP GETs were processed properly. They were all internet requests (without the proxy 'Http://' prefix). I need to make some adjustments to Linreb to properly interface and incorporate your idea. That's where your java code comes in handy. At least now I'll know what your idea is, IN CODE, so that I can try to duplicate/strong-arm it to work for Linreb.

That's not the approach used by Linreb since it doesn't run on port 80 (which needs some privileges).
What you really need is to add some code in Linreb to let it handle requests for domains other than ebooksystem.net and forward the responses back to the reader. In other words, always act as a proxy except when trying to connect to ETI servers.

I will if I find the right code. So far, I've found some web references (in perl) and may try some ideas from there. Here's to hoping it gets me somewhere.

If you don't want to add code to Linreb (maybe for maintenance reasons) you should run a separate proxy that directs all traffic to ebooksystem.net to Linreb's port. (I tried to do that but I could find a way to do a port redirection.)

port redirection?; like port forwarding on a router?

One last thing: I think It'll be a lot of work parsing all the Uploaded book forum section and presenting them to the reader.

If you take a look at the first link on the second page of the content directory in Linreb v0.2, you will see (all) the Mobipocket ebooks available for download from Mobileread.com along with their descriptions. That info was extracted from the Mobipocket Download Guide available here daily. You are correct that that would be a lot for me to parse for .imp ebooks, but not for the moderators or main programmers here. Once the proof-of-concept is acheived, then I'm sure I could get their co-operation preparing the 'IMP Download Guide' the way Linreb needs it. Now, will that listing exhaust the limited memory on-board the EBW1150? Perhaps, but then we will have to work with small lists/subsets; an extra pre-processing step would be required.

Time will tell how successful this becomes but for me it is enough to be a catalyst to make this happen. You have been a great part so far as was the original author of Linreb. Besides I like tweaking/hacking (and maintaining) programs written by others; that's what I do!

In the spirit of sharing, I do want to see this come to fruition.

Anybody else want to help out? Please? Ok, pretty please!

Speak now or forever (you know the drill) .....

[nrapallo]

Originally Posted by Nate the great

That's something that can be automated. We already have a download guide in Mobipocket that is generated automatically.

But it's not worth Alex's time to make one for the Ebw-1150 until we know that it will work.

Nate:

I mentioned this in my previous post, without seeing your repsonse here. Good to see you are 'already in the know'. We know we need this 'proof-of-concept' first, and we are working on it.

Also, he will need two things. What is the largest file size that the 1150 can handle comfortably? Is there a Linux command line tool for generating IMP ebooks? (got a link?)

In the meantime, I'll get you started!

The EBW1150 maxes out its internal memory when many links/styles/images/etc. exist in the ebook. Size and complexity seem to go hand-in-hand. I'll guess that that size limit is under 10 MB, but over 5MB. The REB1200 can surpass that by a lot. It may max out at 30MB and can surely handle most ebooks over 10MB. It has more available memory onboard and its all available for data.

Why do you need to generate those .imp in linux? They already have been uploaded to the IMP ebooks upload forum (and actually are already in that 'mobi' directory used by the Mobipocket Download Guide ).

If you want to generate .imp ebooks 'on-the-fly' from .html files, then my perl scripts for manipulating .imp ebooks will do the trick in linux. It's a sticky in the Fictionwise forum.

Whatever you need most definitely is available here already; one way or another I've made sure of that!. Ask me for anything you need to do.

My original idea was to use an 'IMP Download Guide' that Alex could format in html for our use and then 'wget' or 'get in perl using LWP' the requested file using Linreb. For a great reference, see Web Basics with LWP - Sample Recipes for Common Tasks by Sean M. Burke - author of Perl & LWP (August 20, 2002).

This should be easy enough to set up, but requires Alex's help formatting the listing and the EBW1150 in sending out those HTTP requests to the proxy server. We seem to be close, but are not there yet!

Stay tuned...

[Nate the great]

I was under the impression that the download guide itself would need to be in IMP to be on the 1150.

The best way to make the guide is to automate the process. You click on a link, a script runs and you are offered a file to download. This is how the Mobipocket Download Guide is set up. And in order to execute the script we would first need the command line tool.

BTW, the Mobi Download Guide is still under 2MB, so the new guide has no worries about size problems.

[nrapallo]

Originally Posted by Nate the great

I was under the impression that the download guide itself would need to be in IMP to be on the 1150.

The EBW1150 can receive tabbed info in the form of a booklist without getting an actual .imp ebook. The EBW1150 then can choose one of the listed ebooks and have it transferred in. Do you know what I mean?

The best way to make the guide is to automate the process. You click on a link, a script runs and you are offered a file to download. This is how the Mobipocket Download Guide is set up. And in order to execute the script we would first need the command line tool.

BTW, the Mobi Download Guide is still under 2MB, so the new guide has no worries about size problems.

I think it will have to be handled a bit differently, since the EBW1150 cannot connect to the internet without the Linreb (proxy server) program running in the background. The EBW1150 will have to go through the Linreb program to get the .imp.

I was thinking that Alex could prepare a html listing that could be displayed on the EBW1150 with Linreb and then have the Linreb program intercept the requested .imp ebook, download it "on the fly" and present it back to the EBW1150 by "serving up" the .imp in the local directory. This would be a two-step process. GEBLibrarian does something similar when it allows Categories to be changed by the EBW1150.

If the .imp ebook is the 'IMP Download Guide', then it would be Huge and require the hardware reader to "do the surfing". That's just not possible (I think) and therefore differs slightly to your 'Mobipocket Download Guide' implementation.

Do you see it working better differently? Any insights would be appreciated. Call it brainstorming across reader platforms! Mind you, you almost cover them all, by yourself...

[Nate the great]

I think you should go try the Mobi Download Guide. Then you'll understand why I want to do something similar with the 1150 (if it is technically possible).

I did understand that Linreb was required. It's just that I just got By Schism Rent Asunder so you're not getting my full attention.

[nrapallo]

Originally Posted by Nate the great

I think you should go try the Mobi Download Guide. Then you'll understand why I want to do something similar with the 1150 (if it is technically possible).

I did understand that Linreb was required. It's just that I just got By Schism Rent Asunder so you're not getting my full attention.

I did try using the PC Mobipocket Reader to "fetch" an ebook from the Mobipocket Download Guide a while ago, but it timed out half-way through and failed. Even if I don't have a Mobipocket hardware reader, I understand how it works.

I think instead I would use the html file that was used to create the Mobipocket Download Guide. So BEFORE converting it to .prc (or .imp) Linreb would get from here that html file (with descriptions and links to all .imp ebooks) and send the html file to be displayed on the EBW1150. Then the user would navigate the pages on the reader and when they saw an ebook of interest, they would just click the link in that html file.

That would request the .imp file from the internet but as the ETI servers don't accomodate such requests, the Linreb program would do so; but in a sneaky way. It would intercept that requested .imp file and then go out and get it and place it in the 'shelf' directory on the local machine. There would then be a message displayed on the EBW1150 that should ask the user to request a new Bookshelf (by pressing the Online Bookshelf icon). The Linreb program would then look at the 'shelf' directory and see that newly downloaded ebook and send the updated booklist to the EBW1150. Once the user sees that requested .imp in the Online Bookshelf, they would click it again and have it transferred in. And repeat!

A lot of these terms are specific to the EBW1150 way of transferring ebooks and may not yet be intuitive for you. It's Ok if you're not 100% involved in this project, since we all get distracted with other issues every so often...

Thanks for helping me 'crystalize' what is needed here to make this work! You're a great sound board to bounce (written) thoughts off of!