Feature Request: ipod touch/iphone kindle.app support

ldolse · 06-02-2009, 12:13 AM

Note I didn't title this thread quite right, it should read:
Feature Request: Jailbroken ipod touch/iphone kindle.app support

Hi,

Ever since Amazon allowed iphones/ipod touch users to purchase ebooks I've been using them as my primary source of purchased ebooks. While I really dislike the draconian policies they have with drm and delivery of paid content, I've used them in general for years and their prices are good.

The real problem is that I read these on my Sony, which means I've got to jump through quite a few hoops to get the content there:

Jailbreak the ipod/iphone (one time event)
Acquire software which allows me to browse the device filessytem (one time event)
Launch the above software
find the buried location of kindle.app and it's ebook library
Copy the code named prc files to my local disk
Go through each file one by one until I figure out which one is the latest book I've purchased
Get the serial number of the ipod touch and use it to get the mobipocket ID (one time event)
Strip the DRM
Load the book into Calibre
Send it to the sony

Needless to say that's a bit complicated, and I'd love it if Calibre could uncomplicate it for me along the lines of something like this:

Plug the jailbroken ipod/iphone in, Calibre recognizes it as a device
Calibre allows me to browse whatever books are on the device and transfers them to my library

At that point they would still be DRM'd, but the fact that I could stick them into the library from the device intuitively would make all the difference in the world. Stripping the DRM and replacing the file in the calibre library is relatively trivial.

Of course it would be major bonus points if Calibre could strip the DRM in the process - but I understand the reasons behind not supporting it in the official release or explicitly on these forums. One could always link a plugin modification over on darkreverser's blog or some other location not affiliated with mobileread.

I'm thinking this FR should apply to quite a few people but not entirely certain, so looking for comments.

I also tried looking into downloading the files directly from amazon's site by capturing the ipod's traffic on my network, but amazon uses HTTPS with client certificate auth to download the books. I looked all through the kindle.app directory, but I couldn't find any keypair being used as a client cert, I can only assume that Amazon is embedding the keypair in the actual binary. If I could get the client cert I could actually decrypt the SSL traffic pretty easily with the tools I have to determine what the app is doing.

kovidgoyal · 06-02-2009, 12:47 AM

I don't have an iPhone, so I for one can't do this, but patches are welcome

Incidentally calibre's plugin frameworks makes removing MOBI DRM automatic

Seabound · 06-02-2009, 01:00 AM

Quote:

Originally Posted by kovidgoyal

Incidentally calibre's plugin frameworks makes removing MOBI DRM automatic

Wow, I've been missing some news. When did that automated Mobi de-DRM feature happen?

BTW, I noticed the Ph.D.--congratulations Dr. Goyal!

kovidgoyal · 06-02-2009, 01:13 AM

Quote:

Originally Posted by Seabound

Wow, I've been missing some news. When did that automated Mobi de-DRM feature happen?

BTW, I noticed the Ph.D.--congratulations Dr. Goyal!

Thanks

MOBI DRM removal is automatic in calibre if you can find a DRM removing plugin for calibre

ldolse · 06-02-2009, 04:15 AM

Well, I looked through the files a bit more in the app, I think I can probably take a stab at a start, but not sure how far I'll get. The full books are all listed in the attached xml file. Downloaded samples don't appear in the xml or any other support files for some reason, possibly because they don't have DRM. The filename of the book is the ASIN id plus an extra static extension. So It should be relatively straightforward to parse the xml to show the current books on the iphone.

The big issue is that the iphone doesn't mount as a proper usb disk/filesystem. There are a few filebrowsers that let you access the system using some special tricks, but I don't see a cross-platform library doing this.

I think for now I'm going to try out an approach where I share the filesystem over wifi using Netatalk. It allows you to create an AFP share from the iphone.

If it helps I could post the whole kindle.app directory sans paid books.

ldolse · 06-02-2009, 07:26 AM

Some more info:

After installing NetaTalk the ipod touch is automatically discovered on the network as an AFP share, with two volumes, one called root, one called home. Once root is mounted the path on OS X is the following:

Code:

/Volumes/root/var/mobile/Applications/939E5CA6-B556-4C0E-800C-1A7#A2D

I'm assuming that 939E5CA6-B556-4C0E-800C-1A7#A2D is the universal folder name for the kindle app across all iphones/ipod touches, but I would need some other users who have jailbroken their iphones to corroborate that. If it's not each subfolder in /Applications/ has a name that looks like that, the subfolders would need to be searched for one named Kindle.app to confirm the correct folder for that device.

The xml file I attached previously is located within the app folder at:

Code:

Relative Path from program directory:
/Documents/cache/KindleSyncMetadataCache.xml

Full Path (OS X):
/Volumes/root/var/mobile/Applications/939E5CA6-B556-4C0E-800C-1A7#A2D/Documents/cache/KindleSyncMetadataCache.xml

If you look at single entry from the XML:

Code:

<meta_data><ASIN>B000FBJCJE</ASIN><title>Snow Crash</title><authors><author>Stephenson, Neal</author></authors><publishers><publisher>Spectra</publisher></publishers><publication_date>2003-08-26T00:00:00+0000</publication_date><cde_contenttype>EBOK</cde_contenttype><content_type></content_type></meta_data>

You'll see the ASIN is B000FBJCJE. The book filename can be constructed from this by appending _EBOK.prc:

Code:

Filename:
B000FBJCJE_EBOK.prc

Relative Path:
/Documents/eBooks/B000FBJCJE_EBOK.prc

Full Path (OS X):
/Volumes/root/var/mobile/Applications/939E5CA6-B556-4C0E-800C-1A7#A2D/Documents/eBooks/B000FBJCJE_EBOK.prc

All the ebooks are in that same folder. I suppose another option would be to just parse the metadata of all the files, but that seems like a lot more work.

The biggest problem with the AFP approach is that it's not natively available on Windows or Linux, but I think that it can be installed. The alternative would be to use OpenSSh/sFTP. That's got the benefit of being cross platform, but it would mean integrating a client into Calibre. The paths would be similar in that case, just drop /Volumes/root from the full path.

kovidgoyal · 06-02-2009, 11:27 AM

The first hurdle is device detection. How will calibre know when the iPhone is connected to the computer?

ldolse · 06-02-2009, 12:41 PM

Device detection was one of the stumbling blocks I was hitting, especially as it doesn't mount as a disk. That's what got me thinking along the lines of accessing over the network instead.

Here's the best description I've found so far:
http://wikee.iphwn.org/usb:usbmux

Here are some open source projects that have some detection routines along with filesystem manipulation that can be checked out:
http://code.google.com/p/manzana/
http://code.google.com/p/independence/

I'll keep digging.

kovidgoyal · 06-02-2009, 12:50 PM

calibre's current infrastructure is geared towards managing USB devices, detecting a networked filesystem is a whole other beastie.

donCalpe · 06-02-2009, 03:25 PM

As Idolse already mentioned, with a jailbroken iPhone and Netatalk installed ( under Mac OS X ) your iPhone mounts like any another computer ( it IS a computer

) you have in your network. You can use the finder to search for books ( i.e. ext. ".prc" extensions ). You can copy,transfer, move etc.
Regarding device detection - I add my books fromthe iPhone to the Calibre over the wireless network since I installed Calibre.
Idolse, if you want to automate some of the steps using Automator or Applescript is really straight forward.
The only thing left is the missing plug-in to deDRM your books.

tradewinds · 06-20-2009, 03:02 AM

I got tired of all the manual steps, so wrote a little os x application to help me extract the mobi ebooks I've purchased using the iphone kindle app. Hope this helps someone else. Basically, it scans your iphone backup directory for mobi files, and decrypts them to a directory of your choice. You just have to add your iphone ID which you can get from itunes by clicking on serial # on Summary page for your iphone. Once extracted, you can then drag the mobi file into calibre.

search github for TradeWinds, dmg available on downloads tab

JSWolf · 06-20-2009, 05:43 AM

Quote:

Originally Posted by tradewinds

I got tired of all the manual steps, so wrote a little os x application to help me extract the mobi ebooks I've purchased using the iphone kindle app. Hope this helps someone else. Basically, it scans your iphone backup directory for mobi files, and decrypts them to a directory of your choice. You just have to add your iphone ID which you can get from itunes by clicking on serial # on Summary page for your iphone. Once extracted, you can then drag the mobi file into calibre.

search github for TradeWinds, dmg available on downloads tab

What language did you write this application in? If you use Python or Pearl, it could then be cross platform compatible and also work on Windows.

tradewinds · 06-20-2009, 12:34 PM

Quote:

Originally Posted by JSWolf

What language did you write this application in? If you use Python or Pearl, it could then be cross platform compatible and also work on Windows.

The logic is in ruby, so thats cross platform, but all it really is is a glue layer between apple's interface builder and kindelpid/mobidedrm, so not really portable as is.

ldolse · 06-20-2009, 08:52 PM

Awesome idea! Now if I can only get it to work. GUI itself is straightforward, but there seems to be a path issue with Python. I can browse the ebooks in the backup, but as soon as I attempt to decrypt the books I get this error:

Code:

Unable to get PID from iphone id: sh: python:
Command not found

I do have python of course, Here is the path and permissions info for the python that's active on my system:

Code:

$ which python
/usr/bin/python
$ ls -l /usr/bin/python
lrwxr-xr-x  1 root  wheel  72 Feb 22  2008 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.5/bin/python

One bit of feedback on the GUI - would be nice to have date as a sort option, sorted by date with most recent on top. Not required though, easy enough to find the latest book as the titles are there.

itimpi · 06-20-2009, 10:09 PM

What makes you think you cannot sort on the Data field? I regularily do it to find the most recently added titles.

06-02-2009, 12:13 AM	#1
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Feature Request: ipod touch/iphone kindle.app support Note I didn't title this thread quite right, it should read: Feature Request: Jailbroken ipod touch/iphone kindle.app support Hi, Ever since Amazon allowed iphones/ipod touch users to purchase ebooks I've been using them as my primary source of purchased ebooks. While I really dislike the draconian policies they have with drm and delivery of paid content, I've used them in general for years and their prices are good. The real problem is that I read these on my Sony, which means I've got to jump through quite a few hoops to get the content there: Jailbreak the ipod/iphone (one time event) Acquire software which allows me to browse the device filessytem (one time event) Launch the above software find the buried location of kindle.app and it's ebook library Copy the code named prc files to my local disk Go through each file one by one until I figure out which one is the latest book I've purchased Get the serial number of the ipod touch and use it to get the mobipocket ID (one time event) Strip the DRM Load the book into Calibre Send it to the sony Needless to say that's a bit complicated, and I'd love it if Calibre could uncomplicate it for me along the lines of something like this: Plug the jailbroken ipod/iphone in, Calibre recognizes it as a device Calibre allows me to browse whatever books are on the device and transfers them to my library At that point they would still be DRM'd, but the fact that I could stick them into the library from the device intuitively would make all the difference in the world. Stripping the DRM and replacing the file in the calibre library is relatively trivial. Of course it would be major bonus points if Calibre could strip the DRM in the process - but I understand the reasons behind not supporting it in the official release or explicitly on these forums. One could always link a plugin modification over on darkreverser's blog or some other location not affiliated with mobileread. I'm thinking this FR should apply to quite a few people but not entirely certain, so looking for comments. I also tried looking into downloading the files directly from amazon's site by capturing the ipod's traffic on my network, but amazon uses HTTPS with client certificate auth to download the books. I looked all through the kindle.app directory, but I couldn't find any keypair being used as a client cert, I can only assume that Amazon is embedding the keypair in the actual binary. If I could get the client cert I could actually decrypt the SSL traffic pretty easily with the tools I have to determine what the app is doing. Last edited by ldolse; 06-02-2009 at 01:43 AM.

06-02-2009, 07:26 AM	#6
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Some more info: After installing NetaTalk the ipod touch is automatically discovered on the network as an AFP share, with two volumes, one called root, one called home. Once root is mounted the path on OS X is the following: Code: /Volumes/root/var/mobile/Applications/939E5CA6-B556-4C0E-800C-1A7#A2D I'm assuming that 939E5CA6-B556-4C0E-800C-1A7#A2D is the universal folder name for the kindle app across all iphones/ipod touches, but I would need some other users who have jailbroken their iphones to corroborate that. If it's not each subfolder in /Applications/ has a name that looks like that, the subfolders would need to be searched for one named Kindle.app to confirm the correct folder for that device. The xml file I attached previously is located within the app folder at: Code: Relative Path from program directory: /Documents/cache/KindleSyncMetadataCache.xml Full Path (OS X): /Volumes/root/var/mobile/Applications/939E5CA6-B556-4C0E-800C-1A7#A2D/Documents/cache/KindleSyncMetadataCache.xml If you look at single entry from the XML: Code: <meta_data><ASIN>B000FBJCJE</ASIN><title>Snow Crash</title><authors><author>Stephenson, Neal</author></authors><publishers><publisher>Spectra</publisher></publishers><publication_date>2003-08-26T00:00:00+0000</publication_date><cde_contenttype>EBOK</cde_contenttype><content_type></content_type></meta_data> You'll see the ASIN is B000FBJCJE. The book filename can be constructed from this by appending _EBOK.prc: Code: Filename: B000FBJCJE_EBOK.prc Relative Path: /Documents/eBooks/B000FBJCJE_EBOK.prc Full Path (OS X): /Volumes/root/var/mobile/Applications/939E5CA6-B556-4C0E-800C-1A7#A2D/Documents/eBooks/B000FBJCJE_EBOK.prc All the ebooks are in that same folder. I suppose another option would be to just parse the metadata of all the files, but that seems like a lot more work. The biggest problem with the AFP approach is that it's not natively available on Windows or Linux, but I think that it can be installed. The alternative would be to use OpenSSh/sFTP. That's got the benefit of being cross platform, but it would mean integrating a client into Calibre. The paths would be similar in that case, just drop /Volumes/root from the full path.

06-20-2009, 03:02 AM	#11
tradewinds Junior Member Posts: 2 Karma: 10 Join Date: Jun 2009 Device: none	iphone kindle extraction I got tired of all the manual steps, so wrote a little os x application to help me extract the mobi ebooks I've purchased using the iphone kindle app. Hope this helps someone else. Basically, it scans your iphone backup directory for mobi files, and decrypts them to a directory of your choice. You just have to add your iphone ID which you can get from itunes by clicking on serial # on Summary page for your iphone. Once extracted, you can then drag the mobi file into calibre. search github for TradeWinds, dmg available on downloads tab Last edited by tradewinds; 06-20-2009 at 03:04 AM. Reason: removed direct url

06-20-2009, 08:52 PM	#14
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Awesome idea! Now if I can only get it to work. GUI itself is straightforward, but there seems to be a path issue with Python. I can browse the ebooks in the backup, but as soon as I attempt to decrypt the books I get this error: Code: Unable to get PID from iphone id: sh: python: Command not found I do have python of course, Here is the path and permissions info for the python that's active on my system: Code: $ which python /usr/bin/python $ ls -l /usr/bin/python lrwxr-xr-x 1 root wheel 72 Feb 22 2008 /usr/bin/python -> ../../System/Library/Frameworks/Python.framework/Versions/2.5/bin/python One bit of feedback on the GUI - would be nice to have date as a sort option, sorted by date with most recent on top. Not required though, easy enough to find the latest book as the titles are there.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
All new Nook app for IPhone and IPod touch	boswd	News	1	08-17-2010 01:37 PM
How well does the Kindle app work on the iPod Touch?	shortywriter	Apple Devices	2	06-13-2010 06:54 PM
Sunrise XP-like app for iPhone/iPod Touch?	wildeny	Apple Devices	2	01-18-2010 12:17 AM
Kindle for iPhone/iPod Touch goes International	daffy4u	News	1	12-14-2009 07:20 PM
Kindle app for iPhone and iPod Touch in iTunes Store	cerement	News	303	10-26-2009 06:39 PM

06-02-2009, 12:47 AM	#2
kovidgoyal creator of calibre Posts: 44,704 Karma: 24967300 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I don't have an iPhone, so I for one can't do this, but patches are welcome Incidentally calibre's plugin frameworks makes removing MOBI DRM automatic

06-02-2009, 11:27 AM	#7
kovidgoyal creator of calibre Posts: 44,704 Karma: 24967300 Join Date: Oct 2006 Location: Mumbai, India Device: Various	The first hurdle is device detection. How will calibre know when the iPhone is connected to the computer?

06-02-2009, 12:41 PM	#8
ldolse Wizard Posts: 1,337 Karma: 123455 Join Date: Apr 2009 Location: Malaysia Device: PRS-650, iPhone	Device detection was one of the stumbling blocks I was hitting, especially as it doesn't mount as a disk. That's what got me thinking along the lines of accessing over the network instead. Here's the best description I've found so far: http://wikee.iphwn.org/usb:usbmux Here are some open source projects that have some detection routines along with filesystem manipulation that can be checked out: http://code.google.com/p/manzana/ http://code.google.com/p/independence/ I'll keep digging.

06-02-2009, 12:50 PM	#9
kovidgoyal creator of calibre Posts: 44,704 Karma: 24967300 Join Date: Oct 2006 Location: Mumbai, India Device: Various	calibre's current infrastructure is geared towards managing USB devices, detecting a networked filesystem is a whole other beastie.

06-02-2009, 03:25 PM	#10
donCalpe Junior Member Posts: 9 Karma: 10 Join Date: Apr 2008 Device: iRex DR1000S	As Idolse already mentioned, with a jailbroken iPhone and Netatalk installed ( under Mac OS X ) your iPhone mounts like any another computer ( it IS a computer ) you have in your network. You can use the finder to search for books ( i.e. ext. ".prc" extensions ). You can copy,transfer, move etc. Regarding device detection - I add my books fromthe iPhone to the Calibre over the wireless network since I installed Calibre. Idolse, if you want to automate some of the steps using Automator or Applescript is really straight forward. The only thing left is the missing plug-in to deDRM your books.

06-20-2009, 10:09 PM	#15
itimpi Wizard Posts: 4,553 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	What makes you think you cannot sort on the Data field? I regularily do it to find the most recently added titles.

Advert

Advert