Geekmaster's adventures in K1 spelunking

[geekmaster]

The "Big Buck Bunny" video is now playing on my newest K1. No jailbreak or other hacks needed -- just drop my unsigned Update_RUNME.bin on the kindle, and start UYK (Udate Your Kindle), the the video plays.

My RUNE.sh (launched by Update_RUNME.bin) starts gmplay in the background and waits for the 'R' key, then does "killall gmplay", so I can terminate my viewing pleasure early, if so desired. In fact, I just now did that, and it is restarting my K1.

I need to add that to my KUAL video player extension too, before I publish it.

My current gmplay drops frames if they get more than one frame behind (previously up to one second behind). I did that so I do not need to get too fancy for sound synchronization. I need to test and see how well that works on modern firmware versions -- I had posted in the past that NOT allowing up to one second latency could cause some kindles to drop every-other frame, but there is probably a better way to handle this than allowing a full second of variable video latency (which would be multimedia nastiness if the sound got that far out of sync). For silent films, no problem though. But I want sound...

And using UYK for launching apps is inconvenient -- unlike other kindles, the K1 does not delete update.bin files, so they must delete themselves. I have versions that do NOT delete themselves, but they have a habit of launching whenever a screensaver changes. I first noticed this when "Big Buck Bunny" would begin to play almost immediately after I noticed a new screensaver image flash onto the screen, on may occasions. So I made my update.bin delete all update.bins on both internal and SD card memory. The alternative to dropping a new update.bin before UYK would be to install a launcher script to sniff for a "native menu" hotkey, just my the MyTS terminal app does. But that means modifying the root partition. Though on newer kindles, you must modify the root partition to install the jailbreak custom key (or even deeper hacks on the latest kindles), before you CAN install custom updates or other custom apps.

[geekmaster]

I just "discovered" that I have a ge.tt acount, the same website where ixtab keeps some of his kindle image files shared (at least before using ixtab.tk), and I see that I have had thousands of downloads of debricking images from my mirror as well:
http://ge.tt/1sUhW8Q

I have had tens of thousands of downloads from my mediafire account, so yeah, I thinking my debricking efforts (packaged inside ixtab's kubrick wrapper) have done some good in the world...

I think it is time to update kubrick to handle newer kindles (or at least newer firmware images for kubrick already manages). Perhaps we can add support for newer kindles too, even if that means photographic instructions about how to open a kindle and tape foam and stick-pins to the serial port connections.

And for folks with more than one kindle, we can bend a kindle into being a USB TTL serial adapter, rather than making them purchase such a single-purpose device when they already own a kindle collection (don't we all?)...

[geekmaster]

Originally Posted by knc1

Busybox handles file systems without symlink or hard link support.
...
For the express hint (the esys/bin/bunzip2 file contents):

#!/mnt/us/esys/bin/busybox
bunzip2

I tried replacing the function name (bunzip2 in this case) with $0, and it works fine on my system. That way all the "symlink-replacement" files can be identical, except for their filenames. Is there any reason NOT to do that (other than the fact that busybox config can build equivalent files for you)?

[geekmaster]

I finally got around to running a speed test on the K1 internal userstore partition (shared flash drive) and another speed test on the SD card when accessed by the K1. The test has been running for some time.

I tested the SD card write and read speed while plugged into a host PC, by copying a 100MB zero-filled file. It is class 4, which means that the MINIUMUM write speed should be 4MB/second. I measured 5MB/second writes, and 4MB/second reads.

Inside the kindle, I am creating a file on /mnt/us with "dd if=/dev/zero of=/mnt/us/zero100mb.tst bs=1M count=100", to test write speed to /mnt/us. I then copy that file to /dev/null using the same parameters. I then do both steps using /mnt/mmc instead of /mnt/us.

It is taking a VERY LONG time. I used a "time" command to wrap the "dd" copies, so the log file should tell us the results. My esperience shows that writing to SD card in the K1 (and reading from it) will be a long wait.

My guess is that it is using the SD card in SPI (one-bit serial) mode, which not only explains its slow speed, but also why my SD wifi card did not work (not visible when network scanning). In SPI mode you do not supply power to the card directly. Instead, the card is powered by the serial clock, and you need to send a bunch of clocks before any data to pre-charge the power supply capacitors inside the SD card -- all part of the SD card SPI mode specification. That just adds to the startup time each time you want to communicate with the SD card. Probably not much concern when the kindle is designed to mostly sleep, except during page turns.

I will know with a high degree of certainty when this test completes (perhaps by morning?) and I can view the log file. I am not going to wait any longer -- time will tell (in the morning). It taks an awful long time to assigne a drive letter (and to eject the drive) in windows, when an SD card is installed in the K1. However, with only about 160MB of free space on internal storage, an SD card is essential for anything really useful (like an optware image file, or similar, or a collection of kindle videos for my player).

I want to copy a jffs2 (highly compressed) image file to the SD card tomorrow, and see if I can mount it. As you may know, highly compressed data can compensate for slow communication channels (like a serial interfaced SD card) quite well. If that works, I will install a root filesystem (and a swap file as well) and see if I can do gcc compiles with the aboriginal armv4l build tools, but inside the K1.

Ahh... the test finally completed.

/dev/zero to /mnt/us speed = 471KB/second.
/mnt/us to /dev/null speed = 1.1MB/second.
===> It makes sense that writing to OneNAND flash is slower than reading from it.

Now for the REAL test:
/dev/zero to /mnt/mmc = 257KB/second.
/mnt/mmc to /dev/null = 622KB/second.

Considering that I was doing software bit-bang SPI serial writes to SD card on OpenWRT a decade ago (with my own optimizations) at more than 1MB/second, this is somewhat pathetic. But it is what it is. And I used JFFS2 to speed it up back then. Perhaps once again here. However, unless things have changed, JFFS2 always does an integrity check when you mount it, and back then my SD card was only 1MB. Now I have 4MB at a fraction of the (kindle access) speed. It took minutes to mount back then. I wonder how long to mount JFFS2 on SD inside my K1. We shall see. Perhaps there is a way to NOT make it do that check on every mount -- but is that a good idea?

So, a 4MB/second SD card runs at about 5-percent of its rated speed when mounted in a K1. However, even the internal OneNAND flash memory is not all that much faster, with its SOFTWARE write wear levelling and sector translation overhead. So in that case, I guess it is safer to stick the swap file on the SD card (much easier to replace than the internal OneNAND flash chip)...

[knc1]

Originally Posted by geekmaster

I tried replacing the function name (bunzip2 in this case) with $0, and it works fine on my system. That way all the "symlink-replacement" files can be identical, except for their filenames. Is there any reason NOT to do that (other than the fact that busybox config can build equivalent files for you)?

I don't think it would support:
sh -c bunzip2 .....
situations.

or some other corner case usage of some sort (brain is still asleep).

[geekmaster]

Rob Landley has something interesting to say about why the linux filesystem tree is the way it is:

Note that we skip the various "move stuff from /usr/bin to /bin" steps because /lib, /bin, and /sbin are all symlinks to the appropriate directory under /usr so we've already combined all that stuff already. (That split happened when Ken and Dennis filled up their original root filesystem's RK05 disk pack on the PDP-11 they developed Unix on in 1971, so they let their operating system files leak into the second disk, where all the user directories lived. They eventually added a third disk on /home and moved the user directories to there, but kept the OS straddling two disks. These days between initramfs and cheap multi-terrabyte drives that split is a useless historical artifact kept alive by people blindly following procedures they no longer understand. Don't get me started on /opt.)

For those who weren't paying attention "that day" in Unix history class, Ken and Dennis mentioned above are Ken Thompson and Dennis Ritchie (but real Unix-ish history buffs only use their first names).

So basically, the traditional linux file system tree is just a modern version of "The Pot Roast Story", so there is no sane reason it must remain that way.

Rob Landley also strongly recommended linuxfromscratch.com, but that domain name is now owned by a domain troll offering it up for sale for a mere $499 (they usually try to sell my choicest expired domains back to me to $10,000 each, but mine were GOOD). Anyway, you can still visit linuxfromscratch.org (before it bit-rots away too)...

[knc1]

It has always been *.org - Rob has a typo in his information.
(Hint: Don't waste your time, trying to tell him he made a mistake.)

[geekmaster]

How can you NOT love this little code snippet from the K1 GPL source code?

PHP Code:

#ifndef USE_XSCALE
#error need to define USE_XSCALE
#endif 


I remember a few years back when the kindle GPL source code came with a large number of .patch files, and you needed to apply them yourself. Now the source comes pre-patched (no patch files).

I downloaded the official linux-2.6.10 kernel source, and I am comparing it to the amazon version. Of special interest are the files that are NOT in the official linux source code (i.e. lab126 additions). There are scattered spots where they commented out the "gumstix.h" include, but more interesting is the new code showing how to power up and down the AnyDATA radio at will, and other rather cool-ish code snippets...

[geekmaster]

An interesting comment in the kernel "governor" logic:

PHP Code:

    /* 
     * Every sampling_rate, we check, if current idle time is less
     * than 20% (default), then we try to increase frequency
     * Every sampling_rate*sampling_down_factor, we look for a the lowest
     * frequency which can sustain the load while keeping idle time over
     * 30%. If such a frequency exist, we try to decrease to this frequency.
     *
     * Any frequency increase takes it to the maximum frequency. 
     * Frequency reduction happens at minimum steps of 
     * 5% (default) of current frequency 
     */ 


That comment (and the following source code) are significantly altered from the original linux kernel code -- perhaps for additional time between battery recharge needs (what the eink kindles were famous for, amongst other things).

[geekmaster]

Another interesting K1 kernel comment:

PHP Code:

/* NOTE:  the Fiona OneNAND part only has 20 bytes of space, which is four
   bytes too small for struct jffs2_raw_ebh.  as this is a closed device,
   the EBH compatibility flags will not be used */
#ifndef CONFIG_ARCH_FIONA
...
/* Summary node MAGIC marker */
#define JFFS2_SUM_MAGIC    0x02851885 


Which could be important for decompressing some of those (many) firmware block devices in the K1.

Interestingly, this jffs2.h file contains some comment differences from mainstream linux 2.26.10 kernel source, obviously NOT from lab126 (including an email address, a file version number and date difference in a comment). Perhaps applying whatever "official linux" patch lab126 used to this mainline 2.6.10 source code would make finding lab126-specific changes easier. More specifically (in this case), this:

PHP Code:

 * Created by David Woodhouse <dwmw2@redhat.com>
...
 * $Id: jffs2.h,v 1.33 2004/05/25 11:31:55 havasi Exp $ 


changed to this:

PHP Code:

 * Created by David Woodhouse <dwmw2@infradead.org>
...
 * $Id: jffs2.h,v 1.40 2005/11/07 11:14:51 gleixner Exp $
...
 * Fiona device changes Copyright (C) 2006, Lab126, Inc.  All rights reserved. 


EDIT: I see that linux kernel 2.6.10 has three "testing" release candidates:
https://www.kernel.org/pub/linux/ker.../testing/incr/
I suppose lab126 started with one of those rather than the "stable" code set I am comparing against. Hmm... not likely -- the timestamps in the 2.6.10 stable and rc downloads are much too old for the source code comment shown above. Even the 2.6.11 release candidate timestamps are too old for that comment. I would think if they borrowed code snippets from later kernels, there would be more differences. Though many of the differences are just stripping trailing spaces off the files in the official stable linux source code... One thing of interest is the the main kernel Makefile forces some env vars for cross-compiling "so they are not needed on the command line", AND adds some extra stuff if the cross host is "APPLE". ... And nope. The release candidates are OLDER than the stable release. I wonder if lab126 did mix in bits of source from later kernel versions? The only kernel version with a compressed download file timestamp new enough for that comment above is 2.6.15, and lab126 still calls theirs 2.6.10 for the K1 kernel. Yhe mmc.c code is certainly a LOT newer than 2.6.10, adding io-completion and SDHC card support, plus a lot of newer enhancements. So despite the version name, I think they mixed and matched code from various versions (perhaps borrowed from newer kindles, when they were building new firmware updates for old kindle models).

[geekmaster]

I should have browsed the source code earlier, but more interesting now that I can actually compile code snippets... This will come in really handy (K1 keyboard keycodes, as used by "waitforkey"):

PHP Code:

unsigned char kbd_ioc_xlate_dvt1[FIONA_KEYMAP_SIZE] =  {
    KEY_1, KEY_7, KEY_E, KEY_O, KEY_G, KEY_X, KEY_LEFTALT, KEY_HENKAN,      /* 0x00 - 0x07 */
    KEY_2, KEY_8, KEY_R, KEY_P, KEY_H, KEY_C, KEY_SPACE, KEY_HIRAGANA,      /* 0x08 - 0x0F */
    KEY_3, KEY_9, KEY_T, KEY_A, KEY_J, KEY_V, KEY_PAGEDOWN, KEY_LEFTSHIFT,  /* 0X10 - 0X17 */
    KEY_4, KEY_0, KEY_Y, KEY_S, KEY_K, KEY_B, KEY_PAGEUP, KEY_MUHENKAN,     /* 0X18 - 0X1F */
    KEY_5, KEY_Q, KEY_U, KEY_D, KEY_L, KEY_N, KEY_KPJPCOMMA, KEY_SLASH,     /* 0x20 - 0x27 */
    KEY_6, KEY_W, KEY_I, KEY_F, KEY_Z, KEY_M, KEY_COMPOSE, KEY_KPSLASH,     /* 0X28 - 0X2F */
    KEY_VOLUMEUP, KEY_VOLUMEDOWN, KEY_BACKSPACE, KEY_ENTER,                 /* 0X30 - 0X33 */
    KEY_DOT, KEY_KATAKANA, KEY_YEN,                                         /* 0X34 - 0X36 */

    KEY_NONE, KEY_NONE, KEY_NONE, KEY_NONE, KEY_NONE,                       /* 0x37 - 0x3F */
    KEY_NONE, KEY_NONE, KEY_NONE, KEY_NONE,                                 /* Not defined */
    
    KEY_HANJA, KEY_HANGUEL,                                                 /* 0X40 - 0X41 */
}; 


And perhaps we can setup a keymap for the "myts" terminal. Well, except for that little "no input event devices" problem... I will need to borrow code from the GPL source for reading keys, no simple (obvious) device access (yet)...

Interesting that the key labels are rotated 90-degrees from their sequential keycode mapping on the keyboard layout. And those Japanese keycodes are interesting, considering that the K1 was only released in the USA, with EVDO (not 3G) whispernet.

EDIT: Umm... Except that a firmware install script defined the "R" keycode as decimal 19, and I successfully used that in my scripts, but how does that fit they keycode table above? More research needed... I need to find the VALUES of those key names in that table. But I did find this:

PHP Code:

// Some handy key defs for identifying ALT+FONT key events
#define ALT_KEY_PRESSED     KEY_LEFTALT
#define ALT_KEY_RELEASED    (ALT_KEY_PRESSED | 0x80)
#define FONT_KEY_PRESSED    KEY_KATAKANA
#define FONT_KEY_RELEASED   (KEY_KATAKANA | 0x80) 


Okay, I found where 'R' is keycode 19 -- in kernel source that lab126 did not change, with way more keycodes than any kindle supports. I think they are raw codes for a typical IBM keyboard, and that translation table above is so a ton of existing keyboard support can stay unchanged. Here is an "standard IBM keyboard" keycode table: http://www.penticoff.com/nb/kbds/ibm104kb.htm

EDIT2: FYI, it looks like the "standard" keycodes in the linux headers mostly ascend in column-major QWERTY layout, while the K1 uses row-major QWERTY layout -- just a matter of swapping the row and column lines on the connector cabling (fixed in that software lookup table).

[knc1]

Was gumstick using the mainline kernel?

For a long time, ARM was maintained outside of the mainline kernel repository, in a repository of its own.
AND
The changes where very slow to be pushed to Linus.
(It became a rather 'hot' issue before the workflow among the groups was resolved.)

You will notice the same with the Freescale kernel (in its company repository) vs. the mainline kernel.

Note:
A lot of this "interesting" history was lost when mainline was re-built after kernel.org got hacked.
But perhaps it can be found in the various (original) manufacturers repositories and/or 'network wayback' sites might have copies of "before the hack" mainline repositories.

[geekmaster]

The archive files at kernel.org have sensible timestamps for restoring a backup copy. Though who knows -- maybe somebody editted those timestamps (for directory sorting purposes)... Anyway, manufacturer forks for the ARM series would complicate such comparisons.

It seems that at the 2.6.10 vintage, the lab126 code is bit cleaner (regarding trailing spaces, at least). And the mmc driver lab126 used looks much more "modern" (in interrupt handling and added SD 2.0 support and SDHC support as well, all missing from the mainline "fork" at kernel.org -- with the mmc.c original author (Woodhouse) having different email addresses in those two forks).

[knc1]

Originally Posted by geekmaster

- - - -
It seems that at the 2.6.10 vintage, the lab126 code is bit cleaner (regarding trailing spaces, at least). And the mmc driver lab126 used looks much more "modern" (in interrupt handling and added SD 2.0 support and SDHC support as well, all missing from the mainline "fork" at kernel.org -- with the mmc.c original author (Woodhouse) having different email addresses in those two forks).

Like I posted, they must not have been using the main-line kernel.

That is what you would expect to see if they where pulling from a repository (manufacturer's) that contained changes in advance of what the main-line kernel had accepted.

Check this list, it may include a copy of the repo they had been using:
https://git.kernel.org/cgit/

[geekmaster]

Now that I have examined the lab126 changes to the mmc driver in the K1 kernel source code, I saw that it has additional code for SDHC support (and yes, SDHC cards work in my K1 as long as they are full SD-sized, not microSD in an adapter).

However, most of my microSD cards are large capacity (SDHC), and though the K1 kernel code looks like it TRIES to switch from one-bit mode (not required to be supported by microSD) to 4-bit mode, the slow speed makes me wonder if that code ever works.

I did find something new though -- a 2GB microSD card (not SDHC) that works in my K1, in an adapter (so it fits in the larger SD slot).

So what I have experiences so far, is microSD (not SDHC) seems to work, AND large capacity SDHC (in full SD form factor) also work, but the bulk of my microSD cards are SDHC, and NONE of them have worked so far in a K1.

I have around here somewhere an UNUSUAL card - full-sized SD that is 4GB SD (not SDHC), which sort of violates the SD spec. Many (if not most) devices use signed arithmetic in the mmc driver, which limits is size range between NEGATIVE 2GB and positive 2GB. You need UNSIGNED math to go from 0-4GB, so this 4GB SD card is iffy onto which devices it will work.

The main difference between SD and SDHC protocol is that SD uses BYTE addressing (with the bottom address bits always zero when used as a block device), whereas SDHC uses sector addressing (allowing much more than a mere 2 or 4GB).

The only possible benefit to byte addressing on SD cards would be for tiny devices that do not have enough RAM to hold a 512-byte sector, where they could read and write individual bytes (or smaller blocks of bytes).

I found more of my microSD cards, so I am testing them all again. It was refreshing to find a microSD that works in a K1 (though 2GB is rather smallish). Though really, in my collection in front of me, I have full-sized SD cards with capacities as small as 128MB (yeah, not GB).

And being a "bleeding edge" tech collector, I spent many hundreds of dollars years back on a first-generation USB flash stick, a full 128KB (kilobytes) and it still took over an hour to fill it -- USB 2.0 was not invented yet, and USB was originally meant for slow things like mice and keyboards...

But then again, I still have 5MB hard drives in my collection (which hold the equivalent of about THREE IBM floppy disks), and they only moved the read heads (via stepper motors) at TEN steps per second (just like really old floppy drives). It took several seconds to seek from an inner to an outer cylinder. And we thought that was amazingly FAST back then (when most of our code was stored on boxes full of audio cassette tapes).

And for something more contemporary, the new kindles (like the PW3) are covered with "test points" (little pads on the PCB). I think a lot of those are probably routed to unused GPIO pins on the SoC, and all it takes is a few o them to connect an SD card up and use some custom code to use it as an additional storage device. On my K1, all the GPIO bits are available in the /proc/ tree, so perhaps an SD-card "driver" could even be written in a script (like my eink algorithmic art scripts). But who has enough (unpaid) hobby time to actually do such things?