DIY Kindle voice control (simulating key presses)

[Acorn]

I'm hoping to try and get some kind of a working form of voice control going for the Kindle.

This would be a great help to anyone who doesn't have the use of their hands and can't control their Kindle by pressing buttons.

I specifically wanted to try and get this functionality working for a friend of mine who has Multiple Sclerosis. This would allow her to read on her own, which she hasn't been able to do for a long time.

The practical stuff:

I'm not going to attempt to use the built in microphone, or anything like that. I'm just going to try and get remote execution of key input so that the voice recognition stuff can happen on a computer.

So the main thing I need to work out is, how do you simulate key presses via the command line?

I have a feeling that h1uke's Launchpad hack does exactly this, but I'm having trouble working out exactly what part of his code is responsible for sending the actual command, and how it is doing this.

If anyone could explain to me how one simulates a key press, then I could have a script on the computer trigger those keypresses over SSH when it receives the correct voice commands.

Thanks!

UPDATE:

I've got working remote execution of Kindle key commands that can be interacted with using voice recognition software. Still a work in progress, but give it a go and let me know what you think!

Just remember to put the right settings in config.ini, and make sure you have the usbnetwork hack installed with ssh over wifi.

[lrizzo]

Originally Posted by Acorn

I'm hoping to try and get some kind of a working form of voice control going for the Kindle.

This would be a great help to anyone who doesn't have the use of their hands and can't control their Kindle by pressing buttons.

I specifically wanted to try and get this functionality working for a friend of mine who has Multiple Sclerosis. This would allow her to read on her own, which she hasn't been able to do for a long time.

The practical stuff:

I'm not going to attempt to use the built in microphone, or anything like that. I'm just going to try and get remote execution of key input so that the voice recognition stuff can happen on a computer.

So the main thing I need to work out is, how do you simulate key presses via the command line?

I have a feeling that h1uke's Launchpad hack does exactly this, but I'm having trouble working out exactly what part of his code is responsible for sending the actual command, and how it is doing this.

If anyone could explain to me how one simulates a key press, then I could have a script on the computer trigger those keypresses over SSH when it receives the correct voice commands.

Thanks!

the way to simulate keypresses is to write the string "send NNN" to /proc/keypad or one of the other devices, where NNN is the appropriate keycode. For instance,

echo "send 104" > /proc/keypad

on a Kindle3 is the same as moving to next page.

You can find the device names and keycodes in keydefs.ini and launchpad.ini from my "Kindle Terminal" at http://info.iet.unipi.it/~luigi/kindle/

Finally, the microphone is available e.g. arecord test.wav records a file (you can stop with ctrl-c) that you can later play with aplay test.wav (and in C you can access the device using the ALSA API)

cheers
luigi

[Acorn]

Wow, thanks for the help!

If I'm going to start by just having the voice recognition side of things on a computer, I assume the easiest way would be to just send the commands over SSH?

The idea of getting simple speech recognition working with the built in microphone is really exciting, but it's way over my head. I have no knowledge of C

If it was possible to get Python working on the Kindle, I might be able to give it more of a try, but that's probably a crazy way to go about it.

Thanks again!

Edit:

Is there a simpler way to send commands to the Kindle than SSH? It seems a little bit overkill. It also has the downside that you need to fiddle around with drivers to get SSH over USB working.

Telnet?

[h1uke]

Originally Posted by Acorn

Is there a simpler way to send commands to the Kindle than SSH? It seems a little bit overkill. It also has the downside that you need to fiddle around with drivers to get SSH over USB working.
Telnet?

provided the usbNetwork hack installed, I can easily add a "network" part to the existing
launchpad daemon. This addition will be listening to incoming UDP multicast packets
sent by your desktop application and translate them directly to whatever Kindle shell
commands you want.
You can also use Kindle serial console, this solution will not require any hacks at all.
Some mechanical/soldering work will be required in this case. But this approach can noticeably
simplify the whole thing.

[kindle3zeng]

Originally Posted by Acorn

Wow, thanks for the help!

If I'm going to start by just having the voice recognition side of things on a computer, I assume the easiest way would be to just send the commands over SSH?

The idea of getting simple speech recognition working with the built in microphone is really exciting, but it's way over my head. I have no knowledge of C

If it was possible to get Python working on the Kindle, I might be able to give it more of a try, but that's probably a crazy way to go about it.

Thanks again!

Edit:

Is there a simpler way to send commands to the Kindle than SSH? It seems a little bit overkill. It also has the downside that you need to fiddle around with drivers to get SSH over USB working.

Telnet?

Yes, it's possible to run python on kindle, I copied python from ubuntu arm port, and from my limited tests, it worked fine.

[Acorn]

Originally Posted by h1uke

provided the usbNetwork hack installed, I can easily add a "network" part to the existing
launchpad daemon. This addition will be listening to incoming UDP multicast packets
sent by your desktop application and translate them directly to whatever Kindle shell
commands you want.
You can also use Kindle serial console, this solution will not require any hacks at all.
Some mechanical/soldering work will be required in this case. But this approach can noticeably
simplify the whole thing.

That's an awesome offer h1uke! Do you think it would be fine to not have any authentication, or would there need to be some kind of signing of the commands sent to the Kindle? Do you think it would make sense to have it as part of launchpad? Or as a separate packaged mod (not sure how much effort that takes)? Even just a script that gives this functionality would be really cool.

[Acorn]

Originally Posted by kindle3zeng

Yes, it's possible to run python on kindle, I copied python from ubuntu arm port, and from my limited tests, it worked fine.

That's interesting to know. I'll look into if I ever want to do something more serious on the Kindle itself.

[lrizzo]

Originally Posted by Acorn

That's an awesome offer h1uke! Do you think it would be fine to not have any authentication, or would there need to be some kind of signing of the commands sent to the Kindle? Do you think it would make sense to have it as part of launchpad? Or as a separate packaged mod (not sure how much effort that takes)? Even just a script that gives this functionality would be really cool.

I am missing the point for doing this. ssh (over wifi) is already nicely packaged as part of the usbnetwork package, and you can configure it to have ssh always active. This gives you authentication and reliability without the need of reinventing everything from scratch.

[yifanlu]

Doesn't the kindle 3 have a microphone? Anyone know how to access it?

[Acorn]

Originally Posted by lrizzo

I am missing the point for doing this. ssh (over wifi) is already nicely packaged as part of the usbnetwork package, and you can configure it to have ssh always active. This gives you authentication and reliability without the need of reinventing everything from scratch.

I think UDP would make things a lot simpler. Authentication is probably completely unnecessary. All you need is a way to send simple commands to the Kindle. The fact that Multicast UDP means you don't even need to know the ip address of the kindle is even better. Quite nice to be able to have SSH set-up separately as well so you don't have to have SSH running all the time if you don't need/want it.

I don't know much about networking/socket programming/ssh, so these thoughts are just based on my limited knowledge and what I've read.

[Acorn]

Originally Posted by yifanlu

Doesn't the kindle 3 have a microphone? Anyone know how to access it?

Irizzo explained this further up:

Originally Posted by lrizzo

Finally, the microphone is available e.g. arecord test.wav records a file (you can stop with ctrl-c) that you can later play with aplay test.wav (and in C you can access the device using the ALSA API)

[vdp]

Originally Posted by Acorn

That's interesting to know. I'll look into if I ever want to do something more serious on the Kindle itself.

Why do you think you need to have python in order to do SR on the device?
I think the "command and control" type of application you describe is perfectly doable using CMU's PocketSphinx with a finite state grammar. As far as I know there are already versions for IPhone and Android that can be used as a starting point.

[Acorn]

Originally Posted by vdp

Why do you think you need to have python in order to do SR on the device?
I think the "command and control" type of application you describe is perfectly doable using CMU's PocketSphinx with a finite state grammar. As far as I know there are already versions for IPhone and Android that can be used as a starting point.

I was just saying that Python was the only language I have knowledge of. PocketSphinx looks interesting, I'll definitely look into it! Thanks for the tip.

[Acorn]

It turns out that having to send the commands over SSH is making things a huge headache. I'm wishing I could just send a packet

[Acorn]

Okay, I have a few more questions.

1. What is the command to make the Kindle wake up from sleep? Is it safe to use if the kindle is already awake?

2. How does the Kindle get assigned its IP address? My script needs to know the IP address of the Kindle, so it will need to be static. How can I achieve this?