Programming PW2 using Go language

[rupor]

I am not sure if this will be of interest to anybody, but just in case.

It theory this should allow kindle programming that is as easy as with python and as effective as with c/c++ without requiring installation of any additional components on targeted readers.

I have following:

1. Linaro 20140811 based cross-compiler hosted on Windows 64 bits and targeting PW2 (built with mingw, so no Cygwin or any other *nix like environment is required for development) - it could be used separately.
2. Patches and instructions on how to setup full Go cross environment on Windows targeting PW2 (that includes an ability to use cgo so full set of Go packages could be used, including for example sqllite3 native database driver).

Updates 09/24/2014: I rebuilt gcc to move settings a bit closer to Amazon's original and it now should be possible to target PW1 and PW2 with the same compiler. crosstool-ng now a proper fork of 1.20.0 and leaves in separate github repo.

Updates 10/01/2014: Introducing KAL - kindle auto-lists manager written in go and happily running on my device. Heavily inspired by barsanuphe work. It is work in progress, may serve as an example, but I have some plans for it

Updates 10/03/2014: Changed error processing a bit to print better diagnostics, since I have seen a problem which I cannot understand or reproduce... Also kaznelson created KUAL extension which could be used for testing/cleanup. I rather prefer program to run automatically via even processing, but his extension could give you some peace of mind.

Updates 10/08/2014 - installation and uninstallation packages (with KUAL extension) - here

All sources are here and here

[knc1]

Originally Posted by rupor

- - - -

I have following:

1. Linaro 20140811 based cross-compiler hosted on Windows 64 bits and targeting PW2
- - - -

Ah, are you sure about that?
Please do a:
your-cross-gcc-name -dumpmachine
and
your-cross-gcc-name -v /dev/null
and paste the outputs in code quotes here.

(say what?
you have never tried to cross-compile /dev/null before?
Yes, it does something useful.)

Note 1:
"targeting PW2" requres more than just specifying the compiler's full name unless the compiler (and toolchain) was built specifically to default each and every option to that needed for the PW2.
**and**
not even Freescale's compilers come built that way.
(and neither does that toolchain builder linked to in your readme.md build it that way unless it had a lot of help from yourself (which is what I am asking to see) )

- - - - -

Note 2:
We stress OpenSource software here -
We get enough closed source software from Amazon.

But I don't see the sources to armkpw2.exe
Where are they if you built it yourself?

If you didn't build that self-extracting *.exe yourself (you almost had to - to "target PW2") -
Then please provide the link to the Linaro description page it can be downloaded from.

- - - -

Q: Did you refer to any of the other "Kindle Go" work on this site?

[hawhill]

Nice! But then, I'm a GoLang fanboy. I find it a bit difficult to interface the C world, though. I'm a bit spoiled with Lua and LuaJIT's FFI in that regard.

[rupor]

@knc1 - do you have any reasons to doubt what I am saying?

arm-mingw_kpw2-linux-gnueabi

arm-mingw_kpw2-linux-gnueabi-gcc.exe: error: /dev/null: No such file or directory
Using built-in specs.
COLLECT_GCC=D:\gotools\armkpw2\bin\arm-mingw_kpw2-linux-gnueabi-gcc.exe
COLLECT_LTO_WRAPPER=d:/gotools/armkpw2/bin/../libexec/gcc/arm-mingw_kpw2-linux-gnueabi/4.8.4/lto-wrapper.exe
Target: arm-mingw_kpw2-linux-gnueabi
Configured with: /home/rupor/crosstool-ng/.build/src/gcc-linaro-4.8-2014.08/configure --build=x86_64-build_pc-cygwin --host=x86_64-host_w64-mingw32 --target=arm-mingw_kpw2-linux-gnueabi --prefix=/home/rupor/x-tools/arm-mingw_kpw2-linux-gnueabi --with-sysroot=/home/rupor/x-tools/arm-mingw_kpw2-linux-gnueabi/arm-mingw_kpw2-linux-gnueabi/sysroot --enable-languages=c,c++ --with-arch=armv7-a --with-cpu=cortex-a9 --with-tune=cortex-a9 --with-fpu=neon --with-float=softfp --with-pkgversion='crosstool-NG git+-dirty - rupor' --with-bugurl=http://i1.kym-cdn.com/entries/icons/original/000/002/869/last-page-of-the-internet.jpg --disable-sjlj-exceptions --enable-__cxa_atexit --disable-libmudflap --disable-libgomp --disable-libssp --disable-libquadmath --disable-libquadmath-support --disable-libsanitizer --with-gmp=/home/rupor/crosstool-ng/.build/arm-mingw_kpw2-linux-gnueabi/buildtools/complibs-host --with-mpfr=/home/rupor/crosstool-ng/.build/arm-mingw_kpw2-linux-gnueabi/buildtools/complibs-host --with-mpc=/home/rupor/crosstool-ng/.build/arm-mingw_kpw2-linux-gnueabi/buildtools/complibs-host --with-isl=/home/rupor/crosstool-ng/.build/arm-mingw_kpw2-linux-gnueabi/buildtools/complibs-host --with-cloog=/home/rupor/crosstool-ng/.build/arm-mingw_kpw2-linux-gnueabi/buildtools/complibs-host --with-libelf=/home/rupor/crosstool-ng/.build/arm-mingw_kpw2-linux-gnueabi/buildtools/complibs-host --enable-lto --with-host-libstdcxx='-static-libgcc -Wl,-Bstatic,-lstdc++,-Bdynamic -lm' --enable-threads=posix --disable-libstdcxx-pch --without-long-double-128 --enable-linker-build-id --with-linker-hash-style=gnu --enable-gold --disable-nls --disable-multilib --with-local-prefix=/home/rupor/x-tools/arm-mingw_kpw2-linux-gnueabi/arm-mingw_kpw2-linux-gnueabi/sysroot --enable-c99 --enable-long-long
Thread model: posix
gcc version 4.8.4 20140811 (prerelease) (crosstool-NG git+-dirty - rupor)

I am aware that targeting PW2 requires more, than specifying compiler name. I used configuration settings and versions of the eglibc2 library, kernel and compiler (to the best of my abilities, of course) from Amazon's sources and my kindle dumps - freely available last I checked. I also took a look at NiLuJe's toolchain - but did not follow his example.

The only sources of interest here are configuration settings for crosstool-ng. Those are always included with the toolchain itself as the result of the build (bin\arm-mingw_kpw2-linux-gnueabi-ct-ng.config) and in the full build.log (also available with toolchain). Patches to croostool-ng tool itself (if somebody wants to repeat torture of canadian-cross rebuilding GCC toolchain under cygwin with mingw on Windows 64) are now in github (reluctant fork) and patch to Go 1.3.1 which is submitted to Go community (https://code.google.com/p/go/issues/detail?id=8723). Everything is available on my github.

I verified the results by re-writing small part of barsanuphe librarysync in Go and running it on my device.

I am not aware of any other work of "Kindle Go" on this site - but will appreciate pointers.

[eureka]

Originally Posted by rupor

I am not aware of any other work of "Kindle Go" on this site - but will appreciate pointers.

There was one thread, but it shouldn't discourage you. Your recipe is a Windows-specific, and so it's valuable on its own. It's a unique experience in setting up developer infrastructure under Windows, many thanks for sharing!

[knc1]

Originally Posted by rupor

@knc1 - do you have any reasons to doubt what I am saying?

Nope - nothing personal intended.

Building a cross-compiler is a highly technical undertaking.
Having one presented (as a self-extracting binary) for public use without substantiation, only author claims, spurred the request for machine confirmation.

Originally Posted by rupor

arm-mingw_kpw2-linux-gnueabi

arm-mingw_kpw2-linux-gnueabi-gcc.exe: error: /dev/null: No such file or directory
Using built-in specs.
COLLECT_GCC=D:\gotools\armkpw2\bin\arm-mingw_kpw2-linux-gnueabi-gcc.exe
COLLECT_LTO_WRAPPER=d:/gotools/armkpw2/bin/../libexec/gcc/arm-mingw_kpw2-linux-gnueabi/4.8.4/lto-wrapper.exe
Target: arm-mingw_kpw2-linux-gnueabi
Configured with: 
- - - -
--enable-languages=c,c++ \
--with-arch=armv7-a \
--with-cpu=cortex-a9 \
--with-tune=cortex-a9 \
--with-fpu=neon \
--with-float=softfp 
- - - -

That is "close enough" to call it "targeted at: i.MX6" (both PW1 and PW2) <<- My bad, a think-o while typing.
(Note: use the 'report post' button on your first post of this thread to ask a moderator to correct the title if you wish.)

Originally Posted by rupor

gcc version 4.8.4 20140811 (prerelease) (crosstool-NG git+-dirty - rupor)

Good information to include in your readme.md file - that you used crosstool-NG to generate the toolchain.

Originally Posted by rupor

I used configuration settings and versions of the eglibc2 library, kernel and compiler (to the best of my abilities, of course) from Amazon's sources and my kindle dumps - freely available last I checked. I also took a look at NiLuJe's toolchain - but did not follow his example.

Also good information to include in your readme.md file.
Acknowledgement of prior work reference (credit where credit is due) goes a long way towards getting acceptance of your own work.
(Without insulting sounding questions such as mine.)

Originally Posted by rupor

The only source of interest here are configuration settings for crosstool-ng which are always included with the compiler itself as the result of the build and patches to tool itself (if somebody wants to repeat torture of canadian-cross rebuilding GCC toolchain under cygwin with mingw on Windows 64 - full build.log is included with toolchain) and patch to Go 1.3.1 which is submitted to Go community (https://code.google.com/p/go/issues/detail?id=8723) and is available on my github.

With this, I disagree, I would like to see the output of -dumpmachine also.

All of those specifics in the paragraph above would also be good to include in the "acknowledgements" section of your readme.md

All of my comments about "acknowledgements" and "credit where credit is due" will give the world a much better impression of your work.
They show you did not try to re-invent the entire wheel (which would be an unreasonable thing to attempt) but instead made a very careful and considered attempt to provide a specific tool solution.

In summary:

For this site -
Maybe a tweak to the thread title - your choice.
Our posts and replies (should) already include most of the answers to technical questions.

For the world -
A bit of editing of that readme.md file on the front page of your repo should gain you a larger user base outside of this site.

Overall -
Nice work, thanks a lot for this contribution.
It is always nice to review a good contribution.

- - - -

Notes:

• Perhaps some of our GO users will comment on how the package works for them.
• Perhaps some of our cross-compiler builders will make suggestions to the setting of default options.
  No one should take my word as final, just as a review.

[rupor]

@eureka Yep, it is a bit easier on linux (you do not have to look for a bugs in Go's linker and finding suitable cross toolchain should be easier as well), however I do not want to run VM every time I am hacking something and learning how to build a complete toolchain under cygwin on Windows (which is not using cygwin at the end) has been an interesting challenge.

@knc1 I am sorry, I do not want to get into war of words here with you. Just to make sure my intentions are clear: I am not looking for any recognition of my work. I simply shared my experience in a hope that it may save people some time. This is spare time hobby - had a bit of free time. There is nothing to recognize. Surely knowledgeable people will understand that I did not do anything original, all work is purely derivative at this point and I do not see anybody in particular I have to thank for it - with exception of outstanding work of multiple individuals (you included!) who made jailbraking Kindle possible.

[NiLuJe]

Just one Q: since you appear to be building against the lab126 kernel, did you have to mangle it in any way to get it to behave? I haven't done this in a long time (I usually fall back on the nearest long-term kernel release), but some of them needed some tinkering to behave nicely as a kernel header 'provider' .

As a minor nitpick, I'm partial to either good old fashioned unified diffs, or a real fork, but I find that sneaking in a tweaked third-party library/project/tool as a sub-project in a repo makes it hard to actually diff it to the upstream sources efficiently .

[rupor]

@NiLuJe - I looked at the Amazon's kernel but did not build against it. Saw no point since I was not planning to use C/C++ directly and Go is very conservative when using platform specific calls. Sorry for not being clear enough. I looked at Amazon's code to understand proper target versions and configuration settings for normal GNU sources (building C/C++ was a step - not a final result and I am not interested in using C/C++ toolchain or libraries directly). Was not planning to have crosstool there at all (did not realized the would be any interest), hence the inconvenient format of the contribution.

[knc1]

Ah, right.
(Who?
Me?
Forgot to mention an important point?
Yeah, it happens.)

That cross-compiler was built just to build the GO project's code - -
Its configuration did not include the **gcc** GO build (only c/c++).
I.E: Just a "bootstrap" compiler.

[rupor]

@knc1 this is a full C/C++ compiler, not a bootstrap compiler. It also does not include Java,Fortran,Ada - does it make it a bootstrap? And it does not include Go (gccgo) for a reason - since gccgo version of supported Go language is not 1.3.1. In other words it is not latest. It is not latest even in latest "stable" GCC release 4.9.1 - and I was interested in really stable code base.

[knc1]

I was wondering if the stock, pre-built, linaro toolchain (Ref: https://launchpad.net/linaro-toolchain-binaries ) had the GO language enabled in its configuration.
(There was some past history on including it at that project.)

Short answer: No, it doesn't, only C/C++/Fortran
Short note: They also use crosstool-ng for the builds, read the topmost 'readme' green bar and grab the ct-ng at the second green bar.

The long answer:

Spoiler:

core2quad bin # ./arm-linux-gnueabihf-gcc-4.8.2 -v /dev/null
Using built-in specs.
COLLECT_GCC=./arm-linux-gnueabihf-gcc-4.8.2
COLLECT_LTO_WRAPPER=/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../libexec/gcc/arm-linux-gnueabihf/4.8.2/lto-wrapper
Target: arm-linux-gnueabihf
Configured with: /cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/.build/src/gcc-linaro-4.8-2013.10/configure --build=i686-build_pc-linux-gnu --host=i686-build_pc-linux-gnu --target=arm-linux-gnueabihf --prefix=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/install --with-sysroot=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/install/arm-linux-gnueabihf/libc --enable-languages=c,c++,fortran --enable-multilib --with-arch=armv7-a --with-tune=cortex-a9 --with-fpu=vfpv3-d16 --with-float=hard --with-pkgversion='crosstool-NG linaro-1.13.1-4.8-2013.10 - Linaro GCC 2013.10' --with-bugurl=https://bugs.launchpad.net/gcc-linaro --enable-__cxa_atexit --enable-libmudflap --enable-libgomp --enable-libssp --with-gmp=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/.build/arm-linux-gnueabihf/build/static --with-mpfr=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/.build/arm-linux-gnueabihf/build/static --with-mpc=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/.build/arm-linux-gnueabihf/build/static --with-isl=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/.build/arm-linux-gnueabihf/build/static --with-cloog=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/.build/arm-linux-gnueabihf/build/static --with-libelf=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/.build/arm-linux-gnueabihf/build/static --enable-threads=posix --disable-libstdcxx-pch --enable-linker-build-id --enable-gold --with-local-prefix=/cbuild/slaves/oorts/crosstool-ng/builds/arm-linux-gnueabihf-linux/install/arm-linux-gnueabihf/libc --enable-c99 --enable-long-long --with-mode=thumb
Thread model: posix
gcc version 4.8.2 20131014 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2013.10 - Linaro GCC 2013.10) 
COMPILER_PATH=/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../libexec/gcc/arm-linux-gnueabihf/4.8.2/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../libexec/gcc/arm-linux-gnueabihf/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../libexec/gcc/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../libexec/gcc/arm-linux-gnueabihf/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.8.2/../../../../arm-linux-gnueabihf/bin/
LIBRARY_PATH=/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.8.2/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.8.2/../../../../arm-linux-gnueabihf/lib/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/lib/arm-linux-gnueabihf/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/lib/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/lib/arm-linux-gnueabihf/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/:/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf/
COLLECT_GCC_OPTIONS='-v' '-march=armv7-a' '-mtune=cortex-a9' '-mfloat-abi=hard' '-mfpu=vfpv3-d16' '-mthumb' '-mtls-dialect=gnu'
 /opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../libexec/gcc/arm-linux-gnueabihf/4.8.2/collect2 --sysroot=/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc --build-id --eh-frame-hdr -dynamic-linker /lib/ld-linux-armhf.so.3 -X -m armelf_linux_eabi /opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf/crt1.o /opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf/crti.o /opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.8.2/crtbegin.o -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.8.2 -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.8.2/../../../../arm-linux-gnueabihf/lib -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/lib/arm-linux-gnueabihf -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/lib -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/lib/arm-linux-gnueabihf -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib -L/opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf /dev/null -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../lib/gcc/arm-linux-gnueabihf/4.8.2/crtend.o /opt/gcc-linaro-arm-linux-gnueabihf-4.8-2013.10_linux/bin/../arm-linux-gnueabihf/libc/usr/lib/arm-linux-gnueabihf/crtn.o
/dev/null: file not recognized: File truncated
collect2: error: ld returned 1 exit status

The major difference is that Linaro has adopted hard float (a useful thing to do with the i.MX6 VFP3 co-processor).
The build in this thread is soft float.

My 'ARMhf on Kindle' set of system libraries is built to be compatible, but not specificly for the i.MX6.

[knc1]

Originally Posted by rupor

@knc1 this is a full C/C++ compiler, not a bootstrap compiler. It also does not include Java,Fortran,Ada - does it make it a bootstrap? And it does not include Go (gccgo) for a reason - since gccgo vestion of supported Go language is not 1.3.1. In other words it is not latest. It is not latest even in latest "stable" GCC release 4.9.1 - and I was interested in really stable code base.

Which makes it a "bootstrap compiler" **for the Go language** you built.
Not a "bootstrap compiler" in en_GCC terms.

[rupor]

@knc1

No it does not. Proper Go language does not require cross C/C++ toolchain. Only if your code is using packages which are making calls via cgo interface which in many cases could be avoided. Only because I decided to use "native" sqllite3 driver cross-tool C/C++ toolchain was needed.

I looked at linaro before building my own - 2 points here. First it is 32 bits hosted and I wanted 64 bits (call me nut). Second - I built softfp, not "soft float". And default setting for GOARM is 6 - not 7. It is not the same as ARMel - are you trying to confuse matters on purpose?

[knc1]

Just edited out my error about the PW1 support in my earlier post.
Sorry about that people.