[Kindle Touch] Localization

[ixtab]

Originally Posted by eureka

Man, why are you giving me an attractive task while I didn't finished JS tool yet? OK, I'll try to implement it (but how soon? ) (Typing of detailed instructions will be a bit boring for me compared with programming.)

Sorry, I got a bit carried away because I had the feeling that you were only criticizing my efforts while leaving all the work up to me. Nevermind, and "за дружбу!"

Still, is the JS part so boring? Or is it really so difficult? Come on...

[ixtab]

Originally Posted by eureka

@ixtab, what is a problem? Interspersing of tool changes with translation changes in single commit?

UPD: ah, I see. But adding or changing of files (even untracked) shouldn't a problem for Git. Could you be a bit concrete in Git error messages?

Anyway, I'm sticking with 3. git includes ALL translations. Resources (source and translations) are the main content of Git repo. Tools aren't. (no offence assumed, it's not about quality or unnecessity).

The interspersing of code and translations is one of the issues, but it's more of an "aesthetical" one (and one where the actually relevant information gets buried in a lot of uninteresting changes).
More important is if there are resources which are on transifex, and nobody has pushed them to git yet. So assume a new language "ex_EX" (ex=example) gets registered on transifex.

1. You are only a user (without git commit rights) and you pull resources from transifex using "tx pull -a". You get ex_EX resources. Everything is fine.
2. As long as git knows nothing about the ex_EX locale, everything is fine.
3. Somebody with commit access commits ex_EX resources into git.
4. The next time you try to "git pull", you get an error from git, because it would try to overwrite a previously untracked file with one from the repository. The only way out is to "git clean -d -f", followed by "git pull", which is not nice at all, because it also deletes other local changes.

This is exactly what happened to my "automatic localization snapshot producer" script last night: it failed because of JustAMan's newly committed files, which had been there already (produced by "tx pull", but outside of git's knowledge). The workaround is indeed to call "git clean", then "tx pull" every time.

So, I'm just saying this. Because of the existing workaround, all of the options work, so I'm fine with any of them.

I'm still in favor of option 2, but of course, option 3 also enables folks which are not using Transifex to use the repository. And looking at the title at github, this actually was the intention

[eureka]

Originally Posted by ixtab

Sorry, I got a bit carried away because I had the feeling that you were only criticizing my efforts while leaving all the work up to me.

Oh... Looking back, it now seems for me too like I've pressed you with bla-bla-bla here and there while you were doing real work. Sorry, I'm really ashamed of it (and of my slowness on JS tool part). I need to put more efforts into a programming instead of investigating for various details and nitpicking you. Again, sorry. There was no intention for criticizing and I'm really praising your work.

Originally Posted by ixtab

Nevermind, and "за дружбу!"

За дружбу!

Originally Posted by ixtab

Still, is the JS part so boring? Or is it really so difficult? Come on...

Neither former, nor latter. And I'm on homestretch.

[JustAMan]

Then I suggest keeping those translation files there (in Git) but in separate branch. That way if someone with commit rights would push them from time to time we'll have kind of backup (does Transifex provide such a service already?), but they won't get tampered with tools.

I do not understand git very well either, but I think that this should be done as follows:
Assume you're working on "stuff" (which could be either "tools" string or "src" string).
Then to change things do:
1) git pull
2) git checkout "stuff"
3) work on "stuff"
4) git commit
5) git push

If we go this way I do not volunteer to make the changes to the repo - I'm complete git noob and don't want to ruin things

ixtab, I can add a switch to make this produce old output (no fancy comments) while still sorted.

[eureka]

@ixtab, you should just not pulling from Transifex at local repo clone. Copy .tx/config somewhere and pull resources there. But I understand that then you will be interested only in tools and .tx/config, but not in resources (i.e. only in part of repo). This is valid point for extracting tools in separate repository or branch (or, at other side, for extracting resources and .tx/config in separate repository or branch).

In fact there are three separate entities 1) resources, 2) tools and 3) .tx/config which is required for your "automatic localization snapshot producer" and for any consumer/updater of resources.

.tx/config is a separate entity because any program interested only in current snapshot of translations wants it without any resources (all resources could be pulled straight from Transifex and discarded after processing).

I'm still thinking about what will be better: separate branch or separate repositories (maybe with submodules)... There are should be specific pros/cons in either solution, but I have not enough experience with Git to deduce them with confidence.

So, you are (with JustAMan) speaking about branching. I'm OK with it. I can extract resources and .tx/config to separate branches. Do you want me to do it?

[JustAMan]

ixtab,
I've added option "--nfb" (--no-fancy-blacklist) to show the same style you requested while still sorted by folder & file & property name.

all translators who are interested in,
I've also added two more actions: combine & expand.
Combine is used to generate files like "report/5.0.0/ru_RU.lng" which are essentially all ru_RU .properties combined in one file with source language string showing.
Expand reverses combine effects.

I'm planning to use it in editing the translation as translating via Web interface seems slow and translating via editing individual files not seeing the original seems just bad (and slow, too)

Short usage:
1) do "git clone https://bitbucket.org/katey_hack/kindle-touch-l10n.git"
2) cd kindle-touch-l10n
3) do "tx pull -l <your_lang>" (do "tx pull -f -l <your_lang>" to force freshing your files if you expect them probably out of sync with transifex)
4) do "python tool/checkit.py -a combine -t <your_lang>"
5) edit "report/5.0.0/<your_lang>.lng"
6) do "python tool/checkit.py -a expand -t <your_lang>"
7) do "tx push -r kindle-touch-framework.\* -t -l <your_lang>"

[ixtab]

Totally, completely, absolutely off-topic:

You absolutely MUST have watched this movie in your life.

For native russian speakers: Особенности национальной охоты
For others: Peculiarities of the National Hunt (subtitled in english; partially narrated in russian).

My favorite quote from the movie:
"Russian elephant is best friend of... finnish elephant"

OK, I stop. Just watch the movie, it's hilarious!

[eureka]

@ixtab, it's a well-known comedy in Russia (and other post-USSR republics). A lot of funny scenes and phrases... So I'm signing under your recommendation. Ну, за искусство! (Well, let's drink for art!)

[JustAMan]

I've gone through all ru_RU locale, message by message, and tried fixing every wrongly translated or badly written stuff I can find. I've edited 476 from 946 strings... %) Took me about 3.5-4 hours.
I also tested in on my device to make sure it looks OK.

I've committed changes to transifex and bitbucket.

BTW, ixtab, I had problems trying to build an update.bin with your kt-l10n.jar. Could you provide exact syntax or some help? I tried doing what it said in "--help" but it kept crashing with some strange stack.

[ixtab]

The stack traces should be a pretty good indicator of what went wrong, and if it's caused by "user error", it should also contain some hint as to what you did wrong.

Anyway, here's a snippet of the script that is used to build the daily snapshots:

DATE=`date +%Y%m%d`
BASE=~/kindle-touch
SRC=${BASE}/snapshots/${DATE}
mkdir -p ${SRC}

java -jar ${BASE}/kindle-touch-l10n/tool/kt-l10n.jar dist -f -k ${BASE}/kindletool -s ${BASE}/kindle-touch-l10n/src/5.0.0/ -t ${SRC}/update_5.0.0-${DATE}-LOCALE_TYPE.bin ALL

Only the last line is actually relevant, but since it includes variables, I left in some context.

If this doesn't help, please provide the full stack trace to see what is going wrong.

[JustAMan]

There're many things that could go wrong, like I'm executing your .jar with GNU Java implementation and I've compiled KindleTool myself...
KindleTool also complains and doesn't want to unpack your .bin that I've downloaded saying something like "bad input".

I'll get to getting stacks this night when I'm at home.

[ixtab]

Nah, there aren't so many things that can go wrong I'm pretty sure it's either a problem with kindletool, or a classical case of PEBKAC

... I'm also using OpenJDK, so this should not be an issue... and I've also compiled (and fixed) kindletool to work on my machines.

Kindletool is in a rather awkward state, so yes, it seems normal that it can't unpack the files it has itself generated. You may have realized that there is a previous commit saying "doing half of the KindleTool work myself" now. In fact, the tool is now creating the entire content itself (including all the signatures), kindletool is only used to "encrypt" the .tgz now.

[JustAMan]

Originally Posted by ixtab

or a classical case of PEBKAC

It might be the case However I doubt it tbh

[ixtab]

So... about the system locales. I again looked into the "using precompiled system locale bundles" vs "compiling on the Kindle itself" thing.

Since compiling on the Kindle will need the sources and the charmap, I now have a clear preference for using pre-compiled bundles, even if they require more one-time space in the repo.

The reason is simple: The charmap file itself is 2 MB, and compresses down to around 350kB. This means that the sources will actually be larger than the pre-packaged file, plus this way will require additional tasks on the device itself, plus the effort to produce the bundles is much higher as well. So I'll now modify the tool to work with pre-compiled bundles, and we'll see where we end up.

Concerning the discussions over branching/not branching etc. in the repo, I suggest to just leave it at the current state. Never touch a running system

[eureka]

Originally Posted by ixtab

So... about the system locales. I again looked into the "using precompiled system locale bundles" vs "compiling on the Kindle itself" thing.

Since compiling on the Kindle will need the sources and the charmap, I now have a clear preference for using pre-compiled bundles, even if they require more one-time space in the repo.

The reason is simple: The charmap file itself is 2 MB, and compresses down to around 350kB. This means that the sources will actually be larger than the pre-packaged file, plus this way will require additional tasks on the device itself, plus the effort to produce the bundles is much higher as well. So I'll now modify the tool to work with pre-compiled bundles, and we'll see where we end up.

You've shown that size of one pre-compiled locale definition is mostly about 300kB compressed. And size of sources for one locale definition (common UTF-8 charmap + appropriate input files) is mostly about 400kB compressed. Difference is ~100kB. Let's say all files in glibc's locales directory are for existing locales, now there are 297 of them. So advantage in size of snapshot with current l10n update bundles will be ~30-35MB at most. Now there are update bundles only for 15 locales. So advantage in size of snapshot should be about 1500kb (I didn't checked it, though). In absolute numbers it should be like 4.7MB vs 6.2MB.

And we'll end up with 55MB repo with pre-compiled locales against 30MB repo with just their's sources.

I'm not arguing against your decision with this comparison. Just calculated overall numbers and wanted to share them.

And points about doing of additional work on Kindle and more efforts for update bundle's producing are completely valid.

BTW, JS tool is completed in the sense that it's now can compile .properties back to fully usable JS. I'm polishing it's code now (by restructuring and refactoring, commenting), but it's in working state. (However, I didn't tested resulting JS on KT yet, because I haven't access to KT right now.) Expect merging of js-tool branch into master and uploading of resources to Transifex soon.