Arch Linux content server (500 internal error)

[Joosca77]

Hello everyone,

I am probably overlooking something simple. I cannot, for the life of me, get the Calibre Content Server to allow me access even on the local host if a password is enabled. Everything works fine on all devices (local or over the internet) if I don't use a password.

When I set it up with a password it will act like it is loading and ask for Login:Password but then shows:

500 Internal Server Error
The server encountered an unexpected condition which prevented it from fulfilling the request.
Powerd by CherryPy 3.6.0

If anyone has any insight into this error I would greatly appreciate it.

Thanks!

[eschwartz]

You are using the repo calibre. Yes, I know it is up to date and all... but apparently Kovid patched cherrypy and a few other things, (calibre's internal copy in the binaries is at 3.2.2) so that might have something to do with it.

I can run a content server on arch just fine, using the binaries. Password and all. You can try installing the binaries and see if that helps.

If that is indeed the problem, I will have to see about getting the PKGBUILD corrected. I have just found out that it strips the bundled cherrypy python2-six chardet and html5lib for what probably seemed like good reasons.

[Joosca77]

That worked like a charm. Thank you so much!

[eschwartz]

So as Kovid PMed me previously:

Originally Posted by kovidgoyal

I would not recommend using the Arch calibre package either. Take a look at the PKGBUILD for it, it happily removes various modules from the calibre src folder (probably because they want to use the system versions, despite the fact the if a module is in src, it means it is customized with calibre specific code), patches various bits of calibre source code, does not build the translations and on and on.

Seems like it is indeed causing problems.

[eschwartz]

Speaking of which:

Kovid/personal stream of consciousness,

I guess they assumed you were just bundling it -- I see cherrypy and html5lib are definitely patched, and chardet too when I found the previous location.
six.py is identical to an old version, so it at least could *probably* be unbundled.

Other than that, they change the shebangs to use python2 instead of python (hello, PEP 394 ), forcibly declare the sip_dir, and enforce a bunch of Archlinux name and location conventions, which TBH I don't see why that would be a problem.


Regarding translations, the PKBUILD indicates that building it fails, which corresponds to my attempts at setting up a development install on a direct checkout.
Although IIRC I did manage to get the build to go through that one time, by using the resources from the official binaries.

[eschwartz@arch ~/git/calibre]$ git pull
Already up-to-date.
[eschwartz@arch ~/git/calibre]$ python2 setup.py translations

*
* Running translations
*


Compiling 0 translation files...
Calculating translation statistics...
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    sys.exit(main())
  File "setup.py", line 85, in main
    command.run_all(opts)
  File "/home/eschwartz/git/calibre/setup/__init__.py", line 181, in run_all
    self.run_cmd(self, opts)
  File "/home/eschwartz/git/calibre/setup/__init__.py", line 178, in run_cmd
    cmd.run(opts)
  File "/home/eschwartz/git/calibre/setup/translations.py", line 243, in run
    self.write_stats()
  File "/home/eschwartz/git/calibre/setup/translations.py", line 302, in write_stats
    cPickle.dump(stats, open(dest, 'wb'), -1)
IOError: [Errno 2] No such file or directory: '/home/eschwartz/git/calibre/resources/localization/stats.pickle'

[kovidgoyal]

You meed to run

python setup.py resources

and ensure the directory resources/localization exists, and you should be fine with building the translations.

[eschwartz]

Thanks. And thanks for the fix.

[eschwartz]

Never did end up following up on this.
Recently came back to the front of my mind because the AUR is cleaning up old packages and I might adopt the calibre-git package if I know exactly what needs doing.
(I did notice I need to checkout the calibre-translations.git repo as well, which was not immediately evident. Either the readme in translations/ needs to be updated, or maybe this should be a submodule or something.)

So running python2 setup.py resources errors in:

*
* Running resources
*

	Creating scripts.pickle
	Creating builtin_recipes.xml
	Creating builtin_recipes.zip
	Creating ebook-convert-complete.pickle
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    sys.exit(main())
  File "setup.py", line 85, in main
    command.run_all(opts)
  File "/var/aur/calibre-git/src/calibre/setup/__init__.py", line 181, in run_all
    self.run_cmd(self, opts)
  File "/var/aur/calibre-git/src/calibre/setup/__init__.py", line 178, in run_cmd
    cmd.run(opts)
  File "/var/aur/calibre-git/src/calibre/setup/resources.py", line 283, in run
    from calibre.ebooks.conversion.cli import create_option_parser
  File "/var/aur/calibre-git/src/calibre/src/calibre/ebooks/conversion/cli.py", line 45, in <module>
    ''') + localize_user_manual_link('http://manual.calibre-ebook.com/conversion.html')
  File "/var/aur/calibre-git/src/calibre/src/calibre/utils/localization.py", line 408, in localize_user_manual_link
    lc = lang_as_iso639_1(get_lang())
  File "/var/aur/calibre-git/src/calibre/src/calibre/utils/localization.py", line 393, in lang_as_iso639_1
    code = canonicalize_lang(name_or_code)
  File "/var/aur/calibre-git/src/calibre/src/calibre/utils/localization.py", line 344, in canonicalize_lang
    iso639 = _load_iso639()
  File "/var/aur/calibre-git/src/calibre/src/calibre/utils/localization.py", line 298, in _load_iso639
    with open(ip, 'rb') as f:
IOError: [Errno 2] No such file or directory: '/var/aur/calibre-git/src/calibre/resources/localization/iso639.pickle'

Copying that in from the versioned tarball on the main website gets me a little further:

* Running resources
*

	Creating ebook-convert-complete.pickle
	Creating template-functions.json
	Creating editor-functions.json
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    sys.exit(main())
  File "setup.py", line 85, in main
    command.run_all(opts)
  File "/var/aur/calibre-git/src/calibre/setup/__init__.py", line 181, in run_all
    self.run_cmd(self, opts)
  File "/var/aur/calibre-git/src/calibre/setup/__init__.py", line 178, in run_cmd
    cmd.run(opts)
  File "/var/aur/calibre-git/src/calibre/setup/resources.py", line 319, in run
    from calibre.gui2.tweak_book.function_replace import builtin_functions
  File "/var/aur/calibre-git/src/calibre/src/calibre/gui2/tweak_book/__init__.py", line 13, in <module>
    from calibre.spell.dictionary import Dictionaries, parse_lang_code
  File "/var/aur/calibre-git/src/calibre/src/calibre/spell/dictionary.py", line 82, in <module>
    ul = parse_lang_code(get_system_locale() or 'en-US')
  File "/var/aur/calibre-git/src/calibre/src/calibre/spell/__init__.py", line 33, in parse_lang_code
    ccodes, ccodemap = get_codes()[:2]
  File "/var/aur/calibre-git/src/calibre/src/calibre/spell/__init__.py", line 21, in get_codes
    data = cPickle.loads(P('localization/iso3166.pickle', allow_user_override=False, data=True))
  File "/var/aur/calibre-git/src/calibre/src/calibre/utils/resources.py", line 71, in get_path
    with open(fpath, 'rb') as f:
IOError: [Errno 2] No such file or directory: u'/var/aur/calibre-git/src/calibre/resources/localization/iso3166.pickle'

Ditto. Now:

* Running resources
*

	Creating template-functions.json
	Creating editor-functions.json
Traceback (most recent call last):
  File "setup.py", line 99, in <module>
    sys.exit(main())
  File "setup.py", line 85, in main
    command.run_all(opts)
  File "/var/aur/calibre-git/src/calibre/setup/__init__.py", line 181, in run_all
    self.run_cmd(self, opts)
  File "/var/aur/calibre-git/src/calibre/setup/__init__.py", line 178, in run_cmd
    cmd.run(opts)
  File "/var/aur/calibre-git/src/calibre/setup/resources.py", line 319, in run
    from calibre.gui2.tweak_book.function_replace import builtin_functions
  File "/var/aur/calibre-git/src/calibre/src/calibre/gui2/tweak_book/function_replace.py", line 18, in <module>
    from calibre.gui2.complete2 import EditWithComplete
  File "/var/aur/calibre-git/src/calibre/src/calibre/gui2/complete2.py", line 20, in <module>
    from calibre.gui2.widgets import EnComboBox, LineEditECM
  File "/var/aur/calibre-git/src/calibre/src/calibre/gui2/widgets.py", line 17, in <module>
    from calibre.gui2.filename_pattern_ui import Ui_Form
ImportError: No module named filename_pattern_ui

So, `python2 setup.py gui`
Then another message saying translation stats aren't around.
At this point building the translations works, and then resources finally completes.


Moral of the story -- the git sources are still not enough.



... on the other hand, it doesn't *seem* to be necessary at all if building from the tarballs, since it is all there already AFAICT. Also, on the calibre website it actually says you only need to run `python2 setup.py install` from the source tarball.

[kovidgoyal]

Checkout calibre-translations as a folder named translations in the main calibre source and then run the following sequence:

python setup.py build
python setup.py iso639
python setup.py iso3166
python setup.py translations
python setup.py gui
python setup.py resources
python setup.py install

You need all those commands when building from git as the source tarball already contains the output of many of those, to simplify building from source.

[kovidgoyal]

Here, this should allow you to do just

python setup.py bootstrap

https://github.com/kovidgoyal/calibr...b4497c14243f0c

I haven't really tested it, so let me know if you run into any problems.

[eschwartz]

The bootstrapper works like a charm, thanks!

The PKGBUILD automation separates source downloading from the build functions, so I am manually moving over the already-downloaded translations repo and running the separate commands, instead of bootstrapping and relying on a mandatory internet connection during the build(). I can get away with the extra lines in an automated script.
But outside of a PKGBUILD this shortcut command will be very useful (to the limited crowd that would do a source install, of course).

Thanks for clarifying the necessary steps for me.


One last, minor nit: I noticed when using a --staging-root, there will probably not be a "${staging_root}/lib/python2.7/site-packages/" directory (since python is not installed in the staging_root) so installing the environment module fails (unless it is manually created). Instead of offering a warning, shouldn't the setup create this directory?

[kovidgoyal]

Generally speaking, the installer scripts dont try to create deep folder structures in staging root, the idea is that staging root should already be populated with a skeleton directory structure (staging root is designed for linux package builders, and that assumption typically holds true in that context).

Otherwise one then has to deal with failures to create the directories. Do we warn/abort/ignore? The appropriate action depends on the context.

Still I dont have particularly strong feelings on this issue, so I wont refuse a patch.