container.mi and unicode

[jackie_w]

I'm in the process of creating a User Interface plugin as a calibre version of Doitsu's new Sigil plugin (convert selected epub text files to MP3 using the MS Windows Speech API and the LAME encoder).

I'm still doing initial testing so am still running my .py scripts via calibre-debug in a Windows .bat file rather than via plugin. Consequently, I don't have access to the calibre library metadata ATMO so am trying to extract the few items I want from the container.mi object.

My problem is in understanding whether a container.mi field contains unicode or something else. For example this print statement

print('authors:', container.mi.authors, '\ntitle:', container.mi.title)

results in this in my CMD box

authors: [u'Yrsa Sigur\xf0ard\xf3ttir']
title: A ‘unicode’ title Yrsa Sigurðardóttir

'title' looks like a unicode string, but 'authors' looks like a list of non-unicode items.

Please can you advise how I should be accessing the container.mi data to make sure I always end up with unicode?

I could do what the Sigil plugin does and extract the data directly from container.opf but I'm sure container.mi has already done a far more robust job of that than anything I could come up with.

The metadata will be used to populate the MP3 tags.

I can attach a small test epub if necessary.

[jackie_w]

After further investigation the contents of container.mi may not be my problem. The problem may arise when supplying the args to LAME.exe/subprocess.

I definitely have a problem that Sigil is correctly populating unicode MP3 tags, but when I supply what looks like the same args to LAME.exe/subprocess via calibre I get messed up unicode MP3 tags.

I'll come back when I have a better question to ask.

[kovidgoyal]

subprocess in python 2 on windows cannot handle unicode arguments if you are using a non-utf-8 version of windows. You will need to use CreateProcess see gui2/open_with.py for an example fo doing that.

[jackie_w]

Thanks for pointing me in the right direction.

This low-level OS stuff is way out of my comfort zone. It's taken me forever and a lot of googling to work through it but I think I finally have something which can run lame.exe, handles unicode MP3 tags and produces a returncode using

retcode = win32event.WaitForSingleObject(process_handle, win32event.INFINITE)

I have a few questions:

1. Your example in gui2.open_with has the dwCreationFlags parameter set as win32con.CREATE_DEFAULT_ERROR_MODE | win32con.CREATE_NEW_PROCESS_GROUP | win32con.DETACHED_PROCESS.
   
   I think I need DETACHED_PROCESS for my purposes but please could you explain the other two.
2. I've just copied your use of 'with sanitize_env_vars()' verbatim without understanding it. Is this OK?
3. I didn't even know there were utf-8/non-utf-8 versions of Windows. This new CreateProcess method is working for my Win10 x64 setup but will one-size fit all Windows users.
4. I don't think I need to return stdout/stderr for this particular plugin but I may in the future. Please could you give me a a few pointers how to proceed.

[kovidgoyal]

1. They are documented here: https://msdn.microsoft.com/en-us/lib...(v=vs.85).aspx You should use DEFAULT_ERROR_MODE. You dont need DETACHED_PROCESS

2. Yes that's fine

3. Yes the point of using CreateProcessW is that it uses the unicode windows API which passes all data between the system and the application in utf-16 encoding, regardless of the codepage that windows itself uses

4. Lets cross that bridge when you come to it. Basically the easiest way would be to simply redirect the stdout/stderr of the process to a file. Then after the process finishes read the contents of the file.

[jackie_w]

Originally Posted by kovidgoyal

3. Yes the point of using CreateProcessW is that it uses the unicode windows API which passes all data between the system and the application in utf-16 encoding, regardless of the codepage that windows itself uses

Just when I thought I had this working ... Are you saying I should be using ???.CreateProcessW rather than win32process.CreateProcess as used in open_with.py?

Originally Posted by kovidgoyal

1. They are documented here: https://msdn.microsoft.com/en-us/lib...(v=vs.85).aspx You should use DEFAULT_ERROR_MODE. You dont need DETACHED_PROCESS

Thanks for that.

[kovidgoyal]

No, IIRC win32process uses the correct Createprocess call internally when passed unicode arguments.

[jackie_w]

OK, I'm glad about that. Thanks for your help