Search/Replace from Title Field to Pubdate

[TMSxMR]

Hi

I spent for now more than an hour trying to use search/replace function using regular expression to extract publishing dates from Title or Authors fields and push them into Publishing date field, I used the expression \d{4} to extract the publish year, the preview of the results is OK, but after applying it, it corrupts the library most of the times (and strangly sometimes works just as supposed) , when it corrupts, calibre thinks that there is unrecognized sting in the pubdate field, and some time recognize it just right (all years that I extracted are valid), is there a way to convert or force calibre to understand that the result of regular expression is a number not a string? or I should do that in some other way?

[_Em]

Originally Posted by TMSxMR

Hi

I spent for now more than an hour trying to use search/replace function using regular expression to extract publishing dates from Title or Authors fields and push them into Publishing date field, I used the expression \d{4} to extract the publish year, the preview of the results is OK, but after applying it, it corrupts the library most of the times (and strangly sometimes works just as supposed) , when it corrupts, calibre thinks that there is unrecognized sting in the pubdate field, and some time recognize it just right (all years that I extracted are valid), is there a way to convert or force calibre to understand that the result of regular expression is a number not a string? or I should do that in some other way?

I've spent quite some time trying to figure this out, and I always get a "unicode values" error resulting in too many values of the wrong type reported.

Eg:
Search Field: title
Search Mode: Regex
Search For: (\d{4})-(\d{2}).*
Replace With: \2:::\1 (or \2-\1, which works when IMPORTING books)
Destination Field: pubdate

has these results:
TypeError:QDateTime(): arguments did not match any overloaded call: overload 1: too many arguments overload 2: argument 1 has unexpected type 'unicode' overload 3: argument 1 has unexpected type 'unicode' overload 4: argument 1 has unexpected type 'unicode' overload 5: argument 1 has unexpected type 'unicode'

So, obviously, field pubdate is expecting an ISO date value instead of a unicode value. But importing automatically typecasts 08-2012 into the apropriate iso value -- so what do we have to do to typecast the unicode value in here? Shouldn't Python be able to do that automatically? Shouldn't QDateTime() be doing this? do I have to somehow pre-transform it via regex?

Help please

(oh yes, and this results in my database being corrupted every time; so now I make a copy immediately before attempting this, and replace it after killing the process (the above error continuously loops until I kill the process)).

[theducks]

Peeking into the DB

pubdate is shown as (Type: TIMESTAMP)

2005-02-24 07:00:00
or
2005-02-24 07:00:00+00:00

I don't see the space between the date and time.
Yuo might place a literal time value in the replace eg 01:00+00:00

[chaley]

I was reluctant to make the change to allow search/replace to update date fields precisely because of the possibility of this sort of problem. Calibre was not designed to auto-convert between types and apparently the database calibre uses (SQLite) permits writing trash values into date fields. Sigh ...

The next release (0.8.64) will attempt to convert the string to a date in this case and some other cases that I found.

[_Em]

Originally Posted by chaley

I was reluctant to make the change to allow search/replace to update date fields precisely because of the possibility of this sort of problem. Calibre was not designed to auto-convert between types and apparently the database calibre uses (SQLite) permits writing trash values into date fields. Sigh ...

The next release (0.8.64) will attempt to convert the string to a date in this case and some other cases that I found.

Thanks! It looks like the unicode date string to ISO date value conversion doesn't work either... so that'd be nice.

How come the import conversion works correctly? Could the same code be used for the search/replace?

[chaley]

Originally Posted by _Em

How come the import conversion works correctly? Could the same code be used for the search/replace?

No, at least not in a way that is maintainable. The import code is special cased and handles only pubdate. Search/replace must handle all fields of all types, custom or standard. It knows it has a source and destination field but doesn't care what the base types are; the cross-product of types make such knowledge impractical. Instead it uses metadata for the destination field to look up a db "setter" function for that field, then calls that function hoping that it can handle any necessary type conversion from the source type. There was a path where the pubdate setter did not do the conversion.

FWIW: there was also a path where S/R attempted to set the timestamp. This would never work, so timestamp is no longer a permitted destination field.