Template language changes

[chaley]

7 Mar 2021 (calibre 5.13):

• Fix regression introduced sometime in calibre V5: expression lists as parameters
  
  In calibre versions before V5.something one could use expression lists as parameters. Example:
  

  program:
      list_union(
  	a = 'xxx'; list_union('a', field('tags'), ','), 
  	field('#genre'),
  	','
      )

  This 'feature' was certainly rarely used but it was possible. Fixed in the interest of compatibility.
• Allow expression lists in parenthesized expressions.
  
  As a consequence of the above, you can now put expression lists in parenthesized expressions. Example:
  

  program:
  	b = (
  		a = field('#mytextmult');
  		list_union(a, field('tags'), ',')
  	);
  	if a then
  		list_union(b, field('#genre'), ',')
  	else
  		b
  	fi

[chaley]

11 Mar 2021 (in calibre source):

• Performance improvement of the virtual_libraries() template function
  
  The performance of the virtual_libraries function has been improved substantially. For example, on my Windows machine using a test database with 3900 books, 15 virtual libraries, and one composite column that calls virtual_libraries(), the performance change is:
  

  First call to virtual_libraries()
               old = 368,381 μs
               new = 382,352 μs
  
  Typical time for each call thereafter (once per displayed book), until data changes:
               old = 165,338 μs
               new =       1 μs

  Result: scrolling 20 books goes from 3.3 seconds to 20 microseconds.
  
  On a smaller library of 200 books with 5 virtual libraries the numbers are:
  

  First call to virtual_libraries()
               old = 7,852 μs
               new = 7,846  μs
  
  Typical time for each call thereafter, until data changes:
               old = 195 μs
               new =  <1 μs

  The performance improvement is directly related to the number of virtual libraries.

[chaley]

20 Mar 2021 (in calibre source):

• Template tester improvement
  
  If you select multiple books before launching the template tester then it will show you the template evaluation result for each book.

[chaley]

24 Mar 2021 (in calibre source)

• Addition of field references
  
  You can now use $lookup_key instead of field('lookup_key') and $$lookup_key instead of raw_field('lookup_key'). Examples:
  

    * $authors ==> field('authors')
    * $#genre ==> field('#genre')
    * $$pubdate ==> raw_field('pubdate')
    * $$#my_int ==> raw_field('#my_int')

[davidfor]

Originally Posted by chaley

24 Mar 2021 (in calibre source)

• Addition of field references
  
  You can now use $lookup_key instead of field('lookup_key') and $$lookup_key instead of raw_field('lookup_key'). Examples:
  

    * $authors ==> field('authors')
    * $#genre ==> field('#genre')
    * $$pubdate ==> raw_field('pubdate')
    * $$#my_int ==> raw_field('#my_int')

Not sure if I like this. My templates are going to look like Perl

[chaley]

Originally Posted by davidfor

Not sure if I like this. My templates are going to look like Perl

Oh yes! Next is

 if $line =~ /^firstweek: *(.*)/ then

And for PHP fans

<?GPM echo bgcolor($ForumTableHeaderColor); ?>

And of course

HCF(0x9D)

(Actually, perhaps I should have used =~ instead of the 'in' operator. )

[ownedbycats]

Originally Posted by davidfor

Not sure if I like this. My templates are going to look like Perl

https://www.famicol.in/sigbovik/

[chaley]

26 Mar 2021 (in calibre version 5.14)

• Add new date formats 'to_number' and 'from_number'
  
  You can now convert dates to/from a floating point number representing the number of seconds since some platform-dependent start point. This number is sometimes called a 'timestamp'. Number-format dates are easier to compare and to modify using arithmetic. Details:
  • to_number : convert the date & time into a floating point number (a `timestamp`)
  • from_number : convert a floating point number (a `timestamp`) into an 'iso' formatted date. If you want a different date format then add the desired formatting string after 'from_number' and a colon (':'). Example:
    from_number:MMM dd yyyy

[ownedbycats]

Question: Is there a way to set an arbitrary epoch? I can see some strangeness otherwise, e.g. if to_number is used to store it to a column on *nix environment, then from_number is used on that stored number in Windows environment. Though I am not sure why someone would do this.

Also, does the number work in negative, e.g. someone on a *nix system using to_number on a date earlier than 1970?

[capink]

I think python implementation for this sticks with POSIX timestamp regardless of operating system (might be wrong on this). I don't have Windows to test on. But you can test the ouput of this template:

program:
	format_date('1900-01-01T00:00:00+00:00', 'to_number')

On my linux system it returns: -2208988800.0

[ownedbycats]

I got the same result on a Windows environment. That's good that it stays the same.

[chaley]

Originally Posted by ownedbycats

Question: Is there a way to set an arbitrary epoch? I can see some strangeness otherwise, e.g. if to_number is used to store it to a column on *nix environment, then from_number is used on that stored number in Windows environment. Though I am not sure why someone would do this.

No. You shouldn't store the numbers. Instead store an iso-formatted date, which is guaranteed to be valid cross platform.

Also, does the number work in negative, e.g. someone on a *nix system using to_number on a date earlier than 1970?

Maybe, as long as that is what python does. But you should banish the idea of the number being related to *nix systems. There is zero guarantee of that. The python documents say

• The epoch is the point where the time starts, and is platform dependent. For Unix, the epoch is January 1, 1970, 00:00:00 (UTC).
• The term seconds since the epoch refers to the total number of elapsed seconds since the epoch, typically excluding leap seconds. Leap seconds are excluded from this total on all POSIX-compliant platforms.
• The functions in this module may not handle dates and times before the epoch or far in the future. The cut-off point in the future is determined by the C library; for 32-bit systems, it is typically in 2038.

Originally Posted by capink

I think python implementation for this sticks with POSIX timestamp regardless of operating system (might be wrong on this). I don't have Windows to test on. But you can test the ouput of this template:

program:
	format_date('1900-01-01T00:00:00+00:00', 'to_number')

On my linux system it returns: -2208988800.0

See the above

Originally Posted by ownedbycats

I got the same result on a Windows environment. That's good that it stays the same.

I wouldn't depend on this if I were you.

My take: these numbers are good for comparing dates and doing arithmetic on dates, but not much else.

[ownedbycats]

Also I was experimenting and came across something strange

The highest date available in the Calibre date fields seems to be 9999-12-31 so I put this in:

program:
	format_date('9999-12-31T23:59:59+00:00', 'to_number')

which resulted in 253402300799.0

So I put that back in this:

program:
	format_date('253402300799.0', 'from_number')

and got a BAD DATE error. I wonder why. maybe I put the date in wrong the first template.

[chaley]

Originally Posted by ownedbycats

Also I was experimenting and came across something strange

The highest date available in the Calibre date fields seems to be 9999-12-31 so I put this in:

program:
	format_date('9999-12-31T23:59:59+00:00', 'to_number')

which resulted in 253402300799.0

So I put that back in this:

program:
	format_date('253402300799.0', 'from_number')

and got a BAD DATE error. I wonder why. maybe I put the date in wrong the first template.

You have found one of the fundamental problems with timestamps: they can represent only a subset of dates. Exactly what that subset is depends on the platform.

For example, on windows this expression works:

	format_date(1031*(365*24*60*60), 'from_number')

The number is (approximately) 1032 years in the future from the epoch. It produces 3000-04-26T01:00:00+01:00.

The 2032 year-in-the-future expression

	format_date(1032*(365*24*60*60), 'from_number')

produces "BAD DATE", which means that python threw an exception.

For more bad news see Year 2038 problem.

Bottom line: you can't trust timestamps.

[ownedbycats]

Deep Impact was lost because the date was stored as an unsigned 32-bit integer that overflowed.