Reflecting Calibre Tags as Mac or Windows Tags ("OS-native Tags") ?

[calibre-user]

Hello fellow Calibre users,

I've been using Calibre for a number of years now to manage all my e-books, including technical textbooks.

Lecture notes (pdf, docx) and personal technical notes, which aren't e-books, are not Calibre-ized.

As such, whenever a search for a technical concept, for example the term "homomorph*", is conducted, at least two searches have to be performed – one using Calibre's (impressive) search functionality to look through my e-textbooks, and another to search across documents outside Calibre.

For succinctness, let me call searching using methods outside of the Calibre app "non-Calibre search". This covers searching using the Window's File Explorer, Mac's Finder app, or the Foxtrot Search app, as per my habit.

(I briefly describe the Foxtrot Search app at the end of this post.)

A naïve way to search once instead of twice is to task the non-Calibre search query to also look within Calibre's e-books. This may be done by specifying the inclusion of Calibre's e-book directory in its search efforts.

When this is done, searching for the occurrence of the word "advertise" in a mix of Linear Algebra & Corporate Marketing documents works fine because "advertise" is rarely used in Linear Algebra exposition, resulting in the search only returning Corporate Marketing documents. Note that documents here refer to both Calibre-ized e-books and files outside of Calibre since the single search query was conducted over both Calibre's e-book directory and elsewhere.

However, searching for words that commonly occur in both the communities of Linear Algebra & Corporate Marketing, like "rank", "transformation", "optimization", "projection", "dimension", when one is only concerned with Linear Algebra concepts sees to search results containing irrelevant Corporate Marketing documents. This is why I described this method – the mere inclusion of of Calibre's e-book directory in a non-Calibre search effort – as naïve.

A better approach that reduces irrelevant documents turning up in the search results is to instruct the search effort to focus solely on Linear Algebra documents.

To achieve this, Windows File Explorer, Mac Finder app, and Foxtrot Search allow their users to specify filters. One such filter works with OS-native tags. To my knowledge (and I don't work in the space of Filesystems), an OS-native tag for a file is commonly reflected as an extended attribute in that file. For example, if a file were tagged as "psychology" via the Mac Finder app, running the terminal command $xattr -l prints "psychology" among other things. More precisely, for the programmers reading this, it prints a bunch of text that contains the string "psychology".

So, via a tag filter, one could instruct the search effort to only focus on documents tagged "Linear Algebra", reducing the appearance of irrelevant documents in the search results.

To my knowledge, Calibre 17.9 doesn't reflect its own tags as OS-native tags by default, motivating me to pen this thread with the hope that someone might know how to get Calibre to do so, or to hear from someone who might be able to offer other possible solutions.

To summarize, it'd be quite neat if Calibre could reflect its own tags as OS-native tags so that other search applications could filter on these OS-native tags to reduce users' search efforts. From a certain perspective, OS-native tags used in this manner represents a primitive inter-application communication.

(To clarify, by "reflect Calibre tags as OS-native tags", I mean that if the file, book.pdf, were tagged as "real analysis" in the Calibre app, this file would then have an OS-native tag added, possibly an extended attribute represented by the text "calibre: real analysis", or another text to this effect. And when the user removes this tag using the Calibre app, the corresponding OS-native tag is removed as well.)


****

Foxtrot Search (https://foxtrot-search.com/) is an application that lets users search within documents (.pdf, .doc, etc. files). It is quite sophisticated in my opinion, and has a UI that is really handy for finding textual content occurring several times in documents, across several documents, that may be scattered across many paths.

A thread was recently raised in Foxtrot's forums, seeking suggestions for filtering on Calibre's e-book path conventions: https://forum.foxtrot-search.com/ind...h=612&start=0&

[BetterRed]

Have you tried using the Embed metadata feature, you will probably need to add it to a toolbar/menu, Q.E.D. on Windows:



BR

[calibre-user]

Originally Posted by BetterRed

Have you tried using the Embed metadata feature, you will probably need to add it to a toolbar/menu, Q.E.D. on Windows:

Attachment 211115

BR

First, thank you BR for the suggestion. Am pleasantly surprised at the diverse array of functionality Kovid Goyal has implemented in Calibre.

Moments ago, I ran the "embed metadata" successfully on a pdf file and was able to see the tags with the terminal command mdls (on a Mac). So, Calibre embeds the metadata into what I believe is called the XMP headers in a pdf file but not the extended attributes of a file (viewable with command xattr on a Mac).

Embedded Calibre tags aren't interpreted by Mac OS as tags ~probably~ because Calibre tags are written to a different key in the XMP header. A file's Mac OS tags show up in that file's XMP header under the key "kMDItemUserTags" while Calibre's embed metadata feature writes Calibre tags to the key "kMDItemKeywords". But the real reason might require more digging into how Mac reads tags off a file.

While my Calibre tags have not yet turned into Mac OS tags, this feels like a step in the right direction. I think I should next figure out a means of using the embedded metadata content to set the file's Mac OS tags.

[BetterRed]

I'm surprised that calibre's tags don't show up in MacOS in the same way they do in Windows. Do they show up in the file, e.g.:

Word DOCX



xChange PDF



BR

[kovidgoyal]

calibre does not use/support extended file attributes. They are extremely brittle, platform specific, filesystem specific, dont survive roundtripping via most comman archive formats etc. You can embed metadata inside the file and presumably your search tool can be configured to filter based on that metadata.

[calibre-user]

Originally Posted by BetterRed

I'm surprised that calibre's tags don't show up in MacOS in the same way they do in Windows. Do they show up in the file, e.g.:

Word DOCX

Attachment 211118

xChange PDF

Attachment 211119

BR

Hi BR, thanks once again for the helpful comment and screenshots. It's neat to see that Calibre on Windows is able to write directly to a file's Windows-native tag set.

However, a bit sadly, Calibre on the Mac doesn't seem to be able to write to a file's Mac-native tags.

Five minutes ago, I imported a newly created Microsoft Word file into Calibre 7.19 and applied two tags to it within Calibre, and ran Calibre's "embed metadata" successfully. After that, I was not able to see any tags in the Word app (File > Properties > Summary > "Keywords" field), nor with the Finder app (Finder is the Mac equivalent of Window's File Explorer).

Out of curiosity, I added a tag "test-tag" to this Keyword field in Microsoft Word to see if it would show up as a Mac-native tag – it did not, but it did show up as a keyword in the Finder app. The screenshot below shows this outcome:



I took a few steps back to see if Calibre's tags might also be embedded in this manner am happy to see that it does:



At this stage, I'm looking for some automated way of getting these keywords to be reflected as Mac-native tags.

[calibre-user]

Originally Posted by kovidgoyal

calibre does not use/support extended file attributes. They are extremely brittle, platform specific, filesystem specific, dont survive roundtripping via most comman archive formats etc. You can embed metadata inside the file and presumably your search tool can be configured to filter based on that metadata.

Hi Kovid, glad to hear from you, thank you for taking the time to weigh in on this.

My impression of extended file attributes are similar to yours: that they're brittle, and platform-specific.

Perhaps I've made the mistake of giving an impression that Mac-native tags are implemented solely as extended file attributes. I'm actually not sure about this, especially after discovering that Mac-native tags also show up in the file's XMP headers (in addition to them showing up as extended attributes).

I am now trying to see if the search app is able to filter for tags under the "kMDItemKeywords" field, and will also see if there are means of reflecting "kMDItemKeywords" tags as Mac-native tags.

[JSWolf]

Originally Posted by BetterRed

I'm surprised that calibre's tags don't show up in MacOS in the same way they do in Windows. Do they show up in the file, e.g.:

Word DOCX

Attachment 211118

xChange PDF

Attachment 211119

BR

You attachments are invalid.

[BetterRed]

Originally Posted by JSWolf

You attachments are invalid.

They were fine (hosted here and posted as inline thumbs) until pdurrant tampered with them - for some undisclosed reason.

BR

[JSWolf]

Originally Posted by BetterRed

They were fine (hosted here and posted as inline thumbs) until pdurrant tampered with them - for some undisclosed reason.

BR

Tell him to untamper with them so they work again.

[DNSB]

Originally Posted by JSWolf

Tell him to untamper with them so they work again.

Perhaps it might be better to find out why @pdurrant tampered with them. Copyrighted content comes to mind as one of several possible reasons.

[BetterRed]

BR->pdurrant

Originally Posted by BetterRed

In this thread ==>> Reflecting Calibre Tags as Mac or Windows Tags ("OS-native Tags") ?

Posts 2 and 4

pdurrant->BR

Originally Posted by pdurrant

Accidental. Somehow I managed to delete the thread instead of a post. Unfortunately, restoring the thread didn't restore the images.

I would have ctrl/del'd the screen shots so I'll have to do them again Ψ³

Done

BR

[calibre-user]

Originally Posted by kovidgoyal

calibre does not use/support extended file attributes. They are extremely brittle, platform specific, filesystem specific, dont survive roundtripping via most comman archive formats etc. You can embed metadata inside the file and presumably your search tool can be configured to filter based on that metadata.

Hi Kovid, do you think Calibre developers & maintainers could possibly consider improving the "Embed metadata" feature on the Mac OS by having it write Calibre tags to files' Mac OS native tags instead of writing them to the "kMDItemKeywords" XMP header key?