[GUI Plugin] Media File Importer

DaltonST · 08-09-2016, 03:17 PM

[GUI Plugin] Media File Importer

Summary:

MFI is a tool for those who use Calibre for books and find it a friendly and highly flexible environment for classifying and viewing pictures and/or multimedia files.

---
MFI will import Image, Audio, Video and other Binary files into Calibre by creating a new empty Book with a format of .txt for each media file. Optionally, it will copy the original media file into the same directory as the new empty Book that was just created.

---
MFI adds a 'real' format (e.g. JPG or MP3 or MKV or 7Z or RAR or EXE or PCS or HUS or PDF or EPUB or ZIP or anything else) for each imported media or Other Binary file, enabling copying of the MFI-created 'books' among Calibre libraries. Clicking the Book Detail pane 'format', such as 'JPG', will open the appropriate program for that file extension. This also ensures that a Calibre 'Check Library' action finds no extraneous files. However, it will create a new format only if you specify in the options that you wish to copy the original media file into the same directory as the new empty Book that was just created.

---
MFI allows personal pictures stored in a complex OS directory structure with directory names that have meaning to the user (e.g. 'Vacations' or 'Memorabilia' or the names of children, family members and pets) to be imported into Calibre for viewing while retaining the metadata embedded within the directory structure itself. See the attached simple example.

---
MFI can be used to populate a Calibre library containing, for example, all of your pictures accumulated over a very long period of time (years) in a complex OS directory structure that organizes them in some manner designed to help you find them using the directory names. Afterwards, they can be readily viewed within Calibre as 'covers' since an image's cover will be that image itself, and various Virtual Libraries can be defined based on the various metadata values assigned dynamically upon addition to Calibre.

---
The metadata for each new Book is determined through the use of many 'dynamic variables' and, optionally, user-specified values, for both Standard and Custom Columns.

---
The 'dynamic variables' are mostly OS directory structure oriented. Others are Image EXIF oriented, or embroidery design oriented. New ones can be added if there is a user request to do so.

---
You can indicate to MFI that you want all media files that have the identical paths except for the final file extension to be treated as a single 'collection' and copied to the same Calibre subdirectory.

An advanced "file renamer" utility (e.g. https://www.advancedrenamer.com/) is highly recommended to do mass renaming for this purpose. Make their filenames the same except for their file extension prior to executing MFI to add them to Calibre.

Example: you have the identical photos in several formats (.jpg, .png, .bmp, .tiff), so you want them to be in the same 'book' in Calibre, and not each having their own 'book'. The pictures are identical; only the formats are different.

Example: you could compress using 7Zip (try to avoid a .zip file extension, which is an ebook format) various media files that should be kept with a closely related file (e.g. .pdf or .epub), and MFI will put that .7z file in the same "book" as the .pdf or .epub, and register it with Calibre.

For such "mixed media formats", the MFI processing mode of "Other Binary" must be selected so that MFI will not filter by image, audio or video file extension.

"Other Binary" will process anything with anything else. Example: .pdf with a .jpg with a .mp3 with a .mkv with a .7z (that has anything stored inside of it).

---
Related to just above, a prioritization sequence of file extensions to control which file extensions for the identical basic filename are processed first, since the first processed file in a collection determines the values derived for the Dynamic Variables.

Example: you have .pdf files with associated .mp3 audio files. The "cover" of the "book" will be an image file in the same "collection" as the .pdf if you specify in the prioritization sequence of file extensions that .jpg files have a higher priority than the other file types in that collection. Then, specify the Dynamic Variable {path} in the Cover option in MFI. The path of the first file processed in the collection will be used by Calibre as the "cover" for the "book".

The metadata of the first file in a "collection" is used to update the Calibre "book" metadata. So, use an MFI prioritization sequence for Other Binaries similar to: "Binary=.jpg;Binary=.png;".

The 'cases' (upper, lower, title-case) of the file extensions do not matter; .jpg and .JPG and .Jpg will be handled identically.

---
If you have many formats for the identical ebook, and wish to have all of the formats in the identical folder, MFI will import them as described above. If for some reason you want to reset, reload, or "grab" a particular format's metadata instead of using the Dynamic Variables, simply use the Edit Metadata option to do so. An image displaying that option is attached below.

---
MFI has a custom Identifier type 'zmfi_origpath' that stores the original source file path. That Identifier may be shown in the Book Details pane if desired.

---
MFI has a second custom Identifier type of 'zmfi' that is user-defined. It too can be shown in the Book Details pane.

---
MFI does not move, change, or delete any original files whatsoever.

---
The MFI ToolTips comprise the 'User Guide'. Hover your mouse over each area and widget in the MFI window to view them.

---
Study closely the attached collage for a real example that shows how MFI achieves its goal described in the first paragraph above.

Metadata - Custom Columns:

Up to four (4) Custom Column Search Names (including the leading '#') may be specified.
Leave the name and value blank to deactivate a Custom Column option.
Note that CC datatypes of 'composite' and 'rating' may not be used.

Metadata - Standard Columns:

Author
Title
Publisher
Series
Tags
Comments
Identifier 'zmfi'

Metadata - Dynamic Variables:

'Dynamic variables' may be used in lieu of specific text as values for all of the metadata columns, both Standard and Custom.

Spoiler:

{path} means 'use the full path of the files' as the metadata value.

{directory_path:n} means 'use the full path of the files, but only down to the nth directory above the file itself in the full path' as the metadata value. 0 means the final directory in which the file itself is contained. So, {directory_path:0} = (Path - File Name). Valid values: 0 - 9.

{directory_name:n} means 'use the name of the nth directory above the file itself in the full path' as the metadata value. 0 means the final directory in which the file itself is contained. Valid values: 0 - 9. This variable does not yield a valid path. Example: you have a directory named 'Vacations'.

{file_name} means 'use the file name (with extension) of the file itself' as the metadata value.

{file_name_no_extension} means 'use the file name with no extension' as the metadata value.

{file_extension} means 'use the file extension of the file itself' as the metadata value.

Example:

'{directory_path:0}/{file_name_no_extension}.xyz' means 'use the file path with no extension, then add a new file extension of .xyz' as the metadata value.

Example:

'{directory_path:0}/coverart.jpg' means 'use the file named coverart.jpg that is in the same directory as the source file' as the metadata value.

{image_exif_data} means 'get the EXIF (Exchangeable Image File Format) data from the image (if it exists)' as the metadata value.

{image_exif_date_time} means 'get the EXIF DateTime from the image (if it exists)' as the metadata value.

{image_exif_date_time_original} means 'get the EXIF DateTimeOriginal from the image (if it exists)' as the metadata value.

{image_exif_date_time_digitized} means 'get the EXIF DateTimeDigitized from the image (if it exists)' as the metadata value.

{mp3_info} means 'get the tags and other metadata from any .mp3 files (if any such metadata exists)' as the metadata value.

{embroidery_file_info} is currently functional only for file types .pcs, .hus, and .vip. It means 'get the stitch count (.hus, and .vip only), number of colors, hoop size and colors (.pcs only) from the supported embroidery files' as the metadata value. The .pcs colors are returned as HTML with the appropriate textual coloring, so this Dynamic Variable should be used in the Comments or in a Custom Column of datatype 'comments' that is HTML, not plain text. If you do not use any version of the .pcs format, then it is advised to use a free software utility (e.g. Embroidermodder) to convert from the formats that you do use to .pcs just for the above purpose. Embroidermodder will also create a .csv file that MFI will add to your "book". If you do not care about this particular metadata, then simply skip the use of this specific Dynamic Variable. Note that constantly changing file formats will result in the need to add metadata to your MFI "books" using the information provided by embroidering-specific software such as Embroidermodder rather than relying on a generic tool such as MFI.

{today} means 'the current date in the format YYYY-MM-DD' as the metadata value.

MFI Library Comments: It is highly recommended to use MFI in a "workbench" or "intake" Library, and afterwards to move the new books to your "real" Library. Remember to frequently compress/defragment the MFI Library using the standard Calibre "Library Maintenance > Check Library" menu item. The MFI Library database can grow enormously due to unrecovered SQLite file space.

Requires Minimum Calibre Version: 6.0.0

Version History:

Spoiler:

DaltonST · 08-09-2016, 03:18 PM

Reserved for future use.

DaltonST · 08-10-2016, 09:17 PM

For future use.

BetterRed · 08-11-2016, 08:10 PM

How is the Copy File to Calibre expected to be used. I hoped it would copy the file (MP3, AVI, whatever) to the book folder - but it doesn't seem to do that.

I use View Manager - After this plugin adds a 'book', it is shown in the book list according to the current VM sort sequence, so its hard to find. But when I add a book 'normally' (drop into book list or via File Explorer->Send To) it always appears at the top of the book list (including media files) irrespective of the VM sort sequence, and because I have the relevant setting checked in Prefs->Add books the book is marked. Could the plugin adopt the same behaviour - added books always appear at top of the book list, and they're Marked and Tagged according to Add Books settings.

Why didn't I get an 'MTI BOOK' Identifier with this

Click image for larger version

Name: Clipboard.jpg
Views: 1201
Size: 96.2 KB
ID: 150862

If the Identifiers created by Media File Importer were added into the Preferences->Look and Feel->Book Details->Rules to convert identifiers list, then I think they'd then appear in the Book Details side panel as clickable links. Otherwise I'm not sure what purpose they serve.

Publisher always defaults to 'me' irrespective of what I put there.

BR

BetterRed · 08-11-2016, 08:18 PM

Quote:

Originally Posted by BetterRed

How is the Copy File to Calibre expected to be used. I hoped it would copy the file (MP3, AVI, whatever) to the book folder - but it doesn't seem to do that.

Wrong! It does copy them - but they retain the original file name, and they're not 'registered' in the database, consequently they generate 'problems' in the Check Library function. So I don't think they'll be accessible from calibre server.

BR

DaltonST · 08-11-2016, 09:06 PM

The image in the OP shows in great detail what it does with photographic proof.

DaltonST · 08-11-2016, 09:30 PM

The Calibre server is a book server, not a multimedia server. MFI is for organizing media files in Calibre, especially images. The cover of images will be the images if the dynamic variable is set to {path}. I use Calibre to view all of my pictures organized with the metadata determined dynamically.

BetterRed · 08-11-2016, 10:30 PM

Quote:

MFI will import Image, Audio, Video and other Binary files into Calibre by creating a new empty Book with a format of .txt for each media file. Optionally, it will copy the original media file into the same directory as the new empty Book that was just created.

I think you need to explain in that paragraph that when a media file is copied to the 'book' folder it will retain its original name, which is contrary to normal calibre behaviour, hence:

it will not be accessible via the calibre View menu
it will not have a clickable link in Book Details
it will not be accessible via the calibre server
it will not be copied if the Copy to Library option is used

If the delete after copy option is chosen then it's a hard delete - so the file has gone, ostensibly forever. That option is often used in workflows that import 'books' into an Intake library for preliminary processing, and then move the books to a destination library.
that it will create 'problem' entries in the Check Library report.

FWIW - I prefer documentation to be linear text supported by images. To me the image in post #1 (and elsewhere) is a disparate collection of blocks of text plastered somewhat haphazardly onto a surface - it doesn't tell a story.

Added 1:I use iMatch for image management, can't imagine using calibre for individual images, but calibre's good for 'albums' packaged as a CBZ.

Added 2: I've had a couple of strange errors since I installed the PI. Once on calibre exit with a 'calibre is not responding' msg from Windows (never happened before), and another one - again from Windows - regarding an MP3 not being found, I think it was after I moved a book which had a 'misnamed' MP3. I've removed the PI and its json file permanently, so I won't be trying to repeat the conditions.

BR

davidfor · 08-11-2016, 11:30 PM

Quote:

Originally Posted by DaltonST

The Calibre server is a book server, not a multimedia server. MFI is for organizing media files in Calibre, especially images. The cover of images will be the images if the dynamic variable is set to {path}. I use Calibre to view all of my pictures organized with the metadata determined dynamically.

Calibre is perfectly capable of managing these files if you tell calibre about them. If you do, then changing the titles through the calibre interface will keep the files in the appropriate place. As it is, calibre will not move these files and they will be lost. For an example of this, have a look at the recent discussion in the plugin request about sewing patterns.

What calibre won't do is extract metadata from inside the files. But, it can extract the metadata from the filename during the add process. And someone could write a metadata reader and writer to allow calibre to extract and update the metadata. One already exists for MP3 Audio books. Creating one for video would be possible. And for "covers", a metadata source is needed to pull the cover from an appropriate source.

DaltonST · 08-11-2016, 11:38 PM

@davidfor:

You state the obvious. Nothing new.

DaltonST

DaltonST · 08-11-2016, 11:40 PM

Quote:

I think you need to explain in that paragraph that when a media file is copied to the 'book' folder it will retain its original name, ...

The OP says exactly that. The attached detailed explanation image shows exactly that. That is intentional.

DaltonST

davidfor · 08-11-2016, 11:44 PM

Quote:

Originally Posted by DaltonST

@davidfor:

You state the obvious. Nothing new.

Yes, I probably did. And when I first looked at this plugin, I couldn't work out why you were not taking advantage of calibre function and extending things in ways that worked with calibre rather than seeming to work against it.

Or was that intended as an insult? If so, you need to make it clearer.

DaltonST · 08-12-2016, 08:18 AM

Version 1.0.4 - 2016-08-11 Specified value for Publisher saved to preferences for future recall.

MFI was developed specifically to allow personal pictures stored in a complex OS directory structure with directory names that have meaning to the user (e.g. 'Vacations' or 'Memorabilia' or the names of children, family members and pets) to be imported into Calibre for viewing while maintaining the metadata embedded within the directory structure itself.

DaltonST

DaltonST · 08-12-2016, 07:17 PM

Version 1.0.6 - 2016-08-12 MFI now adds a 'real' format (e.g. JPG) for each imported media file.

This enables copying of the 'books' among Calibre libraries.

Clicking the Book Detail pane 'format', such as 'JPG', will open the appropriate program for that file extension.

The Calibre 'Check Library' action now finds no extraneous files.

DaltonST

DaltonST · 08-13-2016, 01:40 PM

Attached is a very simple example of what a "book" looks like after an image is imported into Calibre from a user's "pictures" directory structure in Windows.

Note the "formats" in the Book Detail pane in the upper right corner, as well as the files shown in the Calibre library sub-directories on the right.

DaltonST

08-10-2016, 09:17 PM	#3
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	For future use. Last edited by DaltonST; 05-29-2017 at 03:58 PM.

08-11-2016, 08:10 PM	#4
BetterRed null operator (he/him) Posts: 21,008 Karma: 27620706 Join Date: Mar 2012 Location: Sydney Australia Device: none	How is the Copy File to Calibre expected to be used. I hoped it would copy the file (MP3, AVI, whatever) to the book folder - but it doesn't seem to do that. I use View Manager - After this plugin adds a 'book', it is shown in the book list according to the current VM sort sequence, so its hard to find. But when I add a book 'normally' (drop into book list or via File Explorer->Send To) it always appears at the top of the book list (including media files) irrespective of the VM sort sequence, and because I have the relevant setting checked in Prefs->Add books the book is marked. Could the plugin adopt the same behaviour - added books always appear at top of the book list, and they're Marked and Tagged according to Add Books settings. Why didn't I get an 'MTI BOOK' Identifier with this If the Identifiers created by Media File Importer were added into the Preferences->Look and Feel->Book Details->Rules to convert identifiers list, then I think they'd then appear in the Book Details side panel as clickable links. Otherwise I'm not sure what purpose they serve. Publisher always defaults to 'me' irrespective of what I put there. BR Last edited by BetterRed; 08-11-2016 at 08:13 PM.

08-12-2016, 08:18 AM	#13
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	Version 1.0.4 Version 1.0.4 - 2016-08-11 Specified value for Publisher saved to preferences for future recall. MFI was developed specifically to allow personal pictures stored in a complex OS directory structure with directory names that have meaning to the user (e.g. 'Vacations' or 'Memorabilia' or the names of children, family members and pets) to be imported into Calibre for viewing while maintaining the metadata embedded within the directory structure itself. DaltonST

08-12-2016, 07:17 PM	#14
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	Version 1.0.6 - MFI now adds a 'real' format (e.g. JPG or MP3 or AVI) Version 1.0.6 - 2016-08-12 MFI now adds a 'real' format (e.g. JPG) for each imported media file. This enables copying of the 'books' among Calibre libraries. Clicking the Book Detail pane 'format', such as 'JPG', will open the appropriate program for that file extension. The Calibre 'Check Library' action now finds no extraneous files. DaltonST

08-13-2016, 01:40 PM	#15
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	Simple Example of a Book Format of "GIF" Attached is a very simple example of what a "book" looks like after an image is imported into Calibre from a user's "pictures" directory structure in Windows. Note the "formats" in the Book Detail pane in the upper right corner, as well as the files shown in the Calibre library sub-directories on the right. DaltonST Attached Thumbnails

08-09-2016, 03:18 PM	#2
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	Reserved for future use.

08-11-2016, 09:06 PM	#6
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	The image in the OP shows in great detail what it does with photographic proof.

08-11-2016, 09:30 PM	#7
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	The Calibre server is a book server, not a multimedia server. MFI is for organizing media files in Calibre, especially images. The cover of images will be the images if the dynamic variable is set to {path}. I use Calibre to view all of my pictures organized with the metadata determined dynamically.

08-11-2016, 11:38 PM	#10
DaltonST Deviser Posts: 2,265 Karma: 2090983 Join Date: Aug 2013 Location: Texas Device: none	@davidfor: You state the obvious. Nothing new. DaltonST

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[GUI Plugin] KindleUnpack - The Plugin	DiapDealer	Plugins	495	10-19-2024 07:06 AM
[GUI Plugin] Zotero Metadata Importer	DaltonST	Plugins	291	08-07-2023 01:38 PM
[GUI Plugin] OPML Importer (RSS feeds from feedly/google reader)	fem	Plugins	14	07-19-2014 11:41 AM
OverDrive Media Console with new Gui and Bookmark Sync	Alexander Turcic	News	6	08-07-2013 09:22 PM
[GUI Plugin] Plugin Updater Deprecated	kiwidude	Plugins	159	06-19-2011 01:27 PM

Advert

Advert