Handling a huge number of files on the SD Card

[Iñigo]

Hi,

I've been very productive this week developing a DR global.db metadata management tool.

DESCRIPTION:
What is this? an evolution of my old "icons" program, a more ambicious command line application to add/update/delete metadata and thumbnails of books. Or if you prefer, an in-progress mdbindex replacement.

Purpose? add thumbnails or other metadata quickly, as mdbindex crashes or is too slow.

What formats does it support? .desktop, .pdf, .fb2, .epub. Take a look at Issues & Bugs section below.


USE:
It can be used from the DR command-line (rxvt or usbnet) or from your own computer.
Note the binary in the package attached is only for the DR, in case you want to run this program on your computer you must compile it before. If anyone wants I can attach a linux x86_64 binary.

As most of the options are self-explaining I will only mention some of them.

Case 1:
Let's suppose you have mounted the DR with the USB cable in /media/DR800 and you have your PC-compiled version of the program in /home/inigo/tmp, which is your current working directory.

Then you type:

$ cd /home/inigo/tmp
$ ./metadata -fs -v -l --sdcard=/media/DR800  --action=report   /media/DR800/Books/Fiction  /media/DR800/Books/NonFiction

-fs: as I'm using DR800+ my thumbnails are stored in file system, not in global.db
-v: show some information on screen while running
-l: generate log file in current directory
--sdcard=/media/DR800 path where to find SD card 'global.db' file and 'System' folder
--action=report: create a report of the files
directories with books

This commands is not fully correct, but we will explain after next case.


Case 2:
You have copied a folder with books from the DR to the current folder of your PC for faster processing. You have also copied global.db (and System/.covers in case of DR800+) to your current folder.

$ cd /home/inigo/tmp
$ ./metadata -v -l --type=pdf,epub  --add-prefix=/media/mmcblk0p1/Books  --action=report   Fiction

--type=pdf,epub only process PDF and EPUB files (not FB2 or DESKTOP)
--add-prefix=/media/mmcblk0p1/Books

No --sdcard or --del-prefix as they default to current directory
With the "--add-prefix=/media/mmcblk0p1/Books" we were modifying filenames path, so "/home/inigo/tmp/Fiction/book1.pdf" will be converted to "/media/mmcblk0p1/Books/Fiction/book1.pdf", as we drop "/home/inigo/tmp" (--del-prefix defaults to current folder, remember) and then add "/media/mmcblk0p1/Books"
Then we can run any other action and copy global.db (and System/.covers) back to the DR.


Back to Case 1: when SD card is inserted into the DR, it is mounted in /media/mmcblk0p1 not in /media/DR800 as in the computer, so the correct command is:

$ cd /home/inigo/tmp
$ ./metadata -fs -v -l --sdcard=/media/DR800  --del-prefix=/media/DR800  -add-prefix=/media/mmcblk0p1  --action=report   /media/DR800/Books/Fiction  /media/DR800/Books/NonFiction

Important:
- There are lots of options, but usually you won't need to specify most of them, only action and directory
- Remember to backup your global.db before running this program
- Also note that depending on the number of files it can take some time to finish

Manage books metadata for iRex DR series eReaders.
Copyright (C) 2011  Iñigo Serna <inigoserna@gmail.com>
Released under GPL v3+

Usage: metadata  [options] directory1 [directory2] [...]

    -h, --help            Display this help
    -v, --verbose         Show additional information on screen
    -l, --log             Write a log in file metadata-{timestamp}.log

    -fs, --thumbs-in-fs   Whether thumbs are stored on file system (default false, in global.db)
                          This option is needed if you are using DR800+ version 4 or higher

    --sdcard=/path/to     Folder where to find SD card 'global.db' file and 'System' folder (default current directory, then /media/mmcblk0p1)
    --del-prefix=/path    Prefix path to be removed from the start of book path in database (default current directory)
    --add-prefix=/path    Prefix path to be added to the start of book path in database (default current directory)

    --type=TYPES          File types to analyze (default all: desktop, pdf, fb2, epub)
                          all
                          item1,item2,item3 (no spaces allowed between items)
                          item

    --action=ACTION       Action to perform (default report)
                          report: show book information
                          add: add new books to database
                          add-metadata: add new books (without thumbnails) to database
                          add-thumbs: add missing thumbs to database
                          overwrite: force update books metadata to database
                          overwrite-metadata: force update books metadata (no thumbnails) to database
                          overwrite-thumbs: force update books thumbs to database
                          update: add if new book or update
                          update-metadata: add if new book or update metadata
                          update-thumbs: add if new book or update thumbs
                          delete: remove entries from database
                          delete-thumbs: remove thumbs from database
                          vacuum: clean database

ISSUES & BUGS:
- overwrite-metadata: update any other field? now only author and title
- does sort by last added work with books added with this program?
- .pdf with first page only contains text -> thumbnail will look blank
- .epub and .fb2 books without images -> no thumbnail will be generated
- .epub with error message "unzip: inflate error": unzip program in DR is old and can't uncompress the file
- .fb2 books not saved with UTF8 encoding fail


SOME IDEAS FOR THE FUTURE:
- a GUI tool
- allow a different source to specify book metadata (title, author)
- add Mackx's "no-update" tag and many tags features to DR800+
- handle more than one tag here
- replace mdbindex?
. we should manage every registered file and dirs
. delete entries and dirs present in db but not in file system


EDIT 2011/07/11 09:00 GMT: slightly modified package

Iñigo