DR800 Long time to index files after upgrading firmware?

[Mackx]

Originally Posted by fekhner

the filenames contain the relevant information information, the tag names are useless,

this implies that the book view is more or less useless while the SD card view is what I really use (I did not have the courage so far to try to implement the possibility offered by Mack's hack not to change the file name after opening a file in a systematic way);.

With tag you mean the title as extracted by the DR file-viewer (uds) or the tag like 'book', 'news' that can be 'attached' by the DR? You can modify the tags that are 'attached' to a document with the hacked firmware. This would allow you to group your documents into smaller groups which makes the non SD-Card view more useful (although there currently is a limit on the number of different tags).
I am creating a new hack that allows to create extra 'drinfo' files, these files (one per document) would be read during indexing and overrule the information that was extracted by the normal indexing process. This allows to change Title, Author, tags (including no_update) of all NEW documents (or you need to 'touch' an existing document to change its modification date).

[Mackx]

Originally Posted by rvs

I assume you mean the no_update tag? This would mean you would have to tag every book on the device and add a tag each time you put a new book on the device. I think a global setting that affects all books would be far easier in use.

For finding the new books the 'Recently Read'-view can be used.
Adding an option to automatically add the no_update tag could indeed be implemented. What about the new feature (drinfo-files) that I described a few messages back. It would not alone allow you to add the no_update-tag, but also allow you to change the Title of a book. I am currently experimenting with a small batch-script that creates drinfo files for all files in a folder that sets the filename as Title and sets the no_update-tag. These can then manually be changed.

Originally Posted by fekhner

I agree of course, but at least Mackx has implemented a viable workaround. Furthermore in the latest version one can give a tag to multiple books, maybe a "select all" option in this context would come quite close to a global setting ...

Indeed a "(de)select all" would be a good addition.

Originally Posted by rvs

It was sort of meant as a subtle hint to Mackx that he should implement this

Very subtle ...

Originally Posted by rvs

Just kidding, but it would be a great feature. The problem with the "select all" option is that most of the documents I read are documents I've only just put on the device. So to be consistent I would have to add tags every time I add documents.

See the alternative described above. Would the drinfo-files be an easier option?

Originally Posted by rvs

I actually wrote a hack today and it works great, but it's just a very quick and dirty hack on Mackx's previous code and I don't know how to make it user configurable. But if you want I can look into this a bit further and see if I can come up with something.

Does the hack add the 'no-update' tag to all new documents? (subtle hint: send me the code:-)

[Mackx]

The problem of Viasheslav (and maybe of others) is not yet addressed. I can see if it is possible to disable the indexing and create a kind of 'lazy-indexing'. This would mean that only the SD-card view would work and that, when entering a folder its current content would be read and added (or replace) the memory-database-view-content. This would add some delay when showing folders.
This would also mean that features like search and the 'views' do not work.

Should this mode then be enabled automatically when there are 'too many' books on the SD-card or manually with an option in Settings?

A hybrid mode would also be possible, limiting the files that are indexed (and thus available in the non SDCard-view) and using the lazy-indexing in the SDCard-view.

Any comments?

[rvs]

Automatic creation of drinfo files is a great idea in itself, but not a very clean solution to my problem I think.

There's already a built-in option to show the filenames instead of titles, except it's only used on the SD-card view. It's easy to extend this to every title (now that I've found the relevant function).

This would have no effect on the global.db file, which means it's possible to switch between the different views without a problem. Tell me if you think it's a useful feature and I'll add a configurable setting and send it to you.

[Mackx]

Originally Posted by rvs

Automatic creation of drinfo files is a great idea in itself, but not a very clean solution to my problem I think.

There's already a built-in option to show the filenames instead of titles, except it's only used on the SD-card view. It's easy to extend this to every title (now that I've found the relevant function).

This would have no effect on the global.db file, which means it's possible to switch between the different views without a problem. Tell me if you think it's a useful feature and I'll add a configurable setting and send it to you.

I would probably not use it myself, maybe fekhner? You can send me the code and I will add it into a future release.

[fekhner]

Originally Posted by Mackx

With tag you mean the title as extracted by the DR file-viewer (uds) or the tag like 'book', 'news' that can be 'attached' by the DR? You can modify the tags that are 'attached' to a document with the hacked firmware. This would allow you to group your documents into smaller groups which makes the non SD-Card view more useful (although there currently is a limit on the number of different tags).
I am creating a new hack that allows to create extra 'drinfo' files, these files (one per document) would be read during indexing and overrule the information that was extracted by the normal indexing process. This allows to change Title, Author, tags (including no_update) of all NEW documents (or you need to 'touch' an existing document to change its modification date).

I meant the title as extracted by the DR file-viewer. I am all in for the new hack

[Viacheslav]

Originally Posted by Mackx

The problem of Viasheslav (and maybe of others) is not yet addressed. I can see if it is possible to disable the indexing and create a kind of 'lazy-indexing'. This would mean that only the SD-card view would work

That would be excellent!
It would be so cool to have an option to turn off all indexing to save the CPU time and make the reader start faster and be more responsive.
We already have a global preference checkbox "Enable SD card view". There could be another checkbox next to it "Enable Bookshelf view" (checked by default). Unchecking it could turn off all indexing and hide the irex's default views because they become empty.

Originally Posted by Mackx

Should this mode then be enabled automatically when there are 'too many' books on the SD-card or manually with an option in Settings?

Err..., YES! If Indexing takes longer than 3 minutes or registers over, say, 5000 files, the indexing process should be aborted with a message "There are too many files on the card. It slows down indexing and delays access to your documents. The Books, News, etc. views will be replaced by direct SD card view."

An optional intelligent "lazy indexing" mode would also be cool, if not too complex or too long to implement.

Originally Posted by Mackx

when entering a folder its current content would be read and added (or replace) the memory-database-view-content. This would add some delay when showing folders.

A hybrid mode would also be possible, limiting the files that are indexed (and thus available in the non SDCard-view) and using the lazy-indexing in the SDCard-view.

I wouldn't like to suffer any delays when entering a folder because I need to open 3-4 folders to get to the needed documents in my Library>Language>Genre>Author(>Series) or Science>Field of study>Author/Conference structure and the folders may contain thousands of files/subfolders. Linux has true multitasking and indexing could still run as an invisible very low priority daemon process, like it works on the desktops (e.g. Beagle search service in GNOME). It must not delay the starting up of the reader, which is slow enough without indexing.

As for the search function, the new thing is that in the case of lazy indexing the database can be not in sync with the filesystem. Therefore, a good searching function must
- find everything in the database;
- verify, if the records found point to really existing files;
- show all verified records / delete invalid ones as verification progresses;
- when verification finishes continue searching by looking through the filesystem unless stopped by the user.

The ability to limit indexing to certain folders on the card might also be nice, but it needs a rather complex GUI part to specify the folders.

The optional indexing daemon could also be compiled as an optional standalone tool for a real Linux-PC. My 2.8Ghz quad-core box can index the card and build a complete DB for the reader in minutes.

[Mackx]

I continued further discussion in the developer's corner. See this thread.

[Iñigo]

Originally Posted by Viacheslav

Yes, I am aware of that. This crazy scheme was definitely conceived while smoking hashish It means that to make the 2.0.x firmware usable someone has to port a real filemanager like Thunar of adapt the file browsing backend part of the 1.x firmware. I would be happy to get rid of the "bookshelf" UI.

As rvs has already mentioned disabling indexing completely is not an option because even SD card view gets the files information from the DB, it doesn't parse the folder contents each time.

Regarding Viacheslav's problem... 100k+ books are too many for what DR developers thought would be common user cases... not only the software, hardware itself (CPU, memory...) is not sized for that amount of books (btw, how much is the size of your global.db file?)... so it's not fair to blame them, in fact it's your own issue because you didn't study the limits of the device ;-)

Me myself use a 4GB SD card with over 5000 books, and I can't use Books view either due to CTB/DB slowness. Thus, I'm using Shortcuts and SD views.

As posible solutions I can think on different combined ways:

1. The ugly hack
Indexing is automatically started by sysd each time it detects a card is mounted, calling ctb which then run mdbindex program. Modifying this behaviour implies lots of ugly source changing, so I don't like it.
But you can rename "mdbindex" program file name to anything else (f.e. "mdbindex.real"), this way the indexing would not be performed. A good thing should be to have a "mdbindex" executable file that accepts same arguments but do nothing to avoid error messages.
Then, each time you add/delete/modify SD card contents, you should run "mdbindex.real" manually and leave it all the night to finish.
Yes, ugly, but quite easy and better than what you have now

2. A new dedicated files browser
This could be nice for everybody... but someone has to get hands dirty and write it.
In fact this is a project in my TODO list for some time ago and I even wrote a somehow working file manager sometime ago for the Iliad in lua language that works for DR as well. Anyway, it lacks DR integration and book opening but it is an advanced working point.

LATER:
- I've read other people has already explained what I say
- Mackx's ideas: I'll comment them in another post

Iñigo

[Iñigo]

Originally Posted by Mackx

[...]
Does the hack add the 'no-update' tag to all new documents? (subtle hint: send me the code:-)

Do you remember my hack INBOX Manager some months ago? [1]
The idea was that every new book added to the DR is stored in an INBOX waiting to be tagged/moved.
This could be improved to be bypass-ed for some kinds of books that would get their proper tags and directory in SD based on their file name.

[1]
https://www.mobileread.com/forums/sho...manager&page=5

[Iñigo]

Hi,

I got time to read and think on all this stuff.
I've written my answers on some ideas commented along this thread.
I've splitted what I don't like and what I think should be easy/powerful.
Please note I don't want to despise other people ideas, just share what I think are pros and cons.

NO:

• Lazy indexing
  . it will stop suspend properly working so it would waste lots of battery
  . complex
• Per-directory local global.db
  . pollutes SD file system with files only useful for some corner cases
  . SD cards don't like writings. I think this could reduce card lifetime adding more file system writings
  . add more complexity to CTB... but (I don't think) benefits are not so big
• Search / file tracker
  . basically it's the same that DR already has in its own DB
  . could we implement a faster alternative? I doubt it

YES:

• Setting to disable indexing
  . option to disable indexing each time system gets "card is mounted" signal
  . also an option to run indexing manually
• A new file browser [1]
  . not a CTB substitute, alternative browsing for advanced users or people with lots of files
  . based on file browser paradigm
  . could use its own DB to speed up searches
  . integrated with DR (menu, toolbar...)
  . DB could be populated/built from an external computer

[1] People who have read my posts in this forum since some time ago already knows that this is/was a project always on my TODO list.
In fact I already have an external book manager able to generate that DB...


Iñigo

[Mackx]

Originally Posted by Iñigo

Do you remember my hack INBOX Manager some months ago? [1]
The idea was that every new book added to the DR is stored in an INBOX waiting to be tagged/moved.
This could be improved to be bypass-ed for some kinds of books that would get their proper tags and directory in SD based on their file name.

[1]
https://www.mobileread.com/forums/sho...manager&page=5

Yes, I remember, I will have a new look at it.

[Mackx]

Originally Posted by Iñigo

NO:

• Per-directory local global.db
  . pollutes SD file system with files only useful for some corner cases
  . SD cards don't like writings. I think this could reduce card lifetime adding more file system writings
  . add more complexity to CTB... but (I don't think) benefits are not so big

I wanted to make this mode optional, so it would not pollute everybody's card. You could be right on the SD-card lifetime, to avoid changing too much code I wanted to copy a new global.db file to /media/mmcblk0p1 every time that a new directory is viewed. (Having a file at that location would allow UDS to add thumbnails to it.) Maybe I have to forget about the global.db and use the content of the directory immediately (loosing thumbnails).

[Viacheslav]

Originally Posted by Iñigo

100k+ books are too many for what DR developers thought would be common user cases ... in fact it's your own issue because you didn't study the limits of the device ;-)

I disagree. I bought a DR1000 to make it my portable library and from day 1 it worked perfectly with any amount of documents under firmware 1.x. There is no indexing and no slowdowns (but there is a limit on the amount of files/subfolders in a folder). The downside of using 1.x, however, is that I am not able to follow the new development for the 2.x branch. I'm stack with a buggy DJVU viewer and my Russian localization for 2.x is frozen.

Thanks for the ugly hack tip! I'll try to substitute the indexer with a dummy script.

[Iñigo]

Originally Posted by Viacheslav

Thanks for the ugly hack tip! I'll try to substitute the indexer with a dummy script.

Note that any file that is not indexed will not appear in SD card view.
So you will need a way to run the indexer after adding new files.