06-23-2011, 09:23 AM | #16 | |
Guru
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
|
Quote:
Code:
SD_card/System/.covers medium/ 00 01 .. FF/ id.png other_id.png ... small/ [similar to medium] If 256 subdirectories are not enough we could use a bigger number. In order to make file loading as fast as possible we have to think in a map function that equally distributes covers in the 256 subdirectories. I can think of "id mod_of_256" (id % 256). Btw, does % operator exist in C? |
|
06-23-2011, 09:41 AM | #17 | |
Guru
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
|
Quote:
But I think that 150k files is far from DR devices capacity, we can't design and write code to support that huge numbers. Even I don't know any other user than you with more than 5k files. I'm not a professional developer as my daily job, it's only a hobby for me, but I've been writing code since almost 30 years ago. One of my current (halted) projects is a book manager. There I was considering to store covers on the db (SQLite) or not. After some tests I finally I preferred to keep them in the file system. I have over 55k files and my machine is a linux AMD64x2 with 6GB RAM, file system is ext3, much powerful than the DRs. And here, on the DRs, maybe SQLite is enough to handle large number of entries, but both CPU and RAM are really not as powerful as we would need. I think in cases like yours is better to use a plain file system browser, not a database-based one. Iņigo |
|
Advert | |
|
06-23-2011, 09:58 AM | #18 | ||
Connoisseur
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
|
Quote:
150K I took from Mack mail: Quote:
And add indexes is job for 5 minutes, a much simpler then change a way all system works Lets checks maybe it would be enough? Secondly: Limit cbt to 2-3 page also much easy then change all I'm not saying "Don't do this.". I say lets check simpler ways to fix our problem. Our time not flexible and the less we will spend for each problem then more we will do. Believe me I also would like to store on 1 SD card all my library, and I have a lot of books, the problem you know Sincerely Yura |
||
06-23-2011, 10:34 AM | #19 | |
Guru
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
|
Quote:
I'm not an expert on DB either, but I think the new indexes can be added now and force a reindex of the db. Could you post the concrete modifications to the DB? I can run them on my computer after copying global.db from the DR800. So we can test the results. As a second step I will investigate about moving covers to outside the DB. |
|
06-23-2011, 12:10 PM | #20 | |
Connoisseur
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
|
Quote:
1) copy global.db in my PC 2) I have sqlite3.exe already so I just execute those commands: 3) sqlite3.exe global.db 4) in opened sqlite shell enter those commands: create index filename_index on file_metadata(filename,tag); create index title_index on file_metadata(title,tag); create index author_index on file_metadata(author,tag); create index type_index on file_metadata(file_type,tag); create index size_index on file_metadata(file_size,tag); create index date_added_index on file_metadata(file_time_added,tag); create index time_viewed_index on file_metadata(file_time_lastread,tag); 5) enter: .quit to exit 6) copy global.db back to device. Right now I don't have all books on my IREX, so find backup and restore those ~3000 files SD will take a while Moreover, as I sad, in case missed global.db IREX copy one empty from some place, so we can just generate one empty with all indexes, and place it as template on this location. Sincerely Yura |
|
Advert | |
|
06-23-2011, 12:51 PM | #21 | |
Guru
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
|
Quote:
|
|
06-23-2011, 12:56 PM | #22 |
Connoisseur
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
|
What speed?
Speed of indexing? or speed of opening page in file view? Indexing should speed up browsing dbase |
06-23-2011, 01:08 PM | #23 |
Guru
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
|
|
06-23-2011, 01:13 PM | #24 |
Connoisseur
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
|
If the speed not increased I see 2 explanations:
1) sql not uses this index, we created wrong index 2) there is additional place that slows all down. |
06-23-2011, 04:21 PM | #25 |
Guru
Posts: 999
Karma: 19985
Join Date: Dec 2008
Location: Netherlands
Device: iRex DR1000S
|
Looking at libermetadb I see the following is used (when sorting on filename):
Code:
ORDER BY sort_priority DESC, (CASE WHEN filename IS NOT NULL THEN 0 ELSE 1 END), filename COLLATE IREX_CASE_INSENSITIVE, directory_path COLLATE IREX_CASE_INSENSITIVE |
06-23-2011, 04:27 PM | #26 |
Guru
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
|
no words here
|
06-23-2011, 04:55 PM | #27 |
Connoisseur
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
|
Sicerely Yura |
06-23-2011, 05:02 PM | #28 |
Connoisseur
Posts: 71
Karma: 592
Join Date: Aug 2010
Device: irex dr800sg DR1000S
|
|
06-24-2011, 04:47 AM | #29 |
Guru
Posts: 730
Karma: 72743
Join Date: Feb 2008
Location: Here or there
Device: iRex iLiad, iRex DR800S. K4NT. Kobo Aura, Aura One, Libra 2.
|
ok, it was a bug or incomplete GtkIconView functionality in erGtkIconView implementation.
Now, after 7 hours of hard work, it looks nicer ;-) |
06-24-2011, 06:31 AM | #30 |
Connoisseur
Posts: 59
Karma: 136
Join Date: Dec 2008
Device: iRex DR1000
|
I was following this thread and it was confirmed several times that FAT slows down file browsing and is suboptimal. What about switching to some other Linux FS e.g. ReiserFS? It was tooted as being fast with many small files. Besides, an Ext* formatted new empty HDD has noticeably less free space than the same HDD with Reiser, because Ext needs more space for system data.
Last edited by Viacheslav; 06-24-2011 at 06:33 AM. |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Mobiperl Mobiperl - Perl tools for handling MobiPocket files | tompe | Kindle Formats | 890 | 01-17-2021 06:16 PM |
Large number of books on memory card | pwalker8 | Sony Reader | 8 | 03-24-2009 02:20 PM |
huge pdf files on iliad | yolle | iRex | 7 | 08-23-2008 10:59 AM |
Fictionwise Handling Fee Against Credit Card Terms of Service? | Gideon | News | 9 | 08-15-2008 02:13 PM |
Huge RTF files after conversion | TheMadBrewer | Sony Reader | 2 | 01-01-2007 02:55 AM |