Handling a huge number of files on the SD Card

[rvs]

Originally Posted by Viacheslav

I was following this thread and it was confirmed several times that FAT slows down file browsing and is suboptimal. What about switching to some other Linux FS e.g. ReiserFS? It was tooted as being fast with many small files. Besides, an Ext* formatted new empty HDD has noticeably less free space than the same HDD with Reiser, because Ext needs more space for system data.

This is perhaps an interesting article on this question. According to the article ext4 is probably the fastest.

I don't think the DR supports ext4 right now, but perhaps this could be added? The SD-card would wear down faster though because of the journaling.

[yuri_b]

I did some analyse for queries:
Actual query is:
SELECT filename, directory_path, is_directory, file_type, file_size, file_time_modified, title, author, thumb_data_small FROM file_metadata AS m LEFT JOIN thumbnails AS t ON m.file_id = t.file_id WHERE tag = 'book' ORDER BY sort_priority DESC, (CASE WHEN title IS NOT NULL THEN 0 ELSE 1 END), title COLLATE IREX_CASE_INSENSITIVE ASC, filename COLLATE IREX_CASE_INSENSITIVE ASC, directory_path COLLATE IREX_CASE_INSENSITIVE ASC;


result of SQL analise of this query is:

SEARCH TABLE file_metadata AS m USING AUTOMATIC COVERING INDEX (tag=?)
SEARCH TABLE thumbnails AS t USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
USE TEMP B-TREE FOR ORDER BY


What I say?
Two search was use not direct indexing and building temporary tree in memory or on disk.
Very bad.

BTW the same for indexed table . Indexes that I added was wrong !!!

[Iñigo]

Originally Posted by Viacheslav

I was following this thread and it was confirmed several times that FAT slows down file browsing and is suboptimal. What about switching to some other Linux FS e.g. ReiserFS? It was tooted as being fast with many small files. Besides, an Ext* formatted new empty HDD has noticeably less free space than the same HDD with Reiser, because Ext needs more space for system data.

reiserfs is not supported by DR linux kernel, your best bet should be ext2 or maybe ext3.

root@qemuarm:/media/mmcblk0p1# cat /etc/filesystems 
minix
fat
vfat
ext2
ext3

In my tests an 4GB ext3 formatted sd card has ~130MB less space available than with vfat.

[rvs]

Originally Posted by Iñigo

reiserfs is not supported by DR linux kernel, your best bet should be ext2 or maybe ext3.

ext2 and ext3 aren't supported either (at the moment)...

[Iñigo]

Originally Posted by rvs

ext2 and ext3 aren't supported either (at the moment)...

aren't DR internal partitions ext2 file systems?

root@qemuarm:/media/mmcblk0p1# mount 
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
/dev/root on /dev/.static/dev type ext2 (rw)
tmpfs on /dev type tmpfs (rw)
devpts on /dev/pts type devpts (rw)
tmpfs on /var type tmpfs (rw)
tmpfs on /media/ram type tmpfs (rw)

Also if you take a look at /etc/filesystems, there appear the file systems supported by the kernel.

Iñigo

[rvs]

The qemu-image uses ext2, but the DR uses JFFS2 for the internal filesystem.

If you look at the kernel config file, you can also see support for ext2/3/4 is not set. The qemu uses a different kernel than the DR.

[Iñigo]

Originally Posted by rvs

The qemu-image uses ext2, but the DR uses JFFS2 for the internal filesystem.

If you look at the kernel config file, you can also see support for ext2/3/4 is not set.

Didn't know it. Thanks for the correction.

[Iñigo]

It's working

There is a lot of work to be done yet, but it works.
I'm including some screenshots and some code (patch against r49 of https://bitbucket.org/inigoserna/dr800, remember to do "hg udpdate -C dr800+ -r49" after cloning the repository).

Could anyone check for memory leaks please? Lot of time since I used to code in C...

CTB: covers in file system instead of global.db
* some comments about the code
- the basic idea is to keep global.db free of book covers, they will be in the file system SD_card/System/.icons/medium/XX/asdfgdgdfh.png
- the original method store all the pixbufs (images) for current view loaded from the db in the model, here we get from file system on demand
- it should be a bit slower than original method in case of few books in the view, but this method use less memory, the difference is huge with lots of books so then it should be much faster too
- id is not stored in the model, so we can't use id % 256 to distribute the covers over the covers root directory
I'm using the first 2 chars of the md5_hex_digest on the fullpath of the book
- erGtkIconView needs to be patched in order to get the images as needed on runtime instead of retrieving from db (look previous screenshots in this thread)
- it handles "system" icons perfectly except a few (see below) => screenshots: test10, test11
- IT WORKS!!! => screenshot: test13: image showing the billiard ball is the presumed cover loaded from the file system

TODO:
* this PoC only works with iconview by now, it needs some code for contentview as well => very easy
* "fix" old code which uses model store
- remove pixbuf column and references to it
- don't query db for thumbnails
* remaining special icons not managed by now
- settings apps => screenshots: test12
- shorcuts?
* we use a cache, a hash table
- i think it stores some entries repeated -> glib's g_str_equal bug?
- clean contents when changedir, change_view
- destroy at end
* write a tool (in vala) to move images from db to fs, remove them from db and do a vacuum
* update other programs to copy new icons to fs instead of db
- mdbindex
- cover from uds
* test in the real DR with lots of books and compare

Kind regard,
Iñigo

[Mackx]

Hi Iñigo,

Why didn't you change filemodel.c: load_items_in_model()? The
for loop:

    for (row=g_item_offset; row < (g_item_offset + items_per_page) && row < g_total_items ; row++)

only fills the gtk_list_store with one-page of icons. The line

        GdkPixbuf *thumbnail = get_thumbnail(cell);

should then only be changed to load the thumbnail from file?

If there is no image in global.db you can check if there is one in the filesystem. This would then work for the settings/global.db also I guess?

I haven't looked at this in much detail so I might be missing some details? Do the Delete-mode overlays work?

[Mackx]

Originally Posted by Iñigo

* update other programs to copy new icons to fs instead of db
- mdbindex
- cover from uds

The function set_file_metadata_impl() in ermetadb_file.c seems to be a good candidate. If I remember well from some past logging UDS will also use this function to update thumbnails (mdbindex used UDS to update thumbnails). The function could be changed to not execute the sql_t-query, but use the data to write images into the filesystem.

[Iñigo]

Originally Posted by Mackx

The
for loop:

    for (row=g_item_offset; row < (g_item_offset + items_per_page) && row < g_total_items ; row++)

only fills the gtk_list_store with one-page of icons.

Ouchhhh!!! silly me... I hadn't noticed this part
I was studying load_items_in_model just now. I thought ctb populated the model with all the results for the view... but not... so no need to get the pixbufs on runtime... so no need to change so much code, even no need to patch erGtkViewIcon.

Originally Posted by Mackx

Do the Delete-mode overlays work?

CTB code was almost finished except ovelays and shortcuts that needed some modifications to work with the new code.

Over 20 hours of hard work...

Ok, I'll create a branch with these code as a monument/example of my silliness.

Anyway, needed changes will be much easier.

[Mackx]

I will be resetting my brain tomorrow, glad I told you today...

[Mackx]

Originally Posted by Iñigo

- don't query db for thumbnails

If keeping thumbnail in the query is not generating too much extra delay it might be better to keep it in, to handle the global.db from settings. Or maybe it is possible to leave thumbnail from the query in filemodel.c:create_query() in case it is not the SETTINGS_VIEW? Although the delay might be caused more by the JOIN in ermetadb_file.c:global_select_common() or the JOIN is removed if the SQL-query is optimised before executing... my database knowledge is not good enough for that.

[Iñigo]

Originally Posted by Mackx

If keeping thumbnail in the query is not generating too much extra delay it might be better to keep it in, to handle the global.db from settings. Or maybe it is possible to leave thumbnail from the query in filemodel.c:create_query() in case it is not the SETTINGS_VIEW? Although the delay might be caused more by the JOIN in ermetadb_file.c:global_select_common() or the JOIN is removed if the SQL-query is optimised before executing... my database knowledge is not good enough for that.

I don't think it has much sense to keep thumbnail in the query as the benefits will come from not having any cover in global.db

Patch is quite simple now. See attached file.
It works without problems in the emulator.

Next steps:
1. write a tool (in vala) to copy images from db to fs, remove them from db and do a vacuum
2. test in the real DR with lots of books and compare
3. update other programs to copy new icons to fs instead of db
- mdbindex
- cover from uds

Btw, have a nice party tomorrow
I have a cooking day

[Mackx]

Originally Posted by Iñigo

I don't think it has much sense to keep thumbnail in the query as the benefits will come from not having any cover in global.db

I thought it might make handling settings and icons for (application) shortcuts and folders easier. But only if the penalty of leaving thumbnail in the query is not too high.

(Also have a lot of fun today.)