Pocketbook Annotations Viewer and Tools (AVATeR), need testing

[wold]

A beta is attacheed, compiled with Debian Buster 10.12. Figured that next to glibc, other libs may also be outdated.

Mainly use it for testing Definitive versions will follow in the next week or so. Some workarounds were needed for the older qt5.11 and libzip, that I'll comment on another time. Docs aren't included, and a log fix was sneaked in, aw well... Had fun fixing it though.

Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.0), libqt5core5a (>= 5.11.0~rc1), libqt5gui5 (>= 5.8.0), libqt5network5 (>= 5.0.2), libqt5sql5 (>= 5.0.2), libqt5widgets5 (>= 5.4.0), libstdc++6 (>= 6), libudev1 (>= 183), libzip4 (>= 1.3.0)

ps: libqt5gui5-gles isn't included yet
ps2. better do a purge when removing, as the dir permissions were fixed.

[Huwaetzel]

Originally Posted by wold

A beta is attacheed, compiled with Debian Buster 10.12. Figured that next to glibc, other libs may also be outdated.

Mainly use it for testing Definitive versions will follow in the next week or so. Some workarounds were needed for the older qt5.11 and libzip, that I'll comment on another time. Docs aren't included, and a log fix was sneaked in, aw well... Had fun fixing it though.

Depends: libc6 (>= 2.14), libgcc1 (>= 1:3.0), libqt5core5a (>= 5.11.0~rc1), libqt5gui5 (>= 5.8.0), libqt5network5 (>= 5.0.2), libqt5sql5 (>= 5.0.2), libqt5widgets5 (>= 5.4.0), libstdc++6 (>= 6), libudev1 (>= 183), libzip4 (>= 1.3.0)

ps: libqt5gui5-gles isn't included yet
ps2. better do a purge when removing, as the dir permissions were fixed.

I had fun testing it

Hi wold, did install without any hick-ups on my debian buster installation. I did some testing and it looks like the functionality is the same like in the Ubuntu or Windows version. Also here, manual mounting necessary, but after that reload and everything works .
As you are probably aware, the same .deb can not be installed in Ubuntu.

I still have this issue with the page numbers and I started to create myself a test book to find out how much page numbers and dates are off assuming the annotations on the device are managed correctly.

For me so far the best solution to manage my annotations. Great!

[wold]

Originally Posted by Huwaetzel

I had fun testing it

Hi wold, did install without any hick-ups on my debian buster installation. I did some testing and it looks like the functionality is the same like in the Ubuntu or Windows version. Also here, manual mounting necessary, but after that reload and everything works .

Windows and Linux use the same functions, except for (mostly) USB related functions and the monitor.

A somewhat iffy area, when thinking about it, are regressions due to older library versions.
For glibc this checked out OK (3 irrelevant changes); libzip seems OK too, but glibcxx reverts to older versions for quite a number of functions. It makes me curious mainly (improvements/bugfixes/etc?), but chasing all those release infos... Using build tests is the better way, but only key points are tested now. I'll reflect some more on this.

Mounting BTW is left to the OS/user. It could be done, but I'd rather stay away from that now.

Originally Posted by Huwaetzel

As you are probably aware, the same .deb can not be installed in Ubuntu.

On what Ubuntu version? 22.x tested OK here, save for a sandboxing warning.
Just to be sure: a right click > open with > installer may be necessary; otherwise use

sudo apt-get install -f ./pathto.deb

In any case, there will be a 'standard' release, and the backward compatible one. I'm OK with supporting one older version, but don't want to burden up-to-date users with any limitations or potential bugs.

I still have this issue with the page numbers and I started to create myself a test book to find out how much page numbers and dates are off assuming the annotations on the device are managed correctly.

The page numbers are a todo: they require a single increment for both PDF and epub that will get into 0.9.9. But you're welcome to do further testing. As for 'universal' page numbers (i.e. Kobo, etc), that's another topic all-together

The dates are spot on compared to my reader's overview. Haven't considered timezone corrections yet for display (raw data are unix epoch). Would you need that?

For me so far the best solution to manage my annotations. Great!

Nice to hear

I'll try to roll out 0.9.9 by next weekend, at worst the one after that.

[Huwaetzel]

Originally Posted by wold

I'll try to roll out 0.9.9 by next weekend, at worst the one after that.

Looking forward to it. Attached as a screenshot the status as of today (0.9.8.1)

The page number is a tricky one and because of the varying results based on reader or application probably difficult to get "right".

Regarding dates: I do not need correction of time-zones. My use-case is the following. When reading scientific books, especially if they are educational, I heavily annotate them in my reader and would like to know, when did I annotate what. Later on I am opening the same book e.g. via my iPad or in the browser in the cloud and may want to add another annotation. Then the date helps to identify in which context I did the new annotation because I can link the annotation to a date and with this e.g. to a new learning situation. For this the date is absolutely sufficient. time would be an overkill for me

While this can be done also via the pocketbook cloud, there are two limitations:
- I can not merge, what your application can do
- I can not keep a reference in my private cloud e.g. in Calibre-Web and would need to rely on a cloud environment I do not own and where I depend on a 3rd party to not take my rights away or do something with my content (as happened with books e.g. in a kindle.

Thanks for taking into account

[wold]

OK, so the current date info is sufficient for you, or do/did you suspect some aberration, etc? That was my concern mainly.

Ideally for you, date would also be added to HTML, which I'll think about. The HTML export was originally meant to be compact for easy consumption, with CSV for re-use of the data.

I'll extend the CSV to include more columns (as it should have): the raw GMT epoch date, date string, color text. See an example below. And probably add hex color as well, from row coloring being added to the HTML export (but that will have to wait).

DateRaw;Date;Author;Title;Page;Highlights;Notes;Color
1613764943;"19 feb. 2021 21:02";"James Mooney";"Myths of the Cherokee / Extract from the Nineteenth Annual Report of the Bureau of American Ethnology";"called Mobilian trade language, a corrupted Choctaw jargon formerly used as the medium of communication among all the tribes of the Gulf states";"";"Yellow;
1613765559;"19 feb. 2021 21:12";"James Mooney";"Myths of the Cherokee / Extract from the Nineteenth Annual Report of the Bureau of American Ethnology";"originated in a land toward the rising sun, where they had been placed by the command of “the four councils sent from above.” ";"";"Yellow";

The page number is a tricky one and because of the varying results based on reader or application probably difficult to get "right".

Tricky indeed. Speaking of which, epubcfi is a close but quirky alternative.
There's a branch that handles epubcfi, but it's still PB flavoured epubcfi, so won't match exactly, but gets close enough for navigating.

Anyway, time-wise now, it's easier for me to supply you a beta. You arrived a few weeks too late as work is ramping up.
I understood you mainly want both Linux versions, correct? (picking one for now would help)

[Huwaetzel]

Originally Posted by wold

OK, so the current date info is sufficient for you, or do/did you suspect some aberration, etc? That was my concern mainly.

I understood you mainly want both Linux versions, correct? (picking one for now would help)

No, I do not expect aberration in the database itself. That looked accurate for me so far.

Please do what ever fits your time and fun for the project - and thanks for your great work.

My main system is kubuntu 22.04 based and would therefore be my preferred test system. Happy to do further tests on windows or debian in case it would help your development work or makes it easier

[wold]

I should really stick to the fixes and not improve too much, as it draws out each release

Attached is a Debian 0.9.8.1 package for your Kubuntu 22.04 installation, and other recent Debian/Ubuntu/etc distros.
Fixed are the dates, page number (quickfix) and log location (/tmp/). Also debug logs were improved to fix filepath text-encoding (UTF8) issues. It's not 100% there yet (windows stdout for one), but close now. Archiving and uploading were tested extensively, now and earlier, so should work OK.

If you want to help with testing, you're more than welcome.
Just the Debian releases should suffice IMHO, as you use those the most.

BTW, any idea what PB firmware you're on? Any v6.x should be OK.

[Huwaetzel]

Originally Posted by wold

I should really stick to the fixes and not improve too much, as it draws out each release

Attached is a Debian 0.9.8.1 package for your Kubuntu 22.04 installation, and other recent Debian/Ubuntu/etc distros.
Fixed are the dates, page number (quickfix) and log location (/tmp/). Also debug logs were improved to fix filepath text-encoding (UTF8) issues. It's not 100% there yet (windows stdout for one), but close now. Archiving and uploading were tested extensively, now and earlier, so should work OK.

Installation works well. Attached find 2 documents
1. A screenshot from the actual installation
2. The html file created with the pocketbook own Notes export.

My firmware version is the latest with U741.6.5.1684

Let my know in case I can help further with testing

[wold]

Originally Posted by Huwaetzel

Installation works well. Attached find 2 documents
1. A screenshot from the actual installation
2. The html file created with the pocketbook own Notes export.

My firmware version is the latest with U741.6.5.1684

Let my know in case I can help further with testing

Thanks. Mhmm, in the DB there are various dates stored. Used now is 'time altered'. A sync could explain them being equal, or manual insertion - but that matters less now.

To properly investigate this the SQL rows are needed. I'll add a debug function to show those ASAP.
In the meantime, if you can run a SQL query on books.db, then see the code below. Ensure the original DB row number are shown under settings > annotation viewer. Added in now are those from the screenshot.

--for the item data
SELECT * FROM Items WHERE OID IN (690, 736, 751, 756, 758, 759)

--for the tag data
SELECT * FROM TAGS WHERE ItemID IN (690, 736, 751, 756, 758, 759)

The UI issues will get also get fixed.

[wold]

Attached is an updated .2 version for Kubuntu.

- edited date was swapped for the creation date.
- added linux help docs (see help > help)
- Ui fixes (except the menu arrow for the viewer menu. It should be disabled, so perhaps a Qt5 thing. For later)

There was one recurring crash that I couldn't pinpoint exactly, but was fixed by editing the settings file, removing the "table_columnsettings" line under [viewer]. It's a recurring cause for me, as I (can) switch between Qt5/6. But I'll mention it in case. Config is at ~/.config/avater.conf

[Huwaetzel]

Originally Posted by wold

Thanks. Mhmm, in the DB there are various dates stored. Used now is 'time altered'. A sync could explain them being equal, or manual insertion - but that matters less now.

To properly investigate this the SQL rows are needed. I'll add a debug function to show those ASAP.
In the meantime, if you can run a SQL query on books.db, then see the code below. Ensure the original DB row number are shown under settings > annotation viewer. Added in now are those from the screenshot.

--for the item data
SELECT * FROM Items WHERE OID IN (690, 736, 751, 756, 758, 759)

--for the tag data
SELECT * FROM TAGS WHERE ItemID IN (690, 736, 751, 756, 758, 759)

The UI issues will get also get fixed.

For the item data I get the following results:
OID ParentID TypeID State TimeAlt HashUUID
788 787 4 0 1654799165 74D7B0C5-57D2-5469-8CDF-C63CAC6E9C40
841 787 4 0 1654799165 062335FC-045A-503A-9AA9-6936EFBEDE1D
863 787 4 0 1654799165 7FC99316-843A-5125-AB56-DB4D6C1BB42B
869 787 4 0 1654799165 88010505-62D2-5842-9B6E-4EE54E95753C
871 787 4 0 1654799165 8a51847e-b269-4e9b-a99a-900d91471664
872 787 4 0 1654847212 919bc2f9-69e0-4d44-8c9d-fd911ca16d7d

For the tags data I get the results:
OID ItemID TagID Val TimeEdt
8636 788 101 {
"created" : 1654358266,
"updated" : "2022-06-04T15:58:23Z"
}
1654358304
8637 788 102 note 1654777571
8639 788 105 {
"text" : "magenta, page 2, 2022-06-04",
"updated" : "2022-06-09T12:26:10Z"
}
1654777571
8640 788 106 magenta 1654507566
8658 788 104 {"begin":"pbr:/page?page=2&offs=19#epubcfi(/6/6!/4/2[toc]/4/2[cover]/2/1:0)","end":"pbr:/page?page=2&offs=24#epubcfi(/6/6!/4/2[toc]/4/2[cover]/2/1:5)","text":"Cover","updated":"2022-06-06T09:24:58Z"} 1654799165
8632 841 102 note 1654777585
8634 841 105 {
"text" : "green, page 1, 2022-06-06",
"updated" : "2022-06-09T12:26:24Z"
}
1654777585
8635 841 106 green 1654507645
8656 841 101 {"anchor":"pbr:/page?page=1&offs=92#epubcfi(/6/4!/4/2/6/1:6)","created":1654507644,"updated":"2022-06-06T09:27:24Z"} 1654799165
8657 841 104 {"begin":"pbr:/page?page=1&offs=92#epubcfi(/6/4!/4/2/6/1:6)","end":"pbr:/page?page=1&offs=102#epubcfi(/6/4!/4/2/6/1:16)","text":"Consortium","updated":"2022-06-06T09:27:24Z"} 1654799165
8642 863 102 note 1654777556
8644 863 105 {
"text" : "yellow, page 3, 2022-06-07",
"updated" : "2022-06-09T12:25:55Z"
}
1654777556
8645 863 106 yellow 1654588179
8659 863 101 {"anchor":"pbr:/page?page=3&offs=185#epubcfi(/6/8!/4/2/4/6/1:5)","created":1654588178,"updated":"2022-06-07T07:49:38Z"} 1654799165
8660 863 104 {"begin":"pbr:/page?page=3&offs=185#epubcfi(/6/8!/4/2/4/6/1:5)","end":"pbr:/page?page=3&offs=196#epubcfi(/6/8!/4/2/4/6/1:16)","text":"publication","updated":"2022-06-07T07:49:38Z"} 1654799165
8647 869 102 note 1654777613
8649 869 105 {
"text" : "cyan, page 4, 2022-06-08",
"updated" : "2022-06-09T12:26:52Z"
}
1654777613
8650 869 106 cian 1654665052
8661 869 101 {"anchor":"pbr:/page?page=4&offs=643#epubcfi(/6/8!/4/2/10/10/1:126)","created":1654665051,"updated":"2022-06-08T05:10:51Z"} 1654799165
8662 869 104 {"begin":"pbr:/page?page=4&offs=643#epubcfi(/6/8!/4/2/10/10/1:126)","end":"pbr:/page?page=4&offs=651#epubcfi(/6/8!/4/2/10/10/1:134)","text":"complete","updated":"2022-06-08T05:10:51Z"} 1654799165
8652 871 102 note 1654777439
8654 871 105 {
"text" : "red, page 5, 2022-06-09",
"updated" : "2022-06-09T12:23:58Z"
}
1654777439
8655 871 106 red 1654777439
8663 871 101 {"anchor":"pbr:/page?page=5&offs=197#epubcfi(/6/10[xhtml-001]!/4/4[visual-010]/4/1:144)","created":1654777438,"updated":"2022-06-09T12:23:58Z"} 1654799165
8664 871 104 {"begin":"pbr:/page?page=5&offs=197#epubcfi(/6/10[xhtml-001]!/4/4[visual-010]/4/1:144)","end":"pbr:/page?page=5&offs=203#epubcfi(/6/10[xhtml-001]!/4/4[visual-010]/4/1:150)","text":"change","updated":"2022-06-09T12:23:58Z"} 1654799165
8668 872 102 note 1654846954
8670 872 105 {
"text" : "blue, page 6, 2022-06-10",
"updated" : "2022-06-10T07:42:34Z"
}
1654846954
8671 872 106 blue 1654846928
8672 872 101 {"anchor":"pbr:/page?page=7&offs=538#epubcfi(/6/12[xhtml-002]!/4/2[sample-text]/6/7:85)","created":1654846929,"updated":"2022-06-10T07:42:08Z"} 1654847212
8673 872 104 {"begin":"pbr:/page?page=7&offs=538#epubcfi(/6/12[xhtml-002]!/4/2[sample-text]/6/7:85)","end":"pbr:/page?page=7&offs=544#epubcfi(/6/12[xhtml-002]!/4/2[sample-text]/6/7:91)","text":"called","updated":"2022-06-10T07:42:08Z"} 1654847212

To get the right data I needed to replace for the queries the original ids as you mentioned.

I hope I could provide you with the the data you were looking for

[Huwaetzel]

Originally Posted by wold

Attached is an updated .2 version for Kubuntu.

- edited date was swapped for the creation date.
- added linux help docs (see help > help)
- Ui fixes (except the menu arrow for the viewer menu. It should be disabled, so perhaps a Qt5 thing. For later)

There was one recurring crash that I couldn't pinpoint exactly, but was fixed by editing the settings file, removing the "table_columnsettings" line under [viewer]. It's a recurring cause for me, as I (can) switch between Qt5/6. But I'll mention it in case. Config is at ~/.config/avater.conf

This version seems to be solving the date issue (screenshot attached. Thanks a lot. Thank you for also updating the documentation. Programming is always more fun, but documentation is core for a good solution to me.

I will dig a bit deeper into the page number issue. I have old books which have way to many annotations listed for the 1st page. I am not sure about the pattern yet (e.g. book not any more in the cloud, book/not downloaded, pdf/epub, ...).

I am already very happy with the version and will be waiting for the next bigger update. while waiting I will do some free testing to find out more

[wold]

Many thanks for checking. I'll get back on your 2nd reply later.

OID ParentID TypeID State TimeAlt HashUUID
788 787 4 0 1654799165 74D7B0C5-57D2-5469-8CDF-C63CAC6E9C40

OID ItemID TagID Val TimeEdt
8636 788 101 {
"created" : 1654358266,
"updated" : "2022-06-04T15:58:23Z"
}

Created is at 1654358266, altered at 1654799165 seconds. That's the cause. Altered is the same for the group: seems these are done in bulk (or a trigger causes this). With the possible advantage of grouping them later, perhaps.

For me all 3 dates are the same (don't edit), but I should have stuck to "created" (it's JSON though). I'll add in the other dates as an option later. If we're being serious about annotations

8658 788 104 {"begin":"pbr:/page?page=2&offs=19#epubcfi(/6/6!/4/2[toc]/4/2[cover]/2/1:0)","end":"pbr:/page?page=2&offs=24#epubcfi(/6/6!/4/2[toc]/4/2[cover]

See the "?page=" part: this is what the program and reader use. These should check out OK on the reader: open a book's menu, tap the page number in the far bottom, and enter this number. A few page turns still count as the same page (that's the offset).

For PDFs, page numbers are absolute, but for EPUBs (as these) it's relative... Now there are developments that implement page-breaks, but perhaps PB readers don't respect these (yet). Backward compatibility when syncing would also be an obstacle. (Unless you're referring to a number-written-in-text/header, or an external page number?)

Originally Posted by wold

It's a recurring cause for me, as I (can) switch between Qt5/6.

To add: actually you can, when syncing the datadir between Windows and Linux.
I'll read up on this, and update my QT6 version. They're at 6.3 by now...

Originally Posted by wold

removing the "table_columnsettings" line under [viewer].

BTW, this (re)stores the column size, order and some other tidbits that seem to break things.

[wold]

v0.9.8.3 was made available yesterday for Windows and Linux.
For Linux .3 fixes a minor issue with stored filepaths and special characters due to Qt5.

The coming weeks I've got some more time (though the weather is tempting...).
I'll try fix up the talked about additions, and the Buster build.

Thank you for also updating the documentation. Programming is always more fun, but documentation is core for a good solution to me.

It's often skipped, yes. Reckoned that if it could prevent a few questions, it was worth it. Time-wise, perhaps less so, but we're well past that point already.
The underlying API is undocumented, but it'll change a bit. FYI, AVATeR is actually the front-end, with a library-ish underlayer that manages the readers, annotations, etc. Ideally that would get opened up one day, but that's still future stuff now. It'll have to offer some added advantages, and require broader support/adaption. We'll see.

I will dig a bit deeper into the page number issue. I have old books which have way to many annotations listed for the 1st page. I am not sure about the pattern yet (e.g. book not any more in the cloud, book/not downloaded, pdf/epub, ...).

I'm curious It are probably highlights edited with the note app. IIRC, those loose their location data. So that's another bug (AKA "you're holding using it wrong"). Easy to solve as well.

I am already very happy with the version and will be waiting for the next bigger update. while waiting I will do some free testing to find out more

Nice to hear. I'm very grateful for your help so far. You mentioned 'free', likely as in 'as it goes'. I'll add that this program is very niche, with PB users being already a minority. If Kobo support is added, that could expand, but I don't really expect it to get large anytime soon, if ever. As for money, it could be sold. And I won't ask much, but I ain't too keen on all the effort with keys, online shop, etc. For an android app that's easier. But perhaps I'm too nice. Then again, the idea was to learn and get work, etc. Again, we'll see. So far, use it freely and enjoy it.

[wold]

And the Buster 'compatible' version is also available. The linux versions will probably stick with GCC/stdlib libstdc in the future.

The manual mounting is not an issue for me, in Linux the next step, in case a device is not detected is a manual mount. => for me an issue with lower priority.

Low prio for me as well. Showing unmounted devices would be something to consider - provided these can be styled properly (i.e. greyed out) to show they're unusable.