Progress Sync across different platforms

[ilovejedd]

Finally got around to installing KOReader on my Kobos (Aura HD and Glo HD).

So far so good except for progress sync. Syncing works fine between PW3 & PW4 and between Aura HD & Glo HD. Alas, syncing between Kobo & Kindle isn't working.

How does progress sync work? The wiki states the following so I assumed it was going by checksum or something. Given it won't sync progress on identical files between Kobo & Kindle, it appears it's using some other identifier that's not intrinsic to the epub. The epubs even have the same path and filename except for obvious reasons the Kindle's on /mnt/us/ and the Kobo's on /mnt/onboard/.

This plugin does not send file name or any file content to the koreader sync server. And all data transferred between the "Progress sync" plugin and the sync server are secured by HTTPS (Hypertext Transfer Protocol Secure) connections.

[NiLuJe]

There's possibly a bit of hashing involved, but take that with a grain of salt, as I'm extremely unfamiliar with that bit of code.

Are the actual files identical on your end? How do you transfer them? (Might be a case of Calibre using a different output template?).

[ilovejedd]

Originally Posted by NiLuJe

There's possibly a bit of hashing involved, but take that with a grain of salt, as I'm extremely unfamiliar with that bit of code.

Are the actual files identical on your end? How do you transfer them? (Might be a case of Calibre using a different output template?).

Files are bit for bit identical. I was able to trace the digest function used.

Spoiler:

-- calculate partial digest of the document and store in its docsettings to avoid document saving
-- feature to change its checksum.
--
-- To the calculating mechanism itself.
-- since only PDF documents could be modified by KOReader by appending data
-- at the end of the files when highlighting, we use a non-even sampling
-- algorithm which samples with larger weight at file head and much smaller
-- weight at file tail, thus reduces the probability that appended data may change
-- the digest value.
-- Note that if PDF file size is around 1024, 4096, 16384, 65536, 262144
-- 1048576, 4194304, 16777216, 67108864, 268435456 or 1073741824, appending data
-- by highlighting in KOReader may change the digest value.
function Document:fastDigest(docsettings)
    if not self.file then return end
    local file = io.open(self.file, 'rb')
    if file then
        local tmp_docsettings = false
        if not docsettings then -- if not provided, open/create it
            docsettings = require("docsettings"):open(self.file)
            tmp_docsettings = true
        end
        local result = docsettings:readSetting("partial_md5_checksum")
        if not result then
            logger.dbg("computing and storing partial_md5_checksum")
            local bit = require("bit")
            local md5 = require("ffi/MD5")
            local lshift = bit.lshift
            local step, size = 1024, 1024
            local m = md5.new()
            for i = -1, 10 do
                file:seek("set", lshift(step, 2*i))
                local sample = file:read(size)
                if sample then
                    m:update(sample)
                else
                    break
                end
            end
            result = m:sum()
            docsettings:saveSetting("partial_md5_checksum", result)
        end
        if tmp_docsettings then
            docsettings:close()
        end
        file:close()
        return result
    end
end

Looks like KOSync uses the partial_md5_checksum as identifier for syncing. My problem was I just overwrote fanfics with updated versions leading to wrong partial_md5_checksum in metadata.epub.lua.

I deleted the sidecar folder forcing KOReader to recalculate the md5 sum and sync worked.