Automatically sync new fanfiction chapters

[Sculptor]

I use Calibre and CC mostly for reading fanfiction with the FanFiction Downloader plugin. If a story gets updated in Calibre, I have to manually redownload the story to get it to update on my device. It's not much of a problem, but if there are a lot of stories, it's a hassle to redownload each one individually. Download all doesn't work because the stories are already there on my device.

I have a custom column in Calibre which shows the number of chapters. The column does appear on CC for each story. I was wondering if it would be possible to add a feature to check for any changes in a custom column like that and if there are, to re-sync the stories from Calibre automatically when it connects as content server and checks for metadata.

[chaley]

Originally Posted by Sculptor

I use Calibre and CC mostly for reading fanfiction with the FanFiction Downloader plugin. If a story gets updated in Calibre, I have to manually redownload the story to get it to update on my device. It's not much of a problem, but if there are a lot of stories, it's a hassle to redownload each one individually. Download all doesn't work because the stories are already there on my device.

I have a custom column in Calibre which shows the number of chapters. The column does appear on CC for each story. I was wondering if it would be possible to add a feature to check for any changes in a custom column like that and if there are, to re-sync the stories from Calibre automatically when it connects as content server and checks for metadata.

This is a rather specialized need. I can't see many people wanting to do the same thing.

However, the more general need, re-downloading books that have changed, is certainly there and has been requested several times. Having calibre + CC optionally cooperate to determine if a particular format has changed wouldn't be terribly difficult. Unfortunately it might be slow, costing the time to look up the changed date of the file both in calibre and in CC, which could total to around 10 milliseconds per book (100 books per second). That is faster than doing it by hand, but it would add visible delay to the connection process.

Thinking about your request, it seems that you could get most of what you want using reading list. My assumption is that you maintain that chapter count column by hand. If when you change that column you add the book to a reading list that is synced with the device then you books would be transferred next time you connect as a wireless device. Ahh, I note that the fanfic downloader plugin already supports automatic maintenance of a reading list, eliminating that manual step (if you use the plugin).

Another way to do the same thing more manually would be to have a yes/no column that you set to "Yes" when you change one of the books. When you connect as a wireless device you would search for ondevice:true and #col:yes, select all the books and send them to the device.

[kaufman]

Originally Posted by chaley

However, the more general need, re-downloading books that have changed, is certainly there and has been requested several times. Having calibre + CC optionally cooperate to determine if a particular format has changed wouldn't be terribly difficult. Unfortunately it might be slow, costing the time to look up the changed date of the file both in calibre and in CC, which could total to around 10 milliseconds per book (100 books per second). That is faster than doing it by hand, but it would add visible delay to the connection process.

What if you didn't do this by default, by added a manual process. You could put "Books with updated files" under "Books with missing files". People wouldn't invoke it all the time, but it would be there when they needed it.

I don't run "Books with missing files" very often, but its not to do once in a while. "Books with updated files" would get run more often, but still not on every connection.

[chaley]

Originally Posted by kaufman

What if you didn't do this by default, by added a manual process. You could put "Books with updated files" under "Books with missing files". People wouldn't invoke it all the time, but it would be there when they needed it.

I don't run "Books with missing files" very often, but its not to do once in a while. "Books with updated files" would get run more often, but still not on every connection.

Interesting idea. The major hurdle to overcome is that when connected as a wireless device, calibre is in control. There is no (straightforward/easy) way for a device to ask calibre to do something. Instead, calibre asks devices to do things. I could add something to calibre to start this sync process, but I am not sure where I could put it. About the only place I can think of would be to add the process to the device menu, but it would need to be done so that it is not there unless the connected device is a wireless device.

Another way to do it would be for the user to indicate that s/he wants the processing by choosing a different "Connect" in CC, something like "As WD and sync books" in addition to "As wireless device" and "To content server". The problem here is the ever-present support question. People would see that menu line, want to know what it does, and contact us. To avoid this we could potentially add an option that enables that menu setting, explaining the feature in the text of the preference.

Hmmmm... I just realized that the performance of the feature wouldn't be as bad as I predicted. Unless you turn off the scan-on-connect option (and I suspect that very few people do) then CC is already in a position to get the book time stamps "for free" during the connection process. Calibre would need to check and compare the book format files but this would take less time than I predicted, probably around 200 to 300 books per second.

Needs thought ...

[kaufman]

Originally Posted by chaley

Hmmmm... I just realized that the performance of the feature wouldn't be as bad as I predicted. Unless you turn off the scan-on-connect option (and I suspect that very few people do) then CC is already in a position to get the book time stamps "for free" during the connection process. Calibre would need to check and compare the book format files but this would take less time than I predicted, probably around 200 to 300 books per second.

Needs thought ...

That could still be a lot for people who have a large library.

If you do that, how about adding a toggle to the menu that says "refresh all changed books on next connect".

You could even add an option to decide if that toggle would reset after the connect. People with small libraries might leave it on all the time, and people with large libraries would turn it on manually once in a while.

Or would this be a support nightmare?

[chaley]

Originally Posted by kaufman

That could still be a lot for people who have a large library.

If you do that, how about adding a toggle to the menu that says "refresh all changed books on next connect".

You could even add an option to decide if that toggle would reset after the connect. People with small libraries might leave it on all the time, and people with large libraries would turn it on manually once in a while.

These both devolve into a second option on the connect menu, one that might have an additional option to turn it on or off.

My feeling is that people who want this would pay the penalty, and people who don't care don't want to pay. That sounds like a preferences setting to me.

Or would this be a support nightmare?

Nightmare is too strong, but the word isn't totally inappropriate.

Under the assumption that whoever is answering the emails is actually working for a living, then the economics are easy to analyze. We get approx $2.10 from Google for a CC sale in the US (much less in Europe because of VAT). If we assume that we are worth at least $40K/year in wages (low for experienced developers), then "salary" is approx $20/hour. After taxes, that $2 pays for 4 minutes, which is less than the time it takes to read, understand, and reply to one email message, and that leaves no money to pay for development. Bottom line: if anything we do increases the number of support requests without increasing the number of sales by 2 to 3 times the support increase, adding the feature is a losing proposition. We have learned over the last 21 months that *no CC development we do increases sales*, a fact that creates a conundrum. What we do might prevent decrease of sales, but that is unquantifiable.

On the other hand, I am retired and CC is a hobby for me, meaning my time "costs" almost nothing other than reduction of the value of efforts of other young developers (and yes, that bothers me). I enjoy interacting with interesting people and I enjoy the challenges of development. My son and daughter-in-law aren't in the same situation. They need income to pay their mortgage, to eat, to save for starting a family, etc, meaning that they must be quite ruthless about how they spend their time.

[Sculptor]

Originally Posted by chaley

This is a rather specialized need. I can't see many people wanting to do the same thing.

Don't know about others but the fanfiction downloader plugin is very popular. In fact, I got calibre just for that. Fanfiction in general is getting very popular and there aren't many ways to download it properly to read on a portable device. And sitting on the PC and reading stories for hours is not really a good option. So there are many other people who uses Calibre for it I'm sure. Don't know how many uses CC though.

Originally Posted by chaley

However, the more general need, re-downloading books that have changed, is certainly there and has been requested several times. Having calibre + CC optionally cooperate to determine if a particular format has changed wouldn't be terribly difficult. Unfortunately it might be slow, costing the time to look up the changed date of the file both in calibre and in CC, which could total to around 10 milliseconds per book (100 books per second). That is faster than doing it by hand, but it would add visible delay to the connection process.

Why not add that in the option which would be turned off by default? That way, if someone wants the feature can use it knowing it will take time. Those who do not need it can ignore it and it won't affect them.

Originally Posted by chaley

Thinking about your request, it seems that you could get most of what you want using reading list. My assumption is that you maintain that chapter count column by hand. If when you change that column you add the book to a reading list that is synced with the device then you books would be transferred next time you connect as a wireless device. Ahh, I note that the fanfic downloader plugin already supports automatic maintenance of a reading list, eliminating that manual step (if you use the plugin).

Yes, the plugin automatically sets the chapter number and page count in their respective custom columns using reading list.

Originally Posted by chaley

Another way to do the same thing more manually would be to have a yes/no column that you set to "Yes" when you change one of the books. When you connect as a wireless device you would search for ondevice:true and #col:yes, select all the books and send them to the device.

How would that work for multiple updates though? Authors usually post one chapter at a time in (usually) random intervals. If I set yes for an update, I would have to set it as no after syncing. Seems more work than as it is right now.

Well, I see you and Kaufman came up with some good ideas. Would you still get a lot of support emails about it if it was added to the FAQ? I don't see many people asking what it does if the name is kept straightforward like 'Check for ebook updates' or something. If it is not too much trouble, I would love to see an option for this. It could possibly help other people too because like I said, fanfiction is getting very popular.

Thanks for your time.

[chaley]

Originally Posted by Sculptor

Why not add that in the option which would be turned off by default? That way, if someone wants the feature can use it knowing it will take time. Those who do not need it can ignore it and it won't affect them.

That is certainly one option, and one that I lean toward.

Yes, the plugin automatically sets the chapter number and page count in their respective custom columns using reading list.

Can't you use the auto-sync features of reading list to transfer changed books to your device?

Well, I see you and Kaufman came up with some good ideas. Would you still get a lot of support emails about it if it was added to the FAQ? I don't see many people asking what it does if the name is kept straightforward like 'Check for ebook updates' or something. If it is not too much trouble, I would love to see an option for this. It could possibly help other people too because like I said, fanfiction is getting very popular.

The issue is really that we can't put enough text onto a menu to explain what the words mean. For example, in your proposal:
- What is an ebook update? Is it metadata? Is it manual editing? Is it something like FFDL?
- What happens if an update is found?
- When is the check done?
and so on.

I realize that I am sounding very negative, which I regret. The idea of offering a way to update book formats under some conditions is a very good one. What we need to find are rules that are extremely easy to understand, a place to enable/explain those rules, a way to avoid increasing confusion (ergo frustration) when people are just starting with CC, and a way for people to discover the feature when they are ready for it.

My guess is that options in the Wireless Device Connection section would work: something like "Update book file in CC if the book file has changed in calibre" and "Update book file in CC if the metadata has changed in calibre". I am not yet convinced that the second is a good idea, although we have been asked for that more often than the first. People want the covers and the jackets in the book to be correct so they see latest info in their reading app, and resending the book on metadata change has the chance of doing that.

[kaufman]

Originally Posted by chaley

My guess is that options in the Wireless Device Connection section would work: something like "Update book file in CC if the book file has changed in calibre" and "Update book file in CC if the metadata has changed in calibre". I am not yet convinced that the second is a good idea, although we have been asked for that more often than the first. People want the covers and the jackets in the book to be correct so they see latest info in their reading app, and resending the book on metadata change has the chance of doing that.

What are your concerns about the second part being a good idea?

Count me in with the people who want the book resent if the metadata changes. I have been e-reading for about 15 years, and I have never once updated a book's content. Covers and other metadata all the time, but I figure the original author's did a better job with the content then I ever could.

Having said that, why would you split this into two options? Couldn't you just have one update command if the metadata or the book has changed?

[chaley]

Originally Posted by kaufman

What are your concerns about the second part being a good idea?

...

Having said that, why would you split this into two options? Couldn't you just have one update command if the metadata or the book has changed?

My concern is that send-on-metadata-change is not actually the best way to solve the user's problem, and in fact can create more problems than it solves. For example, what does "metadata changed" mean? In CC it means that the last mod date changed, which can happen when nothing changes in the metadata. One case: resetting the backup status. Another: manipulating a custom column's definition. A third: the date gets updated because the case of a tag changed. What we would immediately face is the need for some kind of criteria defining what the user considers to be important changes. That is the road to perdition.

Your question about splitting the two options is related. I actively do not want to resend books when metadata changes. I change metadata all the time for my own record keeping. I don't care if the metadata in the book is correct. I never look at it there, so resending the book is a waste of my time.

On the other hand, I have been known to run "polish" on a book to update the jacket (very infrequently, usually for my wife), or to reconvert the book to get better handling of scene breaks, spacing, or paragraph handling (rather more often). In this case the format itself has changed and I want the changes, which I know because I made them. Which leads to the "isn't the best way to solve the problem" assertion. If the user wants to change a book, the best place to do that is in calibre. There are so many tools to do so many things, some of which don't touch metadata. We got here by discussing one of them, the fanfiction downloader, which I think isn't required to change metadata. Triggering the refresh by a change in the format puts the process under the user's control while eliminating the need to remember to transfer the book at next connect. It is the "remembering" that is the real problem.

Bottom line: sending on format change is easier to explain, provides for user control over the update process, and doesn't create an immediate need for feature extension (the criteria).

[kaufman]

I think I have figured out why you and I have differing views on this.

I keep my entire library in CC. I want the CC view to be exactly the same as the Calibre view. For me, my books in CC are "My library" and I always want all the updates. If a change was worth making in Calibre, its worth seeing it in CC too.

I think you have said that you only keep a small part of your library online. Since you are seeing a subset that might change to a different subset, you don't really care if all the metadata always matches you Calibre library.

In any case, while I wouldn't mind if books always got updated on any change, I'm pretty good about doing this myself, so this is more of an academic discussion for me. I've got a couple of things I would much rather see you do first anyhow.

[chaley]

Originally Posted by kaufman

I think you have said that you only keep a small part of your library online. Since you are seeing a subset that might change to a different subset, you don't really care if all the metadata always matches you Calibre library.

I keep my entire library on my device.

I care a *lot* that the metadata is right in CC. What I don't care about is that it is right inside the book files.

In any case, while I wouldn't mind if books always got updated on any change, I'm pretty good about doing this myself, so this is more of an academic discussion for me. I've got a couple of things I would much rather see you do first anyhow.

Uhhh ... Auto-sort from the grouping list?

[kaufman]

Originally Posted by chaley

What I don't care about is that it is right inside the book files.

I like to think of CC as a backup of my book library. In case of a massive system failure, I like knowing I can recreate my library. Which is stupid, because I have very good backups, as well as my entire library in the cloud as well.

Originally Posted by chaley

Uhhh ... Auto-sort from the grouping list?

Exactly. How did you guess?

It's funny, but now that we have discussed the auto-sorting thing, it sticks at me every time I have to manually sort. I'll be very happy if/when you decide to make that addition.

[DoctorOhh]

Originally Posted by chaley

Hmmmm... I just realized that the performance of the feature wouldn't be as bad as I predicted. Unless you turn off the scan-on-connect option (and I suspect that very few people do) then CC is already in a position to get the book time stamps "for free" during the connection process.

I turn this option off since adding a ebook (outside of being a wireless device) to this folder would be extremely rare. What would you estimate the time hit be to leave this option checked? Also what pros are their in leaving this option checked that I might currently be overlooking?

Originally Posted by kaufman

I keep my entire library in CC. I want the CC view to be exactly the same as the Calibre view.

The CC view currently is synced to be exactly the same calibre.

Originally Posted by kaufman

For me, my books in CC are "My library" and I always want all the updates. If a change was worth making in Calibre, its worth seeing it in CC too.

I also have my entire library on my device and updating the books when metadata in calibre's library changes could be disastrous (time wise) because for calibre everything is metadata. Creating a custom column to help answer someone's question could kick off an entire update of all 10,000+ books on my library.

On the other hand if I actually choose to update the metadata in a book using the Polish or Modify Epub plugin the actual book will have changed (not just calibre's metadata and possibly under those circumstances I might wish to have the option to update the book on my device.

Chaley thanks for all of the work you do on this and kaufman thanks for providing well thought out input into CC's operation.

[chaley]

Originally Posted by DoctorOhh

I turn this option off since adding a ebook (outside of being a wireless device) to this folder would be extremely rare. What would you estimate the time hit be to leave this option checked? Also what pros are their in leaving this option checked that I might currently be overlooking?

I am thinking of putting the format mod date into the DB so there wouldn't be any ongoing penalty. On my Galaxy Nexus the scan runs at 500 books per second. If you N7 runs at that speed (I think it might be faster) then for you the first CC start after changing the DB schema would take around 20 seconds.

I don't see any pros for you in leaving it on. For most people it adds a bit of time to the connection cycle for little gain. We originally had it off and asked users if they wanted to turn it on, but changed that after we saw we were getting the dreaded support emails about "what is this?"