Update size is different every attempt?

Wolferl · 09-22-2009, 08:27 AM

Hi folks,

I am just new to SunriseXP (having read the FAQ

)
After AvantGo shut down, I wanted to have another possibility to have a TV guide on my Compaq iPAQ (PPC, Vade Mecum reader). I tried Plucker, but it is not able to handle cookies, which is important for www.klack-channels.de. So the search continued...
SunriseXP did the job well at first sight. Then I discovered, that some links were not downloaded. I increased link depth and retried. Now, even less stuff is created! The pdb file, originally 220kb, is now between 2k and 50kb, it changes on every "update" event I do.

Question 1:
How come it does not update all the stuff?

Question 2:
Is there a setting "maximum file size" or something which prevents SunriseXP downloading ALL the stuff?

I'm a bit lost...

Thanks for help,
Wolferl

Wolferl · 09-23-2009, 07:43 AM

I've done a bit of research and found out that SunriseXP gets errors when retrieving some of the HTML files (notHandled: 403 Forbidden).

It seems there is a problem requesting the files too quickly.

Is there a setting to slow spidering down a bit? Or re-fetching the HTML if it fails?

Cheers,
Wolfgang

Laurens · 09-23-2009, 10:13 AM

No, there is no setting to configure spidering speed. You might want to limit the max. number of connections to get around this. Unfortunately, this is a global setting that affects all sites.

Wolferl · 09-23-2009, 11:24 AM

Hi Laurens,

Quote:

Originally Posted by Laurens

No, there is no setting to configure spidering speed. You might want to limit the max. number of connections to get around this.

You mean "View -> Preferences -> Network tab -> "maximum active updates"?
I already set this to "1". Unfortunately, Sunrise runs two requests at the same time then. There seems to be an "off by one" bug...

But anyways, that does not really solve the problem:
Klack-channels.de seems to block access with a "403 forbidden" if one retrieves pages too quickly.
I tried to use a special proxy software that allows throtteling (Charles Web Debugging Proxy), and using this, it was possible to get the whole thing spidered without any errors. It just takes 20 minutes (only 600kb file size)...and that proxy is not freeware.

If there would be configurable delay "delay before request" in seconds, that would be perfect.

Cheers,
Wolferl

Laurens · 09-23-2009, 05:27 PM

There's not much I can do for you, unfortunately, since I'm no longer maintaining the application. You might want to look around for some other (hopefully free) proxy that can handles bandwidth throttling.

Wolferl · 09-24-2009, 05:32 AM

Hi Laurens,

Quote:

Originally Posted by Laurens

There's not much I can do for you, unfortunately, since I'm no longer maintaining the application.

Since I have lots of programming experience...what do you think about doing this modification myself? I have searched around for the source, but wasn't successful...

Cheers,
Wolferl

Laurens · 09-25-2009, 09:43 AM

Quote:

Originally Posted by Wolferl

Hi Laurens,

Since I have lots of programming experience...what do you think about doing this modification myself? I have searched around for the source, but wasn't successful...

Cheers,
Wolferl

SunriseXP (C++) is unfortunately not open-source. It uses some closed-source libraries, preventing its release as an open-source project.

You might want to consider switching to the older Java version, which is open-source.

http://sourceforge.net/projects/sunrisexp/

Wolferl · 09-26-2009, 06:04 AM

Hi Laurens,

Quote:

Originally Posted by Laurens

SunriseXP (C++) is unfortunately not open-source. It uses some closed-source libraries, preventing its release as an open-source project.

Oh, too bad

Quote:

Originally Posted by Laurens

You might want to consider switching to the older Java version, which is open-source.
http://sourceforge.net/projects/sunrisexp/

OK, I'll stick with this one.

Thanks for your help!

Wolferl

09-22-2009, 08:27 AM	#1
Wolferl Junior Member Posts: 6 Karma: 10 Join Date: Sep 2009 Device: Compaq iPAQ	Update size is different every attempt? Hi folks, I am just new to SunriseXP (having read the FAQ ) After AvantGo shut down, I wanted to have another possibility to have a TV guide on my Compaq iPAQ (PPC, Vade Mecum reader). I tried Plucker, but it is not able to handle cookies, which is important for www.klack-channels.de. So the search continued... SunriseXP did the job well at first sight. Then I discovered, that some links were not downloaded. I increased link depth and retried. Now, even less stuff is created! The pdb file, originally 220kb, is now between 2k and 50kb, it changes on every "update" event I do. Question 1: How come it does not update all the stuff? Question 2: Is there a setting "maximum file size" or something which prevents SunriseXP downloading ALL the stuff? I'm a bit lost... Thanks for help, Wolferl

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Attempt to update JBL - REDUX	steven522	Ectaco jetBook	1	08-24-2010 11:57 PM
anyone else toughing out font size while waiting for firmware update?	BBel	Kobo Reader	33	06-27-2010 11:08 PM
Evince update attempt	tjzeeman	iRex	5	02-07-2010 02:03 PM
My first attempt at writing an original...	JanG	Writers' Corner	13	08-11-2009 06:36 PM
My first upload attempt	lilac_jive	Workshop	1	01-17-2009 11:46 PM

09-23-2009, 07:43 AM	#2
Wolferl Junior Member Posts: 6 Karma: 10 Join Date: Sep 2009 Device: Compaq iPAQ	I've done a bit of research and found out that SunriseXP gets errors when retrieving some of the HTML files (notHandled: 403 Forbidden). It seems there is a problem requesting the files too quickly. Is there a setting to slow spidering down a bit? Or re-fetching the HTML if it fails? Cheers, Wolfgang

09-23-2009, 10:13 AM	#3
Laurens Jah Blessed Posts: 1,295 Karma: 1373 Join Date: Apr 2003 Location: The Netherlands Device: iPod Touch	No, there is no setting to configure spidering speed. You might want to limit the max. number of connections to get around this. Unfortunately, this is a global setting that affects all sites.

09-23-2009, 05:27 PM	#5
Laurens Jah Blessed Posts: 1,295 Karma: 1373 Join Date: Apr 2003 Location: The Netherlands Device: iPod Touch	There's not much I can do for you, unfortunately, since I'm no longer maintaining the application. You might want to look around for some other (hopefully free) proxy that can handles bandwidth throttling.