The Guardian 24 automatic download

[rio]

Hi,

I received my iliad this week and started playing a bit with it...
One thing I wanted was to download automatically the 24h pdf edition of the guardian (a UK newspaper), so I wrote a small script that do just that.

The script starts a network connection, delete any previous downloaded pdf, download the last online pdf from the guardian website, create a folder for the iliad, a manifest (to show an icon and the date of the download), and the stop the network connection.

To install this, download the attached zip, decompress it on your MMC card in newspaper, go in the Guardian24 folder, click on the "Guardian UK" item... and it should just work! (of course you need the shell access for that)

If you want to use it elsewhere, you need to edit the run.sh script, and modify the GUARDIAN_PATH variable to point to the location of your choice.

Note that adapting the script to any other resource is trivial, so have fun.

Here is the script itself:

#!/bin/sh
export GUARDIAN_PATH=/media/card/newspapers/Guardian24/
/usr/bin/wired.sh start
/usr/bin/wireless.sh start
rm -rf $GUARDIAN_PATH/Guardian.pdf
mkdir $GUARDIAN_PATH/Guardian.pdf
echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h edition</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>Guardian.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/Guardian.pdf/manifest.xml
wget -O $GUARDIAN_PATH/Guardian.pdf/Guardian.pdf http://download.guardian.co.uk/g24/World.pdf
cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/Guardian.pdf/icon.png
/usr/bin/wired.sh stop
/usr/bin/wireless.sh stop

[m-reader]

Well done mate, if there ever was a reason to install shell access on my iLiad, this is it!

[carandol]

This is excellent idea, I've been wondering how to download Guardian 24, but lack the programming knowledge...

However, I've given it a go, and there's something not working for me. I changed the path to /media/cf/newspapers/Guardian24/ to suit my iLiad. It seems to go online, and downloads a file with a correct datestamp, but when I try to load the file, the progress bar flashes and then stops with no result. Looking at the file from my linux PC, it shows as an empty file. I've checked my wifi connection, and it's definitely working, I've checked the Guardian 24 file online and it's definitely up, so I don't know what else to try. Any help much appreciated!

[rio]

Originally Posted by carandol

However, I've given it a go, and there's something not working for me. I changed the path to /media/cf/newspapers/Guardian24/ to suit my iLiad. It seems to go online, and downloads a file with a correct datestamp, but when I try to load the file, the progress bar flashes and then stops with no result. Looking at the file from my linux PC, it shows as an empty file. I've checked my wifi connection, and it's definitely working, I've checked the Guardian 24 file online and it's definitely up, so I don't know what else to try. Any help much appreciated!

What you describe happens when the file is not downloaded -- the folder and all the rest will still be created by the script, so it /looks/ like it's fine, but there's no content...

My guess is that the iliad did not manage to get an internet connection. I only tried with a wired connection, maybe that's a reason (although it should have worked anyway...).

Are you sure you configured your network connection on the iliad ? If you open mrxvt and type ifconfig what is shown ? and if you then manualltry try to initiate the internet connection (type "/usr/bin/wired.sh start" or /usr/bin/wireless.sh start) what is displayed ?

[ksira]

Hi, I hope I can help, but I just started playing around with wireless access.

I think you might try and specify the ssid and security when using wireless.sh

edit the run.sh line to:

$/usr/bin/wireless.sh start [network ssid] [encyrption] [key]

BTW the wired version worked fine for me.

[carandol]

Afraid I don't have mrxvt installed (unless it comes as standard and I've not found it?). But iNewstand is definitely connecting, so it can't be a bad connection. I'll try adding ksira's line to the script, and if/when I have time, I'll look up mrxvt.

[carandol]

Just tried ksira's line of code, and now it just sits there with the light flashing and the progress bars cycling. Perhaps I got the line wrong. Should it have been:

$/usr/bin/wireless.sh start mynetwork wep 12345678

or

$/usr/bin/wireless.sh start ssid mynetwork encryption wep key 12345678

?

[cen]

Originally Posted by carandol

Just tried ksira's line of code, and now it just sits there with the light flashing and the progress bars cycling. Perhaps I got the line wrong. Should it have been:

$/usr/bin/wireless.sh start mynetwork wep 12345678

or

$/usr/bin/wireless.sh start ssid mynetwork encryption wep key 12345678

?

I believe if you are trying to put this into rio's script you will need to drop the $ and it should be:

/usr/bin/wireless.sh start mynetwork wep 12345678

[carandol]

Hmm, still no joy.

[cen]

Originally Posted by carandol

Hmm, still no joy.

Does it work if you try it wired? This would give us an idea if it's a problem with the script or the wifi connection (via the script)

[cen]

Problem solved, assuming you had the same issue as me. The script fails if /media/cf/newspapers/Guardian24/ doesn't exist (and I never bothered to create it as I put the script elsewhere )

Updated code to create the root newspaper dir if it doesnt exist (mkdir -p)
To get the wifi working:
uncomment the following lines
#/usr/bin/wireless.sh start <SSID> <ENC_TYPE> <PASS>
#/usr/bin/wireless.sh stop

and replace <...> with your wifi details.

#!/bin/sh
export GUARDIAN_PATH=/media/cf/newspapers/Guardian24/
/usr/bin/wired.sh start
#/usr/bin/wireless.sh start <SSID> <ENC_TYPE> <PASS>
rm -rf $GUARDIAN_PATH/Guardian.pdf
mkdir -p $GUARDIAN_PATH/Guardian.pdf
echo "<?xml version=\"1.0\" encoding=\"utf-8\"?><package><metadata><dc-metadata><Title>The Guardian 24h edition</Title><Description>Downloaded: `echo \`date\``</Description></dc-metadata><y-metadata><startpage>Guardian.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>" > $GUARDIAN_PATH/Guardian.pdf/manifest.xml
wget -O $GUARDIAN_PATH/Guardian.pdf/Guardian.pdf http://download.guardian.co.uk/g24/World.pdf
cp $GUARDIAN_PATH/get-guardian/icon.png $GUARDIAN_PATH/Guardian.pdf/icon.png
/usr/bin/wired.sh stop
#/usr/bin/wireless.sh stop

edit: sort of works, gets to 28% download of the PDF then stalls and craps out (but the PDF is created). YMMV

[carandol]

Still not having any luck. I can't test it wired, because I don't have a network cable. (I mean, what's the point of having a wifi network, then buying wires? ) Has anyone else got this to work with wifi?

[cen]

Not sure how familiar you are with the command line but if you were to fire up a shell on the iLiad and cd into the get-guardian directory and issue the command sh -x run.sh This should give you a better idea of where it is going wrong.

[carandol]

Well, I'm familiar enough with the shell to do that, but not familiar enough to understand the results! Here's the output:

+ export GUARDIAN_PATH=/media/cf/newspapers/Guardian24/
+ GUARDIAN_PATH=/media/cf/newspapers/Guardian24/
+ /usr/bin/wired.sh start
run.sh: line 3: /usr/bin/wired.sh: No such file or directory
+ /usr/bin/wireless.sh start
run.sh: line 4: /usr/bin/wireless.sh: No such file or directory
+ rm -rf /media/cf/newspapers/Guardian24//Guardian.pdf
+ mkdir /media/cf/newspapers/Guardian24//Guardian.pdf
+++ date
++ echo Sat Mar 29 16:03:30 GMT 2008
+ echo '<?xml version="1.0" encoding="utf-8"?><package><metadata><dc-metadata><Title>The Guardian 24h edition</Title><Description>Downloaded: Sat Mar 29 16:03:30 GMT 2008</Description></dc-metadata><y-metadata><startpage>Guardian.pdf</startpage><image>icon.png</image><version>000</version></y-metadata></metadata></package>'
+ wget -O /media/cf/newspapers/Guardian24//Guardian.pdf/Guardian.pdf http://download.guardian.co.uk/g24/World.pdf
--16:03:30--  http://download.guardian.co.uk/g24/World.pdf
           => `/media/cf/newspapers/Guardian24//Guardian.pdf/Guardian.pdf'
Resolving download.guardian.co.uk... 77.91.249.32
Connecting to download.guardian.co.uk|77.91.249.32|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 255,185 (249K) [application/pdf]

100%[====================================>] 255,185      288.43K/s

16:03:31 (287.81 KB/s) - `/media/cf/newspapers/Guardian24//Guardian.pdf/Guardian.pdf' saved [255185/255185]

+ cp /media/cf/newspapers/Guardian24//get-guardian/icon.png /media/cf/newspapers/Guardian24//Guardian.pdf/icon.png
cp: cannot stat `/media/cf/newspapers/Guardian24//get-guardian/icon.png': No such file or directory
+ /usr/bin/wired.sh stop
run.sh: line 10: /usr/bin/wired.sh: No such file or directory
+ /usr/bin/wireless.sh stop
run.sh: line 11: /usr/bin/wireless.sh: No such file or directory

It's definitely downloading the file, but when I look at it, it's only 8kb and doesn't have the icon attached, just a black rectangle. Hopefully, someone will be able to figure out from this what's not working!

[daudi]

Just looking at this quickly:

i) Are you sure the manifest file is being created? You do not seem to be directing the output of the echo command to a file. I would have expected to see something like this at the end of the echo command

> manifest.xml

ii) You can see that the copy command is failing to find the icon to copy.

Can you look at the CF in a PC? Is it possible that the PDF has been downloaded okay and it in the subdirectory, but because there is no manifest file it won't open?