John Murrell
2018-06-15 09:57:20 UTC
Hello,
I have been trying to use WGET to download some files but have been unable
to find a combination of options to achieve what I need to do.
The site I am downloading from publishes a new version of a file every hour
but the exact time it is available is not specified. The replacement has the
same name.
I am running a script using cron to check the site every 10 minutes using
wget -N to download only the new versions of the file.
The problem is that WGET -N clobbers the previous version and what I want
is to keep a copy of each version. I can't find any way of achieving this -
can anyone advise please.
I have tried both renaming and moving the file when WGET -N completes but
then WGET -N 'forgets' the file has already been downloaded and downloads it
again.
-nc only keeps a copy of the original version of the source file - I need
all the versions of the file.
If this is not possible I would like to suggest a new option that either:
* Prevents clobbering files with different contents but duplicate
names this is a combination of the no option and -N option behaviour
* Alternatively WGET should have an option to remember the filename
(+ timestamp and any other comparison information it requires) even though
the file is no longer in the destination directory - I call this the Ghost
option as the ghostly memory of the file exists.
2nd option would be useful if people want to rename or move files on the fly
but I can see problems on storing this information between WGET actuations.
Thanks
John Murrell
I have been trying to use WGET to download some files but have been unable
to find a combination of options to achieve what I need to do.
The site I am downloading from publishes a new version of a file every hour
but the exact time it is available is not specified. The replacement has the
same name.
I am running a script using cron to check the site every 10 minutes using
wget -N to download only the new versions of the file.
The problem is that WGET -N clobbers the previous version and what I want
is to keep a copy of each version. I can't find any way of achieving this -
can anyone advise please.
I have tried both renaming and moving the file when WGET -N completes but
then WGET -N 'forgets' the file has already been downloaded and downloads it
again.
-nc only keeps a copy of the original version of the source file - I need
all the versions of the file.
If this is not possible I would like to suggest a new option that either:
* Prevents clobbering files with different contents but duplicate
names this is a combination of the no option and -N option behaviour
* Alternatively WGET should have an option to remember the filename
(+ timestamp and any other comparison information it requires) even though
the file is no longer in the destination directory - I call this the Ghost
option as the ghostly memory of the file exists.
2nd option would be useful if people want to rename or move files on the fly
but I can see problems on storing this information between WGET actuations.
Thanks
John Murrell