[Bug-wget] "Referer" when using spider mode

[Fernando Gont]

Folks,

I'm using wget in a script to find broken and "moved" links in a web site.

My problem is that, when parsing the output of "wget --spider", I cannot
tell which page triggered the retrieval of a URL (i.e., the "referer" of
such URL) -- so, while I can find that there are broken links, I cannot
easily tell which page contains the broken link.

Any clues on how to obtain such info?

Thanks!
Fernando

[Darshit Shah]

Hi Fernando,

Once again, the answer is quite the same. You could parse the --debug output of
Wget to do this. Though, remember, parsing the debug is not always safe since
we may change it any point.

However, even for this case, I guess, using Wget2 is a better choice for you.

https://gitlab.com/gnuwget/wget2

Post by Fernando Gont
Folks,
I'm using wget in a script to find broken and "moved" links in a web site.
My problem is that, when parsing the output of "wget --spider", I cannot
tell which page triggered the retrieval of a URL (i.e., the "referer" of
such URL) -- so, while I can find that there are broken links, I cannot
easily tell which page contains the broken link.
Any clues on how to obtain such info?
Thanks!
Fernando
--
Fernando Gont
SI6 Networks
PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492

[Dale R. Worley]

Post by Fernando Gont
I'm using wget in a script to find broken and "moved" links in a web site.
My problem is that, when parsing the output of "wget --spider", I cannot
tell which page triggered the retrieval of a URL (i.e., the "referer" of
such URL) -- so, while I can find that there are broken links, I cannot
easily tell which page contains the broken link.
Any clues on how to obtain such info?

I would grep through the files in the file tree that wget constructs.
That isn't perfect, but it's probably close enough for your purposes.

Dale