Yevlempy’s Blog

01.04.2009

11>wget command

SUMMERY:

There are many ways to download files. But there is only one smart way to download from the command line – wget. The wget tool is a non-interactive network download tool that can download single files, recursively download entire directories, and even follow links.
SYNTAX:

-V

–version

Display the version of Wget.

-h

–help

Print a help message describing all of Wget’s command-line options.

-b

–background

Go to background immediately after startup. If no output file is specified via the -o, output is redirected to wget-log.

-e command

–execute command

Execute command as if it were a part of .wgetrc. A command thus invoked will be executed after the commands in .wgetrc, thus taking precedence over them.

EXAMPLES:

Download a single file using wget

$ wget http://www.cyberciti.biz/here/lsst.tar.gz
$ wget ftp://ftp.freebsd.org/pub/sys.tar.gz

Download multiple files on command line using wget

://ftp.redhat.com/pub/xyz-1rc-i386.rpmOR

i) Create variable that holds all urls and later use ‘BASH for loop’ to download all files:
$ URLS=”http://www.cyberciti.biz/download/lsst.tar.gz ftp://ftp.freebsd.org/pub/sys.tar.gz ftp://ftp.redhat.com/pub/xyz-1rc-i386.rpm http://xyz.com/abc.iso" ii) Use for loop as follows:
$ for u in $URLS; do wget $u; doneiii) However, a better way is to put all urls in text file and use -i option to wget to download all files:

(a) Create text file using vi
$ vi /tmp/download.txtAdd list of urls:
http://www.cyberciti.biz/download/lsst.tar.gz
ftp://ftp.freebsd.org/pub/sys.tar.gz
ftp://ftp.redhat.com/pub/xyz-1rc-i386.rpm
http://xyz.com/abc.iso(b) Run wget as follows:
$ wget -i /tmp/download.txt(c) Force wget to resume download
You can use -c option to wget. This is useful when you want to finish up a download started by a previous instance of wget and the net connection was lost. In such case you can add -c option as follows:
$ wget -c http://www.cyberciti.biz/download/lsst.tar.gz
$ wget -c -i /tmp/download.txtPlease note that all ftp/http server does not supports the download resume feature.

Force wget to download all files in background, and log the activity in a file:

$ wget -cb -o /tmp/download.log -i /tmp/download.txtOR$ nohup wget -c -o /tmp/download.log -i /tmp/download.txt &nohup runs the given COMMAND (in this example wget) with hangup signals ignored, so that the command can continue running in the background after you log out.

See man page of wget for more advanced options.