Download web pages in BASH: From binkley's BLOG

BASH is quite amazing. From Bhaskar V. Karambelkar comes Bash shell tricks and this gem (with some small corrections):

function headers()
{
    server=$1
    port=${2:-80}
    exec 5<>/dev/tcp/$server/$port
    echo -ne "HEAD / HTTP/1.0\r\nHost: $server:$port\r\n\r\n" >&5
    cat <&5
    exec 5<&-;
}

I work behind a corporate firewall, so I need web proxy settings. No problem, BASH is still amazing:

function webget()
{
    declare -a parts
    parts=($(echo $1 | tr / ' '))
    protocol=${parts[0]}
    server=${parts[1]}
    path=$(echo $1 | sed "s,$protocol//$server,,")

    exec 5<>/dev/tcp/$http_proxy_server/$http_proxy_port
    echo -ne "GET $path HTTP/1.0\r\nHost: $server\r\n\r\n" >&5
    cat <&5
    exec 5<&-;
}

Usage is obvious:

$ webget http://www.ccil.org/jargon/
# Out pops the top page for Jargon File Resources

UPDATE: Also useful for talking to SMTP (email) servers:

$ exec 5<>/dev/tcp/localhost/smtp
$ read -u 5 line
$ echo $line
220 my.full.host.name ESMTP ...
$ echo QUIT >&5
$ read -u 5 line
$ echo $line
221 2.0.0 my.full.host.name closing connection

This also shows that BASH knows to map the SMTP service to port 25

Share on Facebook Share on Twitter Share on Google Plus

About Unknown