Skip to main content

Curl on Linux - Reference Guide

Source

Curl is a powerful tool that is mainly used to transfer data. It has way more functions, but I won't be able to cover everything. This blog post is mainly a reference for later use and not a step-by-step guide. Therefore I won't cover everything in depth.

Most of it should work on other operating systems too, but I'll use Linux as reference. I'll keep this page up-to-date and add more topics in the future.

General #

Side note: put the URL into single or double quotes if it contains special characters.

By default, curl writes the received data to stdout and does not encode or decode received data.

A quick example to get you public IP:
curl brrl.net/ip
curl -L brrl.net/ip # -L to get through the HTTP>HTTP if necessary
Saving to disk #

You can redirect the content from stdout to another application, save it as a file or download the target file.

Download content to the file:
curl -L -o myip.txt brrl.net/ip # save your public IP to a file called myip.txt in the current directory

If you want to download a file and keep the original name, use the -O (capital 'o') or --remote-name option.

If you want to create a new directory, you can use --create-dirs like this:

curl -L --create-dirs -o path/from/current/dir/myip.txt brrl.net/ip

The permission used is 0750.

Specific interface #

You can use the --interface option to use one specific interface. You are free to use the interface name, the IP address, or the hostname.

Specific DNS server #

You can choose a specific DNS server with the following option. Multiple DNS servers can be chosen and must be separated by a comma.

--dns-servers 9.9.9.9:53,149.112.112.112:53

Redirects #

If you want curl to follow redirects, simply use the -L flag.

Import curl options and targets from the file #

Some tasks require many options. To keep it organized, you can import those options from a file with the -K or --config and followed by the name of the file.

Example:
curl --config curl-options.txt https://example.com
Data tranfer limits #

You can set up- and download limits with --limit-rate. The default are bytes/second, and you can use K,M,G,T for Kilo-,Mega-,Giga- and Terabyte, respectively. 

--limit-rate 10K
--limit-rate 1000
--limit-rate 10M
Parallel function #

To let curl transfer data parallel, you can use the -Z or --parallel and choose --parallel-immediate to start immediately.

-Z --parallel-immediate

The default is 50 parallel transfers but can be set with --parallel-max NUMBER.

Continue downloads automatically
Unreliable connections are a pain, and you can tell curl to retry and continue downloads:
--retry 999 --retry-max-time 0 -C -
--retry 999 # retry it 999 times
--retry-max-time 0 # prevent the default timeout between retries
-C - # continue the transfer when you run the command again, and let curl figure out where to continue

Source from StackExchange

Wildcards / Multiple downloads #

Side note: make sure to put the full URL into single or double quotes if you work with wildcards and sequences.

Sets #

You can tell curl to transfer multiple files by putting the names into curly brac {}

Keep the original name:
curl -O 'http://{domain1,domain2,domain3}.com'
curl -O 'http://domain.com/{uri1,uri2,uri3}'
Rename files:
curl "http://{one,two}.example.com" -o "file_#1.txt"

And you can use multiple sets, as shown in this example:

kuser@pleasejustwork:~/temp/curl$ curl "http://example.com/{1,2}/{3,4}" -o "file_#1_#2.txt"

[1/4]: http://example.com/1/3 --> file_1_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0   6404      0 --:--:-- --:--:-- --:--:--  6375

[2/4]: http://example.com/1/4 --> file_1_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12753      0 --:--:-- --:--:-- --:--:-- 12816

[3/4]: http://example.com/2/3 --> file_2_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12765      0 --:--:-- --:--:-- --:--:-- 12816

[4/4]: http://example.com/2/4 --> file_2_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12804      0 --:--:-- --:--:-- --:--:-- 12816l

kuser@pleasejustwork:~/temp/curl$ ls
file_1_3.txt  file_1_4.txt  file_2_3.txt  file_2_4.txt
Sequence #
Use [] for alphanumeric sequences:
curl -O 'http://example.com/picture-[1-51].img'
curl -O 'http://example.com/attach-[a-z].img'

It works with leading zeros too.

Nested sequences are not supported!

Adding steps:
curl -O 'http://example.com/picture-[1-50:2].img' # every second picture

Proxies #

I am not too familiar with the proxy functions. I normally just use it to download things from Tor.

If you are connected to Tor, you can reach the network through a SOCKS5 socket:
--socks5-hostname localhost:9150 # On Linux
--socks5-hostname localhost:9050 # Windows or Tor in Browser Bundle
with this addition, you can reach the Tor network via curl.

For normal SOCKS5 sockets, you could simply use --socks5 instead of --socks5-hostname, but with --socks5-hostname, the DNS resolution runs on the proxy.

The usual syntax for proxies looks like this, according to the manual:
-x, --proxy [protocol://]host[:port]
curl --proxy http://proxy.example https://example.com
curl --proxy socks5://proxy.example:12345 https://example.com
Another example of HTTP basic auth proxy:
curl --proxy-basic --proxy-user user:password -x http://proxy.example https://example.com

Authentication #

Example for basic authentication:
curl -u name:password --basic https://example.com
Example with bearer token:
curl http://username:password@example.com/api/ -H "Authorization: Bearer reallysecuretoken"
Example with oauth2 bearer:
curl --oauth2-bearer "mF_9.B5f-4.1JqM" https://example.com
Example with ssh public key authentication:
curl --pass secret --key file https://example.com

What else

And there is so much more, but I'll leave it like that for now. Things I am going to add in the future: HTTP post/get requests, certificates troubleshooting, up- and downloading data through FTP, sftp, etc., mail /SMTP. I am unfamiliar with those, so I'll test a bunch before I add those topics here.

--

E-Mail hellofoo@ittafoovern.comcom

More reading: