Curl is a powerful tool that is mainly used to transfer data. It has way more functions, but I won’t be able to cover everything. This blog post is mainly a reference for later use and not a step-by-step guide. Therefore I won’t cover everything in depth.

Most of it should work on other operating systems too, but I’ll use Linux as reference. I’ll keep this page up-to-date and add more topics in the future.

General #

Side note: put the URL into single or double quotes if it contains special characters.

By default, curl writes the received data to stdout and does not encode or decode received data.

A quick example to get you public IP:

curl brrl.net/ip

curl -L brrl.net/ip # -L to get through the HTTP>HTTP if necessary

Saving to disk #

You can redirect the content from stdout to another application, save it as a file or download the target file.

Download content to the file:

curl -L -o myip.txt brrl.net/ip # save your public IP to a file called myip.txt in the current directory

If you want to download a file and keep the original name, use the -O (capital ‘o’) or --remote-name option.

If you want to create a new directory, you can use --create-dirs like this:

curl -L --create-dirs -o path/from/current/dir/myip.txt brrl.net/ip

The permission used is 0750.

Specific interface #

You can use the --interface option to use one specific interface. You are free to use the interface name, the IP address, or the hostname.

Specific DNS server #

You can choose a specific DNS server with the following option. Multiple DNS servers can be chosen and must be separated by a comma.

--dns-servers 9.9.9.9:53,149.112.112.112:53

Redirects #

If you want curl to follow redirects, simply use the -L flag.

Import curl options and targets from the file #

Some tasks require many options. To keep it organized, you can import those options from a file with the -K or --config and followed by the name of the file.

Example:

curl --config curl-options.txt https://example.com

Data tranfer limits #

You can set up- and download limits with --limit-rate. The default are bytes/second, and you can use K,M,G,T for Kilo-,Mega-,Giga- and Terabyte, respectively.

--limit-rate 10K
--limit-rate 1000
--limit-rate 10M

Parallel function #

To let curl transfer data parallel, you can use the -Z or --parallel and choose --parallel-immediate to start immediately.

-Z --parallel-immediate

The default is 50 parallel transfers but can be set with --parallel-max NUMBER.

Continue downloads automatically

Unreliable connections are a pain, and you can tell curl to retry and continue downloads:

--retry 999 --retry-max-time 0 -C -

--retry 999 # retry it 999 times

--retry-max-time 0 # prevent the default timeout between retries

-C - # continue the transfer when you run the command again, and let curl figure out where to continue

Source from StackExchange

Wildcards / Multiple downloads #

Side note: make sure to put the full URL into single or double quotes if you work with wildcards and sequences.

Sets #

You can tell curl to transfer multiple files by putting the names into curly brac {}

Keep the original name:

curl -O 'http://{domain1,domain2,domain3}.com'

curl -O 'http://domain.com/{uri1,uri2,uri3}'

Rename files:

curl "http://{one,two}.example.com" -o "file_#1.txt"

And you can use multiple sets, as shown in this example:

kuser@pleasejustwork:~/temp/curl$ curl "http://example.com/{1,2}/{3,4}" -o "file_#1_#2.txt"

[1/4]: http://example.com/1/3 --> file_1_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0   6404      0 --:--:-- --:--:-- --:--:--  6375

[2/4]: http://example.com/1/4 --> file_1_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12753      0 --:--:-- --:--:-- --:--:-- 12816

[3/4]: http://example.com/2/3 --> file_2_3.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12765      0 --:--:-- --:--:-- --:--:-- 12816

[4/4]: http://example.com/2/4 --> file_2_4.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1256  100  1256    0     0  12804      0 --:--:-- --:--:-- --:--:-- 12816l

kuser@pleasejustwork:~/temp/curl$ ls
file_1_3.txt  file_1_4.txt  file_2_3.txt  file_2_4.txt

Sequence #

Use [] for alphanumeric sequences:

curl -O 'http://example.com/picture-[1-51].img'

curl -O 'http://example.com/attach-[a-z].img'

It works with leading zeros too.

Nested sequences are not supported!

Adding steps:

curl -O 'http://example.com/picture-[1-50:2].img' # every second picture

Proxies #

I am not too familiar with the proxy functions. I normally just use it to download things from Tor.

If you are connected to Tor, you can reach the network through a SOCKS5 socket:

--socks5-hostname localhost:9150 # On Linux

--socks5-hostname localhost:9050 # Windows or Tor in Browser Bundle

with this addition, you can reach the Tor network via curl.

For normal SOCKS5 sockets, you could simply use --socks5 instead of --socks5-hostname, but with --socks5-hostname, the DNS resolution runs on the proxy.

The usual syntax for proxies looks like this, according to the manual:

-x, --proxy [protocol://]host[:port]

curl --proxy http://proxy.example https://example.com

curl --proxy socks5://proxy.example:12345 https://example.com

Another example of HTTP basic auth proxy:

curl --proxy-basic --proxy-user user:password -x http://proxy.example https://example.com

Authentication #

Example for basic authentication:

curl -u name:password --basic https://example.com

Example with bearer token:

curl http://username:password@example.com/api/ -H "Authorization: Bearer reallysecuretoken"

Example with oauth2 bearer:

curl --oauth2-bearer "mF_9.B5f-4.1JqM" https://example.com

Example with ssh public key authentication:

curl --pass secret --key file https://example.com

What else

And there is so much more, but I’ll leave it like that for now. Things I am going to add in the future: HTTP post/get requests, certificates troubleshooting, up- and downloading data through FTP, sftp, etc., mail /SMTP. I am unfamiliar with those, so I’ll test a bunch before I add those topics here.

–