Skip to main content

Getting started with rclone - Data transmission

Source

Rclone is an open-source cross-platform data synchronization application focusing on cloud services. It can act as the CLI for your cloud storage. Rclone provides a broad set of features, from simple data transfer to mounting your cloud storage. It provides so many features that I will work on more posts and concentrate on the initial setup and data transfer in this post.

I wish I had looked into rclone earlier.

I will use Linux as a reference system in this post.

Further information

Instructions for the download and installation can be found in the official documentation.

After the installation, you can get more help via rclone --help, man rclone or the official docs.

Configuration File

With rclone config file you can check the configuration file that rclone would use. In my case, it hasn't been created yet.

$ rclone config file
Configuration file doesn't exist, but rclone will use this path:
/home/kuser/.config/rclone/rclone.conf

You don't have to create this file. Rclone will create it after you've added the first 'Remote'.

Remotes

Remotes as the name implies, are remote storages and rclone supports a large number of providers.

You can check the current list of remotes with rclone listremotes or the interactive rclone config command. 

$ rclone listremotes
hetzner-storagebox:

$ rclone config
Current remotes:

Name                 Type
====                 ====
hetzner-storagebox   sftp

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>
Adding a new remote

I've just added one of my StorageBoxes from Hetzner as a test. You can choose n and add a new 'remote'. Rclone will suggest a list of providers and if the provider of choice is unavailable, you have to choose the protocol like sftp, ftp, webdav, or something else to synchronise your data. Rclone will ask you for all necessary information.

[...]
25 / Pcloud
   \ "pcloud"
26 / Put.io
   \ "putio"
27 / SSH/SFTP Connection
   \ "sftp"
28 / Sugarsync
   \ "sugarsync"
29 / Transparently chunk/split large files
   \ "chunker"
[...]

The information then can be found in the configuration file.

$ rclone config file
Configuration file is stored at:
/home/kuser/.config/rclone/rclone.conf

$ cat ~/.config/rclone/rclone.conf
[hetzner-storagebox]
type = sftp
host = u123123.your-storagebox.de
user = u123123
pass = fdsfasggsghsgsdrgrsgsgsrgsgrg
use_insecure_cipher = true
Removing Remote

You can either use the interactive menu of rclone config or delete the specific remotes section from your configuration file.

$ rclone config
Current remotes:

Name                 Type
====                 ====
hetzner-storagebox   sftp

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> d
Choose a number from below, or type in an existing value
 1 > hetzner-storagebox
remote> 1

Show content of Remote

You can check the content of the root directory of the added 'remote' to verify a successful connection with the ls options.

  ls              List the objects in the path with size and path.
  lsd             List all directories/containers/buckets in the path.
  lsf             List directories and objects in remote:path formatted for parsing.
  lsjson          List directories and objects in the path in JSON format.
  lsl             List the objects in path with modification time, size and path.

As an example:

$ rclone lsd hetzner-storagebox:/
          -1 2023-02-02 14:47:07        -1 023
          -1 2023-11-05 11:09:05        -1 2023
          -1 2023-02-02 15:08:55        -1 cam-2

Basics usage

The basic functions of rclone are to copy, move, sync and check for changes of files and directories.

The basic syntax is:

rclone [options] subcommand <parameters> <parameters...> src:source_path dst:destination_path

Side note: Use the -v flag to get a more verbose output, which I highly recommend! - You can increase the verbosity further by using -vv or -vvv.

Use the --dry-run and --interactive / -i flags before larger transfers to check the connection and the actions rclone would take to prevent data loss. 

$ rclone -v --dry-run sync ./test-dir hetzner-storagebox:/test-dir
2023/11/07 22:52:57 NOTICE: another-test.txt: Skipped copy as --dry-run is set
2023/11/07 22:52:57 NOTICE: test-image.png: Skipped delete as --dry-run is set
2023/11/07 22:52:57 INFO  : 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Deleted:                1
Transferred:            1 / 1, 100%
Elapsed time:         0.3s
Copying data

One of the most used functions is the simple data transfer from a source to a destination.

In the following example, we transfer a test image from our local directory to our 'remote' Hetzner StorageBox.

Usual data transfer:

 ┌──────────────────┐
 │                  │
 │ local─────►cloud │
 │                  │
 └──────────────────┘

rclone copy ./test-image.png hetzner-storagebox:/

I then deleted the file with rclone with the following command rclone deletefile hetzner-storagebox:/test-image.png and transferred it again with a more verbose output:

$ rclone -v copy ./test-image.png hetzner-storagebox:/
2023/11/07 22:22:01 INFO  : test-image.png: Copied (new)
2023/11/07 22:22:01 INFO  : 
Transferred:      261.574k / 261.574 kBytes, 100%, 920.919 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.5s

If you copy a directory, make sure to label/name the folder on the destination, or you'd copy only the content of the source directory.

$ rclone lsd hetzner-storagebox:/
          -1 2023-02-02 14:47:07        -1 023
          -1 2023-11-05 11:09:05        -1 2023
          -1 2023-02-02 15:08:55        -1 cam-2

$ rclone -v copy ./test-dir hetzner-storagebox:/test-dir
2023/11/07 22:29:25 INFO  : test-image.png: Copied (new)
2023/11/07 22:29:25 INFO  : 
Transferred:      261.574k / 261.574 kBytes, 100%, 815.043 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.6s

$ rclone lsd hetzner-storagebox:/          -1 2023-02-02 14:47:07        -1 023
          -1 2023-11-05 11:09:05        -1 2023
          -1 2023-02-02 15:08:55        -1 cam-2
          -1 2023-11-07 22:29:25        -1 test-dir

The source files and directories remain unchanged just to have it written down.

If you want to copy things from a remote storage to your local system, simply swap the source and destination and choose the file or directory in the source.

 ┌──────────────────┐
 │                  │
 │ cloud─────►local │
 │                  │
 └──────────────────┘

rclone copy hetzner-storagebox:/test-image.png ./

Another often used use case for rclone is to copy or sync data from one cloud storage provider to another, which works pretty well!

 ┌──────────────────┐
 │                  │
 │ cloud─────►cloud │
 │                  │
 └──────────────────┘

$ rclone -v copy hetzner-storagebox:/test-dir onedrive-storage:/test-dir-onedrive
2023/11/08 20:01:15 INFO  : another-test.txt: Copied (new)
2023/11/08 20:01:15 INFO  : 
Transferred:      261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.9s
Moving files

I'll keep this one short. Just replace copy with move from the previous section, and rclone will remove the source files after the data transfer.

$ rclone -v move ./test-image.png hetzner-storagebox:/
2023/11/07 22:46:26 INFO  : test-image.png: Copied (new)
2023/11/07 22:46:26 INFO  : test-image.png: Deleted
2023/11/07 22:46:26 INFO  : 
Transferred:      261.574k / 261.574 kBytes, 100%, 975.168 kBytes/s, ETA 0s
Checks:                 2 / 2, 100%
Deleted:                1
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:         0.5s
Syncing files

Syncing - with sync - makes the source and destination identical but only changes the destination. It works on your local device and remote storage.

Side note: As syncing can cause data loss as it deletes data that is not present in the source, use the --dry-run flag beforehand and check what actions rclone would take.

$ rclone -v  sync ./test-dir hetzner-storagebox:/test-dir
2023/11/07 23:06:47 INFO  : another-test.txt: Copied (new)
2023/11/07 23:06:47 INFO  : test-image.png: Deleted
2023/11/07 23:06:47 INFO  : 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Deleted:                1
Transferred:            1 / 1, 100%
Elapsed time:         0.4s
Deleting files

You can delete files and directories with rclone delete and rclone deletefile. 

$ rclone -v delete onedrive-storage:/test-dir-onedrive
2023/11/08 20:16:10 INFO  : another-test.txt: Deleted

E-Mail hellofoo@ittafoovern.comcom

More reading: