it works for public and private buckets after configuration
Configure rclone
Interactive rclone configuration
1.Run rclone config
and follow the interactive configuration for a new remote (basically the location of the bucket on the internet, accessible from a certain endpoint and with specific credentials if private):
Then insert n and follow the instructions:
name
: s3
storage option
: 5
s3 provider
: 22
env_auth
: leave default
access_key_id
: your access key
secret_key_id
: your secret key
region
: leave default
endpoint
: s3 bucket endpint URL https://s3.waw3-1.cloudferro.com
location_constraint
: leave default
acl
: select the one that best fits your use case
Finalize it and save
Basic commands with rclone
The following sections describe some commands that you can perform with rclone once you configure your remote see above. In particular:
- PATH_DIRECTORY is the directory path in the remote
REMOTE_NAME is the name you gave to a remote created with the rclone config command (see above). You can check your configured remotes using the following command:
rclone listremotes
Note: Rclone configuration should be available at the following path: ~/.config/rclone/rclone.conf
List objects in the bucket
rclone ls is used to list the objects in the path with size and path. Use the command below:
rclone ls <REMOTE_NAME>:<PATH_DIRECTORY>
Create a directory in the remote
rclone mkdir is used to create a directory on the path you want in your bucket.
rclone mkdir <REMOTE_NAME>:<PATH_DIRECTORY>
List a directory in the remote
rclone lsd is used to list directories in the remote.
rclone lsd <REMOTE_NAME>:
Copy file to remote
rclone copy is used to copy files from your local machine to the remote bucket.
Consider copying a file to remote:
rclone copy file.txt <REMOTE_NAME>:<PATH_DIRECTORY>
Sync data to remote
rclone sync <SOURCE_DIRECTORY>/ <REMOTE_NAME>:<PATH_DIRECTORY>
where SOURCE_DIRECTORY is a local directory with files you want to sync to remote.
Advanced commands with rclone
Sync large amount of data to remote
rclone sync -P --drive-chunk-size=256M --transfers=24 <SOURCE_DIRECTORY>/ <REMOTE_NAME>:<PATH_DIRECTORY>
where:
- P flag tells to provide statistics to stdout
- --drive-chunk-size=256 The chunk size for a transfer in kilobytes; must be a power of 2 and at least 256. Each chunk is buffered in memory prior to the transfer, so increasing this increases how much memory is used.
- --transfers=24 tells to use 24 threads
Resources
- Rclone official documentation: https://rclone.org/
- Sync files from cloud storage: https://www.tecmint.com/rclone-sync-files-from-cloud-storage/