0%

rclone简介

rclone

Rclone是一款的命令行工具,支持在不同对象存储、网盘间同步、上传、下载数据。

支持的主流对象存储:

  • Google Drive
  • Amazon S3
  • Openstack Swift / Rackspace cloud files / Memset Memstore
  • Dropbox
  • Google Cloud Storage
  • Amazon Drive
  • Microsoft One Drive
  • Hubic
  • Backblaze B2
  • Yandex Disk
  • The local filesystem

安装

https://rclone.org/downloads/

京东云OSS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
C:> rclone config
Current remotes:

Name Type
==== ====
cloud-oss s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n

Enter name for new remote.
name> abc

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
1 / 1Fichier
\ (fichier)
2 / Akamai NetStorage
\ (netstorage)
3 / Alias for an existing remote
\ (alias)
4 / Amazon Drive
\ (amazon cloud drive)
5 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi
\ (s3)
6 / Backblaze B2
\ (b2)
7 / Better checksums for other remotes
\ (hasher)
8 / Box
\ (box)
9 / Cache a remote
\ (cache)
10 / Citrix Sharefile
\ (sharefile)
11 / Combine several remotes into one
\ (combine)
12 / Compress a remote
\ (compress)
13 / Dropbox
\ (dropbox)
14 / Encrypt/Decrypt a remote
\ (crypt)
15 / Enterprise File Fabric
\ (filefabric)
16 / FTP
\ (ftp)
17 / Google Cloud Storage (this is not Google Drive)
\ (google cloud storage)
18 / Google Drive
\ (drive)
19 / Google Photos
\ (google photos)
20 / HTTP
\ (http)
21 / Hadoop distributed file system
\ (hdfs)
22 / HiDrive
\ (hidrive)
23 / Hubic
\ (hubic)
24 / In memory object storage system.
\ (memory)
25 / Internet Archive
\ (internetarchive)
26 / Jottacloud
\ (jottacloud)
27 / Koofr, Digi Storage and other Koofr-compatible storage providers
\ (koofr)
28 / Local Disk
\ (local)
29 / Mail.ru Cloud
\ (mailru)
30 / Mega
\ (mega)
31 / Microsoft Azure Blob Storage
\ (azureblob)
32 / Microsoft OneDrive
\ (onedrive)
33 / OpenDrive
\ (opendrive)
34 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ (swift)
35 / Pcloud
\ (pcloud)
36 / Put.io
\ (putio)
37 / QingCloud Object Storage
\ (qingstor)
38 / SSH/SFTP
\ (sftp)
39 / Sia Decentralized Cloud
\ (sia)
40 / Storj Decentralized Cloud Storage
\ (storj)
41 / Sugarsync
\ (sugarsync)
42 / Transparently chunk/split large files
\ (chunker)
43 / Union merges the contents of several upstream fs
\ (union)
44 / Uptobox
\ (uptobox)
45 / WebDAV
\ (webdav)
46 / Yandex Disk
\ (yandex)
47 / Zoho
\ (zoho)
48 / premiumize.me
\ (premiumizeme)
49 / seafile
\ (seafile)
Storage> 5

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Amazon Web Services (AWS) S3
\ (AWS)
2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
\ (Alibaba)
3 / Ceph Object Storage
\ (Ceph)
4 / China Mobile Ecloud Elastic Object Storage (EOS)
\ (ChinaMobile)
5 / Cloudflare R2 Storage
\ (Cloudflare)
6 / Arvan Cloud Object Storage (AOS)
\ (ArvanCloud)
7 / Digital Ocean Spaces
\ (DigitalOcean)
8 / Dreamhost DreamObjects
\ (Dreamhost)
9 / Huawei Object Storage Service
\ (HuaweiOBS)
10 / IBM COS S3
\ (IBMCOS)
11 / IDrive e2
\ (IDrive)
12 / Seagate Lyve Cloud
\ (LyveCloud)
13 / Minio Object Storage
\ (Minio)
14 / Netease Object Storage (NOS)
\ (Netease)
15 / RackCorp Object Storage
\ (RackCorp)
16 / Scaleway Object Storage
\ (Scaleway)
17 / SeaweedFS S3
\ (SeaweedFS)
18 / StackPath Object Storage
\ (StackPath)
19 / Storj (S3 Compatible Gateway)
\ (Storj)
20 / Tencent Cloud Object Storage (COS)
\ (TencentCOS)
21 / Wasabi Object Storage
\ (Wasabi)
22 / Any other S3 compatible provider
\ (Other)
provider> 22 # 阿里云,就选2

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
1 / Enter AWS credentials in the next step.
\ (false)
2 / Get AWS credentials from the environment (env vars or IAM).
\ (true)
env_auth>

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> ***

生成的配置文件

1
2
3
4
5
6
7
[cloud-oss]
type = s3
provider = Other
env_auth = true
access_key_id = ***
secret_access_key = ***
endpoint = https://**.com

rclone 命令

rclone config

1
2
3
4
5
rclone config - 进入交互式配置选项,进行添加、删除、管理网盘等操作。

rclone config file - 显示配置文件的路径,一般配置文件在 ~/.config/rclone/rclone.conf

rclone config show - 显示配置文件信息

rclone ls
列出bucket下所有文件

1
$ rclone ls cloud-oss:nlp-object-store

列出bucket特定path下所有文件

1
$ rclone ls cloud-oss:nlp-object-store/jr
  • rclone lsl - 比上面多一个显示上传时间。
  • rclone lsd 列出指定路径下的目录
  • rclone lsf - 列出指定路径下的目录和文件

rclone copy

rclone sync

1
2
3
4
5
6
7
8
$ rclone sync --help
$ rclone sync source:path dest:path # 单向的,只改变dest, 双向同步使用 bisync

# 本地 同步到 服务器
$ rclone sync E:/test cloud-oss:nlp-object-store/test

# 服务器 同步到 本地
$ rclone sync cloud-oss:nlp-object-store/test E:/test

Sync the source to the destination, changing the destination only.

注意:

  • 同步数据时,可能会删除目的地址的数据;建议先使用–dry-run 标志来检查要复制、删除的数据。
  • 同步数据出错时,不会删除任何目的地址的数据。
  • 同步的始终是 path 目录下的数据,而不是 path 目录。(空目录将不会被同步)
  • 同步频率? 是一次性同步,不是随时同步

rclone bisync

这是个新增的feature:https://github.com/rclone/rclone/pull/5164

注意:bisync is EXPERIMENTAL. Don’t use in production!

rclone mount:太棒了

rclone mount cloud-oss:nlp-object-store/test X: –cache-dir D:\logs_rcloneCache –vfs-cache-mode writes

这个依赖winfsp

  • 这个挂载,会把数据全部下载,不只是meta信息
  • 是双向同步
  • 同步频率,是不是有点慢?怎么修改

如何开机自动挂载?

rclone md5sum - 为指定路径下的所有文件产生一个md5sum文件。

rclone sha1sum - 为指定路径下的所有文件产生一个sha1sum文件。

参考

  • 官方文档:https://rclone.org/docs/
  • https://www.cnblogs.com/KakuCicada/p/13573132.html 这个文档格式挺不错
  • https://softlns.github.io/2016/11/28/rclone-guide/
  • https://blog.csdn.net/socrates/article/details/104800709 阿里云 亚马逊云配置 rclone