Overview

到目前为止,我们手里已经有了10台服务器了。之前一直说要把这些服务器搭建一个分布式文件系统,现在条件终于成熟了。这些服务器预装的系统大多数是Ubuntu16.04LTS发行版的Linux,少部分是Ubuntu14.04LTS。这次我们选择5台服务器,1feagen(118.138.241.39)服务器作为tracker4台服务器(Bastion3(118.138.240.146),Bastion4(118.138.233.74),Bastion6(118.138.233.26),POSSUM(118.138.233.27))作为storage。这里会记录下FastDFS分布式文件系统在Ubuntu16.04上的安装部署过程的细节。

5台服务器都要安装,只是配置文件稍有不同。

1.必备软件

  • FastDFS
    这个去GitHub上面下载源码:FastDFS
  • libfastcommon
    这个仍然要去GitHub上面下载源码:libfastcommon

这两个Zip包用wget或者在本地下载上传至服务器均可。

2.libfastcommon的安装

首先解压libfastcommon-master.zip包:

1
unzip libfastcommon-master.zip

进入这个文件夹:

1
cd libfastcommon-master/

依次输入如下命令:

1
2
sudo ./make.sh
sudo ./make.sh install

输入第二条命令之后,会有如下显示:

1
2
3
4
5
6
mkdir -p /usr/lib64
mkdir -p /usr/lib
install -m 755 libfastcommon.so /usr/lib64
install -m 755 libfastcommon.so /usr/lib
mkdir -p /usr/include/fastcommon
install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h id_generator.h char_converter.h char_convert_loader.h /usr/include/fastcommon

可以看到libfastcommon.so安装到了/usr/lib64.
下面,我们需要为两个文件创建软链接,指向FastDFS主程序的lib目录:

1
2
3
4
sudo ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
sudo ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
sudo ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
sudo ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so

3.FastDFS的安装

解压fastdfs-master.zip文件:

1
unzip fastdfs-master.zip

进入FastDFS源码根目录:

1
cd fastdfs-master

依次输入下面两条命令:

1
2
sudo ./make.sh
sudo ./make.sh install

安装完成之后,所有的可执行文件都被放在了/usr/bin/目录下面,可以用下面的命令查看:

1
ls /usr/bin/fdfs*

显示共有14个文件:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
/usr/bin/fdfs_appender_test  
/usr/bin/fdfs_download_file 
/usr/bin/fdfs_test1
/usr/bin/fdfs_appender_test1 
/usr/bin/fdfs_file_info     
/usr/bin/fdfs_trackerd
/usr/bin/fdfs_append_file    
/usr/bin/fdfs_monitor       
/usr/bin/fdfs_upload_appender
/usr/bin/fdfs_crc32          
/usr/bin/fdfs_storaged      
/usr/bin/fdfs_upload_file
/usr/bin/fdfs_delete_file    
/usr/bin/fdfs_test

4. 配置FastDFS

配置文件在目录/etc/fdfs下,查看一下:

1
ls /etc/fdfs

显示如下:

1
client.conf.sample  storage.conf.sample  storage_ids.conf.sample  tracker.conf.sample

4.1 创建跟踪器和存储节点的配置文件

1
2
3
cd /etc/fdfs
sudo cp tracker.conf.sample tracker.conf
sudo cp storage.conf.sample storage.conf

4.2 修改tracker配置文件

feagen(118.138.241.39)服务器中修改tracker配置文件如下:

1
sudo vim /etc/fdfs/tracker.conf

可以看到如下显示:

1
2
# the base path to store data and log files
base_path=/home/yuqing/fastdfs

这行改成如下内容:

1
base_path=/feagen/fastdfs/tracker

这种配置文件,注意等号两边不能有空格。且上面的目录必须是真实存在的。
启动tracker

1
sudo fdfs_trackerd /etc/fdfs/tracker.conf start

查看监听端口:

1
sudo netstat -unltp|grep fdfs

如果显示如下,则证明tracker启动成功:

1
tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      6951/fdfs_trackerd

4.3 修改storage配置文件

Bastion3(118.138.240.146)服务器中,修改storage配置文件如下:

1
sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

1
2
3
4
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

1
2
3
4
group_name=group1
base_path=/bastion3_cache/fastdfs/storage
store_path0=/bastion3_cache/fastdfs/storage
tracker_server=118.138.241.39:22122

Bastion4(118.138.233.74)服务器中,修改storage配置文件如下:

1
sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

1
2
3
4
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

1
2
3
4
group_name=group1
base_path=/bastion4_cache/fastdfs/storage
store_path0=/bastion4_cache/fastdfs/storage
tracker_server=118.138.241.39:22122

Bastion6(118.138.233.26)服务器中,修改storage配置文件如下:

1
sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

1
2
3
4
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

1
2
3
4
group_name=group1
base_path=/bastion6_cache/fastdfs/storage
store_path0=/bastion6_cache/fastdfs/storage
tracker_server=118.138.241.39:22122

POSSUM(118.138.233.27)服务器中,修改storage配置文件如下:

1
sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

1
2
3
4
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

1
2
3
4
group_name=group1
base_path=/possum/fastdfs/storage
store_path0=/possum/fastdfs/storage
tracker_server=118.138.241.39:22122

启动storage服务:

1
sudo fdfs_storaged /etc/fdfs/storage.conf start

显示如下,即代表成功启动:

1
process fdfs_storaged already running, pid: 28250

查看监听端口:

1
sudo netstat -unltp|grep fdfs

如果显示如下,则证明storage启动成功:

1
tcp        0      0 0.0.0.0:23000           0.0.0.0:*               LISTEN      28250/fdfs_storaged

如果没有显示,则有可能是storage启动失败。

那么我们以Bastion6服务器为例,查看启动日志:

1
tail /bastion6_cache/fastdfs/storage/logs/storaged.log

如果显示如下,代表启动失败:

1
2
[2018-05-13 19:56:31] ERROR - file: storage_ip_changed_dealer.c, line: 186, connect to tracker server 118.138.2
40.146:22122 fail, errno: 110, error info: Connection timed out

连接超时;
如果显示如下,代表启动成功:

1
2
3
4
[2018-05-14 12:59:40] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 118.138.240.146:22122, as a tracker client, my ip is 118.138.233.26
[2018-05-14 13:00:10] INFO - file: tracker_client_thread.c, line: 1263, tracker server 118.138.240.146:22122, set tracker leader: 118.138.240.146:22122
[2018-05-14 13:03:06] INFO - file: storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000, continuous fail count: 16
[2018-05-14 13:03:41] INFO - file: storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000

连接成功。

超时解读:
之前跟Chris讨论的时候,刚开始以为这几天服务器都是Monash云服务器中心的,属于内网,所以相互之间的端口是不用开的,实验之后才知道不开端口是不行的。后来联系Jerico开开3个端口之后,才连接成功。这3个端口分别是22122888823000

  • 22122:代表tracker服务端口;
  • 8888:代表HTTP协议端口,网页上传下载文件需要这个端口;
  • 23000:代表storage服务端口。

至此,storage存储节点安装成功。

4.4 查看所有存储节点信息

所有存储节点都启动之后,在任意一台storage上面用下面命令,查看集群状态信息:

1
sudo /usr/bin/fdfs_monitor /etc/fdfs/storage.conf

显示如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
[2018-05-24 01:27:18] DEBUG - base_path=/bastion6_cache/fastdfs/storage, connect_timeout=10, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
 
server_count=1, server_index=0
 
tracker server is 118.138.241.39:22122
 
group count: 1
 
Group 1:
group name = group1
disk total space = 100665 MB
disk free space = 72339 MB
trunk free space = 0 MB
storage server count = 4
active server count = 4
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
 
    Storage 1:
        id = 118.138.233.26
        ip_addr = 118.138.233.26 (vm-118-138-233-26.erc.monash.edu.au)  ACTIVE
        http domain =
        version = 5.12
        join time = 2018-05-13 19:55:46
        up time = 2018-05-20 00:51:17
        total storage = 483679 MB
        free storage = 452554 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id =
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 1
        success_upload_count = 1
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 0
        success_set_meta_count = 0
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 87026
        success_upload_bytes = 87026
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 11288
        success_sync_in_bytes = 11288
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 5
        success_file_open_count = 5
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 5
        success_file_write_count = 5
        last_heart_beat_time = 2018-05-24 01:26:54
        last_source_update = 2018-05-17 01:27:29
        last_sync_update = 2018-05-17 01:06:46
        last_synced_timestamp = 1970-01-01 10:00:00 (never synced)
    Storage 2:
        id = 118.138.233.27
        ip_addr = 118.138.233.27 (vm-118-138-233-27.erc.monash.edu.au)  ACTIVE
        http domain =
        version = 5.12
        join time = 2018-05-17 00:39:17
        up time = 2018-05-20 00:52:30
        total storage = 2015737 MB
        free storage = 1689979 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id =
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 0
        success_upload_count = 0
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 0
        success_set_meta_count = 0
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 0
        success_upload_bytes = 0
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 98314
        success_sync_in_bytes = 98314
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 5
        success_file_open_count = 5
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 5
        success_file_write_count = 5
        last_heart_beat_time = 2018-05-24 01:27:08
        last_source_update = 1970-01-01 10:00:00
        last_sync_update = 2018-05-21 16:32:36
        last_synced_timestamp = 2018-05-17 01:06:38 (20m:51s delay)
    Storage 3:
        id = 118.138.233.74
        ip_addr = 118.138.233.74 (vm-118-138-233-74.erc.monash.edu.au)  ACTIVE
        http domain =
        version = 5.12
        join time = 2018-05-13 20:03:06
        up time = 2018-05-20 00:53:40
        total storage = 100665 MB
        free storage = 72339 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id =
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 2
        success_upload_count = 2
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 2
        success_set_meta_count = 2
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 11190
        success_upload_bytes = 11190
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 174142
        success_sync_in_bytes = 87026
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 3
        success_file_open_count = 3
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 3
        success_file_write_count = 3
        last_heart_beat_time = 2018-05-24 01:27:13
        last_source_update = 2018-05-17 01:06:38
        last_sync_update = 2018-05-21 16:32:54
        last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay)
    Storage 4:
        id = 118.138.240.146
        ip_addr = 118.138.240.146 (vm-118-138-240-146.erc.monash.edu.au)  ACTIVE
        http domain =
        version = 5.12
        join time = 2018-05-20 00:53:13
        up time = 2018-05-20 00:53:13
        total storage = 483679 MB
        free storage = 449872 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id = 118.138.233.26
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 0
        success_upload_count = 0
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 0
        success_set_meta_count = 0
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 0
        success_upload_bytes = 0
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 98314
        success_sync_in_bytes = 98314
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 5
        success_file_open_count = 5
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 5
        success_file_write_count = 5
        last_heart_beat_time = 2018-05-24 01:27:03
        last_source_update = 1970-01-01 10:00:00
        last_sync_update = 2018-05-21 16:32:54
        last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay)

可以看到,4storage状态都是ACTIVE,表示都启动成功了。

5. 测试上传文件

需要注意一点,我们之后的特征提取程序会在feagen服务器上,所以,我们暂时将client安装在这台服务器上。测试一下在本台服务器上上传文件,能否在所有storage服务器上面查看得到此文件的备份。

5.1 修改tracker服务器上的client配置文件

先后输入以下命令:

1
2
3
cd /etc/fdfs
sudo cp client.conf.sample client.conf
sudo vim client.conf

base_pathtracker_server两处配置修改为:

1
2
base_path=/feagen/fastdfs/client
tracker_server=118.138.241.39:22122

shell当中输入下面的命令:

1
sudo /usr/bin/fdfs_upload_file /etc/fdfs/client.conf fastdfs-master.zip

返回如下,即可表明上传成功:

1
group1/M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904.zip

这个返回值,在perl当中,以如下方式获取:

1
my $file_name = qx(sudo /usr/bin/fdfs_upload_file /etc/fdfs/client.conf fastdfs-master.zip);

到任意一个storage服务器当中查看,以Bastion3为例,至此处:

1
/bastion3_cache/fastdfs/storage/data/00/00

发现存在如下文件:

1
dorpG1td37yAFSYPAAZ-Sk23ivY904.zip

即表明上传成功。

6. 测试下载文件

当前目录下,运行下面的程序:

1
sudo /usr/bin/fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904.zip

即可在当前目录下面找到dorpG1td37yAFSYPAAZ-Sk23ivY904.zip,表明下载成功。

至此我们的分布式文件服务器算是搭建完成了。

我们参考了这些文章,表示感谢!
FastDFS分布式文件系统集群安装与配置
FastDFS--原理篇
分布式文件系统FastDFS原理介绍
Ubuntu下安装并配置FastDFS