Overview
到目前为止,我们手里已经有了10
台服务器了。之前一直说要把这些服务器搭建一个分布式文件系统,现在条件终于成熟了。这些服务器预装的系统大多数是Ubuntu16.04LTS
发行版的Linux
,少部分是Ubuntu14.04LTS
。这次我们选择5
台服务器,1
台feagen(118.138.241.39)
服务器作为tracker
,4
台服务器(Bastion3(118.138.240.146),Bastion4(118.138.233.74),Bastion6(118.138.233.26),POSSUM(118.138.233.27))
作为storage
。这里会记录下FastDFS
分布式文件系统在Ubuntu16.04
上的安装部署过程的细节。
5
台服务器都要安装,只是配置文件稍有不同。
1.必备软件
FastDFS
这个去GitHub
上面下载源码:FastDFSlibfastcommon
这个仍然要去GitHub
上面下载源码:libfastcommon
这两个Zip
包用wget
或者在本地下载上传至服务器均可。
2.libfastcommon
的安装
首先解压libfastcommon-master.zip
包:
1 | unzip libfastcommon-master.zip |
进入这个文件夹:
1 | cd libfastcommon-master/ |
依次输入如下命令:
1 2 | sudo . /make .sh sudo . /make .sh install |
输入第二条命令之后,会有如下显示:
1 2 3 4 5 6 | mkdir -p /usr/lib64 mkdir -p /usr/lib install -m 755 libfastcommon.so /usr/lib64 install -m 755 libfastcommon.so /usr/lib mkdir -p /usr/include/fastcommon install -m 644 common_define.h hash .h chain.h logger.h base64 .h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h id_generator.h char_converter.h char_convert_loader.h /usr/include/fastcommon |
可以看到libfastcommon.so
安装到了/usr/lib64
.
下面,我们需要为两个文件创建软链接,指向FastDFS
主程序的lib
目录:
1 2 3 4 | sudo ln -s /usr/lib64/libfastcommon .so /usr/local/lib/libfastcommon .so sudo ln -s /usr/lib64/libfastcommon .so /usr/lib/libfastcommon .so sudo ln -s /usr/lib64/libfdfsclient .so /usr/local/lib/libfdfsclient .so sudo ln -s /usr/lib64/libfdfsclient .so /usr/lib/libfdfsclient .so |
3.FastDFS
的安装
解压fastdfs-master.zip
文件:
1 | unzip fastdfs-master.zip |
进入FastDFS
源码根目录:
1 | cd fastdfs-master |
依次输入下面两条命令:
1 2 | sudo . /make .sh sudo . /make .sh install |
安装完成之后,所有的可执行文件都被放在了/usr/bin/
目录下面,可以用下面的命令查看:
1 | ls /usr/bin/fdfs * |
显示共有14
个文件:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | /usr/bin/fdfs_appender_test /usr/bin/fdfs_download_file /usr/bin/fdfs_test1 /usr/bin/fdfs_appender_test1 /usr/bin/fdfs_file_info /usr/bin/fdfs_trackerd /usr/bin/fdfs_append_file /usr/bin/fdfs_monitor /usr/bin/fdfs_upload_appender /usr/bin/fdfs_crc32 /usr/bin/fdfs_storaged /usr/bin/fdfs_upload_file /usr/bin/fdfs_delete_file /usr/bin/fdfs_test |
4. 配置FastDFS
配置文件在目录/etc/fdfs
下,查看一下:
1 | ls /etc/fdfs |
显示如下:
1 | client.conf.sample storage.conf.sample storage_ids.conf.sample tracker.conf.sample |
4.1 创建跟踪器和存储节点的配置文件
1 2 3 | cd /etc/fdfs sudo cp tracker.conf.sample tracker.conf sudo cp storage.conf.sample storage.conf |
4.2 修改tracker
配置文件
在feagen(118.138.241.39)
服务器中修改tracker
配置文件如下:
1 | sudo vim /etc/fdfs/tracker .conf |
可以看到如下显示:
1 2 | # the base path to store data and log files base_path= /home/yuqing/fastdfs |
这行改成如下内容:
1 | base_path= /feagen/fastdfs/tracker |
这种配置文件,注意等号两边不能有空格。且上面的目录必须是真实存在的。
启动tracker
:
1 | sudo fdfs_trackerd /etc/fdfs/tracker .conf start |
查看监听端口:
1 | sudo netstat -unltp| grep fdfs |
如果显示如下,则证明tracker
启动成功:
1 | tcp 0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 6951 /fdfs_trackerd |
4.3 修改storage
配置文件
在Bastion3(118.138.240.146)
服务器中,修改storage
配置文件如下:
1 | sudo vim /etc/fdfs/storage .conf |
可以看到如下显示:
1 2 3 4 | group_name=group1 base_path= /home/yuqing/fastdfs store_path0= /home/yuqing/fastdfs tracker_server=192.168.209.121:22122 |
修改为:
1 2 3 4 | group_name=group1 base_path= /bastion3_cache/fastdfs/storage store_path0= /bastion3_cache/fastdfs/storage tracker_server=118.138.241.39:22122 |
在Bastion4(118.138.233.74)
服务器中,修改storage
配置文件如下:
1 | sudo vim /etc/fdfs/storage .conf |
可以看到如下显示:
1 2 3 4 | group_name=group1 base_path= /home/yuqing/fastdfs store_path0= /home/yuqing/fastdfs tracker_server=192.168.209.121:22122 |
修改为:
1 2 3 4 | group_name=group1 base_path= /bastion4_cache/fastdfs/storage store_path0= /bastion4_cache/fastdfs/storage tracker_server=118.138.241.39:22122 |
在Bastion6(118.138.233.26)
服务器中,修改storage
配置文件如下:
1 | sudo vim /etc/fdfs/storage .conf |
可以看到如下显示:
1 2 3 4 | group_name=group1 base_path= /home/yuqing/fastdfs store_path0= /home/yuqing/fastdfs tracker_server=192.168.209.121:22122 |
修改为:
1 2 3 4 | group_name=group1 base_path= /bastion6_cache/fastdfs/storage store_path0= /bastion6_cache/fastdfs/storage tracker_server=118.138.241.39:22122 |
在POSSUM(118.138.233.27)
服务器中,修改storage
配置文件如下:
1 | sudo vim /etc/fdfs/storage .conf |
可以看到如下显示:
1 2 3 4 | group_name=group1 base_path= /home/yuqing/fastdfs store_path0= /home/yuqing/fastdfs tracker_server=192.168.209.121:22122 |
修改为:
1 2 3 4 | group_name=group1 base_path= /possum/fastdfs/storage store_path0= /possum/fastdfs/storage tracker_server=118.138.241.39:22122 |
启动storage
服务:
1 | sudo fdfs_storaged /etc/fdfs/storage .conf start |
显示如下,即代表成功启动:
1 | process fdfs_storaged already running, pid: 28250 |
查看监听端口:
1 | sudo netstat -unltp| grep fdfs |
如果显示如下,则证明storage
启动成功:
1 | tcp 0 0 0.0.0.0:23000 0.0.0.0:* LISTEN 28250 /fdfs_storaged |
如果没有显示,则有可能是storage
启动失败。
那么我们以Bastion6
服务器为例,查看启动日志:
1 | tail /bastion6_cache/fastdfs/storage/logs/storaged .log |
如果显示如下,代表启动失败:
1 2 | [2018-05-13 19:56:31] ERROR - file : storage_ip_changed_dealer.c, line: 186, connect to tracker server 118.138.2 40.146:22122 fail, errno: 110, error info: Connection timed out |
连接超时;
如果显示如下,代表启动成功:
1 2 3 4 | [2018-05-14 12:59:40] INFO - file : tracker_client_thread.c, line: 310, successfully connect to tracker server 118.138.240.146:22122, as a tracker client, my ip is 118.138.233.26 [2018-05-14 13:00:10] INFO - file : tracker_client_thread.c, line: 1263, tracker server 118.138.240.146:22122, set tracker leader: 118.138.240.146:22122 [2018-05-14 13:03:06] INFO - file : storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000, continuous fail count: 16 [2018-05-14 13:03:41] INFO - file : storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000 |
连接成功。
超时解读:
之前跟Chris
讨论的时候,刚开始以为这几天服务器都是Monash
云服务器中心的,属于内网,所以相互之间的端口是不用开的,实验之后才知道不开端口是不行的。后来联系Jerico
开开3
个端口之后,才连接成功。这3
个端口分别是22122
、8888
和23000
:
22122
:代表tracker
服务端口;8888
:代表HTTP
协议端口,网页上传下载文件需要这个端口;23000
:代表storage
服务端口。
至此,storage
存储节点安装成功。
4.4 查看所有存储节点信息
所有存储节点都启动之后,在任意一台storage
上面用下面命令,查看集群状态信息:
1 | sudo /usr/bin/fdfs_monitor /etc/fdfs/storage .conf |
显示如下:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 | [2018-05-24 01:27:18] DEBUG - base_path= /bastion6_cache/fastdfs/storage , connect_timeout=10, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=1, server_index=0 tracker server is 118.138.241.39:22122 group count: 1 Group 1: group name = group1 disk total space = 100665 MB disk free space = 72339 MB trunk free space = 0 MB storage server count = 4 active server count = 4 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 118.138.233.26 ip_addr = 118.138.233.26 (vm-118-138-233-26.erc.monash.edu.au) ACTIVE http domain = version = 5.12 join time = 2018-05-13 19:55:46 up time = 2018-05-20 00:51:17 total storage = 483679 MB free storage = 452554 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 3 connection.max_count = 3 total_upload_count = 1 success_upload_count = 1 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 87026 success_upload_bytes = 87026 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 11288 success_sync_in_bytes = 11288 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 5 success_file_open_count = 5 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 5 success_file_write_count = 5 last_heart_beat_time = 2018-05-24 01:26:54 last_source_update = 2018-05-17 01:27:29 last_sync_update = 2018-05-17 01:06:46 last_synced_timestamp = 1970-01-01 10:00:00 (never synced) Storage 2: id = 118.138.233.27 ip_addr = 118.138.233.27 (vm-118-138-233-27.erc.monash.edu.au) ACTIVE http domain = version = 5.12 join time = 2018-05-17 00:39:17 up time = 2018-05-20 00:52:30 total storage = 2015737 MB free storage = 1689979 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 3 connection.max_count = 3 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 98314 success_sync_in_bytes = 98314 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 5 success_file_open_count = 5 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 5 success_file_write_count = 5 last_heart_beat_time = 2018-05-24 01:27:08 last_source_update = 1970-01-01 10:00:00 last_sync_update = 2018-05-21 16:32:36 last_synced_timestamp = 2018-05-17 01:06:38 (20m:51s delay) Storage 3: id = 118.138.233.74 ip_addr = 118.138.233.74 (vm-118-138-233-74.erc.monash.edu.au) ACTIVE http domain = version = 5.12 join time = 2018-05-13 20:03:06 up time = 2018-05-20 00:53:40 total storage = 100665 MB free storage = 72339 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 3 connection.max_count = 3 total_upload_count = 2 success_upload_count = 2 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 2 success_set_meta_count = 2 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 11190 success_upload_bytes = 11190 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 174142 success_sync_in_bytes = 87026 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 3 success_file_open_count = 3 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 3 success_file_write_count = 3 last_heart_beat_time = 2018-05-24 01:27:13 last_source_update = 2018-05-17 01:06:38 last_sync_update = 2018-05-21 16:32:54 last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay) Storage 4: id = 118.138.240.146 ip_addr = 118.138.240.146 (vm-118-138-240-146.erc.monash.edu.au) ACTIVE http domain = version = 5.12 join time = 2018-05-20 00:53:13 up time = 2018-05-20 00:53:13 total storage = 483679 MB free storage = 449872 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = 118.138.233.26 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 3 connection.max_count = 3 total_upload_count = 0 success_upload_count = 0 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 0 success_set_meta_count = 0 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 0 success_upload_bytes = 0 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 98314 success_sync_in_bytes = 98314 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 5 success_file_open_count = 5 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 5 success_file_write_count = 5 last_heart_beat_time = 2018-05-24 01:27:03 last_source_update = 1970-01-01 10:00:00 last_sync_update = 2018-05-21 16:32:54 last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay) |
可以看到,4
台storage
状态都是ACTIVE
,表示都启动成功了。
5. 测试上传文件
需要注意一点,我们之后的特征提取程序会在feagen
服务器上,所以,我们暂时将client
安装在这台服务器上。测试一下在本台服务器上上传文件,能否在所有storage
服务器上面查看得到此文件的备份。
5.1 修改tracker
服务器上的client
配置文件
先后输入以下命令:
1 2 3 | cd /etc/fdfs sudo cp client.conf.sample client.conf sudo vim client.conf |
将base_path
和tracker_server
两处配置修改为:
1 2 | base_path= /feagen/fastdfs/client tracker_server=118.138.241.39:22122 |
在shell
当中输入下面的命令:
1 | sudo /usr/bin/fdfs_upload_file /etc/fdfs/client .conf fastdfs-master.zip |
返回如下,即可表明上传成功:
1 | group1 /M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904 .zip |
这个返回值,在perl
当中,以如下方式获取:
1 | my $file_name = qx(sudo /usr/bin/fdfs_upload_file /etc/fdfs/client.conf fastdfs-master.zip) ; |
到任意一个storage
服务器当中查看,以Bastion3
为例,至此处:
1 | /bastion3_cache/fastdfs/storage/data/00/00 |
发现存在如下文件:
1 | dorpG1td37yAFSYPAAZ-Sk23ivY904.zip |
即表明上传成功。
6. 测试下载文件
当前目录下,运行下面的程序:
1 | sudo /usr/bin/fdfs_download_file /etc/fdfs/client .conf group1 /M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904 .zip |
即可在当前目录下面找到dorpG1td37yAFSYPAAZ-Sk23ivY904.zip
,表明下载成功。
至此我们的分布式文件服务器算是搭建完成了。
我们参考了这些文章,表示感谢!
FastDFS分布式文件系统集群安装与配置
FastDFS--原理篇
分布式文件系统FastDFS原理介绍
Ubuntu下安装并配置FastDFS