Overview
到目前为止，我们手里已经有了10台服务器了。之前一直说要把这些服务器搭建一个分布式文件系统，现在条件终于成熟了。这些服务器预装的系统大多数是Ubuntu16.04LTS发行版的Linux，少部分是Ubuntu14.04LTS。这次我们选择5台服务器，1台feagen(118.138.241.39)服务器作为tracker，4台服务器(Bastion3(118.138.240.146)，Bastion4(118.138.233.74)，Bastion6(118.138.233.26)，POSSUM(118.138.233.27))作为storage。这里会记录下FastDFS分布式文件系统在Ubuntu16.04上的安装部署过程的细节。
5台服务器都要安装，只是配置文件稍有不同。
1.必备软件
FastDFS
这个去GitHub上面下载源码：FastDFSlibfastcommon
这个仍然要去GitHub上面下载源码：libfastcommon
这两个Zip包用wget或者在本地下载上传至服务器均可。
2.libfastcommon的安装
首先解压libfastcommon-master.zip包：
unzip libfastcommon-master.zip
进入这个文件夹：
cd libfastcommon-master/
依次输入如下命令：
sudo ./make.sh
sudo ./make.sh install
输入第二条命令之后，会有如下显示：
mkdir -p /usr/lib64
mkdir -p /usr/lib
install -m 755 libfastcommon.so /usr/lib64
install -m 755 libfastcommon.so /usr/lib
mkdir -p /usr/include/fastcommon
install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h id_generator.h char_converter.h char_convert_loader.h /usr/include/fastcommon
可以看到libfastcommon.so安装到了/usr/lib64.
下面，我们需要为两个文件创建软链接，指向FastDFS主程序的lib目录：
sudo ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
sudo ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
sudo ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
sudo ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
3.FastDFS的安装
解压fastdfs-master.zip文件:
unzip fastdfs-master.zip
进入FastDFS源码根目录:
cd fastdfs-master
依次输入下面两条命令：
sudo ./make.sh
sudo ./make.sh install
安装完成之后，所有的可执行文件都被放在了/usr/bin/目录下面，可以用下面的命令查看：
ls /usr/bin/fdfs*
显示共有14个文件：
/usr/bin/fdfs_appender_test
/usr/bin/fdfs_download_file
/usr/bin/fdfs_test1
/usr/bin/fdfs_appender_test1
/usr/bin/fdfs_file_info
/usr/bin/fdfs_trackerd
/usr/bin/fdfs_append_file
/usr/bin/fdfs_monitor
/usr/bin/fdfs_upload_appender
/usr/bin/fdfs_crc32
/usr/bin/fdfs_storaged
/usr/bin/fdfs_upload_file
/usr/bin/fdfs_delete_file
/usr/bin/fdfs_test
4. 配置FastDFS
配置文件在目录/etc/fdfs下，查看一下：
ls /etc/fdfs
显示如下：
client.conf.sample storage.conf.sample storage_ids.conf.sample tracker.conf.sample
4.1 创建跟踪器和存储节点的配置文件
cd /etc/fdfs
sudo cp tracker.conf.sample tracker.conf
sudo cp storage.conf.sample storage.conf
4.2 修改tracker配置文件
在feagen(118.138.241.39)服务器中修改tracker配置文件如下：
sudo vim /etc/fdfs/tracker.conf
可以看到如下显示：
# the base path to store data and log files
base_path=/home/yuqing/fastdfs
这行改成如下内容：
base_path=/feagen/fastdfs/tracker
这种配置文件，注意等号两边不能有空格。且上面的目录必须是真实存在的。
启动tracker：
sudo fdfs_trackerd /etc/fdfs/tracker.conf start
查看监听端口：
sudo netstat -unltp|grep fdfs
如果显示如下，则证明tracker启动成功：
tcp 0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 6951/fdfs_trackerd
4.3 修改storage配置文件
在Bastion3(118.138.240.146)服务器中，修改storage配置文件如下：
sudo vim /etc/fdfs/storage.conf
可以看到如下显示：
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122
修改为：
group_name=group1
base_path=/bastion3_cache/fastdfs/storage
store_path0=/bastion3_cache/fastdfs/storage
tracker_server=118.138.241.39:22122
在Bastion4(118.138.233.74)服务器中，修改storage配置文件如下：
sudo vim /etc/fdfs/storage.conf
可以看到如下显示：
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122
修改为：
group_name=group1
base_path=/bastion4_cache/fastdfs/storage
store_path0=/bastion4_cache/fastdfs/storage
tracker_server=118.138.241.39:22122
在Bastion6(118.138.233.26)服务器中，修改storage配置文件如下：
sudo vim /etc/fdfs/storage.conf
可以看到如下显示：
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122
修改为：
group_name=group1
base_path=/bastion6_cache/fastdfs/storage
store_path0=/bastion6_cache/fastdfs/storage
tracker_server=118.138.241.39:22122
在POSSUM(118.138.233.27)服务器中，修改storage配置文件如下：
sudo vim /etc/fdfs/storage.conf
可以看到如下显示：
group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122
修改为：
group_name=group1
base_path=/possum/fastdfs/storage
store_path0=/possum/fastdfs/storage
tracker_server=118.138.241.39:22122
启动storage服务：
sudo fdfs_storaged /etc/fdfs/storage.conf start
显示如下，即代表成功启动：
process fdfs_storaged already running, pid: 28250
查看监听端口：
sudo netstat -unltp|grep fdfs
如果显示如下，则证明storage启动成功：
tcp 0 0 0.0.0.0:23000 0.0.0.0:* LISTEN 28250/fdfs_storaged
如果没有显示，则有可能是storage启动失败。
那么我们以Bastion6服务器为例，查看启动日志：
tail /bastion6_cache/fastdfs/storage/logs/storaged.log
如果显示如下，代表启动失败：
[2018-05-13 19:56:31] ERROR - file: storage_ip_changed_dealer.c, line: 186, connect to tracker server 118.138.2
40.146:22122 fail, errno: 110, error info: Connection timed out
连接超时；
如果显示如下，代表启动成功：
[2018-05-14 12:59:40] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 118.138.240.146:22122, as a tracker client, my ip is 118.138.233.26
[2018-05-14 13:00:10] INFO - file: tracker_client_thread.c, line: 1263, tracker server 118.138.240.146:22122, set tracker leader: 118.138.240.146:22122
[2018-05-14 13:03:06] INFO - file: storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000, continuous fail count: 16
[2018-05-14 13:03:41] INFO - file: storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000
连接成功。
超时解读：
之前跟Chris讨论的时候，刚开始以为这几天服务器都是Monash云服务器中心的，属于内网，所以相互之间的端口是不用开的，实验之后才知道不开端口是不行的。后来联系Jerico开开3个端口之后，才连接成功。这3个端口分别是22122、8888和23000：
22122:代表tracker服务端口；8888:代表HTTP协议端口，网页上传下载文件需要这个端口；23000:代表storage服务端口。
至此，storage存储节点安装成功。
4.4 查看所有存储节点信息
所有存储节点都启动之后，在任意一台storage上面用下面命令，查看集群状态信息：
sudo /usr/bin/fdfs_monitor /etc/fdfs/storage.conf
显示如下：
[2018-05-24 01:27:18] DEBUG - base_path=/bastion6_cache/fastdfs/storage, connect_timeout=10, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
server_count=1, server_index=0
tracker server is 118.138.241.39:22122
group count: 1
Group 1:
group name = group1
disk total space = 100665 MB
disk free space = 72339 MB
trunk free space = 0 MB
storage server count = 4
active server count = 4
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0
Storage 1:
id = 118.138.233.26
ip_addr = 118.138.233.26 (vm-118-138-233-26.erc.monash.edu.au) ACTIVE
http domain =
version = 5.12
join time = 2018-05-13 19:55:46
up time = 2018-05-20 00:51:17
total storage = 483679 MB
free storage = 452554 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id =
if_trunk_server = 0
connection.alloc_count = 256
connection.current_count = 3
connection.max_count = 3
total_upload_count = 1
success_upload_count = 1
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 0
success_set_meta_count = 0
total_delete_count = 0
success_delete_count = 0
total_download_count = 0
success_download_count = 0
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 87026
success_upload_bytes = 87026
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 0
success_download_bytes = 0
total_sync_in_bytes = 11288
success_sync_in_bytes = 11288
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 5
success_file_open_count = 5
total_file_read_count = 0
success_file_read_count = 0
total_file_write_count = 5
success_file_write_count = 5
last_heart_beat_time = 2018-05-24 01:26:54
last_source_update = 2018-05-17 01:27:29
last_sync_update = 2018-05-17 01:06:46
last_synced_timestamp = 1970-01-01 10:00:00 (never synced)
Storage 2:
id = 118.138.233.27
ip_addr = 118.138.233.27 (vm-118-138-233-27.erc.monash.edu.au) ACTIVE
http domain =
version = 5.12
join time = 2018-05-17 00:39:17
up time = 2018-05-20 00:52:30
total storage = 2015737 MB
free storage = 1689979 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id =
if_trunk_server = 0
connection.alloc_count = 256
connection.current_count = 3
connection.max_count = 3
total_upload_count = 0
success_upload_count = 0
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 0
success_set_meta_count = 0
total_delete_count = 0
success_delete_count = 0
total_download_count = 0
success_download_count = 0
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 0
success_upload_bytes = 0
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 0
success_download_bytes = 0
total_sync_in_bytes = 98314
success_sync_in_bytes = 98314
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 5
success_file_open_count = 5
total_file_read_count = 0
success_file_read_count = 0
total_file_write_count = 5
success_file_write_count = 5
last_heart_beat_time = 2018-05-24 01:27:08
last_source_update = 1970-01-01 10:00:00
last_sync_update = 2018-05-21 16:32:36
last_synced_timestamp = 2018-05-17 01:06:38 (20m:51s delay)
Storage 3:
id = 118.138.233.74
ip_addr = 118.138.233.74 (vm-118-138-233-74.erc.monash.edu.au) ACTIVE
http domain =
version = 5.12
join time = 2018-05-13 20:03:06
up time = 2018-05-20 00:53:40
total storage = 100665 MB
free storage = 72339 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id =
if_trunk_server = 0
connection.alloc_count = 256
connection.current_count = 3
connection.max_count = 3
total_upload_count = 2
success_upload_count = 2
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 2
success_set_meta_count = 2
total_delete_count = 0
success_delete_count = 0
total_download_count = 0
success_download_count = 0
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 11190
success_upload_bytes = 11190
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 0
success_download_bytes = 0
total_sync_in_bytes = 174142
success_sync_in_bytes = 87026
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 3
success_file_open_count = 3
total_file_read_count = 0
success_file_read_count = 0
total_file_write_count = 3
success_file_write_count = 3
last_heart_beat_time = 2018-05-24 01:27:13
last_source_update = 2018-05-17 01:06:38
last_sync_update = 2018-05-21 16:32:54
last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay)
Storage 4:
id = 118.138.240.146
ip_addr = 118.138.240.146 (vm-118-138-240-146.erc.monash.edu.au) ACTIVE
http domain =
version = 5.12
join time = 2018-05-20 00:53:13
up time = 2018-05-20 00:53:13
total storage = 483679 MB
free storage = 449872 MB
upload priority = 10
store_path_count = 1
subdir_count_per_path = 256
storage_port = 23000
storage_http_port = 8888
current_write_path = 0
source storage id = 118.138.233.26
if_trunk_server = 0
connection.alloc_count = 256
connection.current_count = 3
connection.max_count = 3
total_upload_count = 0
success_upload_count = 0
total_append_count = 0
success_append_count = 0
total_modify_count = 0
success_modify_count = 0
total_truncate_count = 0
success_truncate_count = 0
total_set_meta_count = 0
success_set_meta_count = 0
total_delete_count = 0
success_delete_count = 0
total_download_count = 0
success_download_count = 0
total_get_meta_count = 0
success_get_meta_count = 0
total_create_link_count = 0
success_create_link_count = 0
total_delete_link_count = 0
success_delete_link_count = 0
total_upload_bytes = 0
success_upload_bytes = 0
total_append_bytes = 0
success_append_bytes = 0
total_modify_bytes = 0
success_modify_bytes = 0
stotal_download_bytes = 0
success_download_bytes = 0
total_sync_in_bytes = 98314
success_sync_in_bytes = 98314
total_sync_out_bytes = 0
success_sync_out_bytes = 0
total_file_open_count = 5
success_file_open_count = 5
total_file_read_count = 0
success_file_read_count = 0
total_file_write_count = 5
success_file_write_count = 5
last_heart_beat_time = 2018-05-24 01:27:03
last_source_update = 1970-01-01 10:00:00
last_sync_update = 2018-05-21 16:32:54
last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay)
可以看到，4台storage状态都是ACTIVE，表示都启动成功了。
5. 测试上传文件
需要注意一点，我们之后的特征提取程序会在feagen服务器上，所以，我们暂时将client安装在这台服务器上。测试一下在本台服务器上上传文件，能否在所有storage服务器上面查看得到此文件的备份。
5.1 修改tracker服务器上的client配置文件
先后输入以下命令:
cd /etc/fdfs
sudo cp client.conf.sample client.conf
sudo vim client.conf
将base_path和tracker_server两处配置修改为:
base_path=/feagen/fastdfs/client
tracker_server=118.138.241.39:22122
在shell当中输入下面的命令:
sudo /usr/bin/fdfs_upload_file /etc/fdfs/client.conf fastdfs-master.zip
返回如下，即可表明上传成功：
group1/M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904.zip
这个返回值，在perl当中，以如下方式获取：
my $file_name = qx(sudo /usr/bin/fdfs_upload_file /etc/fdfs/client.conf fastdfs-master.zip);
到任意一个storage服务器当中查看，以Bastion3为例，至此处：
/bastion3_cache/fastdfs/storage/data/00/00
发现存在如下文件:
dorpG1td37yAFSYPAAZ-Sk23ivY904.zip
即表明上传成功。
6. 测试下载文件
当前目录下，运行下面的程序:
sudo /usr/bin/fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904.zip
即可在当前目录下面找到dorpG1td37yAFSYPAAZ-Sk23ivY904.zip，表明下载成功。
至此我们的分布式文件服务器算是搭建完成了。
我们参考了这些文章，表示感谢！
FastDFS分布式文件系统集群安装与配置
FastDFS--原理篇
分布式文件系统FastDFS原理介绍
Ubuntu下安装并配置FastDFS