您正在查看: 服务器/数据库 分类下的文章

Python3连接PostgreSQL数据库

Overview

之前项目的人遗留的数据散落在多种数据库中,既有MySQLMongoDB,也有CassandraPostgreSQL。在Python3版本的jupyter中连接PostgreSQL需要安装psycopg2,而psycopg2Python2中则是已经集成好的。

1. 安装python3-psycopg2libpq-dev

先在Linux上安装好这两个包,

sudo apt-get install python3-psycopg2
sudo apt-get install libpq-dev

再安装psycopg2,

pip3 install psycopg2

否则就会报错如下:

ERROR: Command errored out with exit status 1:
     command: /home/yangbingjiao/anaconda3/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wxj7s4lx/psycopg2/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wxj7s4lx/psycopg2/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info
         cwd: /tmp/pip-install-wxj7s4lx/psycopg2/
    Complete output (23 lines):
    running egg_info
    creating pip-egg-info/psycopg2.egg-info
    writing pip-egg-info/psycopg2.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/psycopg2.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/psycopg2.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/psycopg2.egg-info/SOURCES.txt'

    Error: pg_config executable not found.

    pg_config is required to build psycopg2 from source.  Please add the directory
    containing pg_config to the $PATH or specify the full executable path with the
    option:

        python setup.py build_ext --pg-config /path/to/pg_config build ...

    or with the pg_config option in 'setup.cfg'.

    If you prefer to avoid building psycopg2 from source, please install the PyPI
    'psycopg2-binary' package instead.

    For further information please check the 'doc/src/install.rst' file (also at
    <http://initd.org/psycopg/docs/install.html>).

    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

2.连接PostgreSQL数据库

import psycopg2
import pandas as pd

conn = psycopg2.connect(database="database", user="user", password="password", host="192.168.1.230", port="5432")
cur = conn.cursor()
pql = """
SELECT * 
FROM database.table
WHERE  create_time >= '2019-09-23' AND create_time < '2019-09-30'
order by create_time
limit 10
"""
cur.execute(pql)
rows = cur.fetchall()
df = pd.DataFrame(rows)
conn.close()

本文主要参考了以下文章:
Python3连接PostgreSQL(10.5)数据库

php页面不能被解析成网页,而是直接显示了页面代码

Overview

之前安装过很多次的phpMyAdmin,但最近一次安装时,出现了一个之前从来没遇到的问题:访问phpMyAdmin时,主页的php页面会被不能被解析成网页,而是直接显示了页面代码。

问题描述

文字描述已经比较清楚了,就不截图了。如果想看截图,参考 phpmyadmin showing php script [duplicate]

对比之前的安装流程,发现这次安装Apache2时,php还没装(这个一般装Ubuntu系统时都会自动装上,这次居然没装,所以才会碰到这么少见的错误),所以php是在Apache2后面安装的。这样的问题在于,当Apache2安装时,没有检测到系统有php,所以就以为用户安装Apache2不是为了运行php网站(尽管Apache2最常见的应用就是做php网站容器,但实际上可以用作多种语言网站的容器,比如perlCGI程序,但需要手动配置添加专门的模块,使Apache2支持perlCGI程序,不然你通过Apache2访问你的perl页面,也会被直接显示为代码,而不是页面,参见配置Apache2服务器以CGI方式运行Perl程序),也就没有安装配置支持php的模块。

解决方案

  • 先安装php,再安装Apache2,这样Apache2安装时,会检测到php,然后自动安装配置好支持php网站的模块。

  • 如果不幸顺序反了(后面又发现,好像跟顺序无关,还是有可能报这个错误 =。=),那使用下面的命令手动给Apache2安装支持php的模块

    先用下面的命令看看你装的php版本:

    php --version
    

    出现下面的信息:

    PHP 7.0.32-0ubuntu0.16.04.1 (cli) ( NTS )
    Copyright (c) 1997-2017 The PHP Group
    Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies
        with Zend OPcache v7.0.32-0ubuntu0.16.04.1, Copyright (c) 1999-2017, by Zend    Technologies
    

    如果你装的是php 7.0,那通过下面的命令

    sudo apt-get install libapache2-mod-php7.0
    

    然后重启Apache2

    sudo service apache2 restart
    

    就可以了。

参考文献

Python包管理软件pip更新后报错

Overview

今天在服务器(Ubuntu 16.04)使用pip安装深度学习包Keras的时候,发现安装结束之后下面有一行提示小字,大约就是pip有新版本可以更新,使用XXX命令更新,就顺带更新了一下,更新完了pip就不能用了...

问题描述

及时是查看下pip的版本

sudo pip --version

也会报下面的错误:

Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

解决方案

差了很多资料,发现是pip新版本的一个bug,跟平台无关,Windows下也会出现这个问题(可以查看这里ImportError: cannot import name main when running pip --version command in windows7 32 bit)。

有3个解决办法:

  1. 直接修复这个bug,算是临时的解决方案。
    vim编辑/usr/bin/pip文件

     sudo vim /usr/bin/pip
    

    可以看到下面的内容

     from pip import main
     if __name__ == '__main__':
         sys.exit(main())
    

    把这部分内容注释掉,换成下面的内容:

    from pip import __main__
    if __name__ == '__main__':
        sys.exit(__main__._main())
    

    重新运行pip,不在报错了。

  2. Windows下的pip安装位置不太熟悉,所以不知道应该在哪里修改这个文件。如果不知道在哪里修改,也可以考虑用python命令讲pip降级到没有bug的版本:

    python -m pip uninstall pip
    python -m pip install pip==9.0.3
    

    这个没试过,不知道行不行。

  3. 也有人说这个bug是由pip的调用方式引起的,在调用pip时,别用:

    pip install <package>
    

    而是改用(python2):

    python -m pip install <package>
    

    或者(python3):

    python3 -m pip install <package>
    

参考文献

FastDFS分布式文件系统在服务器集群上的安装部署

Overview

到目前为止,我们手里已经有了10台服务器了。之前一直说要把这些服务器搭建一个分布式文件系统,现在条件终于成熟了。这些服务器预装的系统大多数是Ubuntu16.04LTS发行版的Linux,少部分是Ubuntu14.04LTS。这次我们选择5台服务器,1feagen(118.138.241.39)服务器作为tracker4台服务器(Bastion3(118.138.240.146),Bastion4(118.138.233.74),Bastion6(118.138.233.26),POSSUM(118.138.233.27))作为storage。这里会记录下FastDFS分布式文件系统在Ubuntu16.04上的安装部署过程的细节。

5台服务器都要安装,只是配置文件稍有不同。

1.必备软件

  • FastDFS
    这个去GitHub上面下载源码:FastDFS
  • libfastcommon
    这个仍然要去GitHub上面下载源码:libfastcommon

这两个Zip包用wget或者在本地下载上传至服务器均可。

2.libfastcommon的安装

首先解压libfastcommon-master.zip包:

unzip libfastcommon-master.zip

进入这个文件夹:

cd libfastcommon-master/

依次输入如下命令:

sudo ./make.sh
sudo ./make.sh install

输入第二条命令之后,会有如下显示:

mkdir -p /usr/lib64
mkdir -p /usr/lib
install -m 755 libfastcommon.so /usr/lib64
install -m 755 libfastcommon.so /usr/lib
mkdir -p /usr/include/fastcommon
install -m 644 common_define.h hash.h chain.h logger.h base64.h shared_func.h pthread_func.h ini_file_reader.h _os_define.h sockopt.h sched_thread.h http_func.h md5.h local_ip_func.h avl_tree.h ioevent.h ioevent_loop.h fast_task_queue.h fast_timer.h process_ctrl.h fast_mblock.h connection_pool.h fast_mpool.h fast_allocator.h fast_buffer.h skiplist.h multi_skiplist.h flat_skiplist.h skiplist_common.h system_info.h fast_blocked_queue.h php7_ext_wrapper.h id_generator.h char_converter.h char_convert_loader.h /usr/include/fastcommon

可以看到libfastcommon.so安装到了/usr/lib64.
下面,我们需要为两个文件创建软链接,指向FastDFS主程序的lib目录:

sudo ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
sudo ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
sudo ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
sudo ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so

3.FastDFS的安装

解压fastdfs-master.zip文件:

unzip fastdfs-master.zip

进入FastDFS源码根目录:

cd fastdfs-master

依次输入下面两条命令:

sudo ./make.sh
sudo ./make.sh install

安装完成之后,所有的可执行文件都被放在了/usr/bin/目录下面,可以用下面的命令查看:

ls /usr/bin/fdfs*

显示共有14个文件:

/usr/bin/fdfs_appender_test   
/usr/bin/fdfs_download_file  
/usr/bin/fdfs_test1
/usr/bin/fdfs_appender_test1  
/usr/bin/fdfs_file_info      
/usr/bin/fdfs_trackerd
/usr/bin/fdfs_append_file     
/usr/bin/fdfs_monitor        
/usr/bin/fdfs_upload_appender
/usr/bin/fdfs_crc32           
/usr/bin/fdfs_storaged       
/usr/bin/fdfs_upload_file
/usr/bin/fdfs_delete_file     
/usr/bin/fdfs_test

4. 配置FastDFS

配置文件在目录/etc/fdfs下,查看一下:

ls /etc/fdfs

显示如下:

client.conf.sample  storage.conf.sample  storage_ids.conf.sample  tracker.conf.sample

4.1 创建跟踪器和存储节点的配置文件

cd /etc/fdfs
sudo cp tracker.conf.sample tracker.conf
sudo cp storage.conf.sample storage.conf

4.2 修改tracker配置文件

feagen(118.138.241.39)服务器中修改tracker配置文件如下:

sudo vim /etc/fdfs/tracker.conf

可以看到如下显示:

# the base path to store data and log files
base_path=/home/yuqing/fastdfs

这行改成如下内容:

base_path=/feagen/fastdfs/tracker

这种配置文件,注意等号两边不能有空格。且上面的目录必须是真实存在的。
启动tracker

sudo fdfs_trackerd /etc/fdfs/tracker.conf start

查看监听端口:

sudo netstat -unltp|grep fdfs

如果显示如下,则证明tracker启动成功:

tcp        0      0 0.0.0.0:22122           0.0.0.0:*               LISTEN      6951/fdfs_trackerd

4.3 修改storage配置文件

Bastion3(118.138.240.146)服务器中,修改storage配置文件如下:

sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

group_name=group1
base_path=/bastion3_cache/fastdfs/storage
store_path0=/bastion3_cache/fastdfs/storage
tracker_server=118.138.241.39:22122

Bastion4(118.138.233.74)服务器中,修改storage配置文件如下:

sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

group_name=group1
base_path=/bastion4_cache/fastdfs/storage
store_path0=/bastion4_cache/fastdfs/storage
tracker_server=118.138.241.39:22122

Bastion6(118.138.233.26)服务器中,修改storage配置文件如下:

sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

group_name=group1
base_path=/bastion6_cache/fastdfs/storage
store_path0=/bastion6_cache/fastdfs/storage
tracker_server=118.138.241.39:22122

POSSUM(118.138.233.27)服务器中,修改storage配置文件如下:

sudo vim /etc/fdfs/storage.conf

可以看到如下显示:

group_name=group1
base_path=/home/yuqing/fastdfs
store_path0=/home/yuqing/fastdfs
tracker_server=192.168.209.121:22122

修改为:

group_name=group1
base_path=/possum/fastdfs/storage
store_path0=/possum/fastdfs/storage
tracker_server=118.138.241.39:22122

启动storage服务:

sudo fdfs_storaged /etc/fdfs/storage.conf start

显示如下,即代表成功启动:

process fdfs_storaged already running, pid: 28250

查看监听端口:

sudo netstat -unltp|grep fdfs

如果显示如下,则证明storage启动成功:

tcp        0      0 0.0.0.0:23000           0.0.0.0:*               LISTEN      28250/fdfs_storaged

如果没有显示,则有可能是storage启动失败。

那么我们以Bastion6服务器为例,查看启动日志:

tail /bastion6_cache/fastdfs/storage/logs/storaged.log

如果显示如下,代表启动失败:

[2018-05-13 19:56:31] ERROR - file: storage_ip_changed_dealer.c, line: 186, connect to tracker server 118.138.2
40.146:22122 fail, errno: 110, error info: Connection timed out

连接超时;
如果显示如下,代表启动成功:

[2018-05-14 12:59:40] INFO - file: tracker_client_thread.c, line: 310, successfully connect to tracker server 118.138.240.146:22122, as a tracker client, my ip is 118.138.233.26
[2018-05-14 13:00:10] INFO - file: tracker_client_thread.c, line: 1263, tracker server 118.138.240.146:22122, set tracker leader: 118.138.240.146:22122
[2018-05-14 13:03:06] INFO - file: storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000, continuous fail count: 16
[2018-05-14 13:03:41] INFO - file: storage_sync.c, line: 2733, successfully connect to storage server 118.138.233.74:23000

连接成功。

超时解读:
之前跟Chris讨论的时候,刚开始以为这几天服务器都是Monash云服务器中心的,属于内网,所以相互之间的端口是不用开的,实验之后才知道不开端口是不行的。后来联系Jerico开开3个端口之后,才连接成功。这3个端口分别是22122888823000

  • 22122:代表tracker服务端口;
  • 8888:代表HTTP协议端口,网页上传下载文件需要这个端口;
  • 23000:代表storage服务端口。

至此,storage存储节点安装成功。

4.4 查看所有存储节点信息

所有存储节点都启动之后,在任意一台storage上面用下面命令,查看集群状态信息:

sudo /usr/bin/fdfs_monitor /etc/fdfs/storage.conf

显示如下:

[2018-05-24 01:27:18] DEBUG - base_path=/bastion6_cache/fastdfs/storage, connect_timeout=10, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 118.138.241.39:22122

group count: 1

Group 1:
group name = group1
disk total space = 100665 MB
disk free space = 72339 MB
trunk free space = 0 MB
storage server count = 4
active server count = 4
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

    Storage 1:
        id = 118.138.233.26
        ip_addr = 118.138.233.26 (vm-118-138-233-26.erc.monash.edu.au)  ACTIVE
        http domain = 
        version = 5.12
        join time = 2018-05-13 19:55:46
        up time = 2018-05-20 00:51:17
        total storage = 483679 MB
        free storage = 452554 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id = 
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 1
        success_upload_count = 1
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 0
        success_set_meta_count = 0
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 87026
        success_upload_bytes = 87026
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 11288
        success_sync_in_bytes = 11288
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 5
        success_file_open_count = 5
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 5
        success_file_write_count = 5
        last_heart_beat_time = 2018-05-24 01:26:54
        last_source_update = 2018-05-17 01:27:29
        last_sync_update = 2018-05-17 01:06:46
        last_synced_timestamp = 1970-01-01 10:00:00 (never synced)
    Storage 2:
        id = 118.138.233.27
        ip_addr = 118.138.233.27 (vm-118-138-233-27.erc.monash.edu.au)  ACTIVE
        http domain = 
        version = 5.12
        join time = 2018-05-17 00:39:17
        up time = 2018-05-20 00:52:30
        total storage = 2015737 MB
        free storage = 1689979 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id = 
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 0
        success_upload_count = 0
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 0
        success_set_meta_count = 0
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 0
        success_upload_bytes = 0
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 98314
        success_sync_in_bytes = 98314
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 5
        success_file_open_count = 5
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 5
        success_file_write_count = 5
        last_heart_beat_time = 2018-05-24 01:27:08
        last_source_update = 1970-01-01 10:00:00
        last_sync_update = 2018-05-21 16:32:36
        last_synced_timestamp = 2018-05-17 01:06:38 (20m:51s delay)
    Storage 3:
        id = 118.138.233.74
        ip_addr = 118.138.233.74 (vm-118-138-233-74.erc.monash.edu.au)  ACTIVE
        http domain = 
        version = 5.12
        join time = 2018-05-13 20:03:06
        up time = 2018-05-20 00:53:40
        total storage = 100665 MB
        free storage = 72339 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id = 
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 2
        success_upload_count = 2
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 2
        success_set_meta_count = 2
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 11190
        success_upload_bytes = 11190
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 174142
        success_sync_in_bytes = 87026
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 3
        success_file_open_count = 3
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 3
        success_file_write_count = 3
        last_heart_beat_time = 2018-05-24 01:27:13
        last_source_update = 2018-05-17 01:06:38
        last_sync_update = 2018-05-21 16:32:54
        last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay)
    Storage 4:
        id = 118.138.240.146
        ip_addr = 118.138.240.146 (vm-118-138-240-146.erc.monash.edu.au)  ACTIVE
        http domain = 
        version = 5.12
        join time = 2018-05-20 00:53:13
        up time = 2018-05-20 00:53:13
        total storage = 483679 MB
        free storage = 449872 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id = 118.138.233.26
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 3
        connection.max_count = 3
        total_upload_count = 0
        success_upload_count = 0
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 0
        success_set_meta_count = 0
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 0
        success_download_count = 0
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 0
        success_upload_bytes = 0
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 0
        success_download_bytes = 0
        total_sync_in_bytes = 98314
        success_sync_in_bytes = 98314
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 5
        success_file_open_count = 5
        total_file_read_count = 0
        success_file_read_count = 0
        total_file_write_count = 5
        success_file_write_count = 5
        last_heart_beat_time = 2018-05-24 01:27:03
        last_source_update = 1970-01-01 10:00:00
        last_sync_update = 2018-05-21 16:32:54
        last_synced_timestamp = 2018-05-17 01:27:30 (-1s delay)

可以看到,4storage状态都是ACTIVE,表示都启动成功了。

5. 测试上传文件

需要注意一点,我们之后的特征提取程序会在feagen服务器上,所以,我们暂时将client安装在这台服务器上。测试一下在本台服务器上上传文件,能否在所有storage服务器上面查看得到此文件的备份。

5.1 修改tracker服务器上的client配置文件

先后输入以下命令:

cd /etc/fdfs
sudo cp client.conf.sample client.conf
sudo vim client.conf

base_pathtracker_server两处配置修改为:

base_path=/feagen/fastdfs/client
tracker_server=118.138.241.39:22122

shell当中输入下面的命令:

sudo /usr/bin/fdfs_upload_file /etc/fdfs/client.conf fastdfs-master.zip

返回如下,即可表明上传成功:

group1/M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904.zip

这个返回值,在perl当中,以如下方式获取:

my $file_name = qx(sudo /usr/bin/fdfs_upload_file /etc/fdfs/client.conf fastdfs-master.zip);

到任意一个storage服务器当中查看,以Bastion3为例,至此处:

/bastion3_cache/fastdfs/storage/data/00/00

发现存在如下文件:

dorpG1td37yAFSYPAAZ-Sk23ivY904.zip

即表明上传成功。

6. 测试下载文件

当前目录下,运行下面的程序:

sudo /usr/bin/fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/dorpG1td37yAFSYPAAZ-Sk23ivY904.zip

即可在当前目录下面找到dorpG1td37yAFSYPAAZ-Sk23ivY904.zip,表明下载成功。

至此我们的分布式文件服务器算是搭建完成了。

我们参考了这些文章,表示感谢!
FastDFS分布式文件系统集群安装与配置
FastDFS--原理篇
分布式文件系统FastDFS原理介绍
Ubuntu下安装并配置FastDFS

一个Strut2引发的网站安全漏洞

Overview

前几天忽然收到学校云服务器安全中心的邮件,说我们部署的几台服务器流量明显出现异常,可能是被攻击了,并提供了几个可能的问题。大致看了下,问题出在Strut2,这不是个特有的问题,而是涉及到多个版本的安全问题,所以在这里记录一下。

问题描述

Struts2存在远程代码执行的严重漏洞,目前Struts2官方已经确认漏洞(漏洞编号S2-045CVE编号:cve-2017-5638),并定级为高危风险。

  • 漏洞危害程度严重,可造成直接获取应用系统所在服务器的控制权限。
  • 黑客可以利用该漏洞通过浏览器在远程服务器上执行任意系统命令,将会对受影响站点造成严重影响,引发数据泄露、网页篡改、植入后门、成为肉鸡等安全事件。
  • 漏洞利用无任何条件限制,可绕过绝大多数防护设备的通用防护策略!
  • 该漏洞影响范围较广(Struts 2.3.5 - Struts 2.3.31, Struts 2.5 - Struts 2.5.10

我们当前使用的版本号恰好就在上述范围中,从安全性报告大约可以知道,我们的服务器已经变成转发流量的肉鸡了。

解决方案

官方解决方案

升级到不受影响的版本(Struts 2.3.32Struts 2.5.10.1):

如果打不开上面的网页,自己搜索一下应该也很容易下载到。

临时解决方案

如果暂时不方便升级,也可以通过配置目前的版本来避免这次安全漏洞,请参见:一份数据告诉你,被万年漏洞王 Struts2 坑了的网站有哪些

小结

如果服务器已经被攻击,多半需要重新安装服务器以清理原来服务器上的后门程序。我们清空了受攻击的服务器,重新申请了一个新的云服务器挂载到了之前的网址下,现在来看,一切正常。

参考网址