2020年9月

中文网站
https://www.goaccess.cc/

英文网站
http://goaccess.io/

编译安装

#yum安装依赖包
yum install glib2 glib2-devel GeoIP-devel  ncurses-devel zlib zlib-devel
yum install gcc -y
rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm
yum -y install GeoIP-update


$ wget http://tar.goaccess.io/goaccess-1.2.tar.gz
$ tar -xzvf goaccess-1.2.tar.gz
$ cd goaccess-1.2/
$ ./configure --enable-utf8 --enable-geoip=legacy
$ make
# make install


#修改/etc/nginx/nginx.conf文件的日志存储格式
log_format  main  '$remote_addr - $remote_user [$time_local] requesthost:"$http_host"; "$request" requesttime:"$request_time"; '
        '$status $body_bytes_sent "$http_referer" - $request_body'                      
        '"$http_user_agent" "$http_x_forwarded_for"';

        
#修改文件/usr/local/etc/goaccess.conf改成goaccess格式标准对应为
time-format %T
date-format %d/%b/%Y
log-format %h - %^ [%d:%t %^] requesthost:"%v"; "%r" requesttime:"%T"; %s %b "%R" - %^"%u"


#测试生成页面
goaccess -f /var/log/nginx/access.log -c -a>/usr/share/nginx/html/go.html


#后台实时生成数据到goaccess页面
/usr/local/bin/goaccess /var/log/nginx/access.log -o /usr/share/nginx/html/go.html  --real-time-html &

nginx默认日志格式

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

具体参考: https://nginx.org/en/docs/http/ngx_http_log_module.html#example

log_format格式变量:

$remote_addr  #记录访问网站的客户端地址

$remote_user  #远程客户端用户名

$time_local  #记录访问时间与时区

$request  #用户的http请求起始行信息

$status  #http状态码,记录请求返回的状态码,例如:200、301、404等

$body_bytes_sent  #服务器发送给客户端的响应body字节数

$http_referer  #记录此次请求是从哪个连接访问过来的,可以根据该参数进行防盗链设置。

$http_user_agent  #记录客户端访问信息,例如:浏览器、手机客户端等

$http_x_forwarded_for  #当前端有代理服务器时,设置web节点记录客户端地址的配置,此参数生效的前提是代理服务器也要进行相关的x_forwarded_for设置

其他还有:

 '"$upstream_addr" "$upstream_status" "$upstream_response_time" "$request_time"'

注意: 仅仅需要依赖 ncurses 模块。
提示: 如果需要使用实时 HTML 报告, 请确保开放 7890 端口。详见这里。
配置选项
GoAccess 拥有多个配置选项。获取完整的最新配置选项列表,请运行:./configure --help

--enable-debug
使用调试标志编译且关闭编译器优化。
--enable-utf8
宽字符支持。依赖 Ncursesw 模块。
--enable-geoip=<legacy|mmdb>
地理位置支持。依赖 MaxMind GeoIP 模块。legacy 将使用原始 GeoIP 数据库。mmdb 将使用增强版 GeoIP2 数据库。
--enable-tcb=<memhash|btree>
Tokyo Cabinet 存储支持。 memhash 将使用 Tokyo Cabinet 的内存哈希数据库。 btree 将使用 Tokyo Cabinet 的磁盘 B+Tree 数据库。
--disable-zlib
禁止在 B+Tree 数据库上使用 zlib 压缩。
--disable-bzip
禁止在 B+Tree 数据库上使用 bzip2 压缩。
--with-getline
使用动态扩展行缓冲区用来解析完整的行请求,否则将使用固定大小(4096)的缓冲区。
--with-openssl
使 GoAccess 与其 WebSocket 服务器之间的通信能够支持 OpenSSL。

Syntax: log_format name [escape=default|json|none] string ...;
Default:
log_format combined "...";
Context: http
Specifies log format.

The escape parameter (1.11.8) allows setting json or default characters escaping in variables, by default, default escaping is used. The none value (1.13.10) disables escaping.

For default escaping, characters “"”, “”, and other characters with values less than 32 (0.7.0) or above 126 (1.1.6) are escaped as “xXX”. If the variable value is not found, a hyphen (“-”) will be logged.

For json escaping, all characters not allowed in JSON strings will be escaped: characters “"” and “” are escaped as “"” and “\”, characters with values less than 32 are escaped as “n”, “r”, “t”, “b”, “f”, or “u00XX”.

The log format can contain common variables, and variables that exist only at the time of a log write:

$bytes_sent
the number of bytes sent to a client
$connection
connection serial number
$connection_requests
the current number of requests made through a connection (1.1.18)
$msec
time in seconds with a milliseconds resolution at the time of the log write
$pipe
“p” if request was pipelined, “.” otherwise
$request_length
request length (including request line, header, and request body)
$request_time
request processing time in seconds with a milliseconds resolution; time elapsed between the first bytes were read from the client and the log write after the last bytes were sent to the client
$status
response status
$time_iso8601
local time in the ISO 8601 standard format
$time_local
local time in the Common Log Format
In the modern nginx versions variables $status (1.3.2, 1.2.2), $bytes_sent (1.3.8, 1.2.5), $connection (1.3.8, 1.2.5), $connection_requests (1.3.8, 1.2.5), $msec (1.3.9, 1.2.6), $request_time (1.3.9, 1.2.6), $pipe (1.3.12, 1.2.7), $request_length (1.3.12, 1.2.7), $time_iso8601 (1.3.12, 1.2.7), and $time_local (1.3.12, 1.2.7) are also available as common variables.
Header lines sent to a client have the prefix “sent_http_”, for example, $sent_http_content_range.

The configuration always includes the predefined “combined” format:

log_format combined '$remote_addr - $remote_user [$time_local] '

                '"$request" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent"';

优化,http 200 300 记录, 其余不记录

map $status $loggable {
    ~^[23]  0;
    default 1;
}

access_log /path/to/access.log combined if=$loggable;

定时任务

0 0 1 * * goaccess -a -d -f /var/log/nginx/access.log -p /etc/goaccess.conf -o /var/log/nginx/goaccess.html 2> /var/log/nginx/goaccess.log

问题场景: SPA单页面应用, vuejs router, 微信WebView网页下给 JS SDK API 做 签名,报错

  1. 项目基于vue,在微信上的web应用
  2. 要使用原生系统某些功能(不同系统的兼容真的很多BUG,最后只能使用微信方案),支付功能,定位, 设置分享....
  3. 签名一直存在bug和问题,那么后面的功能都会唤不起(偶尔也可以)

具体表现:
项目使用 vue 开发微信手机网页端,获取从后端传来的 微信配置信息, android 端 能够配置正确并能 正常调用 微信提供的js api。 但是 在 ios 上 报 签名错误,重新刷新页面又能正确配置 微信的 config, 再次刷新页面又报 签名错误,

直接上解决代码

if (window.__wxjs_is_wkwebview) { // IOS, 签名地址,固定是入口地址,不是路由跳转地址
  window.entryUrl = window.location.href
}
router.afterEach((to, from) => {
  // window.__wxjs_is_wkwebview
  // true 时 为 IOS 设备
  // false时 为 安卓 设备
  if (window.__wxjs_is_wkwebview) {  // IOS
    // 什么也不做
  }else {       // 安卓, 签名地址,每次都变动
    window.entryUrl = `${window.location.origin}${to.fullPath}`
  }
})

后端做签名的时候, url encode 一下, 使用 window.entryUrl 而不是 window.location.href


问题的原因:

苹果微信App, 签名失败是因为: 苹果在微信中浏览器机制和安卓不同,有 IOS缓存问题,和IOS对单页面的优化问题,

安卓进行页面跳转分享时会刷新当前的history url,而苹果不会;

如果苹果下的微信App是通过vue router即H5的historyState历史记录进来的,不会刷新url所以会导致签名失败

问题来了: vue的history在IOS的微信不会刷新,但是我们又要做js签名。

有很多做法, 比如 beforerouteEnter的路由判断,或者写在Vue.prototype.Wxshare()的原型链中


微信下能检测内核的判断:window.__wxjs_is_wkwebview(是否为webview内核),如果是的情况下就变成true

因为SPA应用下,会有一定的问题,路由采用的是history模式(不带#号)。因为在页面每次进入到路由之后才去获取签名授权,所以将方法公用写在路由的模块下

window.entryUrl这个是什么鬼?这个是自己定义的全局属性,就是为了获取IOS第一次进入页面的时候存储起来的,如果IOS的签名的路径不是第一次进入的页面,那么就一定会失败,所以这个路由第一次进入就要储存起来

为什么要写在router.afterEach,因为页面进入了再去做申请和签名,如果在beforeEach,会导致页面申请签名的时候还是上一个页面,但是到了新页面却没有注册签名,或者因为签名的路径不同,导致invalid signature

安卓会存在一些情况,就是即便签名成功,但是还是会唤不起功能, 增加一个延时器或者页面mounted之后再签名一次比较稳妥


题外话

分享的跳转地址会携带一下参数

对于IOS系统会自动增加如下参数:
朋友圈 from=timeline&isappinstalled=0
微信群 from=groupmessage&isappinstalled=0
好友分享 from=singlemessage&isappinstalled=0

对于安卓系统会自动添加如下参数:
朋友圈 from=timeline
微信群 from=groupmessage
好友分享 from=singlemessage

任务: 更新

yum -y update
reboot

任务: ssh连接终端

查看IP地址
ifconfig
检查服务端口情况
netstat -natp |grep 22

任务: 删除 CentOS7 更新后产生的多余的内核

1.首先列出系统中正在使用的内核:

# uname -a 或者 uname -r

2.查询系统中全部的内核:

# rpm -qa | grep kernel

3.将你想删除的内核删除掉:(需要把所有含有kernel且相同版本号的全部删掉)

# yum remove -y kernel-modules-4.18.0-193.el8.x86_64
# yum remove -y kernel-4.18.0-193.el8.x86_64
# yum remove -y kernel-core-4.18.0-193.el8.x86_64

4.重启后就可以看到,内核被删掉了,同时多余的启动项也自动被删掉了。
reboot

任务: 补充安装常用命令

yum install -y wget

任务: 安装基础编译器,默认依赖Perl5

yum install -y gcc gcc-c++ make automake

任务: 安装CMake3

yum remove cmake
wget https://github.com/Kitware/CMake/releases/download/v3.17.5/cmake-3.17.5-Linux-x86_64.tar.gz
tar zxvf cmake-3.17.5-Linux-x86_64.tar.gz
vi /etc/profile.d/cmake.sh
export CMAKE_HOME=/root/cmake-3.17.5-Linux-x86_64
export PATH=$PATH:$CMAKE_HOME/bin
chmod +x /etc/profile.d/cmake.sh
source /etc/profile

(跳过)任务: 安装Perl5,官网教程 https://www.cpan.org/src/

wget https://www.cpan.org/src/5.0/perl-5.32.0.tar.gz
tar -xzf perl-5.32.0.tar.gz
cd perl-5.32.0
此时依赖make或gmake,还有gcc
./Configure -des -Dprefix=/usr/local/perl
make
make test
make install

软链接,替换掉旧的perl命令
mv /usr/bin/perl /usr/bin/perl.old
ln -s /usr/local/perl/bin/perl /usr/bin/perl

验证安装完成
perl -v

任务: 安装最新版openssl至/usr/local/openssl-1.1.1g

wget https://github.com/openssl/openssl/archive/OpenSSL_1_1_1g.tar.gz
tar zxvf OpenSSL_1_1_1g.tar.gz
cd openssl-OpenSSL_1_1_1g
此时需要Perl5
./config enable-weak-ssl-ciphers -fPIC --prefix=/usr/local/openssl-1.1.1g --openssldir=/usr/local/openssl-1.1.1g
make depend
make
make install

任务: 创建禁止登录终端的www用户组和用户

groupadd www
useradd -s /sbin/nologin -g www www

任务: 安装nginx
依赖 pcre openssl zlib

wget https://nginx.org/download/nginx-1.19.2.tar.gz
tar -xzf nginx-1.19.2.tar.gz
cd nginx-1.19.2
./configure --user=www --group=www --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_v2_module --with-http_gzip_static_module --with-http_sub_module  --with-http_realip_module --with-pcre=/ --with-stream --with-stream_ssl_module --with-openssl=/usr/local/openssl-1.1.1g --with-openssl-opt='enable-weak-ssl-ciphers' --with-http_addition_module --with-http_image_filter_module --with-http_flv_module --with-http_mp4_module --with-http_secure_link_module
make
make install

下载安装编译工具

yum groupinstall 'Development Tools'

安装依赖包

yum install libxml2 libxml2-devel openssl openssl-devel bzip2 bzip2-devel libcurl libcurl-devel libjpeg libjpeg-devel libpng libpng-devel freetype freetype-devel gmp gmp-devel libmcrypt libmcrypt-devel readline readline-devel libxslt libxslt-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel ncurses curl gdbm-devel db4-devel libXpm-devel libX11-devel gd-devel gmp-devel expat-devel xmlrpc-c xmlrpc-c-devel libicu-devel libmcrypt-devel libmemcached-devel

安装CMake3

yum remove cmake
wget https://github.com/Kitware/CMake/releases/download/v3.17.5/cmake-3.17.5-Linux-x86_64.tar.gz
tar zxvf cmake-3.17.5-Linux-x86_64.tar.gz
vi /etc/profile.d/cmake.sh
export CMAKE_HOME=/root/cmake-3.17.5-Linux-x86_64
export PATH=$PATH:$CMAKE_HOME/bin
chmod +x /etc/profile.d/cmake.sh
source /etc/profile

简单介绍: CMake的好处是方便跨平台,在Linux环境下,请先创建build目录,并到目录build下,执行cmake:

mkdir build && cd build
cmake ../

安装libzip库(要求CMake版本至少3.0.0,而2.8.12.2太低), 1.3.1和1.7.0(1.7.3)都被禁止了,我换1.3.2

yum remove -y libzip
wget https://libzip.org/download/libzip-1.7.3.tar.gz
tar -zxvf libzip-1.7.3.tar.gz
cd libzip-1.7.3
mkdir build && cd build/
选项一 cmake -DCMAKE_INSTALL_PREFIX=/usr/local/libzip ..
[选这个]选项二 cmake ..
[1.3.2低版本] ./configure
make
make install
选项一 export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
[选这个]选项二 export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/:/usr/lib/pkgconfig/
ldconfig

错误libxml

No package 'libxml-2.0' found
rpm -qa |grep  libxml2
libxml2-2.9.1-6.el7.4.x86_64

安装libxml2和libxml2-devel包
yum install -y libxml2
yum install -y libxml2-devel

错误sqlite3

No package 'sqlite3' found
error: Package requirements (sqlite3 > 3.7.4) were not met

yum install libsqlite3x-devel -y

错误libcurl

configure: error: Package requirements (libcurl >= 7.15.5) were not met:
No package 'libcurl' found

安装
yum install -y libcurl libcurl-devel

错误libjpeg

configure: error: Package requirements (libjpeg) were not met:
No package 'libjpeg' found

error: Package requirements (oniguruma) were not met

由于PHP 7.4.x中mbstring的正则功能需要oniguruma的支持,编译PHP的时候使用了 --enable-mbstring 参数,系统中却没有oniguruma库,

规避方式一: 不使用mbstring的正则功能,即在“--enable-mbstring”后再添加“--disable-mbregex”参数

规避方式二: 安装oniguruma库

yum install -y epel-release
yum install oniguruma-devel -y

编译PHP7.4 默认查找64位的动态链接库,
centos系统默认的动态链接库配置文件/etc/ld.so.conf里并没有将64位库文件加入搜索路径,
需要将 /usr/local/lib64 /usr/lib64 这些针对64位的库文件路径加进去。

添加搜索路径到配置文件

echo '/usr/local/lib64
/usr/local/lib
/usr/lib
/usr/lib64'>>/etc/ld.so.conf

更新配置
ldconfig -v

其他安装源

yum -y install http://mirror.centos.org/centos-7/7.7.1908/cloud/x86_64/openstack-queens/oniguruma-6.7.0-1.el7.x86_64.rpm
yum -y install http://mirror.centos.org/centos-7/7.7.1908/cloud/x86_64/openstack-queens/oniguruma-devel-6.7.0-1.el7.x86_64.rpm

编译安装oniguruma

wget https://github.com/kkos/oniguruma/releases/download/v6.9.5_rev1/onig-6.9.5-rev1.tar.gz
tar zxvf onig-6.9.5-rev1.tar.gz
cd onig-6.9.5
./configure --prefix=/usr --libdir=/lib64
make
make install

编译安装freetype, centos8开始要安装freetype2.10.x, 旧版系统可以是2.8.1

wget https://download.savannah.gnu.org/releases/freetype/freetype-2.10.2.tar.gz
tar zxvf freetype-2.10.2.tar.gz
cd freetype-2.10.2
./configure --prefix=/usr/local/freetype
make
make install
mkdir /usr/lib/pkgconfig
cp /usr/local/freetype/lib/pkgconfig/freetype2.pc /usr/lib/pkgconfig/
cat > /etc/ld.so.conf.d/freetype.conf<<EOF
/usr/local/freetype/lib
EOF
ldconfig
ln -sf /usr/local/freetype/include/freetype2/* /usr/include/


export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/lib/pkgconfig/
ldconfig

心累,依然报错缺少libjpeg和libpng
yum install -y libjpeg libjpeg-devel libpng libpng-devel
ldconfig

编译安装openssl

wget https://github.com/openssl/openssl/archive/OpenSSL_1_1_1f.tar.gz
./config enable-weak-ssl-ciphers -fPIC --prefix=/usr/local/openssl1.1.1 --openssldir=/usr/local/ope        nssl1.1.1
make depend
make
make install

wget https://github.com/openssl/openssl/archive/OpenSSL_1_0_2u.tar.gz
tar zxvf OpenSSL_1_0_2u.tar.gz
cd openssl-OpenSSL_1_0_2u/
./config -fPIC --prefix=/usr/local/openssl --openssldir=/usr/local/openssl
make depend
make
make install
ldconfig

编译安装curl

wget https://github.com/curl/curl/releases/download/curl-7_72_0/curl-7.72.0.tar.gz
tar zxvf curl-7.72.0.tar.gz
cd curl-7.72.0
./configure --prefix=/usr/local/curl --enable-ares --without-nss --with-zlib --with-ssl=/usr/local/openssl
make
make install
ldconfig

安装libiconv

wget http://ftp.gnu.org/pub/gnu/libiconv/libiconv-1.16.tar.gz
tar zxvf libiconv-1.16.tar.gz
cd libiconv-1.16
./configure --prefix=/usr/local/libiconv --enable-static
make
make install

报错缺少icu

configure: error: Package requirements (icu-uc >= 50.1 icu-io icu-i18n) were not met:

No package 'icu-uc' found
No package 'icu-io' found
No package 'icu-i18n' found

yum install -y libicu-devel

安装PHP7.4


wget https://www.php.net/distributions/php-7.4.10.tar.gz
tar zxvf php-7.4.10.tar.gz
cd php-7.4.10
./configure --prefix=/usr/local/php --with-config-file-path=/usr/local/php/etc --with-config-file-scan-dir=/usr/local/php/conf.d --enable-fpm --with-fpm-user=www --with-fpm-group=www --enable-mysqlnd --with-mysqli=mysqlnd --with-pdo-mysql=mysqlnd --with-iconv=/usr/local/libiconv --with-freetype=/usr/local/freetype --with-jpeg --with-png --with-zlib --enable-xml --disable-rpath --enable-bcmath --enable-shmop --enable-sysvsem --enable-inline-optimization --with-curl=/usr/local/curl --enable-mbregex --enable-mbstring --enable-intl --enable-pcntl --enable-ftp --enable-gd --with-openssl=/usr/local/openssl --with-mhash --enable-pcntl --enable-sockets --with-xmlrpc --with-zip --without-libzip --enable-soap --with-gettext --enable-opcache --with-xsl --with-pear

make
make install

依然报错: configure: WARNING: unrecognized options: --with-png, --without-libzip

安装Zip库

http://pecl.php.net/get/zip-1.19.0.tgz
tar xvf zip-1.19.0.tgz
./configure  --with-php-config=/usr/bin/php-config

ln -sf /usr/local/php/bin/php /usr/bin/php
ln -sf /usr/local/php/bin/phpize /usr/bin/phpize
ln -sf /usr/local/php/bin/pear /usr/bin/pear
ln -sf /usr/local/php/bin/pecl /usr/bin/pecl
ln -sf /usr/local/php/sbin/php-fpm /usr/bin/php-fpm
rm -f /usr/local/php/conf.d/*

mkdir -p /usr/local/php/{etc,conf.d}
cp php.ini-production /usr/local/php/etc/php.ini

cd /root/php-7.4.10

sed -i 's/post_max_size =.*/post_max_size = 50M/g' /usr/local/php/etc/php.ini
sed -i 's/upload_max_filesize =.*/upload_max_filesize = 50M/g' /usr/local/php/etc/php.ini
sed -i 's/;date.timezone =.*/date.timezone = PRC/g' /usr/local/php/etc/php.ini
sed -i 's/short_open_tag =.*/short_open_tag = On/g' /usr/local/php/etc/php.ini
sed -i 's/;cgi.fix_pathinfo=.*/cgi.fix_pathinfo=0/g' /usr/local/php/etc/php.ini
sed -i 's/max_execution_time =.*/max_execution_time = 300/g' /usr/local/php/etc/php.ini
sed -i 's/disable_functions =.*/disable_functions = passthru,exec,system,chroot,chgrp,chown,shell_exec,proc_open,proc_get_status,popen,ini_alter,ini_restore,dl,openlog,syslog,readlink,symlink,popepassthru,stream_socket_server/g' /usr/local/php/etc/php.ini

pear config-set php_ini /usr/local/php/etc/php.ini
pecl config-set php_ini /usr/local/php/etc/php.ini

wget --prefer-family=IPv4 --no-check-certificate -T 120 -t3 ${Download_Mirror}/web/php/composer/composer.phar -O /usr/local/bin/composer
if [ $? -eq 0 ]; then
    echo "Composer install successfully."
    chmod +x /usr/local/bin/composer
else
    echo "Composer install failed, try to from composer official website..."
    curl -sS --connect-timeout 30 -m 60 https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
    if [ $? -eq 0 ]; then
        echo "Composer install successfully."
    fi
fi
if [ "${country}" = "CN" ]; then
    composer config -g repo.packagist composer https://mirrors.aliyun.com/composer/
fi

cat >/usr/local/php/etc/php-fpm.conf<<EOF
[global]
pid = /usr/local/php/var/run/php-fpm.pid
error_log = /usr/local/php/var/log/php-fpm.log
log_level = notice

[www]
listen = /tmp/php-cgi.sock
listen.backlog = -1
listen.allowed_clients = 127.0.0.1
listen.owner = www
listen.group = www
listen.mode = 0666
user = www
group = www
pm = dynamic
pm.max_children = 10
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 6
pm.max_requests = 1024
pm.process_idle_timeout = 10s
request_terminate_timeout = 100
request_slowlog_timeout = 0
slowlog = var/log/slow.log
EOF

cp ${cur_dir}/src/${Php_Ver}/sapi/fpm/init.d.php-fpm /etc/init.d/php-fpm
cp ${cur_dir}/init.d/php-fpm.service /etc/systemd/system/php-fpm.service
chmod +x /etc/init.d/php-fpm

groupadd www
useradd -s /sbin/nologin -g www www

yum -y install libxml2
yum -y install libxml2-devel
yum install sqlite-devel
yum install bzip2 bzip2-devel
yum install libxslt-devel
yum install libpng-devel