架构说明
rsyslog client上应用程序的原生日志不做处理，直接通过relp协议发送到rsyslog server
rsyslog server接收日志队列，通过配置tag+msg配合设施（管道）识别原生日志再写到自定义的文件中分类保存。
一、Rsyslog Client
CentOS 7.6 为例
1.安装relp协议模块 1 2 3 4 5 # CentOS yum install rsyslog-relp # Ubuntu apt-get install rsyslog-relp
2.配置rsyslog.conf 加载模块 1 2 3 4 5 6 7 8 9 10 11 12 # 加载输出模块 omrelp $ModLoad omrelp # 加载输入模块 三选一，我这边用的是 imtcp #$ModLoad imudp #$UDPServerRun 514 $ModLoad imtcp $InputTCPServerRun 514 #$ModLoad imrelp #$InputRELPServerRun 514
参数优化 1 2 3 $SystemLogRateLimitInterval 0 # Interval 设置率计算的时间间隔，0 表示关闭 $SystemLogRateLimitBurst 0 # Burst 设置该间隔内允许的日志数，0 表示关闭 # $MaxMessageSize 16k # 日志最大大小，太大的值需要考虑传输协议，如 UDP
配置输入设施
这边要看应用程序生产日志配置的是什么设施，推荐使用自定义设施local0-local6
但是java log4j2模块即便配置了local3，也无法识别，生产设置走的是默认的user.notice
所以这边以user.notice为设施
禁止user.*设施生产的日志写到本地
user.*设施生产的所有级别日志发送到远端rsyslog server
1 2 3 4 5 6 # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none;user.none /var/log/messages # Save boot messages also to boot.log user.* :omrelp:10.10.0.19:20514
3.检查语法 1 2 3 rsyslogd -N1 或 rsyslogd -f /etc/rsyslog.conf -N1
4.重启rsyslogd 1 systemctl restart rsyslog.service
一、Rsyslog Server
Ubuntu 16.04
1.安装relp协议模块 1 2 3 4 5 # CentOS yum install rsyslog-relp # Ubuntu apt-get install rsyslog-relp
2.配置rsyslog.conf 加载模块 加载输入模块，主要是在tcp20514端口接收日志
1 2 3 4 5 $ModLoad imrelp $InputRELPServerRun 20514 $ModLoad imtcp $InputTCPServerRun 514
参数优化 1 2 3 4 5 6 $SystemLogRateLimitInterval 0 # Interval 设置率计算的时间间隔，0 表示关闭 $SystemLogRateLimitBurst 0 # Burst 设置该间隔内允许的日志数，0 表示关闭 # $MaxMessageSize 16k # 日志最大大小，太大的值需要考虑传输协议，如 UDP # $InputTCPMaxSessions 1024 $EscapeControlCharactersOnReceive off
3.配置/etc/rsyslog.d/50-default.conf 禁止来自设施user.none的日志写到本地文件。
1 2 #*.*;auth,authpriv.none -/var/log/syslog *.*;auth,authpriv.none;local3.none;local4.none;user.none -/var/log/syslog
4.过滤切割 应用程序原生日志 1 Aug 14 17:05:59 xxxx xxxx 2019-08-14 17:05:59,386 [LogDataUtil.java:44][INFO]:{"account_info":{"age":"","aid":"yyt_201908226050558812_518","area":"","gamekey":"notSdk","sex":"0","sid":0,"uid":55397},"create_time":1565773559380,"device_info":{"device_model":"","device_type":"","os_info":"","pt":0,"uuid":""},"ip":"220.249.166.153","log_id":"06b8090e-93ee-4478-b0d2-8dc8749e41e2","log_type":"register_log","role_info":{"job":"","nickname":"yyt_201908226050558812_518","role_id":55397,"sex":""}}
过滤切割/etc/rsyslog.d/gamelog-pxmbd.conf 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 # $template cocsFormat, "%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg%\n" # $template DEBUG, "/data/rsyslog/%fromhost-ip%/DEBUG_%$year%%$month%%$day%.log" $template xxxx_REGISTER_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_register_%$year%%$month%%$day%.log" $template xxxx_LOGIN_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_login_%$year%%$month%%$day%.log" $template xxxx_RESOURCE_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_resource_%$year%%$month%%$day%.log" $template xxxx_PAYMENT_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_payment_%$year%%$month%%$day%.log" $template xxxx_ACTION_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_action_%$year%%$month%%$day%.log" $template xxxx_UNKNOWN_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_unknown_%$year%%$month%%$day%.log" $template xxxx_BUGS_LOG, "/data/rsyslog/%fromhost-ip%/%$year%%$month%%$day%/xxxx_bugs_%$year%%$month%%$day%.log" if $programname startswith 'xxxx' and $msg contains 'register_log' then ?xxxx_REGISTER_LOG & stop if $programname startswith 'xxxx' and $msg contains 'login_log' then ?xxxx_LOGIN_LOG & stop if $programname startswith 'xxxx' and $msg contains 'resource_log' then ?xxxx_RESOURCE_LOG & stop if $programname startswith 'xxxx' and $msg contains 'payment_log' then ?xxxx_PAYMENT_LOG & stop if $programname startswith 'xxxx' and $msg contains 'action_log' then ?xxxx_ACTION_LOG & stop if $programname startswith 'xxxx' then ?xxxx_BUGS_LOG & stop if $programname startswith 'xxxx' and $syslogfacility-text == 'user' and $syslogseverity <= '5' then ?xxxx_UNKNOWN_LOG & stop
5.重启rsyslogd 1 2 3 4 5 6 7 8 9 10 11 # 检查语法 rsyslogd -N1 # 或 rsyslogd -f /etc/rsyslog.conf -N1 # 授权 chown -R syslog.syslog /data/rsyslog # 重启rsyslogd systemctl restart rsyslog.service # 开机自启动 echo '/etc/init.d/rsyslog start' >> /etc/rc.local
三、Client logger命令测试 client 1 2 3 4 5 logger -it xxxx -p user.warning 111111111register_log111111111 logger -it xxxx -p user.warning 111111111login_log111111111 logger -it xxxx -p user.warning 111111111resource_log111111111 logger -it xxxx -p user.warning 111111111payment_log111111111 logger -it xxxx -p user.warning 111111111action_log111111111
rsyslog server 1 2 3 4 5 6 ls -rw-r----- 1 syslog adm 72 Aug 14 15:54 xxxx_user_4_xxxx_action_20190814.log -rw-r----- 1 syslog adm 142 Aug 14 15:54 xxxx_user_4_xxxx_login_20190814.log -rw-r----- 1 syslog adm 73 Aug 14 15:54 xxxx_user_4_xxxx_payment_20190814.log -rw-r----- 1 syslog adm 74 Aug 14 15:54 xxxx_user_4_xxxx_register_20190814.log -rw-r----- 1 syslog adm 74 Aug 14 15:54 xxxx_user_4_xxxx_resource_20190814.log
