如何使用logstash同步nginx日志到数据库
概述
伊春ssl适用于网站、小程序/APP、API接口等需要进行数据传输应用场景,ssl证书未来市场广阔!成为成都创新互联的ssl证书销售渠道,可以享受市场价格4-6折优惠!如果有意向欢迎电话联系或者加微信:13518219792(备注:SSL证书合作)期待与您的合作!
logstash
logstash就是一根具备实时数据传输能力的管道,负责将数据信息从管道的输入端传输到管道的输出端;与此同时这根管道还可以让你根据自己的需求在中间加上滤网,Logstash提供里很多功能强大的滤网以满足你的各种应用场景。
Nginx
Nginx(engine x) 是一个高性能的HTTP和反向代理服务器,也是一个IMAP/POP3/SMTP服务器。其将源代码以类BSD许可证的形式发布,因它的稳定性、丰富的功能集、示例配置文件和低系统资源的消耗而闻名。Nginx是一款轻量级的Web服务器/反向代理服务器及电子邮件(IMAP/POP3)代理服务器,并在一个BSD-like 协议下发行。其特点是占有内存少,并发能力强,事实上nginx的并发能力确实在同类型的网页服务器中表现较好。
1.logstash安装(jdk提前安装1.8)
rpm -ivh logstash-6.6.2.rpm
2.上传数据库驱动jar包
wget https://dev.MySQL.com/get/Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz
mkdir -p /usr/share/logstash/vendor/jar/jdbc cd /usr/share/logstash/vendor/jar/jdbc [root@localhost soft]# cd /usr/share/logstash/vendor/jar/jdbc [root@localhost jdbc]# ll total 984 -rw-r--r--. 1 logstash logstash 1006959 Jul 11 19:43 mysql-connector-java-5.1.48-bin.jar [root@localhost jdbc]#
3.数据库创建库 并授权用户
create database nginxlog; use nginxlog; CREATE TABLE `consumerlog` ( `client_ip` varchar(128) DEFAULT NULL, `log_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `status` int(6) DEFAULT NULL, `http_referer` text, `AgentVersion` varchar(512) DEFAULT NULL, ) grant all on nginxlog.* to nginxlog@'%' identified by '123456';
4.安装 logstash-output-jdbc插件
vim /usr/share/logstash/Gemfile # source "https://rubygems.org" 将国外的源注释,换成国内的 source "https://gems.ruby-china.com/"/usr/share/logstash/bin/logstash-plugin install logstash-output-jdbc
Validating logstash-output-jdbc Installing logstash-output-jdbc Installation successful /usr/share/logstash/bin/logstash-plugin list | grep jdbc logstash-input-jdbc logstash-output-jdbc
5.nginx日志格式设置
log_format access_log_json '{"client_ip":"$remote_addr","log_time":"$time_local","request":"$request","status":"$status","body_bytes_sent":"$body_bytes_sent","http_referer":"$http_referer","AgentVersion":"$http_user_agent","upstream_addr":"$upstream_addr","request_time":"$request_time","upstream_response_time":"$upstream_response_time"}';
6.
[root@localhost conf.d]# cat consumer_log.conf
/etc/logstash/conf.d
input{ file{ path => "/usr/local/tengine-2.1.2/logs/sxt-consumer.log" start_position => "beginning" stat_interval => "2" codec => "json" } } filter { if [status] != "200" { drop{} } } output{ jdbc{ connection_string => "jdbc:mysql://192.168.14.61/nginxlog?user=nginxlog&password=123456&useUnicode=true&characterEncoding=UTF8" statement => ["insert into consumerlog(client_ip,status,http_referer,AgentVersion) VALUES(?,?,?,?)", "client_ip","status","http_referer","AgentVersion"] } }
7.启动logstash
systemctl start logstash
8.验证数据
网页名称:如何使用logstash同步nginx日志到数据库
标题来源:http://pwwzsj.com/article/pdioig.html