Nginx+Flume+Hive日志采集

原标题:Nginx+Flume+Hive日志采集

流程简介:

1.Nginx数据格式

Nginx定义日志格式:

$remote_addr 客户端IP

$time_local 通用日志格式下的本地时间

$status 状态码

$body_bytes_sent 发送给客户端的字节数,不包括响应头的大小

$http_user_agent 客户端浏览器信息

$http_referer 请求的referer地址。

$request 完整的原始请求

$request_method #HTTP请求方法,通常为"GET"或"POST"

$request_time 请求处理时长

$request_uri 完整的请求地址

$server_protocol #服务器的HTTP版本,通常为 "HTTP/1.0" 或 "HTTP/1.1"

$request_body POST请求参数,参数需放form中

token $http_token (自定义header字段前加http_,即可将指定的自定义header字段打印到log中。)

version $arg_version (自定义body字段前加arg_,即可将指定的自定义header字段打印到log中。)

Nginx配置文件中配置输出日志格式:

log_format user_log_format "$remote_addr,$time_local,$status,$body_bytes_sent,$http_user_agent,$http_referer,$request_method,$request_time,$request_uri,$server_protocol,$request_body,$http_token";

示例:

1.119.140.194,29/Dec/2018:02:08:50 +0000,200,556,okhttp/3.8.1,-,GET,0.028,/tv/phone/resource_types?type=3,HTTP/1.1,-,-

2.Flume采集清洗

flume依赖的包:

flume-conf.properties:

# agent

a1.sources = s1

a1.channels = c1

a1.sinks = k1

# sources

a1.sources.s1.type = spooldir

a1.sources.s1.channels = c1

# FTP路径

a1.sources.s1.spoolDir = /var/ftp/

# 自定义拦截器

a1.sources.s1.interceptors = f1

a1.sources.s1.interceptors.f1.type = com.hx.common.flume.FlumeBuilder

# sink

a1.sinks.k1.type=hive

a1.sinks.k1.channel = c1

# hive地址

a1.sinks.k1.hive.metastore=thrift://*.*.*.*:9083

a1.sinks.k1.hive.database=hive_test

a1.sinks.k1.hive.table=nginx_acc_log

a1.sinks.k1.serializer=delimited

# 输入分隔符

a1.sinks.k1.serializer.delimiter=","

# 输出分隔符

a1.sinks.k1.serializer.serdeSeparator=','

a1.sinks.k1.serializer.fieldnames=remote_addr,time_local,status,body_bytes_sent,http_user_agent,http_referer,request_method,request_time,request_uri,server_protocol,request_body,http_token,id,appkey,sing,version

# channel

a1.channels.c1.type = memory

a1.channels.c1.capacity = 100

启动flume:

nohup ./flume-ng agent -c /opt/flume/apache-flume/conf -f /opt/flume/apache-flume/conf/flume-conf.properties -n a1 -Dflume.root.logger=INFO,console &

经过Flume拦截器处理后字段:

remote_addr,time_local,status,body_bytes_sent,http_user_agent,http_referer,request_method,request_time,request_uri,server_protocol,request_body,http_token,id,appkey,sing,version

3.Hive操作

先切换账户:

su hadoop

启动hive:

nohup hive --service metastore >> ~/metastore.log 2>&1 &

nohup hive --service hiveserver2 >> ~/hiveserver2.log 2>&1 &

修改权限:

hdfs dfs -ls /user/hive/warehouse

hadoop dfs -chmod 777 /user/hive/warehouse/hive_test.db

建表语句:(与flume输出一致)

DROP TABLE IF EXISTS nginx_acc_log;

create table nginx_acc_log(remote_addr string,time_local string,status string,body_bytes_sent string,http_user_agent string,http_referer string,request_method string,request_time string,request_uri string,server_protocol string,request_body string,http_token string,id string,appkey string,sing string,version string) clustered by (id) into 5 buckets stored as orc TBLPROPERTIES ('transactional'='true');

设置hive事务:

<!--支持事务-->

<property>

<name>hive.support.concurrency</name>

<value>true</value>

</property>

<property>

<name>hive.exec.dynamic.partition.mode</name>

<value>nonstrict</value>

</property>

<property>

<name>hive.txn.manager</name>

<value>org.apache.hadoop.hive.ql.lockmgr.DbTxnManager</value>

</property>

<property>

<name>hive.compactor.initiator.on</name>

<value>true</value>

</property>

<property>

<name>hive.compactor.worker.threads</name>

<value>5</value>

</property>

<property>

<name>hive.enforce.bucketing</name>

<value>true</value>

</property>

显示表:

hive> show databases;

hive> use hive_test;

hive> show tables;

hive> select * from nginx_acc_log;

---------------------

作者:@凭栏

原文:https://blog.csdn.net/xiaoxudong666/article/details/85335207

版权声明:本文为博主原创文章,转载请附上博文链接!返回搜狐,查看更多

责任编辑:

声明:该文观点仅代表作者本人,搜狐号系信息发布平台,搜狐仅提供信息存储空间服务。
阅读 ()
免费获取
今日搜狐热点
今日推荐