CDH6集群部署
大数据集群管理方式分为手工方式(Apache hadoop)和工具方式(Ambari + hdp 和Cloudera Manger + CDH)
桦川网站建设公司成都创新互联,桦川网站设计制作,有大型网站制作公司丰富经验。已为桦川上千多家提供企业网站建设服务。企业网站搭建\成都外贸网站建设要多少钱,请找那个售后服务好的桦川做网站的公司定做!
主机:
192.168.0.32:Server+agent
192.168.0.33:agent
192.168.0.34:agent
安装之前准备环境:
网络名称
CDH需要IPv4。不支持IPv6。
按如下方式配置群集中的每个主机,以确保所有成员可以相互通信:
将主机名设置为唯一名称(不是 本地主机)。
sudo hostnamectl set-hostname hadoop01.youe.com
编辑 / etc / hosts文件使用群集中每个主机的IP地址和完全限定的域名(FQDN)。您也可以添加非限定名称。
192.168.0.32 hadoop01.youe.com
192.168.0.33 hadoop02.youe.com
192.168.0.34 hadoop03.youe.com
每个主机的规范名称 / etc / hosts文件 必须是FQDN(例如myhost-1.example.com),而不是不合格的主机名(例如 为myhost-1)。规范名称是IP地址后的第一个条目。
不要使用别名 / etc / hosts文件 或者配置DNS。
不合格的主机名(短名称)在Cloudera Manager实例中必须是唯一的。例如,您不能同时拥有由同一Cloudera Manager Server管理的host01.example.com和host01.standby.example.com。
编辑 /etc/sysconfig/network 仅限此主机的FQDN:
HOSTNAME = hadoop01.youe.com
禁用防火墙和selinux(所有会话)
NTP(所有会话)
CDH要求您在群集中的每台计算机上配置网络时间协议(NTP)服务。要启动NTP并将其配置为在重新引导时自动运行,请在群集中的每个节点上执行以下步骤。
yum -y install ntp
编辑 /etc/ntp.conf文件添加NTP服务器
server 0.pool.ntp.org
server 1.pool.ntp.org
server 2.pool.ntp.org
启动ntp
systemctl enable ntpd
systemctl start ntpd
同步节点
ntpdate -u 0.pool.ntp.org
同步系统时钟
hwclock --systohc
python版本
python2.7(centos7默认python2.7)
ssh免秘钥登录:生成私钥和公钥(所有节点)
ssh-keygen -t rsa -P ''
ssh-copy-id root@192.168.0.33
ssh-copy-id root@192.168.0.34
ssh-copy-id root@192.168.0.32
Master可以通过ssh连接到32 33 34
安装cloudera manager server
安装存储库(所有节点)
wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
导入存储库签名GPG密钥:
rpm --import https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPM-GPG-KEY-cloudera
安装jdk配置环境变量(所有会话)
* The JDK must be 64-bit. Do not use a 32-bit JDK.
* The installed JDK must be a supported version as documented in Java Requirements.
* The same version of the Oracle JDK must be installed on each cluster host.
* The JDK must be installed at /usr/java/jdk-version.
为Cloudera Manager配置存储库后,可以使用软件包管理器在Cloudera Manager Server主机上安装Oracle JDK,如下所示:
yum -y install oracle-j2sdk1.8
如果手动安装jdk,必须安装到/usr/java/目录下
安装cloudera manager包
Master节点:
yum -y install cloudera-manager-agent cloudera-manager-daemons cloudera-manager-server
Node节点:
yum -y install cloudera-manager-agent cloudera-manager-daemons
cloudera-manager-daemons是server和agent必须安装的,大概有1.3G大小。
也可以提前去下载rpm包进行安装:https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/
wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-daemons-6.0.0-530873.el7.x86_64.rpm
wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-server-6.0.0-530873.el7.x86_64.rpm
wget https://archive.cloudera.com/cm6/6.0.0/redhat7/yum/RPMS/x86_64/cloudera-manager-agent-6.0.0-530873.el7.x86_64.rpm
如果用rpm包安装,先安装cloudera-manager-daemons,然后在安装cloudera-manager-server和cloudera-manager-agent
启用Auto-TLS以自动创建证书:(后面配置麻烦,如果不开启https,可以跳过此项)
https://www.cloudera.com/documentation/enterprise/6/latest/topics/auto_tls.html#auto_tls
注意: Auto-TLS支持两种选择:
* 选项1:使用Cloudera Manager生成内部证书颁发机构和相应的证书
* 选项2:使用现有的证书颁发机构和相应的证书
这里使用选项1,使auto-TLS能够使用由Cloudera Manager创建和管理的内部证书颁发机构(CA)。要使用受信任的公共CA(选项2),必须首先获取群集主机的证书。
必须在新安装过程中启用Auto-TLS 。无法在现有群集上启用自动TLS。在将任何主机添加到Cloudera Manager之前,必须启用Auto-TLS。
在Cloudera Manager Server主机上安装 Cloudera Manager Agent 之后,在启动Cloudera Manager之前,请按如下所示启用auto-TLS。
JAVA_HOME=/usr/java/jdk1.8.0_141 /opt/cloudera/cm-agent/bin/certmanager setup --configure-services
将JDK版本替换为适合您环境的正确路径。如果要将文件存储在默认目录以外的目录中( /var/lib/cloudera-scm-server/certmanager),添加 --location选项如下:
JAVA_HOME=/usr/java/jdk1.8.0_121 /opt/cloudera/cm-agent/bin/certmanager --location /opt/cloudera/CMCA setup --configure-services
/opt/cloudera/CMCA此目录是自动创建的,在运行该命令之前不得存在。检查/var/log/cloudera-scm-agent/certmanager.log 日志文件以确认目录已创建。
安装MySQL:
https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_mysql.html#cmig_topic_5_5_2
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
yum -y install mysql-community-release-el7-5.noarch.rpm
yum -y update
yum -y install mysql-server
* 对于MySQL 5.6和5.7,必须安装MySQL-shared-compat或MySQL-shared软件包。这是Cloudera Manager Agent软件包安装所必需的。
* 如果在MySQL中启用了基于GTID的复制,则Cloudera Manager安装将失败。
1. Move old InnoDB log files /var/lib/mysql/ib_logfile0 and /var/lib/mysql/ib_logfile1 out of /var/lib/mysql/ to a backup location.
rm -rf /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile1
如果要对现有数据库进行更改,请确保在继续之前停止使用该数据库的任何服务。
vim /etc/my.cnf
#官方推荐配置文件
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
log_bin=/var/lib/mysql/mysql_binary_log
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
systemctl enable mysqld
systemctl start mysqld
初始化Mysql
/usr/bin/mysql_secure_installation
[...]
Enter current password for root (enter for none):Asd@12345
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] Y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] Y
[...]
Disallow root login remotely? [Y/n] N
[...]
Remove test database and access to it [Y/n] Y
[...]
Reload privilege tables now? [Y/n] Y
All done!
安装MySQL JDBC驱动程序
wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
为Cloudera软件创建数据库:https://www.cloudera.com/documentation/enterprise/6/latest/topics/prepare_cm_database.html
必须配置数据库以支持MySQL UTF8 字符集编码。
mysql -u root -pAsd@12345
create database scm default character set utf8 default collate utf8_general_ci;
grant all on scm.* to 'scm'@'%' identified by 'Admin@123';
create database amon default character set utf8 default collate utf8_general_ci;
grant all on amon.* to 'amon'@'%' identified by 'Admin@123';
create database rman default character set utf8 default collate utf8_general_ci;
grant all on rman.* to 'rman'@'%' identified by 'Admin@123';
create database hue default character set utf8 default collate utf8_general_ci;
grant all on hue.* to 'hue'@'%' identified by 'Admin@123';
create database metastore default character set utf8 default collate utf8_general_ci;
grant all on metastore.* to 'hive'@'%' identified by 'Admin@123';
create database sentry default character set utf8 default collate utf8_general_ci;
grant all on sentry.* to 'sentry'@'%' identified by 'Admin@123';
create database nav default character set utf8 default collate utf8_general_ci;
grant all on nav.* to 'nav'@'%' identified by 'Admin@123';
create database oozie default character set utf8 default collate utf8_general_ci;
grant all on oozie.* to 'oozie'@'%' identified by 'Admin@123';
flush privileges;
设置Cloudera Manager数据库
mysql在本地时执行:
rm -rf /etc/cloudera-scm-server/db.mgmt.properties
/opt/cloudera/cm/schema/scm_prepare_database.sh
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
会生成/etc/cloudera-scm-server/db.properties,里面是数据库的连接信息,如果这个文件有误,或者不存在,后面的服务无法启动。
可能会报错:无法在这个主机上找到兼容的Java版本,要么是因为javahome没有被设置,要么是因为Java的兼容版本没有安装。
解决方法:JAVA_HOME路径以/opt/cloudera/cm-agent/service/common/cloudera-config.sh里面配置的路径为准
mysql不在本地时执行:
在Cloudera Manager Server主机上运行脚本(cm01.example.com)并连接到远程MySQL主机(db01.example.com):
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm
如果要安装CDH 6,请不要选择KAFKA,KUDU或SPARK包裹,因为它们包含在CDH 6中
手动下载地址:
wget https://archive.cloudera.com/cdh7/6.0.0/parcels/CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel
wget https://archive.cloudera.com/cdh7/6.0.0/parcels/CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel.sha256
wget https://archive.cloudera.com/cdh7/6.0.0/parcels/manifest.json
将CHD6相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中,如果没有此目录,可以自己创建。
注意:最后将CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel.sha256,重命名为CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel.sha,这点必须注意,否则系统会重新下载CDH-6.0.0-1.cdh7.0.0.p0.537114-el7.parcel文件。
重要!--->在manifest.json文件中,找到对应版本的秘钥,复制到.sha文件中。如果manifest.json文件中秘钥错误,系统也会重新下载CDH包。
chown cloudera-scm:cloudera-scm /opt/cloudera/ -R
chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-agent -R
禁用透明大页面压缩
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
并将上面的两条命令写入开机自启动
vim /etc/rc.local
优化交换分区:https://blog.csdn.net/lufeisan/article/details/53339991
vim /etc/sysctl.conf
vm.swappiness = 10
sysctl -p /etc/sysctl.conf
安装CDH:https://www.cloudera.com/documentation/enterprise/6/latest/topics/install_software_cm_wizard.html
设置Cloudera Manager数据库后,启动Cloudera Manager Server,然后登录Cloudera Manager Admin Console:
启动Cloudera Manager Server:
systemctl start cloudera-scm-server
可以查看启动日志
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
看到此条信息,说明启动完成
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
登录
在Web浏览器中,转到 192.168.0.32:7180,其中
如果您启用了自动TLS,则会重定向到https://
登录Cloudera Manager Admin Console。默认凭据为:
Username: admin
Password: admin
安装过程中报错1: could not contact scm server at localhost:7182, giving up
BEGIN python -c 'import socket; import sys; s = socket.socket(socket.AF_INET); s.settimeout(5.0); s.connect((sys.argv[1], int(sys.argv[2]))); s.close();' localhost 7182
解决方法: 将连不上的机器 /usr/bin/host 文件删掉, 这样Cloudera Manager就会直接使用 ip进行连接。
报错2:
解决方法:查看这个进程,然后杀死进程,重新启动就可以了
ps -ef | grep supervisord
kill -9
systemctl restart cloudera-scm-agent
如果部署失败,可以卸载删除Cloudera Manger环境,重新部署。
systemctl stop cloudera-scm-server
systemctl stop cloudera-scm-agent
yum -y remove 'cloudera-manager-*'
yum clean all
umount cm_processes
umount /var/run/cloudera-scm-agent/process
rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera*
rm -rf /tmp/.scm_prepare_node.lock
rm -Rf /var/lib/flume-ng /var/lib/hadoop* /var/lib/hue /var/lib/navigator /var/lib/oozie /var/lib/solr /var/lib/sqoop* /var/lib/zookeeper
rm -Rf data_drive_path/dfs data_drive_path/mapred data_drive_path/yarn
systemctl stop mysqld
yum -y remove mysql
rm -rf /var/lib/mysql
rm -rf /var/log/mysqld.log
rm -rf /var/lib/mysql/mysql
rm -rf /usr/lib64/mysql
rm -rf /usr/share/mysql
rm -rf /opt/cloudera/cm/cloudera-navigator-audit-server/schema/mysql
rm -rf /opt/cloudera/cm/cloudera-navigator-server/schema/mysql
rm -rf /opt/cloudera/cm/schema/mysql
rpm -qa | grep -i mysql
yum -y remove MySQL-python-1.2.5-1.el7.x86_64
yum -y remove mysql-community-libs-5.7.23-1.el7.x86_64
yum -y remove mysql57-community-release-el7-10.noarch
yum -y remove mysql-community-common-5.7.23-1.el7.x86_64
yum -y remove mysql-community-libs-compat-5.7.23-1.el7.x86_64
yum -y remove mysql-connector-java-5.1.25-3.el7.noarch
本文标题:CDH6集群部署
链接地址:http://pwwzsj.com/article/jehpdj.html