Centos7搭建Ambari-易微帮

Ambari管理群集

Apache Ambari项目旨在通过开发用于配置，管理和监视Apache Hadoop集群的软件来简化Hadoop管理。Ambari通过其RESTful API提供了直观，易于使用的Hadoop管理Web UI。

Ambari使系统管理员可以：

设置Hadoop集群

Ambari提供了用于在任意数量的主机上安装Hadoop服务的分步向导。
Ambari处理群集的Hadoop服务的配置。
管理Hadoop集群

Ambari提供了用于在整个集群中启动，停止和重新配置Hadoop服务的集中管理。
监控Hadoop集群

Ambari提供了一个仪表板，用于监视Hadoop集群的运行状况和状态。
Ambari利用Ambari Metrics System收集指标。
Ambari利用Ambari Alert Framework进行系统警报，并在需要您关注时（例如，节点故障，剩余磁盘空间不足等）通知您。
Ambari使应用程序开发人员和系统集成商能够：

使用Ambari REST API轻松将Hadoop的配置，管理和监视功能集成到自己的应用程序中。

准备环境

使用的centos7（内存硬盘根据自身环境加大空间）安装时尽量使用英文版的不然会有不知名的报错

ambari-server(master)
提供外部访问的API
接受Ambari-agent的心跳信息和管理Ambari-agent
元数据的管理和数据库的访问
提供可视化的操作界面

ambari-agent（slave1）
采集所在节点的信息并且汇总发心跳汇报给ambari-server
处理ambari-server的执行请求,安装、启动、停止服务等

Ambari所在的公司已经被Cloudera公司收购所以相关的文档都在clodera公司的官方网站：HDP

我们选择的安装方式是:Ambari+HDP+HDP-UTILS,因为Ambari本身只是一个大数据平台自动化部署和管理的工具，所以需要配合HDP(大数据软件安装包集合)一起使用。

一,配置基础环境 master&slave1

1.主机名
hostnamectl set-hostname master

hostnamectl set-hostname slave1

2.修改网卡设置（我这里使用的桥接网卡）

vim /etc/sysconfig/network-scripts/ifcfg-ens33
master:
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=150dbab6-ce79-4f40-b0dc-776217768249
DEVICE=ens33
ONBOOT=yes
IPADDR=172.16.1.131
NETMASK=255.255.255.0
GATEWAY=172.16.1.254
DNS1=8.8.8.8          


slave1:
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
UUID=150dbab6-ce79-4f40-b0dc-776217768249
DEVICE=ens33
ONBOOT=yes
IPADDR=172.16.1.133
NETMASK=255.255.255.0
GATEWAY=172.16.1.254
DNS1=8.8.8.8

3.修改host文件 
master
172.16.1.131 master.hadoop master
172.16.1.133 slave1.hadoop

slave1
172.16.1.131 master.hadoop
172.16.1.133 slave1.hadoop slave1

4.更改yum源
我这里使用的华为源
cp -a /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
wget -O /etc/yum.repos.d/CentOS-Base.repo https://repo.huaweicloud.com/repository/conf/CentOS-7-reg.repo
yum clean all
yum makecache  如果有的常用软件包没有，就使用yum update更新一下
reboot

二，安装HTTPD 用于存放HDP master

注：这里的HDP包用HDP-3.1.5.0 和 HDP-UTILS-1.1.0.22

HTTPD

yum install -y httpd

mkdir /var/www/html/ambari-hdp
使用第三方终端工具 将文件拖动到/var/www/html/ambari-hdp

systemctl enable httpd.service | systemctl status httpd.service

完成后reboot重启回显这个即为成功

三，时间同步，OPENSSH ，禁用khugepaged master&slave1

这里不搭建ntp,过于麻烦

openssh 是免密登录

timedatectl set-timezone Asia/Shanghai    #两台都需要同步

yum install -yopenssh-clients             
ssh-keygen          #生成密钥   这里不需要输入什么 默认回车即可
ssh-copy-id master.hadoop           #验证密钥
ssh-copy-id slave1.hadoop           #验证密钥
ssh master.hadoop                   #连接对端
exit                                #退出 
ssh slave1.hadoop                   #连接对端
exit                                #退出

禁用进程khugepaged

khugepaged 这个程序会不间断地吃性能 所以需要关掉（重启失效 需要再次执行此操作）
echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

这里顺手就把防火墙一起关了
systemctl disable firewalld.service 两台都需要禁用

四，安装配置ambari server mariadb master

这里先安装ambari server 需要将ambari-2.7.5.0 拖动到家目录进行解压（可先在Windows解压完成后再进行拖动也可以使用 tar -zxvf “文件名”进行压缩）

配置JDK环境变量

jdk 环境变量
先将ambari-2.7.5.0放到家目录（解压完成的）
然后才能通过目录进行离线安装
这个方法在JDK版本不兼容或者找不到 可以这样使用这里包内是有这个版本的JDK的

yum -y install ambari-server mariadb mariadb-server mysql-connector-java 

[root@master ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'enforcing'
Temporarily disabling SELinux
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms found at http://www.oracle.com/technetwork/java/javase/terms/license/index.html and not accepting will cancel the Ambari Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y
Downloading JDK from http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz
jdk-8u112-linux-x64.tar.gz... 22% (39.6 MB of 174.7 MB)^C
Aborting ... Keyboard Interrupt.
看到链接后 就ctrl+c 强制停止
通过这个命令获得下载链接http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz

wget http://public-repo-1.hortonworks.com/ARTIFACTS/jdk-8u112-linux-x64.tar.gz 进行压缩

mkdir /usr/jdk64

mv jdk1.8.0_112/  /usr/jdk64  移到指定目录

vim /etc/profile

export JAVA_HOME=/usr/jdk64/jdk1.8.0_112         #添加以下两行
export PATH=$JAVA_HOME/bin:$PATH

source /etc/profile   #快速生效

java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

mariadb配置

配置mariadb
先systemctl enable mariadb  |   systemctl start mariadb

[root@master ~]# mysql_secure_installation (这里敲完后会让你输入root密码 然后回车输入新密码为“bigdata” 不然后面大概率会报错)

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user.  If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!


By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network·

Disallow root login remotely? [Y/n] n
 ... skipping.

By default, MariaDB comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!
[root@master ~]#
[root@master ~]#
[root@master ~]# mysql -uroot -pbigdata
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 10
Server version: 5.5.68-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>
MariaDB [(none)]>
MariaDB [(none)]> create database ambari;
Query OK, 1 row affected (0.00 sec)

MariaDB [(none)]>
MariaDB [(none)]>
MariaDB [(none)]> grant all privileges on ambari. * to 'ambari'@'localhost'identified by 'bigdata';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]>
MariaDB [(none)]> grant all privileges on ambari. * to 'ambari'@'%'identified by 'bigdata';
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]>
MariaDB [(none)]>
MariaDB [(none)]>
MariaDB [(none)]> use ambari
Database changed
MariaDB [ambari]>
MariaDB [ambari]>
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)
...

Ambari 配置

ambari

[root@master ~]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'enabled'
SELinux mode is 'permissive'
WARNING: SELinux is set to 'permissive' mode and temporarily disabled.
OK to continue [y/n] (y)? y
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/jdk64/jdk1.8.0_112
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)? y
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.5.0.72.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
[root@master ~]# ambari-server start
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.......................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

ambari-server start  进行启动

yum install 2.7.5.0-72/ambari/ambari-agent-2.7.5.0-72.x86_64.rpm 进行安装

vim /etc/ambari-agent/conf/ambari-agent.ini
[server]
hostname=172.16.1.131   #这里指向ambari服务器
url_port=8440
secured_url_port=8441
connect_retry_delay=10
max_reconnect_retry_delay=30