欧拉操作系统下安装hadoop集群

背景：欧拉操作系统下安装CDH集群的时候，需要安装python2.7.5，但是本身欧拉系统对python2的支持可能没有那么好，所以考虑搭建原生的hadoop集群。

基础环境如下

一、jdk的安装

openjdk的下载地址

https://jdk.java.net/java-se-ri/8-MR6

二、mysql的安装

mysql安装包的下载地址
https://dev.mysql.com/downloads/mysql/

三、hadoop的安装

hadoop安装的前置条件是系统中已经有Java的环境
还需要将免密登录配置好

创建Hadoop用户

#创建Hadoop的用户
sudo useradd hadoop

#设置Hadoop用户的密码
sudo passwd hadoop   

#设置Hadoop的用户有sudo权限            
sudo usermod -aG sudo hadoop     

解压Hadoop的安装包

#解压这个压缩包到/opt的目录下，这里最好别放在/root下
sudo tar -xzf hadoop-3.4.1.tar.gz -C /opt

#给这个解压完的目录，修改一个目录名
sudo mv /opt/hadoop-3.4.1 /opt/hadoop

#设置这个路径的用户、用户组
sudo chown -R hadoop:hadoop /opt/hadoop  

创建环境变量在配置文件中

#将配置写入到配置文件中
echo 'export HADOOP_HOME=/opt/hadoop' | sudo tee -a /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin' | sudo tee -a /etc/profile

#让配置文件立即生效
source /etc/profile

修改这几个配置文件

这几个配置文件的位置如下；
$HADOOP_HOME/etc/hadoop/

1.hadoop-env.sh

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 

2.core-site.xml

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>  # 默认文件系统地址
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/hadoop/tmp</value>        # 临时文件目录:
  </property>
</configuration>

3.hdfs-site.xml

<property>
  <name>dfs.replication</name>
  <value>1</value>                        # 数据副本数（单机设置为1）
</property>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>/opt/hadoop/hdfs/namenode</value>  # NameNode数据存储路径
</property>

4.mapred-site.xml

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>  # 启用YARN框架
</property>

5.yarn-site.xml

<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>  # 指定NodeManager附加服务
</property>

格式化HDFS文件系统

hdfs namenode -format  # 初始化NameNode

启动hadoop

1.启动hdfs
start-dfs.sh

2.启动yarn
start-yarn.sh

查看集群启动状态及进程

jps

hdfs dfsadmin -report

启动hadoop报错

./start-dfs.sh
Starting namenodes on [localhost]
ERROR: Attempting to operate on hdfs namenode as root
ERROR: but there is no HDFS_NAMENODE_USER defined. Aborting operation.
Starting datanodes
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined. Aborting operation.
Starting secondary namenodes [localhost.localdomain]
ERROR: Attempting to operate on hdfs secondarynamenode as root
ERROR: but there is no HDFS_SECONDARYNAMENODE_USER defined. Aborting operation

解决方案，在配置文件中配置如下，即可用root用户进行启动了

四、hive的安装

https://dlcdn.apache.org/hive/hive-4.0.1/

在MySQL中创建Hive元数据库及用户

CREATE DATABASE hive_meta;
CREATE USER 'hive'@'%' IDENTIFIED BY 'Jky1234!@#$';
GRANT ALL ON hive_meta.* TO 'hive'@'%';
FLUSH PRIVILEGES;

修改Hive配置文件

cd $HIVE_HOME/conf
cp hive-default.xml.template hive-site.xml
vi hive-site.xml  # 替换以下关键参数

hive的配置文件内容添加如下

<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>com.mysql.cj.jdbc.Driver</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hive</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>Jky1234!@#$</value>
</property>

下载mysql-connector-java-8.0.27.jar并拷贝到Hive的lib目录

cp mysql-connector-java-8.0.27.jar /opt/hive/lib

/opt/hadoop/share/hadoop/common/lib
cp guava-27.0-jre.jar /opt/hive/lib

在hive的bin目录下执行初始化数据库的操作

./schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Initializing the schema to: 4.0.0
Metastore connection URL:	 jdbc:mysql://localhost:3306/hive_meta?createDatabaseIfNotExist=true
Metastore connection Driver :	 com.mysql.cj.jdbc.Driver
Metastore connection User:	 hive
Starting metastore schema initialization to 4.0.0
Initialization script hive-schema-4.0.0.mysql.sql

Initialization script completed

是否成功？ 去MySQL中

use hive_meta;
show tables;

启动hive的客户端，显示如下；
仔细看日志打印的信息，应该是提示日志冲突了

./hive
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 4.0.1 by Apache Hive
beeline> show databases;
No current connection

ll log4j
-rw-r–r–. 1 root root 349845 9月 25 2024 log4j-1.2-api-2.18.0.jar
-rw-r–r–. 1 root root 315115 9月 25 2024 log4j-api-2.18.0.jar
-rw-r–r–. 1 root root 1861441 9月 25 2024 log4j-core-2.18.0.jar
-rw-r–r–. 1 root root 24801 9月 25 2024 log4j-slf4j-impl-2.18.0.jar
-rw-r–r–. 1 root root 36166 9月 25 2024 log4j-web-2.18.0.jar

mv log4j-slf4j-impl-2.18.0.jar log4j-slf4j-impl-2.18.0-bak.jar0526
然后在执行，发现多余的日志信息没有了

执行连接hive的命令，报错如下；

!connect jdbc:hive2://localhost:10000

./beeline --verbose
Default hs2 connection config file not found
Beeline version 4.0.1 by Apache Hive

查看1000的端口是否被监听
ss -tulnp | grep 9870
tcp LISTEN 0 500 0.0.0.0:9870 0.0.0.0:* users:((“java”,pid=13987,fd=329))

./hive --service hiveserver2 --hiveconf hive.log.dir=/var/log/hive --hiveconf hive.log.file=hiveserver2-daemon.log
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/java/java-se-8u44-ri/bin:/opt/hadoop/bin:/opt/hadoop/sbin:/opt/hive/bin:/root/bin)
2025-05-26 18:16:09: Starting HiveServer2
Exception in thread “main” java.lang.IllegalArgumentException: Logs will be split in two files if the commandline argument hive.log.file is used. To prevent this use to HADOOP_CLIENT_OPTS -Dhive.log.file=hiveserver2-daemon.log or use the set the value in the configuration file (see HIVE-19886)
at org.apache.hive.service.server.HiveServer2$ServerOptionsProcessor.parse(HiveServer2.java:1366)
at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1270)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:330)
at org.apache.hadoop.util.RunJar.main(RunJar.java:245)

./beeline --verbose
Overriding connection url property url_prefix from user connection configuration file
Connecting to jdbc:hive2://localhost:10000/default
Enter username for jdbc:hive2://localhost:10000/default: hive
Enter password for jdbc:hive2://localhost:10000/default: ****
25/05/26 18:23:26 [main]: WARN jdbc.HiveConnection: Failed to connect to localhost:10000
Error: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive (state=08S01,code=0)
java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default: Failed to open new session: java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): User: root is not allowed to impersonate hive
解决方案

hadoop.proxyuser.root.hosts



hadoop.proxyuser.root.groups