8. sqoop-1.4.7安装部署

环境说明：

主机名：cmcc01为例 

操作系统：centos7

1.下载sqoop: 

   官网：http://archive.apache.org/dist/sqoop/

   解压：

# 解压
tar /opt/package/sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /opt/software/

# 修改目录名称
mv /opt/software/sqoop-1.4.7.bin__hadoop-2.6.0 /opt/software/sqoop-1.4.7

2.配置环境变量

vim ~/.bash_profile
 
# sqoop
export SQOOP_HOME=/opt/software/sqoop-1.4.7
export CLASSPATH=$CLASSPATH:$SQOOP_HOME/lib
export PATH=$PATH:${SQOOP_HOME}/bin

# 使用环境变量生效
source ~/.bash_profile

 3.修改配置信息

cd /opt/software/sqoop-1.4.7/conf
# 复制配置文件
cp sqoop-env-template.sh sqoop-env.sh
# 修改 sqoop-env.sh
vim  sqoop-env.sh
# 修改以下配置

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/opt/software/hadoop-3.1.3

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/opt/software/hadoop-3.1.3

#set the path to where bin/hbase is available
# export HBASE_HOME=

#Set the path to where bin/hive is available
export HIVE_HOME=/opt/software/hive-3.1.3-bin
export HIVE_CONF_DIR=/opt/software/hive-3.1.3-bin/conf

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/opt/software/zookeeper-3.4.10

 4.整合Hive

cd /opt/software/sqoop-1.4.7/conf
# 创建hive配置文件的软连接
ln -s /opt/software/hive-3.1.3-bin/conf/hive-site.xml /opt/software/sqoop-1.4.7/conf/hive-site.xml

# 复制mysql jar包到sqoop的lib目录下
cp /opt/package/mysql-connector-java-8.0.20.jar /opt/software/sqoop-1.4.7/lib

5.测试sqoop是否安装成功

# 执行下面令会列出所有mysql的所有数据库
sqoop list-databases  --connect jdbc:mysql://localhost:3306/  --username root --password 123qwe

6.sqoop从mysql同步数据到hive测试

  1.创建hive stg库

hive -e "create database stg"

  2.创建mysql测试表 flinkcdc.all_flink_cdc_config（表字段数据自己定义）

  3.抽取测试

sqoop import --connect jdbc:mysql://localhost:3306/flinkcdc \
 --username root \
 --password 123qwe \
 --table all_flink_cdc_config \
 --hive-import \
 --delete-target-dir \
 --target-dir /all_flink_cdc_config \
 --hive-database stg \
 --m 1