Hive详解（1）

发布于：2024-03-29 ⋅ 阅读:(18) ⋅ 点赞:(0)

Hive

简介

概述

  1. Hive是由Facobook开发的后来贡献给了Apache的一套用于进行数据仓库管理的工具，使用类SQL语言来对分布式文件系统中的PB级别的数据来进行读写、管理以及分析

  2. Hive基于Hadoop来使用的，底层的默认计算引擎使用的是MapReduce。Hive利用类SQL(HQL，Hive Query Language)语言来操作数据，但是底层是将SQL转化为MapReduce来执行，也因此，Hive更适合于处理离线分析场景

  3. Hive可以屏蔽掉不同语言之间的差异性，降低了大数据的入门门槛

  4. Hive的支持的计算引擎：MapReduce、Tez、Spark。底层默认使用的是MapReduce，但是效率最高的是Spark

  5. 在Hive中，库名、表名、字段名、字段类型、分区、分桶等信息属于元数据，是存储在数据库中，默认支持的是Derby。但是Derby本身是一个单连接的数据库，所以会将Hive的元数据库替换为MySQL。注意：Hive的元数据是存储在数据库的，Hive的数据是放在HDFS上！

版本

  1. 到目前为止，Hive一共有4个版本：Hive1.X和Hive4.X

  2. Hive4.X一共有3个小版本(alpha1、alpha2、beta1)，目前都是测试版

  3. 目前，在市面上，Hive1.x到Hive3.x都在使用，其中Hive1.x逐渐被抛弃，现在市面上使用较多的是Hive2.x和Hive3.x

  4. Hive3.x需要基于Hadoop3.x使用，Hive2.x需要基于Hadoop2.x使用

安装

安装MySQL

  1. CentOS7自带MySQL，需要先卸载这个MySQL：

    <span style="background-color:#f8f8f8">rpm <span style="color:#0000cc">-qa</span> | <span style="color:#3300aa">grep</span> <span style="color:#0000cc">-i</span> mysql | xargs rpm <span style="color:#0000cc">-ev</span> <span style="color:#0000cc">--nodeps</span>
rpm <span style="color:#0000cc">-qa</span> | <span style="color:#3300aa">grep</span> <span style="color:#0000cc">-i</span> mariadb | xargs rpm <span style="color:#0000cc">-ev</span> <span style="color:#0000cc">--nodeps</span></span>

  2. 扇出MySQL可能残留的配置文件

    <span style="background-color:#f8f8f8"><span style="color:#3300aa">find</span> / <span style="color:#0000cc">-name</span> mysql | xargs <span style="color:#3300aa">rm</span> <span style="color:#0000cc">-rf</span>
<span style="color:#3300aa">find</span> / <span style="color:#0000cc">-name</span> my.cnf | xargs <span style="color:#3300aa">rm</span> <span style="color:#0000cc">-rf</span>
<span style="color:#3300aa">rm</span> <span style="color:#0000cc">-rf</span> /var/lib/mysql/</span>

  3. 进入预安装目录，上传或者下载MySQL的安装包

    <span style="background-color:#f8f8f8"><span style="color:#3300aa">cd</span> /opt/presoftware/
<span style="color:#aa5500"># 上传/下载MySQL的安装包</span></span>

  4. 解压

    <span style="background-color:#f8f8f8">tar <span style="color:#0000cc">-xvf</span> mysql-5.7.33-1.el7.x86_64.rpm-bundle.tar</span>

  5. 安装MySQL，并且顺序不能颠倒！！！

    <span style="background-color:#f8f8f8">rpm <span style="color:#0000cc">-ivh</span> mysql-community-common-5.7.33-1.el7.x86_64.rpm
rpm <span style="color:#0000cc">-ivh</span> mysql-community-libs-5.7.33-1.el7.x86_64.rpm
rpm <span style="color:#0000cc">-ivh</span> mysql-community-devel-5.7.33-1.el7.x86_64.rpm 
rpm <span style="color:#0000cc">-ivh</span> mysql-community-libs-compat-5.7.33-1.el7.x86_64.rpm
rpm <span style="color:#0000cc">-ivh</span> mysql-community-client-5.7.33-1.el7.x86_64.rpm
rpm <span style="color:#0000cc">-ivh</span> mysql-community-server-5.7.33-1.el7.x86_64.rpm</span>

  6. 启动MySQL

    <span style="background-color:#f8f8f8">systemctl <span style="color:#3300aa">start</span> mysqld</span>

  7. 第一次安装MySQL，会自动生成初始密码

    <span style="background-color:#f8f8f8"><span style="color:#3300aa">grep</span> <span style="color:#aa1111">"temporary password"</span> /var/log/mysqld.log</span>

  8. 登陆MySQL

    <span style="background-color:#f8f8f8">mysql <span style="color:#0000cc">-u</span> root <span style="color:#0000cc">-p</span>
<span style="color:#aa5500"># 输入初始密码</span></span>

  9. 默认情况下，MySQL的规则要求，密码至少要包含12个字符，至少包含1个大写字母、1个小写字母、1个数字和1个特殊符号。在实际开发过程中，密码肯定要求复杂度；学习环境下，密码不需要这么复杂，所以需要修改MySQL的密码规则

    <span style="background-color:#f8f8f8"><span style="color:#aa5500">-- 修改MySQL关于密码长度的限制</span>
<span style="color:#770088">set</span> <span style="color:#770088">global</span> validate_password_length = <span style="color:#116644">4</span>;
<span style="color:#aa5500">-- 修改MySQL关于密码字符类型的限制</span>
<span style="color:#770088">set</span> <span style="color:#770088">global</span> validate_password_policy = <span style="color:#116644">0</span>;</span>

  10. 修改MySQL的密码

    <span style="background-color:#f8f8f8"><span style="color:#aa5500">-- 为root@localhost设置密码为root</span>
<span style="color:#770088">set</span> <span style="color:#770088">PASSWORD</span> <span style="color:#770088">FOR</span> <span style="color:#aa1111">'root'</span><span style="color:#0055aa">@'localhost'</span> = <span style="color:#aa1111">"root"</span>;</span>

  11. 配置MySQL的远程连接

    <span style="background-color:#f8f8f8"><span style="color:#aa5500">-- 开放权限</span>
<span style="color:#770088">grant</span> <span style="color:#770088">all</span> <span style="color:#770088">privileges</span> <span style="color:#770088">on</span> *.* <span style="color:#770088">to</span> <span style="color:#aa1111">'root'</span><span style="color:#0055aa">@'%' identified by 'root'</span> <span style="color:#770088">with</span> <span style="color:#770088">grant</span> <span style="color:#770088">option</span>;
<span style="color:#aa5500">-- 策略生效</span>
<span style="color:#770088">flush</span> <span style="color:#770088">privileges</span>;</span>

  12. 测试

    <span style="background-color:#f8f8f8"><span style="color:#aa5500"># 退出MySQL</span>
quit;
<span style="color:#aa5500"># 重新进入MySQL</span>
mysql <span style="color:#0000cc">-u</span> root <span style="color:#0000cc">-p</span></span>

安装Hive

  1. 进入预安装目录，上传或者下载Hive的安装包。注意：上课使用的Hive的安装包不是官网提供的，而是我们自己编译好的。官网提供的Hive安装包，最多支持Spark2，但是现在使用的比较多的Spark3版本，也因此，实际过程中，如果想要让Hive支持Spark3，就意味着需要自己下载Hive的源代码并且修改，自己编译！！！

    <span style="background-color:#f8f8f8"><span style="color:#3300aa">cd</span> /opt/presoftware/
<span style="color:#aa5500"># 上传/下载安装包</span></span>

  2. 解压

    <span style="background-color:#f8f8f8">tar <span style="color:#0000cc">-xvf</span> apache-hive-3.1.3-bin.tar.gz <span style="color:#0000cc">-C</span> /opt/software/</span>

  3. 重命名

    <span style="background-color:#f8f8f8"><span style="color:#3300aa">cd</span> ../software/
<span style="color:#3300aa">mv</span> apache-hive-3.1.3-bin/ hive-3.1.3</span>

  4. 配置环境变量

    <span style="background-color:#f8f8f8"><span style="color:#aa5500"># 编辑文件</span>
<span style="color:#3300aa">vim</span> /etc/profile.d/hivehome.sh
<span style="color:#aa5500"># 在文件中添加</span>
<span style="color:#770088">export</span> <span style="color:#0000ff">HIVE_HOME</span><span style="color:#981a1a">=</span>/opt/software/hive-3.1.3
<span style="color:#770088">export</span> <span style="color:#0000ff">PATH</span><span style="color:#981a1a">=</span><span style="color:#0000ff">$PATH</span>:<span style="color:#0000ff">$HIVE_HOME</span>/bin
<span style="color:#aa5500"># 保存退出，生效</span>
<span style="color:#3300aa">source</span> /etc/profile.d/hivehome.sh
<span style="color:#aa5500"># 测试</span>
 hive <span style="color:#0000cc">--version</span></span>

  5. 解决Hive的日志jar包冲突的问题

    <span style="background-color:#f8f8f8"><span style="color:#3300aa">cd</span> hive-3.1.3/lib/
<span style="color:#3300aa">mv</span> log4j-slf4j-impl-2.17.1.jar log4j-slf4j-impl-2.17.1.bak</span>

  6. 给Hive添加MySQL的连接驱动

  7. 修改Hive的配置

    <span style="background-color:#f8f8f8"><span style="color:#3300aa">cd</span> ../conf
<span style="color:#aa5500"># 编辑文件</span>
<span style="color:#3300aa">vim</span> hive-site.xml</span>

    在文件中添加

    <span style="background-color:#f8f8f8"><span style="color:#555555"><?xml</span> <span style="color:#555555">version="1.0"?></span>
<span style="color:#555555"><?xml-stylesheet</span> <span style="color:#555555">type="text/xsl" href="configuration.xsl"?></span>
<span style="color:#117700"><</span><span style="color:#117700">configuration</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--Hive元数据库--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>hive.metastore.db.type<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>mysql<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--MySQL连接地址--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>javax.jdo.option.ConnectionURL<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>jdbc:mysql://hadoop01:3306/hive?useSSL=false<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--MySQL驱动--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>javax.jdo.option.ConnectionDriverName<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>com.mysql.jdbc.Driver<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--MySQL用户名--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>javax.jdo.option.ConnectionUserName<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>root<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--MySQL密码--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>javax.jdo.option.ConnectionPassword<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>root<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--Hive元数据的存储位置--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>hive.metastore.warehouse.dir<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>/user/hive/warehouse<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--Hive元数据的约束--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>hive.metastore.schema.verification<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>false<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--Hive元数据的访问位置--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>hive.metastore.uris<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>thrift://hadoop01:9083<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--Hive服务的访问端口--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>hive.server2.thrift.port<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>10000<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--Hive服务监听的主机--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>hive.server2.thrift.bind.host<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>hadoop01<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#aa5500"><!--Hive的自动认证--></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>hive.metastore.event.db.notification.api.auth<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>false<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
    <span style="color:#117700"><</span><span style="color:#117700">property</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">name</span><span style="color:#117700">></span>datanucleus.schema.autoCreateAll<span style="color:#117700"></</span><span style="color:#117700">name</span><span style="color:#117700">></span>
        <span style="color:#117700"><</span><span style="color:#117700">value</span><span style="color:#117700">></span>true<span style="color:#117700"></</span><span style="color:#117700">value</span><span style="color:#117700">></span>
    <span style="color:#117700"></</span><span style="color:#117700">property</span><span style="color:#117700">></span>
<span style="color:#117700"></</span><span style="color:#117700">configuration</span><span style="color:#117700">></span></span>

  8. 初始化元数据库

    <span style="background-color:#f8f8f8"><span style="color:#aa5500"># 进入MySQL</span>
mysql <span style="color:#0000cc">-u</span> root <span style="color:#0000cc">-p</span></span>

    在MySQL中建立元数据库，注意元数据库的编码必须是latin1

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> <span style="color:#770088">database</span> hive <span style="color:#770088">character</span> <span style="color:#770088">set</span> latin1;
<span style="color:#aa5500">-- 退出</span>
<span style="color:#ff5500">quit</span>;</span>

    初始化元数据库

    <span style="background-color:#f8f8f8">schematool <span style="color:#0000cc">-initSchema</span> <span style="color:#0000cc">-dbType</span> mysql <span style="color:#0000cc">--verbose</span></span>

  9. 因为Hive基于Hadoop的(利用HDFS存储数据，利用MapReduce计算)，所以需要启动Hadoop

    <span style="background-color:#f8f8f8">start-all.sh</span>

  10. 启动Hive的元数据服务

    <span style="background-color:#f8f8f8">hive <span style="color:#0000cc">--service</span> metastore &</span>

  11. 启动Hive

    <span style="background-color:#f8f8f8">hive</span>

基础语法

注意问题

  1. Hive中，如果不指定，默认只有一个库default

  2. 在Hive中，每一个database对应了HDFS上的一个目录。默认情况下，database是放在HDFS的/user/hive/warehouse下

  3. 在Hive中，没有主键的概念，也不支持主键

  4. Hive中的每一个表也对应了HDFS上的一个目录

  5. Hive中建表的时候需要指定字段之间的间隔符号，表一旦建好，字段之间的间隔符号就不能变化！

  6. Hive的数据默认支持insert和select，默认不支持delete和update。如果需要支持delete和update操作，那么在建表的时候，需要指定Hive数据在HDFS上的文件存储类型必须是orc和parquet格式

库操作

  1. 查看所有的库

    <span style="background-color:#f8f8f8">show databases;</span>

  2. 建库

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> database demo;</span>

  3. 删除库。要求库必须是空的

    <span style="background-color:#f8f8f8"><span style="color:#770088">drop</span> database demo;</span>

  4. 建库的时候指定库的存储位置

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> database demo location <span style="color:#aa1111">'/demo.db'</span>;</span>

  5. 利用正则表达式过滤符合要求的库

    <span style="background-color:#f8f8f8">show databases <span style="color:#770088">like</span> <span style="color:#aa1111">'d*'</span>;</span>

  6. 建库的时候，判断库是否存在。如果不存在，那么建库

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> database if <span style="color:#770088">not</span> exists demo;</span>

  7. 强制删除库

    <span style="background-color:#f8f8f8"><span style="color:#770088">drop</span> database demo cascade;</span>

  8. 使用库

    <span style="background-color:#f8f8f8">use demo;</span>

  9. 建库的时候还给库指定属性

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> database demo2 with dbproperties(<span style="color:#aa1111">'create_time'</span>=<span style="color:#aa1111">'2024-03-26'</span>);</span>

  10. 描述库

    <span style="background-color:#f8f8f8"><span style="color:#770088">desc</span> database demo2;</span>

  11. 描述库的详细信息

    <span style="background-color:#f8f8f8"><span style="color:#770088">desc</span> database extended demo2;</span>

  12. 修改库的属性

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> database demo2 <span style="color:#770088">set</span> dbproperties(<span style="color:#aa1111">'create_time'</span>=<span style="color:#aa1111">'024-03-27'</span>);</span>

表及数据操作

  1. 创建表

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> <span style="color:#770088">table</span> person ( id <span style="color:#3300aa">int</span>, name string, age <span style="color:#3300aa">int</span>);</span>

  2. 插入数据

    <span style="background-color:#f8f8f8"><span style="color:#770088">insert</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> person <span style="color:#770088">values</span>(<span style="color:#116644">1</span>, <span style="color:#aa1111">'Amy'</span>, <span style="color:#116644">15</span>);</span>

    转化为MapReduce执行

  3. 查询数据

    <span style="background-color:#f8f8f8"><span style="color:#770088">select</span> * <span style="color:#770088">from</span> person;</span>

  4. 加载数据 - 实际上就是将文件上传到HDFS上，加载速度和文件的上传速度相关

    <span style="background-color:#f8f8f8">load data local inpath <span style="color:#aa1111">'/opt/hive_data/person'</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> person;
<span style="color:#aa5500">-- load data 加载数据</span>
<span style="color:#aa5500">-- local 本地</span>
<span style="color:#aa5500">-- inpath 输入路径</span></span>

  5. 删除表

    <span style="background-color:#f8f8f8"><span style="color:#770088">drop</span> <span style="color:#770088">table</span> person;</span>

  6. 建立person表，指定字段之间的间隔符号是空格

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> <span style="color:#770088">table</span> person ( id <span style="color:#3300aa">int</span>, name string, age <span style="color:#3300aa">int</span>) row format delimited fields terminated <span style="color:#770088">by</span> <span style="color:#aa1111">' '</span>; 
<span style="color:#aa5500">-- row format 对数据进行按行的处理</span>
<span style="color:#aa5500">-- delimited 对什么数据进行拆分</span>
<span style="color:#aa5500">-- fields 字段、属性</span>
<span style="color:#aa5500">-- terminated by ' ' 用什么拆分</span></span>

  7. 查看所有的表

    <span style="background-color:#f8f8f8">show tables;</span>

  8. 描述表

    <span style="background-color:#f8f8f8"><span style="color:#770088">desc</span> person</span>

  9. 创建p2表，p2的表结构和person一样

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> <span style="color:#770088">table</span> p2 <span style="color:#770088">like</span> person;</span>

  10. 如果p3表不存在，那么创建p3表，表结构和person一样

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> <span style="color:#770088">table</span> if <span style="color:#770088">not</span> exists p3 <span style="color:#770088">like</span> person;</span>

  11. 创建p4表，表结构以及数据都和person一样(复制person表)

    <span style="background-color:#f8f8f8"><span style="color:#aa5500">-- 将person的数据全部查询出来，根据查询数据的结构来建立p4表并放入数据</span>
<span style="color:#770088">create</span> <span style="color:#770088">table</span> p4 <span style="color:#770088">as</span> <span style="color:#770088">select</span> * <span style="color:#770088">from</span> person;
<span style="color:#770088">select</span> * <span style="color:#770088">from</span> p4;</span>

  12. 将person表中age>=18的数据查询出来，放入p2表中

    <span style="background-color:#f8f8f8"><span style="color:#770088">insert</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> p2 <span style="color:#770088">select</span> * <span style="color:#770088">from</span> person <span style="color:#770088">where</span> age >= <span style="color:#116644">18</span>;</span>

    注意：insert into表示向表中追加数据，insert overwrite先将表中的数据清空掉，然后再覆盖写入

  13. 从person表中查询数据，将id<5的数据覆盖到p2表中，将age<18的数据写入到p3表中

    <span style="background-color:#f8f8f8"><span style="color:#770088">from</span> person
 <span style="color:#770088">insert</span> overwrite <span style="color:#770088">table</span> p2 <span style="color:#770088">select</span> * <span style="color:#770088">where</span> id < <span style="color:#116644">5</span>
 <span style="color:#770088">insert</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> p3 <span style="color:#770088">select</span> * <span style="color:#770088">where</span> age < <span style="color:#116644">18</span>;</span>

  14. 将person表中id>5的数据查询出来放到HDFS的/person目录下，字段之间用,间隔

    <span style="background-color:#f8f8f8"><span style="color:#770088">insert</span> overwrite directory <span style="color:#aa1111">'/person'</span> row format delimited fields terminated <span style="color:#770088">by</span> <span style="color:#aa1111">','</span> <span style="color:#770088">select</span> * <span style="color:#770088">from</span> person <span style="color:#770088">where</span> id > <span style="color:#116644">5</span>;</span>

    如果是将数据写到Hive之外的地方，此时只能使用insert overwrite，所以要求输出路径必须为空。如果输出路径不为空，那么会将输出路径下的所有文件给清空！！！

  15. Hive除了支持将结果写到HDFS上，还支持写到本地磁盘上

    <span style="background-color:#f8f8f8"><span style="color:#770088">insert</span> overwrite local directory <span style="color:#aa1111">'/opt/hive_demo'</span> row format delimited fields terminated <span style="color:#770088">by</span> <span style="color:#aa1111">'\t'</span> <span style="color:#770088">select</span> * <span style="color:#770088">from</span> person <span style="color:#770088">where</span> age >= <span style="color:#116644">18</span>;</span>

    同样，要求输出路径必须为空！！！

  16. 将person表重命名为p1

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> person rename to p1;</span>

  17. 将p1表的id字段改为pid，类型改为string

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> p1 change column id pid string;</span>

  18. 在p1表中新增字段：gender，类型是string

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> p1 add columns(gender string);</span>

表结构

内部表和外部表

  1. 在Hive中，所有的表分为内部表(又叫管理表，Managed Table)和外部表(External Table)

  2. 一般而言，在Hive中，手动建表手动添加数据的表，大部分是内部表；在Hive中，建表管理HDFS上已经存在的数据，这个表称之为外部表

  3. 可以通过

    <span style="background-color:#f8f8f8"><span style="color:#770088">desc</span> extended p1;
<span style="color:#aa5500">-- 或者</span>
<span style="color:#770088">desc</span> formatted p1;</span>

    如果是内部表，那么tableType的值MANAGED_TABLE；如果是外部表，那么tableType的值EXTERNAL_TABLE

  4. 建立外部表

    <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> external <span style="color:#770088">table</span> flows (
    phone string,
    city string,
    name string,
    upFlow <span style="color:#3300aa">int</span>,
    downFlow <span style="color:#3300aa">int</span>
) row format delimited fields terminated <span style="color:#770088">by</span> <span style="color:#aa1111">' '</span>
 location <span style="color:#aa1111">'/flow'</span>;</span>

  5. 查询数据

    <span style="background-color:#f8f8f8"><span style="color:#aa5500">-- tablesample，抽样函数</span>
<span style="color:#770088">select</span> * <span style="color:#770088">from</span> flows tablesample(<span style="color:#116644">5</span> rows);</span>

  6. 删除内部表，那么对应的元数据和HDFS上的目录都会删除；删除外部表，那么只删除元数据但是不删除HDFS上的目录

  7. 在实际过程中，项目前期使用外部表，中后期使用内部表

  8. 内部表转化为外部表

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> p3 <span style="color:#770088">set</span> tblproperties ( <span style="color:#aa1111">'EXTERNAL'</span> = <span style="color:#aa1111">'true'</span>);</span>

  9. 外部表转化为内部表

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> p3 <span style="color:#770088">set</span> tblproperties ( <span style="color:#aa1111">'EXTERNAL'</span> = <span style="color:#aa1111">'false'</span>);</span>

分区表

  1. 分区表的最常见的作用也是对数据进行分类

  2. 案例

    1. 原始数据

      <span style="background-color:#f8f8f8"><span style="color:#aa5500">-- wei</span>
<span style="color:#116644">1</span> 荀彧
<span style="color:#116644">2</span> 荀攸
<span style="color:#116644">3</span> 贾诩
<span style="color:#116644">4</span> 郭嘉
<span style="color:#aa5500">-- shu</span>
<span style="color:#116644">1</span> 孙乾
<span style="color:#116644">2</span> 简雍
<span style="color:#116644">3</span> 糜竺
<span style="color:#116644">4</span> 庞统
<span style="color:#aa5500">-- wu</span>
<span style="color:#116644">1</span> 陆逊
<span style="color:#116644">2</span> 张昭
<span style="color:#116644">3</span> 鲁肃
<span style="color:#116644">4</span> 周瑜</span>

    2. 在Hive中建表，管理数据，同时区分数据对应的国家

      <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> <span style="color:#770088">table</span> heros (
    id <span style="color:#3300aa">int</span>, <span style="color:#aa5500">-- 编号</span>
    name string <span style="color:#aa5500">-- 姓名</span>
) partitioned <span style="color:#770088">by</span> (country string)
 row format delimited fields terminated <span style="color:#770088">by</span> <span style="color:#aa1111">' '</span>;</span>

    3. 查看表

      <span style="background-color:#f8f8f8"><span style="color:#770088">desc</span> heros;</span>

    4. 将数据加载到表中。当向分区表中加载数据的时候，如果没有指定分区，那么会将数据放入默认分区__HIVE_DEFAULT_PARTITION__

      <span style="background-color:#f8f8f8">load data local inpath <span style="color:#aa1111">'/opt/hive_data/wei'</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> heros partition(country = <span style="color:#aa1111">'wei'</span>);
load data local inpath <span style="color:#aa1111">'/opt/hive_data/shu'</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> heros partition(country = <span style="color:#aa1111">'shu'</span>);
load data local inpath <span style="color:#aa1111">'/opt/hive_data/wu'</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> heros partition(country = <span style="color:#aa1111">'wu'</span>);</span>

  3. 在Hive中，每一个分区对应HDFS上一个单独的目录！！！

  4. 在分区表中，当查询数据的时候以分区作为查询条件，可以提高查询效率；如果进行跨分区查询，那么此时效率会降低

  5. 手动在HDFS上创建的目录，默认不会自动加载到Hive的元数据中

  6. 手动建立分区

    <span style="background-color:#f8f8f8"><span style="color:#aa5500"># 在HDFS上建立目录，上传文件</span>
hadoop fs <span style="color:#0000cc">-mkdir</span> /user/hive/warehouse/demo.db/heros<span style="color:#0000ff">/country</span><span style="color:#981a1a">=</span>qun
hadoop fs <span style="color:#0000cc">-put</span> qun /user/hive/warehouse/demo.db/heros<span style="color:#0000ff">/country</span><span style="color:#981a1a">=</span>qun</span>

    在Hive中手动添加分区

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> heros add partition(country = <span style="color:#aa1111">'qun'</span>) location <span style="color:#aa1111">'/user/hive/warehouse/demo.db/heros/country=qun'</span>;
<span style="color:#aa5500">-- 或者 修复分区 - 有可能会修复失败，如果修复失败，那么此时只能手动add</span>
msck repair <span style="color:#770088">table</span> heros;</span>

  7. 修改分区名

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> heros partition(country = <span style="color:#aa1111">'qun'</span>) rename to partition(country = <span style="color:#aa1111">'other'</span>);</span>

  8. 删除分区

    <span style="background-color:#f8f8f8"><span style="color:#770088">alter</span> <span style="color:#770088">table</span> heros <span style="color:#770088">drop</span> partition(country = <span style="color:#aa1111">'__HIVE_DEFAULT_PARTITION__'</span>);</span>

  9. 需要注意的是，在Hive中，分区字段在原始数据中是不存在的，而是在加载数据的时候手动指定，所以分区实际上可以认为是数据的"伪列"。如果原始数据中包含了分区字段，那么此时就需要进行动态分区

  10. 动态分区

    1. 原始数据

      <span style="background-color:#f8f8f8"><span style="color:#116644">1</span> other 吕布
<span style="color:#116644">2</span> wei 甘宁
<span style="color:#116644">3</span> shu 马岱
<span style="color:#116644">4</span> wei 王凌
<span style="color:#116644">5</span> shu 糜芳
<span style="color:#116644">6</span> shu 魏延
<span style="color:#116644">7</span> other 刘璋
<span style="color:#116644">8</span> wei 丁仪
<span style="color:#116644">9</span> wu 周平
<span style="color:#116644">10</span> shu 马忠</span>

    2. 建立临时表

      <span style="background-color:#f8f8f8"><span style="color:#770088">create</span> <span style="color:#770088">table</span> heros_tmp (
    hid <span style="color:#3300aa">int</span>,
    h_country string,
    h_name string
) row format delimited fields terminated <span style="color:#770088">by</span> <span style="color:#aa1111">' '</span>;</span>

    3. 加载数据

      <span style="background-color:#f8f8f8">load data local inpath <span style="color:#aa1111">'/opt/hive_data/heros'</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> heros_tmp;
<span style="color:#aa5500">-- 测试</span>
<span style="color:#770088">select</span> * <span style="color:#770088">from</span> heros_tmp tablesample(<span style="color:#116644">5</span> rows);</span>

    4. 动态分区默认是不开启

      <span style="background-color:#f8f8f8"><span style="color:#aa5500">-- 只对本次session有效，如果退出Hive重新进入，那么依然是严格模式</span>
<span style="color:#770088">set</span> hive<span style="color:#0055aa">.exec.dynamic.partition.mode</span> = nonstrict;</span>

    5. 将未分区表中的数据插入到分区表中 - 动态分区

      <span style="background-color:#f8f8f8"><span style="color:#333333"><span style="color:#770088">insert</span> <span style="color:#770088">into</span> <span style="color:#770088">table</span> heros partition(country) <span style="color:#770088">select</span> hid, h_name, h_country <span style="color:#770088">from</span> heros_tmp distribute <span style="color:#770088">by</span> h_country;</span></span>
本文含有隐藏内容，请 开通VIP 后查看

网站公告

今日签到

点亮在社区的每一天
去签到

热门文章

最新发布