JavaAPI操作hdfs

12.JavaAPI操作hdfs

1）下载maven 解压

2）配置环境变量 使用这个目录来放本地的jar包 我们在maven下新建一个名为repositories的文件夹 复制其路径添加到下面的文件位置

3）下面我们打开idea 新建工程 

输入组id 工程名 版本号 名字可以随便起 通过组id （工程名 版本号 就可以确定一个子项目）

配置idea与maven关联（不能自动关联）

点击欧克以后 我们需要等待一会 等jar包下载完毕

我们添加一下hadoop依赖jar的坐标 将下面的内容复制粘贴进去 pom.xml

 <properties>        <maven.compiler.source>8</maven.compiler.source>        <maven.compiler.target>8</maven.compiler.target>        <mysql.version>5.1.6</mysql.version>        <hadoop.version>2.7.1</hadoop.version>        <hbase.version>1.1.2</hbase.version>        <lzo_hadoop_verion>1.0.0</lzo_hadoop_verion>    </properties>    <repositories>        <repository>            <id>alimaven</id>            <name>aliyun maven</name>            <url>http://maven.aliyun.com/nexus/content/groups/public/</url>            <releases>                <enabled>true</enabled>            </releases>            <snapshots>                <enabled>false</enabled>            </snapshots>        </repository>    </repositories>    <pluginRepositories>        <pluginRepository>            <id>maven-net-cn</id>            <name>Maven China Mirror</name>            <url>http://maven.net.cn/content/groups/public/</url>            <releases>                <enabled>true</enabled>            </releases>            <snapshots>                <enabled>false</enabled>            </snapshots>        </pluginRepository>    </pluginRepositories>    <dependencies><!--groupid:-->        <dependency>            <groupId>mysql</groupId>            <artifactId>mysql-connector-java</artifactId>            <version>${mysql.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-common</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-hdfs</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-mapreduce-client-core</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hadoop</groupId>            <artifactId>hadoop-mapreduce-client-common</artifactId>            <version>${hadoop.version}</version>        </dependency>        <dependency>            <groupId>org.apache.hbase</groupId>            <artifactId>hbase-it</artifactId>            <version>${hbase.version}</version>            <exclusions>                <exclusion>                    <artifactId>jdk.tools</artifactId>                    <groupId>jdk.tools</groupId>                </exclusion>            </exclusions>        </dependency><!--        <dependency>--><!--            <groupId>org.anarres.lzo</groupId>--><!--            <artifactId>lzo-hadoop</artifactId>--><!--            <version>${lzo_hadoop_verion}</version>--><!--        </dependency>--><!--        <dependency>--><!--            <groupId>org.anarres.lzo</groupId>--><!--            <artifactId>lzo-core</artifactId>--><!--            <version>${lzo_hadoop_verion}</version>--><!--        </dependency>-->​        <dependency>            <groupId>org.apache.maven.plugins</groupId>            <artifactId>maven-assembly-plugin</artifactId>            <version>3.6.0</version>        </dependency>    </dependencies>    <build>        <sourceDirectory>src/main/java</sourceDirectory>        <testSourceDirectory>src/test</testSourceDirectory>        <plugins>            <plugin>                <groupId>org.apache.maven.plugins</groupId>                <artifactId>maven-assembly-plugin</artifactId>                <version>3.6.0</version>                <configuration>                    <descriptorRefs>                        <descriptorRef>jar-with-dependencies</descriptorRef>                    </descriptorRefs>                    <archive>                        <manifest>                            <mainClass>com.oracle.Test</mainClass>                        </manifest>                    </archive>                </configuration>                <executions>                    <execution>                        <id>make-assembly</id>                        <phase>package</phase>                        <goals>                            <goal>single</goal>                        </goals>                    </execution>                </executions>            </plugin>        </plugins>    </build>

添加完以后出现报错 我们刷新一下

刷新完毕

Repositories：它是设置下载jar包的网址的

4）利用JavaAPI把d:/a.txt文件上传到Hadoop上

package com.pracle.example;​​import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.fs.FSDataOutputStream;import org.apache.hadoop.fs.FileSystem;import org.apache.hadoop.fs.Path;​import java.io.IOException;import java.net.URI;import java.net.URISyntaxException;import java.nio.charset.StandardCharsets;​public class Test1 {    public static void main(String[] args) throws URISyntaxException, IOException {        // 配置文件对象类 用于加载Hadoop的配置        Configuration configuration=new Configuration();        // 获取Hadoop文件系统 hdfs://192.168.67.90:9000 namenode        FileSystem fs = FileSystem.get(new URI("hdfs://192.168.67.90:9000"), configuration);        // 创建一个指向b.txt的输出流        FSDataOutputStream out = fs.create(new Path("/b.txt"));// hadoop路径        byte[] c="aaaaaaaaaa".getBytes(); // 把字符串转换成字节数组        out.write(c);        out.close();​​    }}​

点击运行 查看证实