Hive3.1.3加载paimon-hive-connector-3.1-1.1.1.jar报错UnsatisfiedLinkError

发布于:2025-06-27 ⋅ 阅读:(20) ⋅ 点赞:(0)

mkdir /app/apache-hive-3.1.3-sqg-bin/auxlib

cp paimon-hive-connector-3.1-1.1.1.jar /app/apache-hive-3.1.3-sqg-bin/auxlib/

查询paimon表时报错

Exception in thread "main" java.lang.UnsatisfiedLinkError: com.github.luben.zstd.ZstdInputStreamNoFinalizer.recommendedDInSize()J
        at com.github.luben.zstd.ZstdInputStreamNoFinalizer.recommendedDInSize(Native Method)
        at com.github.luben.zstd.ZstdInputStreamNoFinalizer.<clinit>(ZstdInputStreamNoFinalizer.java:35)
        at org.apache.paimon.shade.org.apache.avro.file.ZstandardLoader.input(ZstandardLoader.java:38)
        at org.apache.paimon.shade.org.apache.avro.file.ZstandardCodec.decompress(ZstandardCodec.java:84)
        at org.apache.paimon.shade.org.apache.avro.file.DataFileStream$DataBlock.decompressUsing(DataFileStream.java:392)
        at org.apache.paimon.shade.org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:226)
        at org.apache.paimon.format.avro.AvroBulkFormat.replaceAvroRuntimeException(AvroBulkFormat.java:156)
        at org.apache.paimon.format.avro.AvroBulkFormat.access$200(AvroBulkFormat.java:43)
        at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.readNextBlock(AvroBulkFormat.java:119)
        at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.readBatch(AvroBulkFormat.java:103)
        at org.apache.paimon.format.avro.AvroBulkFormat$AvroReader.readBatch(AvroBulkFormat.java:57)
        at org.apache.paimon.reader.RecordReaderIterator.<init>(RecordReaderIterator.java:37)
        at org.apache.paimon.reader.RecordReader.toCloseableIterator(RecordReader.java:210)
        at org.apache.paimon.utils.ObjectsFile.createIterator(ObjectsFile.java:186)
        at org.apache.paimon.utils.ObjectsFile.readWithIOException(ObjectsFile.java:153)
        at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:134)
        at org.apache.paimon.utils.ObjectsFile.read(ObjectsFile.java:105)
        at org.apache.paimon.manifest.ManifestList.readDataManifests(ManifestList.java:90)
        at org.apache.paimon.operation.ManifestsReader.readManifests(ManifestsReader.java:128)
        at org.apache.paimon.operation.ManifestsReader.read(ManifestsReader.java:114)
        at org.apache.paimon.operation.AbstractFileStoreScan.readManifests(AbstractFileStoreScan.java:417)
        at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:257)
        at org.apache.paimon.table.source.snapshot.SnapshotReaderImpl.read(SnapshotReaderImpl.java:318)
        at org.apache.paimon.table.source.snapshot.ReadPlanStartingScanner.scan(ReadPlanStartingScanner.java:45)
        at org.apache.paimon.table.source.DataTableBatchScan.plan(DataTableBatchScan.java:77)
        at org.apache.paimon.hive.utils.HiveSplitGenerator.generateSplits(HiveSplitGenerator.java:100)
        at org.apache.paimon.hive.mapred.PaimonInputFormat.getSplits(PaimonInputFormat.java:45)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:425)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:395)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:314)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:540)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:509)
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:146)
        at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2691)
        at org.apache.hadoop.hive.ql.reexec.ReExecDriver.getResults(ReExecDriver.java:229)
        at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
        at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

解决办法:
由于paimon-hive-connector-3.1-1.1.1.jar中zstd版本冲突,需要将zstd-jni-1.3.2-2.jar移出hive lib目录

mv /app/apache-hive-3.1.3-sqg-bin/lib/zstd-jni-1.3.2-2.jar /app/apache-hive-3.1.3-sqg-bin/ 

如果还报错要检查是否安装zstd


网站公告

今日签到

点亮在社区的每一天
去签到