Package org.apache.iceberg.hive

Class HiveTableOperations

  • All Implemented Interfaces:
    org.apache.iceberg.TableOperations
    public class HiveTableOperations
extends org.apache.iceberg.BaseMetastoreTableOperations
    TODO we should be able to extract some more commonalities to BaseMetastoreTableOperations to avoid code duplication between this class and Metacat Tables.

    • Nested Class Summary

      • Nested classes/interfaces inherited from class org.apache.iceberg.BaseMetastoreOperations

        org.apache.iceberg.BaseMetastoreOperations.CommitStatus

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected HiveTableOperations​(org.apache.hadoop.conf.Configuration conf, org.apache.iceberg.ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,​org.apache.thrift.TException> metaClients, org.apache.iceberg.io.FileIO fileIO, java.lang.String catalogName, java.lang.String database, java.lang.String table)  

    • Method Summary

      All Methods Static Methods Instance Methods Concrete Methods Default Methods 
      Modifier and Type Method Description
      java.lang.String database()  
      protected void doCommit​(org.apache.iceberg.TableMetadata base, org.apache.iceberg.TableMetadata metadata)  
      protected void doRefresh()  
      default boolean exposeInHmsProperties()  
      default java.util.Map<java.lang.String,​java.lang.String> hmsEnvContext​(java.lang.String metadataLocation)  
      org.apache.iceberg.io.FileIO io()  
      default org.apache.hadoop.hive.metastore.api.Table loadHmsTable()  
      long maxHiveTablePropertySize()  
      org.apache.iceberg.ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,​org.apache.thrift.TException> metaClients()  
      default org.apache.hadoop.hive.metastore.api.Table newHmsTable​(java.lang.String hmsTableOwner)  
      default void persistTable​(org.apache.hadoop.hive.metastore.api.Table hmsTable, boolean updateHiveTable, java.lang.String metadataLocation)  
      default void setField​(java.util.Map<java.lang.String,​java.lang.String> parameters, java.lang.String key, java.lang.String value)  
      default void setSchema​(org.apache.iceberg.Schema schema, java.util.Map<java.lang.String,​java.lang.String> parameters)  
      java.lang.String table()  
      protected java.lang.String tableName()  
      org.apache.hadoop.hive.metastore.TableType tableType()  
      static java.lang.String translateToIcebergProp​(java.lang.String hmsProp)
      Provides key translation where necessary between Iceberg and HMS props.

      • Methods inherited from class org.apache.iceberg.BaseMetastoreTableOperations

        checkCommitStatus, commit, current, currentMetadataLocation, currentVersion, disableRefresh, locationProvider, metadataFileLocation, refresh, refreshFromMetadataLocation, refreshFromMetadataLocation, refreshFromMetadataLocation, refreshFromMetadataLocation, requestRefresh, temp, writeNewMetadata, writeNewMetadataIfRequired

      • Methods inherited from class org.apache.iceberg.BaseMetastoreOperations

        checkCommitStatus

      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

      • Methods inherited from interface org.apache.iceberg.TableOperations

        encryption, newSnapshotId, requireStrictCleanup

    • Field Detail

      • HIVE_TABLE_PROPERTY_MAX_SIZE

        public static final java.lang.String HIVE_TABLE_PROPERTY_MAX_SIZE
        See Also:
        Constant Field Values

      • HIVE_TABLE_PROPERTY_MAX_SIZE_DEFAULT

        public static final long HIVE_TABLE_PROPERTY_MAX_SIZE_DEFAULT
        See Also:
        Constant Field Values

      • NO_LOCK_EXPECTED_KEY

        public static final java.lang.String NO_LOCK_EXPECTED_KEY
        See Also:
        Constant Field Values

      • NO_LOCK_EXPECTED_VALUE

        public static final java.lang.String NO_LOCK_EXPECTED_VALUE
        See Also:
        Constant Field Values

      • ICEBERG_VIEW_TYPE_VALUE

        public static final java.lang.String ICEBERG_VIEW_TYPE_VALUE
        See Also:
        Constant Field Values

    • Constructor Detail

      • HiveTableOperations

        protected HiveTableOperations​(org.apache.hadoop.conf.Configuration conf,
                              org.apache.iceberg.ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,​org.apache.thrift.TException> metaClients,
                              org.apache.iceberg.io.FileIO fileIO,
                              java.lang.String catalogName,
                              java.lang.String database,
                              java.lang.String table)

    • Method Detail

      • translateToIcebergProp

        public static java.lang.String translateToIcebergProp​(java.lang.String hmsProp)
        Provides key translation where necessary between Iceberg and HMS props. This translation is needed because some properties control the same behaviour but are named differently in Iceberg and Hive. Therefore changes to these property pairs should be synchronized.

        Example: Deleting data files upon DROP TABLE is enabled using gc.enabled=true in Iceberg and external.table.purge=true in Hive. Hive and Iceberg users are unaware of each other's control flags, therefore inconsistent behaviour can occur from e.g. a Hive user's point of view if external.table.purge=true is set on the HMS table but gc.enabled=false is set on the Iceberg table, resulting in no data file deletion.

        Parameters:
        hmsProp - The HMS property that should be translated to Iceberg property
        Returns:
        Iceberg property equivalent to the hmsProp. If no such translation exists, the original hmsProp is returned

      • tableName

        protected java.lang.String tableName()
        Specified by:
        tableName in class org.apache.iceberg.BaseMetastoreTableOperations

      • io

        public org.apache.iceberg.io.FileIO io()
        Specified by:
        io in interface org.apache.iceberg.TableOperations

      • doRefresh

        protected void doRefresh()
        Overrides:
        doRefresh in class org.apache.iceberg.BaseMetastoreTableOperations

      • doCommit

        protected void doCommit​(org.apache.iceberg.TableMetadata base,
                        org.apache.iceberg.TableMetadata metadata)
        Overrides:
        doCommit in class org.apache.iceberg.BaseMetastoreTableOperations

      • maxHiveTablePropertySize

        public long maxHiveTablePropertySize()

      • database

        public java.lang.String database()

      • table

        public java.lang.String table()

      • tableType

        public org.apache.hadoop.hive.metastore.TableType tableType()

      • metaClients

        public org.apache.iceberg.ClientPool<org.apache.hadoop.hive.metastore.IMetaStoreClient,​org.apache.thrift.TException> metaClients()

      • loadHmsTable

        public default org.apache.hadoop.hive.metastore.api.Table loadHmsTable()
                                                                throws org.apache.thrift.TException,
                                                                       java.lang.InterruptedException
        Throws:
        org.apache.thrift.TException
        java.lang.InterruptedException

      • hmsEnvContext

        public default java.util.Map<java.lang.String,​java.lang.String> hmsEnvContext​(java.lang.String metadataLocation)

      • exposeInHmsProperties

        public default boolean exposeInHmsProperties()

      • setSchema

        public default void setSchema​(org.apache.iceberg.Schema schema,
                              java.util.Map<java.lang.String,​java.lang.String> parameters)

      • setField

        public default void setField​(java.util.Map<java.lang.String,​java.lang.String> parameters,
                             java.lang.String key,
                             java.lang.String value)

      • persistTable

        public default void persistTable​(org.apache.hadoop.hive.metastore.api.Table hmsTable,
                                 boolean updateHiveTable,
                                 java.lang.String metadataLocation)
                          throws org.apache.thrift.TException,
                                 java.lang.InterruptedException
        Throws:
        org.apache.thrift.TException
        java.lang.InterruptedException

      • newHmsTable

        public default org.apache.hadoop.hive.metastore.api.Table newHmsTable​(java.lang.String hmsTableOwner)