org.apache.hudi.common.table.log.block

Class HoodieLogBlock

java.lang.Object

org.apache.hudi.common.table.log.block.HoodieLogBlock

Direct Known Subclasses:

HoodieCommandBlock, HoodieCorruptBlock, HoodieDataBlock, HoodieDeleteBlock

public abstract class HoodieLogBlock
extends Object

Abstract class defining a block in HoodieLogFile.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	HoodieLogBlock.FooterMetadataType Log Metadata footers abstraction for a HoodieLogBlock WARNING : This enum is serialized as the ordinal.
static class	HoodieLogBlock.HeaderMetadataType Log Metadata headers abstraction for a HoodieLogBlock WARNING : This enum is serialized as the ordinal.
static class	HoodieLogBlock.HoodieLogBlockContentLocation This class is used to store the Location of the Content of a Log Block.
static class	HoodieLogBlock.HoodieLogBlockType Type of the log block WARNING: This enum is serialized as the ordinal.

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	readBlockLazily
static int	version The current version of the log block.

Constructor Summary

Constructors

Constructor and Description
HoodieLogBlock(Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader, Map<HoodieLogBlock.HeaderMetadataType,String> logBlockFooter, Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation, Option<byte[]> content, org.apache.hadoop.fs.FSDataInputStream inputStream, boolean readBlockLazily)

Method Summary

Modifier and Type	Method and Description
protected void	deflate() After the content bytes is converted into the required DataStructure by a logBlock, deflate the content to release byte [] and relieve memory pressure when GC kicks in.
Option<HoodieLogBlock.HoodieLogBlockContentLocation>	getBlockContentLocation()
abstract HoodieLogBlock.HoodieLogBlockType	getBlockType()
Option<byte[]>	getContent()
byte[]	getContentBytes()
Map<HoodieLogBlock.HeaderMetadataType,String>	getLogBlockFooter()
Map<HoodieLogBlock.HeaderMetadataType,String>	getLogBlockHeader()
long	getLogBlockLength()
static Map<HoodieLogBlock.HeaderMetadataType,String>	getLogMetadata(DataInputStream dis) Convert bytes to LogMetadata, follow the same order as getLogMetadataBytes(java.util.Map<org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType, java.lang.String>).
static byte[]	getLogMetadataBytes(Map<HoodieLogBlock.HeaderMetadataType,String> metadata) Convert log metadata to bytes 1.
byte[]	getMagic()
protected void	inflate() When lazyReading of blocks is turned on, inflate the content of a log block from disk.
static Option<byte[]>	tryReadContent(org.apache.hadoop.fs.FSDataInputStream inputStream, Integer contentLength, boolean readLazily) Read or Skip block content of a log block in the log file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

version

public static int version

The current version of the log block. Anytime the logBlock format changes this version needs to be bumped and corresponding changes need to be made to HoodieLogBlockVersion TODO : Change this to a class, something like HoodieLogBlockVersionV1/V2 and implement/override operations there

readBlockLazily

protected boolean readBlockLazily

Constructor Detail

HoodieLogBlock

public HoodieLogBlock(@Nonnull
                      Map<HoodieLogBlock.HeaderMetadataType,String> logBlockHeader,
                      @Nonnull
                      Map<HoodieLogBlock.HeaderMetadataType,String> logBlockFooter,
                      @Nonnull
                      Option<HoodieLogBlock.HoodieLogBlockContentLocation> blockContentLocation,
                      @Nonnull
                      Option<byte[]> content,
                      @Nullable
                      org.apache.hadoop.fs.FSDataInputStream inputStream,
                      boolean readBlockLazily)

Method Detail

getContentBytes

public byte[] getContentBytes()
                       throws IOException

Throws:

IOException

getMagic

public byte[] getMagic()

getBlockType

public abstract HoodieLogBlock.HoodieLogBlockType getBlockType()

getLogBlockLength

public long getLogBlockLength()

getBlockContentLocation

public Option<HoodieLogBlock.HoodieLogBlockContentLocation> getBlockContentLocation()

getLogBlockHeader

public Map<HoodieLogBlock.HeaderMetadataType,String> getLogBlockHeader()

getLogBlockFooter

public Map<HoodieLogBlock.HeaderMetadataType,String> getLogBlockFooter()

getContent

public Option<byte[]> getContent()

getLogMetadataBytes

public static byte[] getLogMetadataBytes(Map<HoodieLogBlock.HeaderMetadataType,String> metadata)
                                  throws IOException

Convert log metadata to bytes 1. Write size of metadata 2. Write enum ordinal 3. Write actual bytes

Throws:

IOException

getLogMetadata

public static Map<HoodieLogBlock.HeaderMetadataType,String> getLogMetadata(DataInputStream dis)
                                                                    throws IOException

Convert bytes to LogMetadata, follow the same order as getLogMetadataBytes(java.util.Map<org.apache.hudi.common.table.log.block.HoodieLogBlock.HeaderMetadataType, java.lang.String>).

Throws:

IOException

tryReadContent

public static Option<byte[]> tryReadContent(org.apache.hadoop.fs.FSDataInputStream inputStream,
                                            Integer contentLength,
                                            boolean readLazily)
                                     throws IOException

Read or Skip block content of a log block in the log file. Depends on lazy reading enabled in HoodieMergedLogRecordScanner

Throws:

IOException

inflate

protected void inflate()
                throws HoodieIOException

When lazyReading of blocks is turned on, inflate the content of a log block from disk.

Throws:

HoodieIOException

deflate

protected void deflate()

After the content bytes is converted into the required DataStructure by a logBlock, deflate the content to release byte [] and relieve memory pressure when GC kicks in. NOTE: This still leaves the heap fragmented