org.apache.hadoop.util

Class LineReader

java.lang.Object

org.apache.hadoop.util.LineReader

All Implemented Interfaces:

java.io.Closeable, java.lang.AutoCloseable

@InterfaceAudience.LimitedPrivate(value="MapReduce")
 @InterfaceStability.Unstable
public class LineReader
extends java.lang.Object
implements java.io.Closeable

A class that provides a line reader from an input stream. Depending on the constructor used, lines will either be terminated by:

• one of the following: '\n' (LF) , '\r' (CR), or '\r\n' (CR+LF).
• or, a custom byte sequence delimiter

In both cases, EOF also terminates an otherwise unterminated line.

Constructor Summary

Constructors

Constructor and Description
LineReader(java.io.InputStream in) Create a line reader that reads from the given stream using the default buffer-size (64k).
LineReader(java.io.InputStream in, byte[] recordDelimiterBytes) Create a line reader that reads from the given stream using the default buffer-size, and using a custom delimiter of array of bytes.
LineReader(java.io.InputStream in, org.apache.hadoop.conf.Configuration conf) Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration.
LineReader(java.io.InputStream in, org.apache.hadoop.conf.Configuration conf, byte[] recordDelimiterBytes) Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration, and using a custom delimiter of array of bytes.
LineReader(java.io.InputStream in, int bufferSize) Create a line reader that reads from the given stream using the given buffer-size.
LineReader(java.io.InputStream in, int bufferSize, byte[] recordDelimiterBytes) Create a line reader that reads from the given stream using the given buffer-size, and using a custom delimiter of array of bytes.

Method Summary

Modifier and Type	Method and Description
void	close() Close the underlying stream.
protected int	fillBuffer(java.io.InputStream in, byte[] buffer, boolean inDelimiter)
protected int	getBufferPosn()
protected int	getBufferSize()
int	readLine(org.apache.hadoop.io.Text str) Read from the InputStream into the given Text.
int	readLine(org.apache.hadoop.io.Text str, int maxLineLength) Read from the InputStream into the given Text.
int	readLine(org.apache.hadoop.io.Text str, int maxLineLength, int maxBytesToConsume) Read one line from the InputStream into the given Text.
protected void	unsetNeedAdditionalRecordAfterSplit()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

LineReader

public LineReader(java.io.InputStream in)

Create a line reader that reads from the given stream using the default buffer-size (64k).

Parameters:

in - The input stream

Throws:

java.io.IOException

LineReader

public LineReader(java.io.InputStream in,
                  int bufferSize)

Create a line reader that reads from the given stream using the given buffer-size.

Parameters:

in - The input stream

bufferSize - Size of the read buffer

Throws:

java.io.IOException

LineReader

public LineReader(java.io.InputStream in,
                  org.apache.hadoop.conf.Configuration conf)
           throws java.io.IOException

Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration.

Parameters:

in - input stream

conf - configuration

Throws:

java.io.IOException

LineReader

public LineReader(java.io.InputStream in,
                  byte[] recordDelimiterBytes)

Create a line reader that reads from the given stream using the default buffer-size, and using a custom delimiter of array of bytes.

Parameters:

in - The input stream

recordDelimiterBytes - The delimiter

LineReader

public LineReader(java.io.InputStream in,
                  int bufferSize,
                  byte[] recordDelimiterBytes)

Create a line reader that reads from the given stream using the given buffer-size, and using a custom delimiter of array of bytes.

Parameters:

in - The input stream

bufferSize - Size of the read buffer

recordDelimiterBytes - The delimiter

Throws:

java.io.IOException

LineReader

public LineReader(java.io.InputStream in,
                  org.apache.hadoop.conf.Configuration conf,
                  byte[] recordDelimiterBytes)
           throws java.io.IOException

Create a line reader that reads from the given stream using the io.file.buffer.size specified in the given Configuration, and using a custom delimiter of array of bytes.

Parameters:

in - input stream

conf - configuration

recordDelimiterBytes - The delimiter

Throws:

java.io.IOException

Method Detail

close

public void close()
           throws java.io.IOException

Close the underlying stream.

Specified by:

close in interface java.io.Closeable

Specified by:

close in interface java.lang.AutoCloseable

Throws:

java.io.IOException

readLine

public int readLine(org.apache.hadoop.io.Text str,
                    int maxLineLength,
                    int maxBytesToConsume)
             throws java.io.IOException

Read one line from the InputStream into the given Text.

Parameters:

str - the object to store the given line (without newline)

maxLineLength - the maximum number of bytes to store into str; the rest of the line is silently discarded.

maxBytesToConsume - the maximum number of bytes to consume in this call. This is only a hint, because if the line cross this threshold, we allow it to happen. It can overshoot potentially by as much as one buffer length.

Returns:

the number of bytes read including the (longest) newline found.

Throws:

java.io.IOException - if the underlying stream throws

fillBuffer

protected int fillBuffer(java.io.InputStream in,
                         byte[] buffer,
                         boolean inDelimiter)
                  throws java.io.IOException

Throws:

java.io.IOException

readLine

public int readLine(org.apache.hadoop.io.Text str,
                    int maxLineLength)
             throws java.io.IOException

Read from the InputStream into the given Text.

Parameters:

str - the object to store the given line

maxLineLength - the maximum number of bytes to store into str.

Returns:

the number of bytes read including the newline

Throws:

java.io.IOException - if the underlying stream throws

readLine

public int readLine(org.apache.hadoop.io.Text str)
             throws java.io.IOException

Read from the InputStream into the given Text.

Parameters:

str - the object to store the given line

Returns:

the number of bytes read including the newline

Throws:

java.io.IOException - if the underlying stream throws

getBufferPosn

protected int getBufferPosn()

getBufferSize

protected int getBufferSize()

unsetNeedAdditionalRecordAfterSplit

protected void unsetNeedAdditionalRecordAfterSplit()