org.netpreserve.jwarc.cdx

Class CdxWriter

java.lang.Object

org.netpreserve.jwarc.cdx.CdxWriter

All Implemented Interfaces:

Closeable, AutoCloseable

public class CdxWriter
extends Object
implements Closeable

Writes CDX records.

Constructor Summary

Constructors

Constructor and Description
CdxWriter(Writer writer)

Method Summary

Modifier and Type	Method and Description
void	close()
void	onWarning(java.util.function.Consumer<String> warningHandler) Sets a handler for warnings emitted during processing.
void	process(List<Path> warcFiles, boolean useAbsolutePaths) Processes a list of WARC files writing CDX records for each response or resource record.
void	process(WarcReader reader, String filename) Writes CDX records for each response or resource record in a WARC file.
void	setFormat(CdxFormat format) Sets the CDX output format to use.
void	setPostAppend(boolean postAppend) Appends an encoded version of the request to the URL key for POST and PUT requests.
void	setRecordFilter(java.util.function.Predicate<WarcRecord> recordFilter) Records that don't match this filter will not be emitted.
void	write(WarcCaptureRecord capture, String filename, long position, long length) Writes a CDX record.
void	write(WarcCaptureRecord capture, String filename, long position, long length, String encodedRequest) Writes a CDX record with an encoded request body appended to the urlkey field.
void	writeHeaderLine() Writes a CDX header line.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CdxWriter

public CdxWriter(Writer writer)

Method Detail

writeHeaderLine

public void writeHeaderLine()
                     throws IOException

Writes a CDX header line.

Throws:

IOException

write

public void write(WarcCaptureRecord capture,
                  String filename,
                  long position,
                  long length)
           throws IOException

Writes a CDX record.

Throws:

IOException

write

public void write(WarcCaptureRecord capture,
                  String filename,
                  long position,
                  long length,
                  String encodedRequest)
           throws IOException

Writes a CDX record with an encoded request body appended to the urlkey field.

Throws:

IOException

See Also:

CdxRequestEncoder.encode(HttpRequest)

process

public void process(List<Path> warcFiles,
                    boolean useAbsolutePaths)
             throws IOException

Processes a list of WARC files writing CDX records for each response or resource record.

Throws:

IOException

process

public void process(WarcReader reader,
                    String filename)
             throws IOException

Writes CDX records for each response or resource record in a WARC file.

Throws:

IOException

setRecordFilter

public void setRecordFilter(java.util.function.Predicate<WarcRecord> recordFilter)

Records that don't match this filter will not be emitted. Accepts all records if null.

setFormat

public void setFormat(CdxFormat format)

Sets the CDX output format to use.

See Also:

CdxFormat.Builder

setPostAppend

public void setPostAppend(boolean postAppend)

Appends an encoded version of the request to the URL key for POST and PUT requests.

onWarning

public void onWarning(java.util.function.Consumer<String> warningHandler)

Sets a handler for warnings emitted during processing. Setting it to null disables warnings.

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException