GenomicsDBImporter (com.intel:genomicsdb-gatk4 0.4.0 API)

java.lang.Object
- com.intel.genomicsdb.GenomicsDBImporter

```
public class GenomicsDBImporter
extends Object
```
Java wrapper for vcf2tiledb - imports VCFs into TileDB/GenomicsDB. All vid information is assumed to be set correctly by the user (JSON files)

Field Summary

Fields
Modifier and Type	Field and Description
`private String`	`DEFAULT_ARRAYNAME`
`private long`	`DEFAULT_SIZE_PER_COLUMN_PARTITION`
`private int`	`DEFAULT_TILEDB_CELLS_PER_TILE`
`private ArrayList<GenomicsDBImporterStreamWrapper>`	`mBufferStreamWrapperVector`
`private GenomicsDBCallsetsMapProto.CallsetMappingPB`	`mCallsetMap`
`private org.json.simple.JSONObject`	`mCallsetMappingJSON`
`private ChromosomeInterval`	`mChromosomeInterval`
`private boolean`	`mContainsBufferStreams`
`(package private) static long`	`mDefaultBufferCapacity`
`private boolean`	`mDone`
`private long[]`	`mExhaustedBufferStreamIdentifiers`
`private long`	`mGenomicsDBImporterObjectHandle`
`private boolean`	`mIsLoaderSetupDone`
`private long`	`mLbRowIdx`
`private String`	`mLoaderJSONFile`
`private long`	`mNumExhaustedBufferStreams`
`private int`	`mRank`
`private static String`	`mTempLoaderJSONFileName`
`private long`	`mUbRowIdx`
`private boolean`	`mUsingVidMappingProtoBuf`
`private GenomicsDBVidMapProto.VidMappingPB`	`mVidMap`

Constructor Summary

Constructors
Constructor and Description
`GenomicsDBImporter()` Default Constructor
`GenomicsDBImporter(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> sampleToVCMap, Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader, ChromosomeInterval chromosomeInterval, String workspace, String arrayname, Long sizePerColumnPartition, Long segmentSize)` Constructor to create required data structures from a list of GVCF files and a chromosome interval.
`GenomicsDBImporter(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> sampleToVCMap, Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader, ChromosomeInterval chromosomeInterval, String workspace, String arrayname, Long sizePerColumnPartition, Long segmentSize, boolean useSamplesInOrderProvided)` Constructor to create required data structures from a list of GVCF files and a chromosome interval.
`GenomicsDBImporter(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> sampleToVCMap, Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader, ChromosomeInterval chromosomeInterval, String workspace, String arrayname, Long sizePerColumnPartition, Long segmentSize, String outputVidMapJSONFilePath)` Constructor to create required data structures from a list of GVCF files and a chromosome interval.
`GenomicsDBImporter(String loaderJSONFile)` Constructor
`GenomicsDBImporter(String loaderJSONFile, int rank)` Constructor
`GenomicsDBImporter(String loaderJSONFile, int rank, long lbRowIdx, long ubRowIdx)` Constructor

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`boolean`	`add(htsjdk.variant.variantcontext.VariantContext vc, int streamIdx)` Write VariantContext object to stream - may fail if the buffer is full It's the caller's responsibility keep track of the VC object that's not written
`private int`	`addBufferStream(String streamName, htsjdk.variant.vcf.VCFHeader vcfHeader, long bufferCapacity, htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType, Iterator<htsjdk.variant.variantcontext.VariantContext> vcIterator, Map<Integer,SampleInfo> sampleIndexToInfo)` Add a buffer stream or VC iterator - internal function
`int`	`addBufferStream(String streamName, htsjdk.variant.vcf.VCFHeader vcfHeader, long bufferCapacity, htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType, Map<Integer,SampleInfo> sampleIndexToInfo)` Add a buffer stream as the data source - caller must: 1.
`int`	`addSortedVariantContextIterator(String streamName, htsjdk.variant.vcf.VCFHeader vcfHeader, Iterator<htsjdk.variant.variantcontext.VariantContext> vcIterator, long bufferCapacity, htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType, Map<Integer,SampleInfo> sampleIndexToInfo)` Add a sorted VC iterator as the data source - caller must: 1.
`<SOURCE> MultiChromosomeIterator<SOURCE>`	`columnPartitionIterator(htsjdk.tribble.AbstractFeatureReader<htsjdk.variant.variantcontext.VariantContext,SOURCE> reader)` Utility function that returns a MultiChromosomeIterator given an AbstractFeatureReader that will iterate over the VariantContext objects provided by the reader belonging to the column partition specified by this object's loader JSON file and rank/partition index
`static <SOURCE> MultiChromosomeIterator<SOURCE>`	`columnPartitionIterator(htsjdk.tribble.AbstractFeatureReader<htsjdk.variant.variantcontext.VariantContext,SOURCE> reader, String loaderJSONFile, int partitionIdx)` Utility function that returns a MultiChromosomeIterator given an AbstractFeatureReader that will iterate over the VariantContext objects provided by the reader belonging to the column partition specified by the loader JSON file and rank/partition index
`private GenomicsDBImportConfiguration.ImportConfiguration`	`createImportConfiguration(String workspace, String arrayname, Long sizePerColumnPartition, Long segmentSize)`
`private GenomicsDBCallsetsMapProto.CallsetMappingPB`	`generateSortedCallSetMap(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> variants, boolean useSamplesInOrderProvided)` Creates a sorted list of callsets and generates unique TileDB row indices for them.
`private GenomicsDBVidMapProto.VidMappingPB`	`generateVidMapFromMergedHeader(Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader)` Generate the ProtoBuf data structure for vid mapping Also remember, contigs are 1-based which means TileDB column offsets should start from 1
`private static ArrayList<ChromosomeInterval>`	`getChromosomeIntervalsForColumnPartition(String loaderJSONFile, int partitionIdx)` Utility function that returns a list of ChromosomeInterval objects for the column partition specified by the loader JSON file and rank/partition index
`int`	`getExhaustedBufferStreamIndex(long i)` Get buffer stream index of i-th exhausted stream There are mNumExhaustedBufferStreams and the caller must provide data for streams with indexes getExhaustedBufferStreamIndex(0), getExhaustedBufferStreamIndex(1),..., getExhaustedBufferStreamIndex(mNumExhaustedBufferStreams-1)
`private String`	`getLength(htsjdk.variant.vcf.VCFHeaderLine headerLine)` Maps the "Number" from INFO or FORMAT fields in VCF header to GenomicsDB lengths.
`long`	`getNumExhaustedBufferStreams()`
`boolean`	`importBatch()`
`private void`	`initialize(String loaderJSONFile, int rank, long lbRowIdx, long ubRowIdx)` Initialize variables
`static long`	`initializeSampleInfoMapFromHeader(Map<Integer,SampleInfo> sampleIndexToInfo, htsjdk.variant.vcf.VCFHeader vcfHeader, long rowIdx)` Static function that reads sample names from the vcfHeader and adds entries to the map.
`boolean`	`isDone()` Is the import process completed
`private void`	`jniAddBufferStream(long genomicsDBImporterHandle, String streamName, boolean isBCF, long bufferCapacity, byte[] buffer, long numValidBytesInBuffer)` Notify importer object that a new stream is to be added
`private long`	`jniCopyCallsetMap(long genomicsDBImporterHandle, byte[] callsetMapAsByteArray)` Copy the callset map protocol buffer to C++ through JNI
`private long`	`jniCopyVidMap(long genomicsDBImporterHandle, byte[] vidMapAsByteArray)` Copy the vid map protocol buffer to C++ through JNI
`private int`	`jniGenomicsDBImporter(String loaderJSONFile, int rank, long lbRowIdx, long ubRowIdx)` Creates GenomicsDBImporter object when importing VCF files (no streams)
`private static String`	`jniGetChromosomeIntervalsForColumnPartition(String loaderJSONFile, int rank)` Obtain the chromosome intervals for the column partition specified in the loader JSON file identified by the rank.
`private boolean`	`jniImportBatch(long genomicsDBImporterHandle, long[] exhaustedBufferIdentifiers)` Import the next batch of data into TileDB/GenomicsDB
`private long`	`jniInitializeGenomicsDBImporterObject(String loaderJSONFile, int rank, long lbRowIdx, long ubRowIdx)` Creates GenomicsDBImporter object when importing VCF files (no streams)
`private long`	`jniSetupGenomicsDBLoader(long genomicsDBImporterHandle, String callsetMappingJSON, boolean usingVidMappingProtoBuf)` Setup loader after all the buffer streams are added
`private void`	`jniWriteDataToBufferStream(long handle, int streamIdx, int partitionIdx, byte[] buffer, long numValidBytesInBuffer)`
`static File`	`printLoaderJSONFile(GenomicsDBImportConfiguration.ImportConfiguration importConfiguration, String filename)` Create a JSON file from the import configuration
`private GenomicsDBVidMapProto.InfoField`	`remove(List<GenomicsDBVidMapProto.InfoField> infoFields, int dpIndex)`
`int`	`setSortedVariantContextIterator(String streamName, htsjdk.variant.vcf.VCFHeader vcfHeader, Iterator<htsjdk.variant.variantcontext.VariantContext> vcIterator, long bufferCapacity, htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType, Map<Integer,SampleInfo> sampleIndexToInfo)` Sets sorted VC iterator as the data source and calls setupGenomicsDBImporter().
`void`	`setupGenomicsDBImporter()` Setup the importer after all the buffer streams are added, but before any data is inserted into any stream No more buffer streams can be added once setupGenomicsDBImporter() is called
`void`	`write()` Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
`void`	`write(int rank, long lbRowIdx)` Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
`void`	`write(int rank, long lbRowIdx, long ubRowIdx)` Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
`void`	`write(long lbRowIdx)` Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
`void`	`write(String loaderJSONFile, int rank, long lbRowIdx, long ubRowIdx)` Write to TileDB/GenomicsDB

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - mDefaultBufferCapacity
```
static long mDefaultBufferCapacity
```
  - mTempLoaderJSONFileName
```
private static final String mTempLoaderJSONFileName
```
    See Also:
    
    Constant Field Values
  - DEFAULT_ARRAYNAME
```
private final String DEFAULT_ARRAYNAME
```
    See Also:
    
    Constant Field Values
  - DEFAULT_SIZE_PER_COLUMN_PARTITION
```
private final long DEFAULT_SIZE_PER_COLUMN_PARTITION
```
    See Also:
    
    Constant Field Values
  - DEFAULT_TILEDB_CELLS_PER_TILE
```
private final int DEFAULT_TILEDB_CELLS_PER_TILE
```
    See Also:
    
    Constant Field Values
  - mLoaderJSONFile
```
private String mLoaderJSONFile
```
  - mRank
```
private int mRank
```
  - mLbRowIdx
```
private long mLbRowIdx
```
  - mUbRowIdx
```
private long mUbRowIdx
```
  - mContainsBufferStreams
```
private boolean mContainsBufferStreams
```
  - mGenomicsDBImporterObjectHandle
```
private long mGenomicsDBImporterObjectHandle
```
  - mBufferStreamWrapperVector
```
private ArrayList<GenomicsDBImporterStreamWrapper> mBufferStreamWrapperVector
```
  - mIsLoaderSetupDone
```
private boolean mIsLoaderSetupDone
```
  - mExhaustedBufferStreamIdentifiers
```
private long[] mExhaustedBufferStreamIdentifiers
```
  - mNumExhaustedBufferStreams
```
private long mNumExhaustedBufferStreams
```
  - mDone
```
private boolean mDone
```
  - mCallsetMappingJSON
```
private org.json.simple.JSONObject mCallsetMappingJSON
```
  - mUsingVidMappingProtoBuf
```
private boolean mUsingVidMappingProtoBuf
```
  - mVidMap
```
private GenomicsDBVidMapProto.VidMappingPB mVidMap
```
  - mChromosomeInterval
```
private ChromosomeInterval mChromosomeInterval
```
  - mCallsetMap
```
private GenomicsDBCallsetsMapProto.CallsetMappingPB mCallsetMap
```
- Constructor Detail
  - GenomicsDBImporter
```
public GenomicsDBImporter()
```
    Default Constructor
  - GenomicsDBImporter
```
public GenomicsDBImporter(String loaderJSONFile)
```
    Constructor
    
    Parameters:
    
    loaderJSONFile - GenomicsDB loader JSON configuration file
  - GenomicsDBImporter
```
public GenomicsDBImporter(String loaderJSONFile,
                          int rank)
```
    Constructor
    
    Parameters:
    
    loaderJSONFile - GenomicsDB loader JSON configuration file
    
    rank - Rank of this process (TileDB/GenomicsDB partition idx)
  - GenomicsDBImporter
```
public GenomicsDBImporter(String loaderJSONFile,
                          int rank,
                          long lbRowIdx,
                          long ubRowIdx)
```
    Constructor
    
    Parameters:
    
    loaderJSONFile - GenomicsDB loader JSON configuration file
    
    rank - Rank of this process (TileDB/GenomicsDB partition idx)
    
    lbRowIdx - Smallest row idx which should be imported by this object
    
    ubRowIdx - Largest row idx which should be imported by this object
  - GenomicsDBImporter
```
public GenomicsDBImporter(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> sampleToVCMap,
                          Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader,
                          ChromosomeInterval chromosomeInterval,
                          String workspace,
                          String arrayname,
                          Long sizePerColumnPartition,
                          Long segmentSize)
                   throws IOException
```
    Constructor to create required data structures from a list of GVCF files and a chromosome interval. This constructor is developed specifically for GATK4 GenomicsDBImport tool.
    
    Parameters:
    
    sampleToVCMap - Variant Readers objects of the input GVCF files
    
    chromosomeInterval - Chromosome interval to traverse input VCFs
    
    workspace - TileDB workspace
    
    arrayname - TileDB array name
    
    sizePerColumnPartition - sizePerColumnPartition in bytes
    
    segmentSize - segmentSize in bytes
    
    Throws:
    
    IOException
  - GenomicsDBImporter
```
public GenomicsDBImporter(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> sampleToVCMap,
                          Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader,
                          ChromosomeInterval chromosomeInterval,
                          String workspace,
                          String arrayname,
                          Long sizePerColumnPartition,
                          Long segmentSize,
                          boolean useSamplesInOrderProvided)
                   throws IOException
```
    Constructor to create required data structures from a list of GVCF files and a chromosome interval. This constructor is developed specifically for GATK4 GenomicsDBImport tool.
    
    Parameters:
    
    sampleToVCMap - Variant Readers objects of the input GVCF files
    
    chromosomeInterval - Chromosome interval to traverse input VCFs
    
    workspace - TileDB workspace
    
    arrayname - TileDB array name
    
    sizePerColumnPartition - sizePerColumnPartition in bytes
    
    segmentSize - segmentSize in bytes
    
    useSamplesInOrderProvided - if true, don't sort samples, instead use in the the order provided
    
    Throws:
    
    IOException
  - GenomicsDBImporter
```
public GenomicsDBImporter(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> sampleToVCMap,
                          Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader,
                          ChromosomeInterval chromosomeInterval,
                          String workspace,
                          String arrayname,
                          Long sizePerColumnPartition,
                          Long segmentSize,
                          String outputVidMapJSONFilePath)
                   throws IOException
```
    Constructor to create required data structures from a list of GVCF files and a chromosome interval. This constructor is developed specifically for GATK4 GenomicsDBImport tool.
    
    Parameters:
    
    sampleToVCMap - Variant Readers objects of the input GVCF files
    
    chromosomeInterval - Chromosome interval to traverse input VCFs
    
    Throws:
    
    IOException
- Method Detail
  - jniGenomicsDBImporter
```
private int jniGenomicsDBImporter(String loaderJSONFile,
                                  int rank,
                                  long lbRowIdx,
                                  long ubRowIdx)
```
    Creates GenomicsDBImporter object when importing VCF files (no streams)
    
    Parameters:
    
    loaderJSONFile - Path to loader JSON file
    
    rank - Rank of object - corresponds to the partition index in the loader for which this object will import data
    
    lbRowIdx - Smallest row idx which should be imported by this object
    
    ubRowIdx - Largest row idx which should be imported by this object
    
    Returns:
    
    status - 0 if everything was ok, -1 otherwise
  - jniInitializeGenomicsDBImporterObject
```
private long jniInitializeGenomicsDBImporterObject(String loaderJSONFile,
                                                   int rank,
                                                   long lbRowIdx,
                                                   long ubRowIdx)
```
    Creates GenomicsDBImporter object when importing VCF files (no streams)
    
    Parameters:
    
    loaderJSONFile - Path to loader JSON file
    
    rank - Rank of object - corresponds to the partition index in the loader for which this object will import data
    
    lbRowIdx - Smallest row idx which should be imported by this object
    
    ubRowIdx - Largest row idx which should be imported by this object
    
    Returns:
    
    "pointer"/address to GenomicsDBImporter object in memory, if 0, then something went wrong
  - jniCopyVidMap
```
private long jniCopyVidMap(long genomicsDBImporterHandle,
                           byte[] vidMapAsByteArray)
```
    Copy the vid map protocol buffer to C++ through JNI
    
    Parameters:
    
    genomicsDBImporterHandle - Reference to a C++ GenomicsDBImporter object
    
    vidMapAsByteArray - INFO, FORMAT, FILTER header lines and contig positions
    
    Returns:
    
    Reference to a C++ GenomicsDBImporter object as long
  - jniCopyCallsetMap
```
private long jniCopyCallsetMap(long genomicsDBImporterHandle,
                               byte[] callsetMapAsByteArray)
```
    Copy the callset map protocol buffer to C++ through JNI
    
    Parameters:
    
    genomicsDBImporterHandle - Reference to a C++ GenomicsDBImporter object
    
    callsetMapAsByteArray - Callset name and row index map
    
    Returns:
    
    Reference to a C++ GenomicsDBImporter object as long
  - jniAddBufferStream
```
private void jniAddBufferStream(long genomicsDBImporterHandle,
                                String streamName,
                                boolean isBCF,
                                long bufferCapacity,
                                byte[] buffer,
                                long numValidBytesInBuffer)
```
    Notify importer object that a new stream is to be added
    
    Parameters:
    
    genomicsDBImporterHandle - "pointer" returned by jniInitializeGenomicsDBImporterObject
    
    streamName - name of the stream
    
    isBCF - use BCF format to pass data to C++ layer
    
    bufferCapacity - in bytes
    
    buffer - initialization buffer containing the VCF/BCF header
    
    numValidBytesInBuffer - num valid bytes in the buffer (length of the header)
  - jniSetupGenomicsDBLoader
```
private long jniSetupGenomicsDBLoader(long genomicsDBImporterHandle,
                                      String callsetMappingJSON,
                                      boolean usingVidMappingProtoBuf)
```
    Setup loader after all the buffer streams are added
    
    Parameters:
    
    genomicsDBImporterHandle - "pointer" returned by jniInitializeGenomicsDBImporterObject
    
    callsetMappingJSON - JSON formatted string containing globally consistent callset name to row index mapping
    
    Returns:
    
    maximum number of buffer stream identifiers that can be returned in mExhaustedBufferStreamIdentifiers later (this depends on the number of partitions and the number of buffer streams)
  - jniWriteDataToBufferStream
```
private void jniWriteDataToBufferStream(long handle,
                                        int streamIdx,
                                        int partitionIdx,
                                        byte[] buffer,
                                        long numValidBytesInBuffer)
```
    Parameters:
    
    handle - "pointer" returned by jniInitializeGenomicsDBImporterObject
    
    streamIdx - stream index
    
    partitionIdx - partition index (unused now)
    
    buffer - buffer containing data
    
    numValidBytesInBuffer - num valid bytes in the buffer
  - jniImportBatch
```
private boolean jniImportBatch(long genomicsDBImporterHandle,
                               long[] exhaustedBufferIdentifiers)
```
    Import the next batch of data into TileDB/GenomicsDB
    
    Parameters:
    
    genomicsDBImporterHandle - "pointer" returned by jniInitializeGenomicsDBImporterObject
    
    exhaustedBufferIdentifiers - contains the list of exhausted buffer stream identifiers - the number of exhausted streams is stored in the last element of the array
    
    Returns:
    
    true if the whole import process is completed, false otherwise
  - jniGetChromosomeIntervalsForColumnPartition
```
private static String jniGetChromosomeIntervalsForColumnPartition(String loaderJSONFile,
                                                                  int rank)
```
    Obtain the chromosome intervals for the column partition specified in the loader JSON file identified by the rank. The information is returned as a string in JSON format { "contigs": [ { "chr1": [ 100, 200] }, { "chr2": [ 500, 600] } ] }
    
    Parameters:
    
    loaderJSONFile - path to loader JSON file
    
    rank - rank/partition index
    
    Returns:
    
    chromosome intervals for the queried column partition in JSON format
  - createImportConfiguration
```
private GenomicsDBImportConfiguration.ImportConfiguration createImportConfiguration(String workspace,
                                                                                    String arrayname,
                                                                                    Long sizePerColumnPartition,
                                                                                    Long segmentSize)
```
  - printLoaderJSONFile
```
public static File printLoaderJSONFile(GenomicsDBImportConfiguration.ImportConfiguration importConfiguration,
                                       String filename)
```
    Create a JSON file from the import configuration
    
    Parameters:
    
    importConfiguration - The configuration object
    
    Returns:
    
    New file (with the specified name) written to local storage
  - generateSortedCallSetMap
```
private GenomicsDBCallsetsMapProto.CallsetMappingPB generateSortedCallSetMap(Map<String,htsjdk.tribble.FeatureReader<htsjdk.variant.variantcontext.VariantContext>> variants,
                                                                             boolean useSamplesInOrderProvided)
```
    Creates a sorted list of callsets and generates unique TileDB row indices for them. Sorted to maintain order between distributed share-nothing load processes. Assume one sample per input GVCF file
    
    Parameters:
    
    variants - Variant Readers objects of the input GVCF files
    
    Returns:
    
    Mappings of callset (sample) names to TileDB rows
  - initialize
```
private void initialize(String loaderJSONFile,
                        int rank,
                        long lbRowIdx,
                        long ubRowIdx)
```
    Initialize variables
    
    Parameters:
    
    loaderJSONFile - GenomicsDB loader JSON configuration file
    
    rank - Rank of this process (TileDB/GenomicsDB partition idx)
    
    lbRowIdx - Smallest row idx which should be imported by this object
    
    ubRowIdx - Largest row idx which should be imported by this object
  - generateVidMapFromMergedHeader
```
private GenomicsDBVidMapProto.VidMappingPB generateVidMapFromMergedHeader(Set<htsjdk.variant.vcf.VCFHeaderLine> mergedHeader)
```
    Generate the ProtoBuf data structure for vid mapping Also remember, contigs are 1-based which means TileDB column offsets should start from 1
    
    Parameters:
    
    mergedHeader - Header from all input GVCFs are merged to create one vid map for all
    
    Returns:
    
    a vid map containing all field names, lengths and types from the merged GVCF header
  - remove
```
private GenomicsDBVidMapProto.InfoField remove(List<GenomicsDBVidMapProto.InfoField> infoFields,
                                               int dpIndex)
```
  - getLength
```
private String getLength(htsjdk.variant.vcf.VCFHeaderLine headerLine)
```
    Maps the "Number" from INFO or FORMAT fields in VCF header to GenomicsDB lengths. If unbounded, the field becomes a variable length attribute (discussed in more detail in TileDB tutorial tiledb.org). If "A", "R" or "G", the field also becomes a variable length attribute, but the order and length depends on order/number of alleles or genotypes. Integers remain integers
    
    Parameters:
    
    headerLine - Info or Format header line from VCF
    
    Returns:
    
    VAR, A, R, G, or integer values from VCF
  - initializeSampleInfoMapFromHeader
```
public static long initializeSampleInfoMapFromHeader(Map<Integer,SampleInfo> sampleIndexToInfo,
                                                     htsjdk.variant.vcf.VCFHeader vcfHeader,
                                                     long rowIdx)
```
    Static function that reads sample names from the vcfHeader and adds entries to the map. The function assumes that the samples will be assigned row indexes beginning at rowIdx and that the sample names specified in the header are globally unique (across all streams/files)
    
    Parameters:
    
    sampleIndexToInfo - map: key=sampleIndex in vcfHeader: value=SampleInfo
    
    vcfHeader - VCF header
    
    rowIdx - Starting row index from which to assign
    
    Returns:
    
    rowIdx+#samples in the header
  - addBufferStream
```
public int addBufferStream(String streamName,
                           htsjdk.variant.vcf.VCFHeader vcfHeader,
                           long bufferCapacity,
                           htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType,
                           Map<Integer,SampleInfo> sampleIndexToInfo)
                    throws GenomicsDBException
```
    Add a buffer stream as the data source - caller must: 1. Call setupGenomicsDBImporter() once all streams are added 2. Provide VC objects using the add() function 3. Call importBatch() 4. Get list of exhausted buffer streams using getNumExhaustedBufferStreams(), getExhaustedBufferStreamIndex() 5. If !isDone() goto 2
    
    Parameters:
    
    streamName - Name of the stream being added - must be unique with respect to this GenomicsDBImporter object
    
    vcfHeader - VCF header for the stream
    
    bufferCapacity - Capacity of the stream buffer in bytes
    
    streamType - BCF_STREAM or VCF_STREAM
    
    sampleIndexToInfo - map from sample index in the vcfHeader to SampleInfo object which contains row index and globally unique name can be set to null, which implies that the mapping is stored in a callsets JSON file
    
    Returns:
    
    returns buffer stream index
    
    Throws:
    
    GenomicsDBException
  - addSortedVariantContextIterator
```
public int addSortedVariantContextIterator(String streamName,
                                           htsjdk.variant.vcf.VCFHeader vcfHeader,
                                           Iterator<htsjdk.variant.variantcontext.VariantContext> vcIterator,
                                           long bufferCapacity,
                                           htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType,
                                           Map<Integer,SampleInfo> sampleIndexToInfo)
                                    throws GenomicsDBException
```
    Add a sorted VC iterator as the data source - caller must: 1. Call setupGenomicsDBImporter() once all iterators are added 2. Call importBatch() 3. Done!
    
    Parameters:
    
    streamName - Name of the stream being added - must be unique with respect to this GenomicsDBImporter object
    
    vcfHeader - VCF header for the stream
    
    vcIterator - Iterator over VariantContext objects
    
    bufferCapacity - Capacity of the stream buffer in bytes
    
    streamType - BCF_STREAM or VCF_STREAM
    
    sampleIndexToInfo - map from sample index in the vcfHeader to SampleInfo object which contains row index and globally unique name can be set to null, which implies that the mapping is stored in a callsets JSON file
    
    Returns:
    
    returns the stream index
    
    Throws:
    
    GenomicsDBException
  - setSortedVariantContextIterator
```
public int setSortedVariantContextIterator(String streamName,
                                           htsjdk.variant.vcf.VCFHeader vcfHeader,
                                           Iterator<htsjdk.variant.variantcontext.VariantContext> vcIterator,
                                           long bufferCapacity,
                                           htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType,
                                           Map<Integer,SampleInfo> sampleIndexToInfo)
                                    throws GenomicsDBException,
                                           IOException
```
    Sets sorted VC iterator as the data source and calls setupGenomicsDBImporter(). No more streams/iterators can be added after this function is called. The caller must: 1. Call importBatch() 2. Done!
    
    Parameters:
    
    streamName - Name of the stream being added - must be unique with respect to this GenomicsDBImporter object
    
    vcfHeader - VCF header for the stream
    
    vcIterator - Iterator over VariantContext objects
    
    bufferCapacity - Capacity of the stream buffer in bytes
    
    streamType - BCF_STREAM or VCF_STREAM
    
    sampleIndexToInfo - map from sample index in the vcfHeader to SampleInfo object which contains row index and globally unique name can be set to null, which implies that the mapping is stored in a callsets JSON file
    
    Returns:
    
    returns the stream index
    
    Throws:
    
    GenomicsDBException - thrown if incorrect iterator or missing JSON configuration
    
    IOException - thrown if incorrect iterator or missing JSON configuration files
  - addBufferStream
```
private int addBufferStream(String streamName,
                            htsjdk.variant.vcf.VCFHeader vcfHeader,
                            long bufferCapacity,
                            htsjdk.variant.variantcontext.writer.VariantContextWriterBuilder.OutputType streamType,
                            Iterator<htsjdk.variant.variantcontext.VariantContext> vcIterator,
                            Map<Integer,SampleInfo> sampleIndexToInfo)
                     throws GenomicsDBException
```
    Add a buffer stream or VC iterator - internal function
    
    Parameters:
    
    streamName - Name of the stream being added - must be unique with respect to this GenomicsDBImporter object
    
    vcfHeader - VCF header for the stream
    
    bufferCapacity - Capacity of the stream buffer in bytes
    
    streamType - BCF_STREAM or VCF_STREAM
    
    vcIterator - Iterator over VariantContext objects - can be null
    
    sampleIndexToInfo - map from sample index in the vcfHeader to SampleInfo object which contains row index and globally unique name can be set to null, which implies that the mapping is stored in a callsets JSON file
    
    Returns:
    
    returns the stream index
    
    Throws:
    
    GenomicsDBException
  - setupGenomicsDBImporter
```
public void setupGenomicsDBImporter()
                             throws IOException
```
    Setup the importer after all the buffer streams are added, but before any data is inserted into any stream No more buffer streams can be added once setupGenomicsDBImporter() is called
    
    Throws:
    
    IOException - throws IOException if modified callsets JSON cannot be written
  - add
```
public boolean add(htsjdk.variant.variantcontext.VariantContext vc,
                   int streamIdx)
            throws GenomicsDBException,
                   htsjdk.samtools.util.RuntimeIOException
```
    Write VariantContext object to stream - may fail if the buffer is full It's the caller's responsibility keep track of the VC object that's not written
    
    Parameters:
    
    vc - VariantContext object
    
    streamIdx - index of the stream returned by the addBufferStream() call
    
    Returns:
    
    true if the vc object was written successfully, false otherwise
    
    Throws:
    
    GenomicsDBException
    
    htsjdk.samtools.util.RuntimeIOException
  - importBatch
```
public boolean importBatch()
                    throws IOException
```
    Returns:
    
    true if the import process is done
    
    Throws:
    
    IOException - if the wimport fails
  - getNumExhaustedBufferStreams
```
public long getNumExhaustedBufferStreams()
```
    Returns:
    
    get number of buffer streams for which new data must be supplied
  - getExhaustedBufferStreamIndex
```
public int getExhaustedBufferStreamIndex(long i)
```
    Get buffer stream index of i-th exhausted stream There are mNumExhaustedBufferStreams and the caller must provide data for streams with indexes getExhaustedBufferStreamIndex(0), getExhaustedBufferStreamIndex(1),..., getExhaustedBufferStreamIndex(mNumExhaustedBufferStreams-1)
    
    Parameters:
    
    i - i-th exhausted buffer stream
    
    Returns:
    
    the buffer stream index of the i-th exhausted stream
  - isDone
```
public boolean isDone()
```
    Is the import process completed
    
    Returns:
    
    true if complete, false otherwise
  - getChromosomeIntervalsForColumnPartition
```
private static ArrayList<ChromosomeInterval> getChromosomeIntervalsForColumnPartition(String loaderJSONFile,
                                                                                      int partitionIdx)
                                                                               throws org.json.simple.parser.ParseException
```
    Utility function that returns a list of ChromosomeInterval objects for the column partition specified by the loader JSON file and rank/partition index
    
    Parameters:
    
    loaderJSONFile - path to loader JSON file
    
    partitionIdx - rank/partition index
    
    Returns:
    
    list of ChromosomeInterval objects for the specified partition
    
    Throws:
    
    org.json.simple.parser.ParseException - when there is a bug in the JNI interface and a faulty JSON is returned
  - columnPartitionIterator
```
public static <SOURCE> MultiChromosomeIterator<SOURCE> columnPartitionIterator(htsjdk.tribble.AbstractFeatureReader<htsjdk.variant.variantcontext.VariantContext,SOURCE> reader,
                                                                               String loaderJSONFile,
                                                                               int partitionIdx)
                                                                        throws org.json.simple.parser.ParseException,
                                                                               IOException
```
    Utility function that returns a MultiChromosomeIterator given an AbstractFeatureReader that will iterate over the VariantContext objects provided by the reader belonging to the column partition specified by the loader JSON file and rank/partition index
    
    Type Parameters:
    
    SOURCE - LineIterator for VCFs, PositionalBufferedStream for BCFs
    
    Parameters:
    
    reader - AbstractFeatureReader over VariantContext objects - SOURCE can vary - BCF v/s VCF for example
    
    loaderJSONFile - path to loader JSON file
    
    partitionIdx - rank/partition index
    
    Returns:
    
    MultiChromosomeIterator that iterates over VariantContext objects in the reader belonging to the specified column partition
    
    Throws:
    
    IOException - when the reader's query method throws an IOException
    
    org.json.simple.parser.ParseException - when there is a bug in the JNI interface and a faulty JSON is returned
  - columnPartitionIterator
```
public <SOURCE> MultiChromosomeIterator<SOURCE> columnPartitionIterator(htsjdk.tribble.AbstractFeatureReader<htsjdk.variant.variantcontext.VariantContext,SOURCE> reader)
                                                                 throws org.json.simple.parser.ParseException,
                                                                        IOException
```
    Utility function that returns a MultiChromosomeIterator given an AbstractFeatureReader that will iterate over the VariantContext objects provided by the reader belonging to the column partition specified by this object's loader JSON file and rank/partition index
    
    Type Parameters:
    
    SOURCE - LineIterator for VCFs, PositionalBufferedStream for BCFs
    
    Parameters:
    
    reader - AbstractFeatureReader over VariantContext objects - SOURCE can vary - BCF v/s VCF for example
    
    Returns:
    
    MultiChromosomeIterator that iterates over VariantContext objects in the reader belonging to the specified column partition
    
    Throws:
    
    IOException - when the reader's query method throws an IOException
    
    org.json.simple.parser.ParseException - when there is a bug in the JNI interface and a faulty JSON is returned
  - write
```
public void write()
           throws GenomicsDBException
```
    Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
    
    Throws:
    
    GenomicsDBException
  - write
```
public void write(long lbRowIdx)
           throws GenomicsDBException
```
    Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
    
    Parameters:
    
    lbRowIdx - Minimum row idx from which new data will be added
    
    Throws:
    
    GenomicsDBException
  - write
```
public void write(int rank,
                  long lbRowIdx)
           throws GenomicsDBException
```
    Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
    
    Parameters:
    
    rank - Rank of this process (TileDB/GenomicsDB partition idx)
    
    lbRowIdx - Minimum row idx from which new data will be added
    
    Throws:
    
    GenomicsDBException
  - write
```
public void write(int rank,
                  long lbRowIdx,
                  long ubRowIdx)
           throws GenomicsDBException
```
    Write to TileDB/GenomicsDB using the configuration specified in the loader file passed to constructor
    
    Parameters:
    
    rank - Rank of this process (TileDB/GenomicsDB partition idx)
    
    lbRowIdx - Minimum row idx from which new data will be added
    
    ubRowIdx - Maximum row idx upto which new data will be added
    
    Throws:
    
    GenomicsDBException
  - write
```
public void write(String loaderJSONFile,
                  int rank,
                  long lbRowIdx,
                  long ubRowIdx)
           throws GenomicsDBException
```
    Write to TileDB/GenomicsDB
    
    Parameters:
    
    loaderJSONFile - GenomicsDB loader JSON configuration file
    
    rank - Rank of this process (TileDB/GenomicsDB partition idx)
    
    lbRowIdx - Minimum row idx from which new data will be added
    
    ubRowIdx - Maximum row idx upto which new data will be added
    
    Throws:
    
    GenomicsDBException

Class GenomicsDBImporter

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

mDefaultBufferCapacity

mTempLoaderJSONFileName

DEFAULT_ARRAYNAME

DEFAULT_SIZE_PER_COLUMN_PARTITION

DEFAULT_TILEDB_CELLS_PER_TILE

mLoaderJSONFile

mRank

mLbRowIdx

mUbRowIdx

mContainsBufferStreams

mGenomicsDBImporterObjectHandle

mBufferStreamWrapperVector

mIsLoaderSetupDone

mExhaustedBufferStreamIdentifiers

mNumExhaustedBufferStreams

mDone

mCallsetMappingJSON

mUsingVidMappingProtoBuf

mVidMap

mChromosomeInterval

mCallsetMap

Constructor Detail

GenomicsDBImporter

GenomicsDBImporter

GenomicsDBImporter

GenomicsDBImporter

GenomicsDBImporter

GenomicsDBImporter

GenomicsDBImporter

Method Detail

jniGenomicsDBImporter

jniInitializeGenomicsDBImporterObject

jniCopyVidMap

jniCopyCallsetMap

jniAddBufferStream

jniSetupGenomicsDBLoader

jniWriteDataToBufferStream

jniImportBatch

jniGetChromosomeIntervalsForColumnPartition

createImportConfiguration

printLoaderJSONFile

generateSortedCallSetMap

initialize

generateVidMapFromMergedHeader

remove

getLength

initializeSampleInfoMapFromHeader

addBufferStream

addSortedVariantContextIterator

setSortedVariantContextIterator

addBufferStream

setupGenomicsDBImporter

add

importBatch

getNumExhaustedBufferStreams

getExhaustedBufferStreamIndex

isDone

getChromosomeIntervalsForColumnPartition

columnPartitionIterator

columnPartitionIterator

write

write

write

write

write