package accumulo
- Alphabetic
- By Inheritance
- accumulo
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
- class AccumuloAttributeStore extends DiscreteLayerAttributeStore
- class AccumuloCollectionLayerReader extends CollectionLayerReader[LayerId]
- trait AccumuloEncoder[T] extends AnyRef
- trait AccumuloInstance extends Serializable
- class AccumuloLayerCopier extends LayerCopier[LayerId]
- class AccumuloLayerDeleter extends LayerDeleter[LayerId]
- case class AccumuloLayerHeader(keyClass: String, valueClass: String, tileTable: String, layerType: LayerType = AvroLayerType) extends LayerHeader with Product with Serializable
- class AccumuloLayerManager extends LayerManager[LayerId]
-
class
AccumuloLayerProvider extends AttributeStoreProvider with LayerReaderProvider with LayerWriterProvider with ValueReaderProvider with CollectionLayerReaderProvider
Provides AccumuloAttributeStore instance for URI with
accumuloscheme.Provides AccumuloAttributeStore instance for URI with
accumuloscheme. ex:accumulo://[user[:password]@]zookeeper/instance-name[?attributes=table1[&layers=table2]]Attributes table name is optional, not provided default value will be used. Layers table name is required to instantiate a LayerWriter
- class AccumuloLayerReader extends FilteringLayerReader[LayerId]
- class AccumuloLayerReindexer extends LayerReindexer[LayerId]
- class AccumuloLayerWriter extends LayerWriter[LayerId]
- class AccumuloValueReader extends OverzoomingValueReader
- sealed trait AccumuloWriteStrategy extends AnyRef
- case class BaseAccumuloInstance(instanceName: String, zookeeper: String, user: String, tokenBytes: (String, Array[Byte])) extends AccumuloInstance with Product with Serializable
-
case class
HdfsWriteStrategy(ingestPath: Path) extends AccumuloWriteStrategy with Product with Serializable
This strategy will perfom Accumulo bulk ingest.
This strategy will perfom Accumulo bulk ingest. Bulk ingest requires that sorted records be written to the filesystem, preferbly HDFS, before Accumulo is able to ingest them. After the ingest is finished the nodes will likely go through a period of high load as they perform major compactions.
Note: Giving relative URLs will cause HDFS to use the
fs.defaultFSproperty incore-site.xml. If not specified this will default to local ('file:/') system, this is undesriable.- ingestPath
Path where spark will write RDD records for ingest
-
case class
SocketWriteStrategy(config: BatchWriterConfig = ..., threads: Int = ...) extends AccumuloWriteStrategy with Product with Serializable
This strategy will create one BatchWriter per partition and attempt to stream the records to the target tablets.
This strategy will create one BatchWriter per partition and attempt to stream the records to the target tablets. In order to gain some parallism this strategy will create a number of splits in the target table equal to the number of tservers in the cluster. This is suitable for smaller ingests, or where HdfsWriteStrategy is otherwise not possible.
This strategy will not create splits before starting to write. If you wish to do that use AccumuloUtils.getSplits first.
There is a problem in Accumulo 1.6 (fixed in 1.7) where the split creation does not wait for the resulting empty tablets to distribute through the cluster before returning. This will create a warm-up period where the pressure the ingest writers on that node will delay tablet re-balancing.
The speed of the ingest can be improved by setting
tserver.wal.sync.method=hflushin accumulo shell. Note: this introduces higher chance of data loss due to sudden node failure.BatchWriter is notified of the tablet migrations and will follow them around the cluster.
- config
Configuration for the BatchWriters
- implicit class connectorWriter extends AnyRef
- implicit class scannerIterator extends Iterator[(Key, Value)]
Value Members
- def columnFamily(id: LayerId): String
- implicit def stringToText(s: String): Text
- object AccumuloAttributeStore extends Serializable
- object AccumuloCollectionLayerReader
- object AccumuloCollectionReader
- object AccumuloInstance extends Serializable
- object AccumuloKeyEncoder
- object AccumuloLayerCopier
- object AccumuloLayerDeleter
- object AccumuloLayerHeader extends Serializable
- object AccumuloLayerMover
- object AccumuloLayerReader
- object AccumuloLayerReindexer
- object AccumuloLayerWriter
- object AccumuloRDDReader
- object AccumuloRDDWriter
- object AccumuloUtils
- object AccumuloValueReader
- object AccumuloWriteStrategy
- object HdfsWriteStrategy extends Serializable