S3DatasourceBackfill

abstract class S3DatasourceBackfill<R : Any, P : Any> : Backfill

Functions

getBucket

abstract fun getBucket(config: PrepareBackfillConfig): String

Bucket where the S3 files are located

getPrefix

open fun getPrefix(config: PrepareBackfillConfig): String

Calculates the prefix of S3 objects to process.

runBatch

open fun runBatch(records: List<R>, config: BackfillConfig)

Called for each batch of records.

runOne

open fun runOne(record: R, config: BackfillConfig)

Called for each record.

validate

open fun validate(config: PrepareBackfillConfig)

Override this and throw an exception to prevent the backfill from being created.

recordStrategy

Produces records from the S3 file.

staticPrefix

Static element of the prefix, this is a helper used by the default getPrefix.