Instance Constructors

1.  new StatsCollector(dataSchema: StructType, statsSchema: StructType, parquetRebaseMode: String, ignoreMissingStats: Boolean, stringTruncateLength: Option[Int])
   dataSchema

   The data schema from table metadata, which is the logical schema with logical to physical mapping per schema field. It is used to map statsSchema to parquet metadata.

   statsSchema

   The schema of stats to be collected, statsSchema should follow the physical schema and must be generated by StatisticsCollection.

   parquetRebaseMode

   The parquet rebase mode used to parse date and timestamp.

   ignoreMissingStats

   Indicate whether to return partial result by ignoring missing stats or throw an exception.

   stringTruncateLength

   The optional max length of string stats to be truncated into. Scala Example:

   import org.apache.spark.sql.delta.stats.StatsCollector
   
   val stringTruncateLength =
     spark.sessionState.conf.getConf(DeltaSQLConf.DATA_SKIPPING_STRING_PREFIX_LENGTH)
   
   val statsCollector = StatsCollector(
     snapshot.metadata.columnMappingMode, snapshot.metadata.dataSchema, snapshot.statsSchema,
     ignoreMissingStats = false, Some(stringTruncateLength))
   
   val filesWithStats = snapshot.allFiles.map { file =>
     val path = DeltaFileOperations.absolutePath(dataPath, file.path)
     val fileSystem = path.getFileSystem(hadoopConf)
     val fileStatus = fileSystem.listStatus(path).head
   
     val footer = ParquetFileReader.readFooter(hadoopConf, fileStatus)
     val (stats, _) = statsCollector.collect(footer)
     file.copy(stats = JsonUtils.toJson(stats))
   }