Packages

class JsonFilters extends StructFilters

The class provides API for applying pushed down source filters to rows with a struct schema parsed from JSON records. The class should be used in this way: 1. Before processing of the next row, JacksonParser (parser for short) resets the internal state of JsonFilters by calling the reset() method. 2. The parser reads JSON fields one-by-one in streaming fashion. It converts an incoming field value to the desired type from the schema. After that, it sets the value to an instance of InternalRow at the position according to the schema. Order of parsed JSON fields can be different from the order in the schema. 3. Per every JSON field of the top-level JSON object, the parser calls skipRow by passing an InternalRow in which some of fields can be already set, and the position of the JSON field according to the schema. 3.1 skipRow finds a group of predicates that refers to this JSON field. 3.2 Per each predicate from the group, skipRow decrements its reference counter. 3.2.1 If predicate reference counter becomes 0, it means that all predicate attributes have been already set in the internal row, and the predicate can be applied to it. skipRow invokes the predicate for the row. 3.3 skipRow applies predicates until one of them returns false. In that case, the method returns true to the parser. 3.4 If all predicates with zero reference counter return true, the final result of the method is false which tells the parser to not skip the row. 4. If the parser gets true from JsonFilters.skipRow, it must not call the method anymore for this internal row, and should go the step 1.

Besides of StructFilters assumptions, JsonFilters assumes that:

  • skipRow() can be called for any valid index of the struct fields, and in any order.
  • After skipRow() returns true, the internal state of JsonFilters can be inconsistent, so, skipRow() must not be called for the current row anymore without reset().
Linear Supertypes
StructFilters, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. JsonFilters
  2. StructFilters
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new JsonFilters(pushedFilters: Seq[Filter], schema: StructType)

    pushedFilters

    The pushed down source filters. The filters should refer to the fields of the provided schema.

    schema

    The required schema of records from datasource files.

Type Members

  1. case class JsonPredicate(predicate: BasePredicate, totalRefs: Int) extends Product with Serializable

    Stateful JSON predicate that keeps track of its dependent references in the current row via refCount.

    Stateful JSON predicate that keeps track of its dependent references in the current row via refCount.

    predicate

    The predicate compiled from pushed down source filters.

    totalRefs

    The total amount of all filters references which the predicate compiled from.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. val filters: Array[Filter]
    Attributes
    protected
    Definition Classes
    StructFilters
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  10. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. def reset(): Unit

    Reset states of all predicates by re-initializing reference counters.

    Reset states of all predicates by re-initializing reference counters.

    Definition Classes
    JsonFilters → StructFilters
  17. def skipRow(row: InternalRow, index: Int): Boolean

    Applies predicates (compiled filters) associated with the row field value at the position index only if other predicates dependencies are already set in the given row.

    Applies predicates (compiled filters) associated with the row field value at the position index only if other predicates dependencies are already set in the given row.

    Note: If the function returns true, refCount of some predicates can be not decremented.

    row

    The row with fully or partially set values.

    index

    The index of already set value.

    returns

    true if at least one of applicable predicates (all dependent row values are set) return false. It returns false if all predicates return true.

    Definition Classes
    JsonFilters → StructFilters
  18. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  19. def toPredicate(filters: Seq[Filter]): BasePredicate

    Compiles source filters to a predicate.

    Compiles source filters to a predicate.

    Definition Classes
    StructFilters
  20. def toString(): String
    Definition Classes
    AnyRef → Any
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  22. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  23. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from StructFilters

Inherited from AnyRef

Inherited from Any

Ungrouped