Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark
    Definition Classes
    apache
  • package sql
    Definition Classes
    spark
  • package catalyst

    Catalyst is a library for manipulating relational query plans.

    Catalyst is a library for manipulating relational query plans. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

    Definition Classes
    sql
  • package json
    Definition Classes
    catalyst
  • class JsonFilters extends StructFilters

    The class provides API for applying pushed down source filters to rows with a struct schema parsed from JSON records.

    The class provides API for applying pushed down source filters to rows with a struct schema parsed from JSON records. The class should be used in this way: 1. Before processing of the next row, JacksonParser (parser for short) resets the internal state of JsonFilters by calling the reset() method. 2. The parser reads JSON fields one-by-one in streaming fashion. It converts an incoming field value to the desired type from the schema. After that, it sets the value to an instance of InternalRow at the position according to the schema. Order of parsed JSON fields can be different from the order in the schema. 3. Per every JSON field of the top-level JSON object, the parser calls skipRow by passing an InternalRow in which some of fields can be already set, and the position of the JSON field according to the schema. 3.1 skipRow finds a group of predicates that refers to this JSON field. 3.2 Per each predicate from the group, skipRow decrements its reference counter. 3.2.1 If predicate reference counter becomes 0, it means that all predicate attributes have been already set in the internal row, and the predicate can be applied to it. skipRow invokes the predicate for the row. 3.3 skipRow applies predicates until one of them returns false. In that case, the method returns true to the parser. 3.4 If all predicates with zero reference counter return true, the final result of the method is false which tells the parser to not skip the row. 4. If the parser gets true from JsonFilters.skipRow, it must not call the method anymore for this internal row, and should go the step 1.

    Besides of StructFilters assumptions, JsonFilters assumes that:

    • skipRow() can be called for any valid index of the struct fields, and in any order.
    • After skipRow() returns true, the internal state of JsonFilters can be inconsistent, so, skipRow() must not be called for the current row anymore without reset().
    Definition Classes
    json
  • JsonPredicate

case class JsonPredicate(predicate: BasePredicate, totalRefs: Int) extends Product with Serializable

Stateful JSON predicate that keeps track of its dependent references in the current row via refCount.

predicate

The predicate compiled from pushed down source filters.

totalRefs

The total amount of all filters references which the predicate compiled from.

Linear Supertypes
Serializable, Serializable, Product, Equals, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. JsonPredicate
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JsonPredicate(predicate: BasePredicate, totalRefs: Int)

    predicate

    The predicate compiled from pushed down source filters.

    totalRefs

    The total amount of all filters references which the predicate compiled from.

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  8. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  9. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  10. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  12. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  13. val predicate: BasePredicate
  14. var refCount: Int
  15. def reset(): Unit
  16. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  17. val totalRefs: Int
  18. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AnyRef

Inherited from Any

Ungrouped