Type Members

1.  class DeltaSparkSessionExtension extends (SparkSessionExtensions) ⇒ Unit

   An extension for Spark SQL to activate Delta SQL parser to support Delta SQL grammar.

   An extension for Spark SQL to activate Delta SQL parser to support Delta SQL grammar.

   Scala example to create a SparkSession with the Delta SQL parser:

   import org.apache.spark.sql.SparkSession
   
   val spark = SparkSession
      .builder()
      .appName("...")
      .master("...")
      .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
      .getOrCreate()

   Java example to create a SparkSession with the Delta SQL parser:

   import org.apache.spark.sql.SparkSession;
   
   SparkSession spark = SparkSession
                .builder()
                .appName("...")
                .master("...")
                .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
                .getOrCreate();

   Python example to create a SparkSession with the Delta SQL parser (PySpark doesn't pick up the SQL conf "spark.sql.extensions" in Apache Spark 2.4.x, hence we need to activate it manually in 2.4.x. However, because SparkSession has been created and everything has been materialized, we need to clone a new session to trigger the initialization. See SPARK-25003):

   from pyspark.sql import SparkSession
   
   spark = SparkSession \
       .builder \
       .appName("...") \
       .master("...") \
       .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
       .getOrCreate()
   if spark.sparkContext().version < "3.":
       spark.sparkContext()._jvm.io.delta.sql.DeltaSparkSessionExtension() \
           .apply(spark._jsparkSession.extensions())
       spark = SparkSession(spark.sparkContext(), spark._jsparkSession.cloneSession())

   Since

   0.4.0