object DateTimeUtils extends SparkDateTimeUtils

Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.

Linear Supertypes
SparkDateTimeUtils, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DateTimeUtils
  2. SparkDateTimeUtils
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final val JULIAN_DAY_OF_EPOCH: Int(2440588)
  5. val TIMEZONE_OPTION: String
  6. final val TimeZoneUTC: TimeZone
    Definition Classes
    SparkDateTimeUtils
  7. def anyToDays(obj: Any): Int
    Definition Classes
    SparkDateTimeUtils
  8. def anyToMicros(obj: Any): Long
    Definition Classes
    SparkDateTimeUtils
  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def cleanLegacyTimestampStr(s: UTF8String): UTF8String
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  12. def convertSpecialDate(input: String, zoneId: ZoneId): Option[Int]

    Converts notational shorthands that are converted to ordinary dates.

    Converts notational shorthands that are converted to ordinary dates.

    input

    A string to parse. It can contain trailing or leading whitespaces.

    zoneId

    Zone identifier used to get the current date.

    returns

    Some of days since the epoch if the conversion completed successfully otherwise None.

  13. def convertSpecialTimestamp(input: String, zoneId: ZoneId): Option[Long]

    Converts notational shorthands that are converted to ordinary timestamps.

    Converts notational shorthands that are converted to ordinary timestamps.

    input

    A string to parse. It can contain trailing or leading whitespaces.

    zoneId

    Zone identifier used to get the current timestamp.

    returns

    Some of microseconds since the epoch if the conversion completed successfully otherwise None.

  14. def convertSpecialTimestampNTZ(input: String, zoneId: ZoneId): Option[Long]

    Converts notational shorthands that are converted to ordinary timestamps without time zone.

    Converts notational shorthands that are converted to ordinary timestamps without time zone.

    input

    A string to parse. It can contain trailing or leading whitespaces.

    zoneId

    Zone identifier used to get the current local timestamp.

    returns

    Some of microseconds since the epoch if the conversion completed successfully otherwise None.

  15. def convertTimestampNtzToAnotherTz(sourceTz: String, targetTz: String, micros: Long): Long

    Converts a timestamp without time zone from a source to target time zone.

    Converts a timestamp without time zone from a source to target time zone.

    sourceTz

    The time zone for the input timestamp without time zone.

    targetTz

    The time zone to which the input timestamp should be converted.

    micros

    The offset in microseconds represents a local timestamp.

    returns

    The timestamp without time zone represents the same moment (physical time) as the input timestamp in the input time zone, but in the destination time zone.

  16. def convertTz(micros: Long, fromZone: ZoneId, toZone: ZoneId): Long
    Definition Classes
    SparkDateTimeUtils
  17. def currentDate(zoneId: ZoneId): Int

    Obtains the current date as days since the epoch in the specified time-zone.

  18. def currentTimestamp(): Long

    Obtains the current instant as microseconds since the epoch at the UTC time zone.

  19. def dateAddInterval(start: Int, interval: CalendarInterval): Int

    Adds the interval's months and days to a date expressed as days since the epoch.

    Adds the interval's months and days to a date expressed as days since the epoch.

    returns

    A date value, expressed in days since 1970-01-01.

    Exceptions thrown

    DateTimeException if the result exceeds the supported date range

    IllegalArgumentException if the interval has microseconds part

  20. def dateAddMonths(days: Int, months: Int): Int

    Adds an year-month interval to a date represented as days since 1970-01-01.

    Adds an year-month interval to a date represented as days since 1970-01-01.

    returns

    a date value, expressed in days since 1970-01-01.

  21. def daysToLocalDate(days: Int): LocalDate
    Definition Classes
    SparkDateTimeUtils
  22. def daysToMicros(days: Int, zoneId: ZoneId): Long
    Definition Classes
    SparkDateTimeUtils
  23. def doubleToTimestampAnsi(d: Double, context: SQLQueryContext): Long
  24. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  25. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  26. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  27. def fromJavaDate(date: Date): Int
    Definition Classes
    SparkDateTimeUtils
  28. def fromJavaTimestamp(timeZoneId: String, t: Timestamp): Long
    Definition Classes
    SparkDateTimeUtils
  29. def fromJavaTimestamp(t: Timestamp): Long
    Definition Classes
    SparkDateTimeUtils
  30. def fromJavaTimestampNoRebase(t: Timestamp): Long
    Definition Classes
    SparkDateTimeUtils
  31. def fromJulianDay(days: Int, nanos: Long): Long

    Returns the number of microseconds since epoch from Julian day and nanoseconds in a day.

  32. def fromUTCTime(micros: Long, timeZone: String): Long

    Returns a timestamp of given timezone from UTC timestamp, with the same string representation in their timezone.

  33. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  34. def getDayInYear(days: Int): Int

    Returns the 'day in year' value for the given number of days since 1970-01-01.

  35. def getDayOfMonth(days: Int): Int

    Returns the 'day of month' value for the given number of days since 1970-01-01.

  36. def getDayOfWeek(days: Int): Int

    Returns the day of the week for the given number of days since 1970-01-01 (1 = Sunday, 2 = Monday, ..., 7 = Saturday).

  37. def getDayOfWeekFromString(string: UTF8String): Int

    Returns day of week from String.

    Returns day of week from String. Starting from Thursday, marked as 0. (Because 1970-01-01 is Thursday).

    Exceptions thrown

    IllegalArgumentException if the input is not a valid day of week.

  38. def getHours(micros: Long, zoneId: ZoneId): Int

    Returns the hour value of a given timestamp value.

    Returns the hour value of a given timestamp value. The timestamp is expressed in microseconds.

  39. def getLastDayOfMonth(days: Int): Int

    Returns last day of the month for the given number of days since 1970-01-01.

  40. def getLocalDateTime(micros: Long, zoneId: ZoneId): LocalDateTime
    Attributes
    protected
    Definition Classes
    SparkDateTimeUtils
  41. def getMicroseconds(micros: Long, zoneId: ZoneId): Int

    Returns local seconds, including fractional parts, multiplied by 1000000.

    Returns local seconds, including fractional parts, multiplied by 1000000.

    micros

    The number of microseconds since the epoch.

    zoneId

    The time zone id which milliseconds should be obtained in.

  42. def getMinutes(micros: Long, zoneId: ZoneId): Int

    Returns the minute value of a given timestamp value.

    Returns the minute value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.

  43. def getMonth(days: Int): Int

    Returns the month value for the given number of days since 1970-01-01.

    Returns the month value for the given number of days since 1970-01-01. January is month 1.

  44. def getNextDateForDayOfWeek(startDay: Int, dayOfWeek: Int): Int

    Returns the first date which is later than startDate and is of the given dayOfWeek.

    Returns the first date which is later than startDate and is of the given dayOfWeek. dayOfWeek is an integer ranges in [0, 6], and 0 is Thu, 1 is Fri, etc,.

  45. def getQuarter(days: Int): Int

    Returns the quarter for the given number of days since 1970-01-01.

  46. def getSeconds(micros: Long, zoneId: ZoneId): Int

    Returns the second value of a given timestamp value.

    Returns the second value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.

  47. def getSecondsWithFraction(micros: Long, zoneId: ZoneId): Decimal

    Returns the seconds part and its fractional part with microseconds.

  48. def getTimeZone(timeZoneId: String): TimeZone
    Definition Classes
    SparkDateTimeUtils
  49. def getWeekBasedYear(days: Int): Int

    Returns the year which conforms to ISO 8601.

    Returns the year which conforms to ISO 8601. Each ISO 8601 week-numbering year begins with the Monday of the week containing the 4th of January.

  50. def getWeekDay(days: Int): Int

    Returns the day of the week for the given number of days since 1970-01-01 (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).

  51. def getWeekOfYear(days: Int): Int

    Returns the week of the year of the given date expressed as the number of days from 1970-01-01.

    Returns the week of the year of the given date expressed as the number of days from 1970-01-01. A week is considered to start on a Monday and week 1 is the first week with > 3 days.

  52. def getYear(days: Int): Int

    Returns the year value for the given number of days since 1970-01-01.

  53. def getZoneId(timeZoneId: String): ZoneId
    Definition Classes
    SparkDateTimeUtils
  54. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  55. def instantToMicros(instant: Instant): Long
    Definition Classes
    SparkDateTimeUtils
  56. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  57. def localDateTimeToMicros(localDateTime: LocalDateTime): Long
    Definition Classes
    SparkDateTimeUtils
  58. def localDateToDays(localDate: LocalDate): Int
    Definition Classes
    SparkDateTimeUtils
  59. def microsToDays(micros: Long, zoneId: ZoneId): Int
    Definition Classes
    SparkDateTimeUtils
  60. def microsToInstant(micros: Long): Instant
    Definition Classes
    SparkDateTimeUtils
  61. def microsToLocalDateTime(micros: Long): LocalDateTime
    Definition Classes
    SparkDateTimeUtils
  62. def microsToMillis(micros: Long): Long
    Definition Classes
    SparkDateTimeUtils
  63. def millisToMicros(millis: Long): Long
    Definition Classes
    SparkDateTimeUtils
  64. def monthsBetween(micros1: Long, micros2: Long, roundOff: Boolean, zoneId: ZoneId): Double

    Returns number of months between micros1 and micros2.

    Returns number of months between micros1 and micros2. micros1 and micros2 are expressed in microseconds since 1970-01-01. If micros1 is later than micros2, the result is positive.

    If micros1 and micros2 are on the same day of month, or both are the last day of month, returns, time of day will be ignored.

    Otherwise, the difference is calculated based on 31 days per month. The result is rounded to 8 decimal places if roundOff is set to true.

  65. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  66. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  67. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  68. def parseTimestampString(s: UTF8String): (Array[Int], Option[ZoneId], Boolean)
    Definition Classes
    SparkDateTimeUtils
  69. def parseTruncLevel(format: UTF8String): Int

    Returns the truncate level, could be from TRUNC_TO_MICROSECOND to TRUNC_TO_YEAR, or TRUNC_INVALID, TRUNC_INVALID means unsupported truncate level.

  70. def stringToDate(s: UTF8String): Option[Int]
    Definition Classes
    SparkDateTimeUtils
  71. def stringToDateAnsi(s: UTF8String, context: SQLQueryContext): Int
    Definition Classes
    SparkDateTimeUtils
  72. def stringToTimestamp(s: UTF8String, timeZoneId: ZoneId): Option[Long]
    Definition Classes
    SparkDateTimeUtils
  73. def stringToTimestampAnsi(s: UTF8String, timeZoneId: ZoneId, context: SQLQueryContext): Long
    Definition Classes
    SparkDateTimeUtils
  74. def stringToTimestampWithoutTimeZone(s: UTF8String): Option[Long]

    Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch.

    Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch. The result is independent of time zones. Zone id component will be ignored. The return type is Option in order to distinguish between 0L and null. Please refer to parseTimestampString for the allowed formats.

  75. def stringToTimestampWithoutTimeZone(s: UTF8String, allowTimeZone: Boolean): Option[Long]
    Definition Classes
    SparkDateTimeUtils
  76. def stringToTimestampWithoutTimeZoneAnsi(s: UTF8String, context: SQLQueryContext): Long
  77. def subtractDates(endDay: Int, startDay: Int): CalendarInterval

    Subtracts two dates expressed as days since 1970-01-01.

    Subtracts two dates expressed as days since 1970-01-01.

    endDay

    The end date, exclusive

    startDay

    The start date, inclusive

    returns

    An interval between two dates. The interval can be negative if the end date is before the start date.

  78. def subtractTimestamps(endMicros: Long, startMicros: Long, zoneId: ZoneId): Long

    Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.

    Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.

    endMicros

    The end timestamp as microseconds since the epoch, exclusive

    startMicros

    The end timestamp as microseconds since the epoch, inclusive

    zoneId

    The time zone ID in which the subtraction is performed

    returns

    The difference in microseconds between local timestamps corresponded to the input instants end and start.

  79. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  80. def timestampAdd(unit: String, quantity: Int, micros: Long, zoneId: ZoneId): Long

    Adds the specified number of units to a timestamp.

    Adds the specified number of units to a timestamp.

    unit

    A keyword that specifies the interval units to add to the input timestamp.

    quantity

    The amount of units to add. It can be positive or negative.

    micros

    The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

    zoneId

    The time zone ID at which the operation is performed.

    returns

    A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

  81. def timestampAddDayTime(micros: Long, dayTime: Long, zoneId: ZoneId): Long

    Adds a day-time interval expressed in microseconds to a timestamp at the given time zone.

    Adds a day-time interval expressed in microseconds to a timestamp at the given time zone. It converts the input timestamp to a local timestamp, and adds the interval by:

    • Splitting the interval to days and microsecond adjustment in a day, and
    • First of all, it adds days and then the time part. The resulted local timestamp is converted back to an instant at the given time zone.
    micros

    The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

    dayTime

    The amount of microseconds to add. It can be positive or negative.

    zoneId

    The time zone ID at which the operation is performed.

    returns

    A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

  82. def timestampAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long

    Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.

    Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.

    returns

    A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

  83. def timestampAddMonths(micros: Long, months: Int, zoneId: ZoneId): Long

    Adds months to a timestamp at the given time zone.

    Adds months to a timestamp at the given time zone. It converts the input timestamp to a local timestamp at the given time zone, adds months, and converts the resulted local timestamp back to a timestamp, expressed in microseconds since 1970-01-01 00:00:00Z.

    micros

    The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z

    months

    The amount of months to add. It can be positive or negative.

    zoneId

    The time zone ID at which the operation is performed.

    returns

    A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.

  84. def timestampDiff(unit: String, startTs: Long, endTs: Long, zoneId: ZoneId): Long

    Gets the difference between two timestamps.

    Gets the difference between two timestamps.

    unit

    Specifies the interval units in which to express the difference between the two timestamp parameters.

    startTs

    A timestamp which the function subtracts from endTs.

    endTs

    A timestamp from which the function subtracts startTs.

    zoneId

    The time zone ID at which the operation is performed.

    returns

    The time span between two timestamp values, in the units specified.

  85. def timestampNTZAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long

    Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.

    Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.

    returns

    A timestamp without time zone value, expressed in range [0001-01-01T00:00:00.000000, 9999-12-31T23:59:59.999999].

  86. def toJavaDate(days: Int): Date
    Definition Classes
    SparkDateTimeUtils
  87. def toJavaTimestamp(timeZoneId: String, micros: Long): Timestamp
    Definition Classes
    SparkDateTimeUtils
  88. def toJavaTimestamp(micros: Long): Timestamp
    Definition Classes
    SparkDateTimeUtils
  89. def toJavaTimestampNoRebase(micros: Long): Timestamp
    Definition Classes
    SparkDateTimeUtils
  90. def toJulianDay(micros: Long): (Int, Long)

    Returns Julian day and nanoseconds in a day from the number of microseconds

    Returns Julian day and nanoseconds in a day from the number of microseconds

    Note: support timestamp since 4717 BC (without negative nanoseconds, compatible with Hive).

  91. def toString(): String
    Definition Classes
    AnyRef → Any
  92. def toUTCTime(micros: Long, timeZone: String): Long

    Returns a utc timestamp from a given timestamp from a given timezone, with the same string representation in their timezone.

  93. def truncDate(days: Int, level: Int): Int

    Returns the trunc date from original date and trunc level.

    Returns the trunc date from original date and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 6 and 9.

  94. def truncTimestamp(micros: Long, level: Int, zoneId: ZoneId): Long

    Returns the trunc date time from original date time and trunc level.

    Returns the trunc date time from original date time and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 0 and 9.

  95. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  96. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  97. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from SparkDateTimeUtils

Inherited from AnyRef

Inherited from Any

Ungrouped