DateTimeUtils

object DateTimeUtils

Helper functions for converting between internal and external date and time representations. Dates are exposed externally as java.sql.Date and are represented internally as the number of dates since the Unix epoch (1970-01-01). Timestamps are exposed externally as java.sql.Timestamp and are stored internally as longs, which are capable of storing timestamps with microsecond precision.

Linear Supertypes

AnyRef, Any

Ordering

Alphabetic
By Inheritance

Inherited

DateTimeUtils
AnyRef
Any

Hide All
Show All

Visibility

Public
Protected

Value Members

final def !=(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final def ##: Int
Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final val JULIAN_DAY_OF_EPOCH: Int(2440588)
val TIMEZONE_OPTION: String
final val TimeZoneUTC: TimeZone
final def asInstanceOf[T0]: T0
Definition Classes
Any
def cleanLegacyTimestampStr(s: UTF8String): UTF8String
def clone(): AnyRef
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.CloneNotSupportedException]) @native()
def convertSpecialDate(input: String, zoneId: ZoneId): Option[Int]
Converts notational shorthands that are converted to ordinary dates.
Converts notational shorthands that are converted to ordinary dates.
input
A string to parse. It can contain trailing or leading whitespaces.
zoneId
Zone identifier used to get the current date.
returns
Some of days since the epoch if the conversion completed successfully otherwise None.
def convertSpecialTimestamp(input: String, zoneId: ZoneId): Option[Long]
Converts notational shorthands that are converted to ordinary timestamps.
Converts notational shorthands that are converted to ordinary timestamps.
input
A string to parse. It can contain trailing or leading whitespaces.
zoneId
Zone identifier used to get the current timestamp.
returns
Some of microseconds since the epoch if the conversion completed successfully otherwise None.
def convertSpecialTimestampNTZ(input: String, zoneId: ZoneId): Option[Long]
Converts notational shorthands that are converted to ordinary timestamps without time zone.
Converts notational shorthands that are converted to ordinary timestamps without time zone.
input
A string to parse. It can contain trailing or leading whitespaces.
zoneId
Zone identifier used to get the current local timestamp.
returns
Some of microseconds since the epoch if the conversion completed successfully otherwise None.
def convertTz(micros: Long, fromZone: ZoneId, toZone: ZoneId): Long
Converts the timestamp micros from one timezone to another.
Converts the timestamp micros from one timezone to another.
Time-zone rules, such as daylight savings, mean that not every local date-time is valid for the toZone time zone, thus the local date-time may be adjusted.
def currentDate(zoneId: ZoneId): Int
Obtains the current date as days since the epoch in the specified time-zone.
def currentTimestamp(): Long
Obtains the current instant as microseconds since the epoch at the UTC time zone.
def dateAddInterval(start: Int, interval: CalendarInterval): Int
Adds the interval's months and days to a date expressed as days since the epoch.
Adds the interval's months and days to a date expressed as days since the epoch.
returns
A date value, expressed in days since 1970-01-01.
Exceptions thrown
DateTimeException if the result exceeds the supported date range
IllegalArgumentException if the interval has microseconds part
def dateAddMonths(days: Int, months: Int): Int
Adds an year-month interval to a date represented as days since 1970-01-01.
Adds an year-month interval to a date represented as days since 1970-01-01.
returns
a date value, expressed in days since 1970-01-01.
def daysToLocalDate(days: Int): LocalDate
Obtains an instance of java.time.LocalDate from the epoch day count.
def daysToMicros(days: Int, zoneId: ZoneId): Long
Converts days since 1970-01-01 at the given zone ID to microseconds since 1970-01-01 00:00:00Z.
final def eq(arg0: AnyRef): Boolean
Definition Classes
AnyRef
def equals(arg0: AnyRef): Boolean
Definition Classes
AnyRef → Any
def finalize(): Unit
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.Throwable])
def fromJavaDate(date: Date): Int
Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part.
Converts a local date at the default JVM time zone to the number of days since 1970-01-01 in the hybrid calendar (Julian + Gregorian) by discarding the time part. The resulted days are rebased from the hybrid to Proleptic Gregorian calendar. The days rebasing is performed via UTC time zone for simplicity because the difference between two calendars is the same in any given time zone and UTC time zone.
Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions. The goal of the shift is to get a local date derived from the number of days that has the same date fields (year, month, day) as the original date at the default JVM time zone.
date
It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.
returns
The number of days since the epoch in Proleptic Gregorian calendar.
def fromJavaTimestamp(t: Timestamp): Long
Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z.
Converts an instance of java.sql.Timestamp to the number of microseconds since 1970-01-01T00:00:00.000000Z. It extracts date-time fields from the input, builds a local timestamp in Proleptic Gregorian calendar from the fields, and binds the timestamp to the system time zone. The resulted instant is converted to microseconds since the epoch.
The conversion is performed via the system time zone because it is used internally in java.sql.Timestamp while extracting date-time fields.
The goal of the function is to have the same local date-time in the original calendar - the hybrid calendar (Julian + Gregorian) and in the target calendar which is Proleptic Gregorian calendar, see SPARK-26651.
t
It represents a specific instant in time based on the hybrid calendar which combines Julian and Gregorian calendars.
returns
The number of micros since epoch from java.sql.Timestamp.
def fromJulianDay(days: Int, nanos: Long): Long
Returns the number of microseconds since epoch from Julian day and nanoseconds in a day.
def fromUTCTime(micros: Long, timeZone: String): Long
Returns a timestamp of given timezone from UTC timestamp, with the same string representation in their timezone.
final def getClass(): Class[_ <: AnyRef]
Definition Classes
AnyRef → Any
Annotations
@native()
def getDayInYear(days: Int): Int
Returns the 'day in year' value for the given number of days since 1970-01-01.
def getDayOfMonth(days: Int): Int
Returns the 'day of month' value for the given number of days since 1970-01-01.
def getDayOfWeek(days: Int): Int
Returns the day of the week for the given number of days since 1970-01-01 (1 = Sunday, 2 = Monday, ..., 7 = Saturday).
def getDayOfWeekFromString(string: UTF8String): Int
Returns day of week from String.
Returns day of week from String. Starting from Thursday, marked as 0. (Because 1970-01-01 is Thursday).
Exceptions thrown
IllegalArgumentException if the input is not a valid day of week.
def getHours(micros: Long, zoneId: ZoneId): Int
Returns the hour value of a given timestamp value.
Returns the hour value of a given timestamp value. The timestamp is expressed in microseconds.
def getLastDayOfMonth(days: Int): Int
Returns last day of the month for the given number of days since 1970-01-01.
def getMicroseconds(micros: Long, zoneId: ZoneId): Int
Returns local seconds, including fractional parts, multiplied by 1000000.
Returns local seconds, including fractional parts, multiplied by 1000000.
micros
The number of microseconds since the epoch.
zoneId
The time zone id which milliseconds should be obtained in.
def getMinutes(micros: Long, zoneId: ZoneId): Int
Returns the minute value of a given timestamp value.
Returns the minute value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.
def getMonth(days: Int): Int
Returns the month value for the given number of days since 1970-01-01.
Returns the month value for the given number of days since 1970-01-01. January is month 1.
def getNextDateForDayOfWeek(startDay: Int, dayOfWeek: Int): Int
Returns the first date which is later than startDate and is of the given dayOfWeek.
Returns the first date which is later than startDate and is of the given dayOfWeek. dayOfWeek is an integer ranges in [0, 6], and 0 is Thu, 1 is Fri, etc,.
def getQuarter(days: Int): Int
Returns the quarter for the given number of days since 1970-01-01.
def getSeconds(micros: Long, zoneId: ZoneId): Int
Returns the second value of a given timestamp value.
Returns the second value of a given timestamp value. The timestamp is expressed in microseconds since the epoch.
def getSecondsWithFraction(micros: Long, zoneId: ZoneId): Decimal
Returns the seconds part and its fractional part with microseconds.
def getTimeZone(timeZoneId: String): TimeZone
def getWeekBasedYear(days: Int): Int
Returns the year which conforms to ISO 8601.
Returns the year which conforms to ISO 8601. Each ISO 8601 week-numbering year begins with the Monday of the week containing the 4th of January.
def getWeekDay(days: Int): Int
Returns the day of the week for the given number of days since 1970-01-01 (0 = Monday, 1 = Tuesday, ..., 6 = Sunday).
def getWeekOfYear(days: Int): Int
Returns the week of the year of the given date expressed as the number of days from 1970-01-01.
Returns the week of the year of the given date expressed as the number of days from 1970-01-01. A week is considered to start on a Monday and week 1 is the first week with > 3 days.
def getYear(days: Int): Int
Returns the year value for the given number of days since 1970-01-01.
def getZoneId(timeZoneId: String): ZoneId
def hashCode(): Int
Definition Classes
AnyRef → Any
Annotations
@native()
def instantToMicros(instant: Instant): Long
Gets the number of microseconds since the epoch of 1970-01-01 00:00:00Z from the given instance of java.time.Instant.
Gets the number of microseconds since the epoch of 1970-01-01 00:00:00Z from the given instance of java.time.Instant. The epoch microsecond count is a simple incrementing count of microseconds where microsecond 0 is 1970-01-01 00:00:00Z.
final def isInstanceOf[T0]: Boolean
Definition Classes
Any
def localDateTimeToMicros(localDateTime: LocalDateTime): Long
def localDateToDays(localDate: LocalDate): Int
Converts the local date to the number of days since 1970-01-01.
def microsToDays(micros: Long, zoneId: ZoneId): Int
Converts microseconds since 1970-01-01 00:00:00Z to days since 1970-01-01 at the given zone ID.
def microsToInstant(micros: Long): Instant
Obtains an instance of java.time.Instant using microseconds from the epoch of 1970-01-01 00:00:00Z.
def microsToLocalDateTime(micros: Long): LocalDateTime
def microsToMillis(micros: Long): Long
Converts the timestamp to milliseconds since epoch.
Converts the timestamp to milliseconds since epoch. In Spark timestamp values have microseconds precision, so this conversion is lossy.
def millisToMicros(millis: Long): Long
Converts milliseconds since the epoch to microseconds.
def monthsBetween(micros1: Long, micros2: Long, roundOff: Boolean, zoneId: ZoneId): Double
Returns number of months between micros1 and micros2.
Returns number of months between micros1 and micros2. micros1 and micros2 are expressed in microseconds since 1970-01-01. If micros1 is later than micros2, the result is positive.
If micros1 and micros2 are on the same day of month, or both are the last day of month, returns, time of day will be ignored.
Otherwise, the difference is calculated based on 31 days per month. The result is rounded to 8 decimal places if roundOff is set to true.
final def ne(arg0: AnyRef): Boolean
Definition Classes
AnyRef
final def notify(): Unit
Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit
Definition Classes
AnyRef
Annotations
@native()
def parseTimestampString(s: UTF8String): (Array[Int], Option[ZoneId], Boolean)
Trims and parses a given UTF8 timestamp string to the corresponding timestamp segments, time zone id and whether it is just time without a date.
Trims and parses a given UTF8 timestamp string to the corresponding timestamp segments, time zone id and whether it is just time without a date. value. The return type is Option in order to distinguish between 0L and null. The following formats are allowed:
[+-]yyyy* [+-]yyyy*-[m]m [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] [+-]yyyy*-[m]m-[d]dT[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] [h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id] T[h]h:[m]m:[s]s.[ms][ms][ms][us][us][us][zone_id]
where zone_id should have one of the forms:
- Z - Zulu time zone UTC+0
- +|-[h]h:[m]m
- A short id, see https://docs.oracle.com/javase/8/docs/api/java/time/ZoneId.html#SHORT_IDS
- An id with one of the prefixes UTC+, UTC-, GMT+, GMT-, UT+ or UT-, and a suffix in the formats:
  - +|-h[h]
  - +|-hh[:]mm
  - +|-hh:mm:ss
  - +|-hhmmss
- Region-based zone IDs in the form area/city, such as Europe/Paris
returns
timestamp segments, time zone id and whether the input is just time without a date. If the input string can't be parsed as timestamp, the result timestamp segments are empty.
def parseTruncLevel(format: UTF8String): Int
Returns the truncate level, could be from TRUNC_TO_MICROSECOND to TRUNC_TO_YEAR, or TRUNC_INVALID, TRUNC_INVALID means unsupported truncate level.
def stringToDate(s: UTF8String): Option[Int]
Trims and parses a given UTF8 date string to a corresponding Int value.
Trims and parses a given UTF8 date string to a corresponding Int value. The return type is Option in order to distinguish between 0 and null. The following formats are allowed:
[+-]yyyy* [+-]yyyy*-[m]m [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d [+-]yyyy*-[m]m-[d]d * [+-]yyyy*-[m]m-[d]dT*
def stringToDateAnsi(s: UTF8String): Int
def stringToTimestamp(s: UTF8String, timeZoneId: ZoneId): Option[Long]
Trims and parses a given UTF8 timestamp string to the corresponding a corresponding Long value.
Trims and parses a given UTF8 timestamp string to the corresponding a corresponding Long value. The return type is Option in order to distinguish between 0L and null. Please refer to parseTimestampString for the allowed formats
def stringToTimestampAnsi(s: UTF8String, timeZoneId: ZoneId): Long
def stringToTimestampWithoutTimeZone(s: UTF8String): Option[Long]
Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch.
Trims and parses a given UTF8 string to a corresponding Long value which representing the number of microseconds since the epoch. The result is independent of time zones, which means that zone ID in the input string will be ignored. The return type is Option in order to distinguish between 0L and null. Please refer to parseTimestampString for the allowed formats.
def stringToTimestampWithoutTimeZoneAnsi(s: UTF8String): Long
def subtractDates(endDay: Int, startDay: Int): CalendarInterval
Subtracts two dates expressed as days since 1970-01-01.
Subtracts two dates expressed as days since 1970-01-01.
endDay
The end date, exclusive
startDay
The start date, inclusive
returns
An interval between two dates. The interval can be negative if the end date is before the start date.
def subtractTimestamps(endMicros: Long, startMicros: Long, zoneId: ZoneId): Long
Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.
Subtracts two timestamps expressed as microseconds since 1970-01-01 00:00:00Z, and returns the difference in microseconds between local timestamps at the given time zone.
endMicros
The end timestamp as microseconds since the epoch, exclusive
startMicros
The end timestamp as microseconds since the epoch, inclusive
zoneId
The time zone ID in which the subtraction is performed
returns
The difference in microseconds between local timestamps corresponded to the input instants end and start.
final def synchronized[T0](arg0: => T0): T0
Definition Classes
AnyRef
def timestampAddDayTime(micros: Long, dayTime: Long, zoneId: ZoneId): Long
Adds a day-time interval expressed in microseconds to a timestamp at the given time zone.
Adds a day-time interval expressed in microseconds to a timestamp at the given time zone. It converts the input timestamp to a local timestamp, and adds the interval by:
- Splitting the interval to days and microsecond adjustment in a day, and
- First of all, it adds days and then the time part. The resulted local timestamp is converted back to an instant at the given time zone.
micros
The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
dayTime
The amount of microseconds to add. It can be positive or negative.
zoneId
The time zone ID at which the operation is performed.
returns
A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
def timestampAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long
Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.
Adds a full interval (months, days, microseconds) to a timestamp represented as the number of microseconds since 1970-01-01 00:00:00Z.
returns
A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
def timestampAddMonths(micros: Long, months: Int, zoneId: ZoneId): Long
Adds months to a timestamp at the given time zone.
Adds months to a timestamp at the given time zone. It converts the input timestamp to a local timestamp at the given time zone, adds months, and converts the resulted local timestamp back to a timestamp, expressed in microseconds since 1970-01-01 00:00:00Z.
micros
The input timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z
months
The amount of months to add. It can be positive or negative.
zoneId
The time zone ID at which the operation is performed.
returns
A timestamp value, expressed in microseconds since 1970-01-01 00:00:00Z.
def timestampNTZAddInterval(start: Long, months: Int, days: Int, microseconds: Long, zoneId: ZoneId): Long
Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.
Adds a full interval (months, days, microseconds) to a timestamp without time zone represented as a local time in microsecond precision, which is independent of time zone.
returns
A timestamp without time zone value, expressed in range [0001-01-01T00:00:00.000000, 9999-12-31T23:59:59.999999].
def toJavaDate(days: Int): Date
Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian).
Converts days since the epoch 1970-01-01 in Proleptic Gregorian calendar to a local date at the default JVM time zone in the hybrid calendar (Julian + Gregorian). It rebases the given days from Proleptic Gregorian to the hybrid calendar at UTC time zone for simplicity because the difference between two calendars doesn't depend on any time zone. The result is shifted by the time zone offset in wall clock to have the same date fields (year, month, day) at the default JVM time zone as the input daysSinceEpoch in Proleptic Gregorian calendar.
Note: The date is shifted by the offset of the default JVM time zone for backward compatibility with Spark 2.4 and earlier versions.
days
The number of days since 1970-01-01 in Proleptic Gregorian calendar.
returns
A local date in the hybrid calendar as java.sql.Date from number of days since epoch.
def toJavaTimestamp(micros: Long): Timestamp
Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.
Converts microseconds since the epoch to an instance of java.sql.Timestamp via creating a local timestamp at the system time zone in Proleptic Gregorian calendar, extracting date and time fields like year and hours, and forming new timestamp in the hybrid calendar from the extracted fields.
The conversion is based on the JVM system time zone because the java.sql.Timestamp uses the time zone internally.
The method performs the conversion via local timestamp fields to have the same date-time representation as year, month, day, ..., seconds in the original calendar and in the target calendar.
micros
The number of microseconds since 1970-01-01T00:00:00.000000Z.
returns
A java.sql.Timestamp from number of micros since epoch.
def toJulianDay(micros: Long): (Int, Long)
Returns Julian day and nanoseconds in a day from the number of microseconds
Returns Julian day and nanoseconds in a day from the number of microseconds
Note: support timestamp since 4717 BC (without negative nanoseconds, compatible with Hive).
def toString(): String
Definition Classes
AnyRef → Any
def toUTCTime(micros: Long, timeZone: String): Long
Returns a utc timestamp from a given timestamp from a given timezone, with the same string representation in their timezone.
def truncDate(days: Int, level: Int): Int
Returns the trunc date from original date and trunc level.
Returns the trunc date from original date and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 6 and 9.
def truncTimestamp(micros: Long, level: Int, zoneId: ZoneId): Long
Returns the trunc date time from original date time and trunc level.
Returns the trunc date time from original date time and trunc level. Trunc level should be generated using parseTruncLevel(), should be between 0 and 9.
final def wait(): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long, arg1: Int): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException]) @native()

Packages

DateTimeUtils

object DateTimeUtils

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

Packages

DateTimeUtils

object DateTimeUtils

Value Members

Inherited from AnyRef

Inherited from Any

Ungrouped

DateTimeUtils