com.conveyal.gtfs.loader

Class JdbcGtfsLoader

java.lang.Object

com.conveyal.gtfs.loader.JdbcGtfsLoader

public class JdbcGtfsLoader
extends Object

This class loads CSV tables from zipped GTFS into an SQL relational database management system with a JDBC driver. By comparing the GTFS specification for a table with the headers present in the GTFS CSV, it dynamically builds up table definitions and SQL statements to interact with those tables. It retains all columns present in the GTFS, including optional columns, known extensions, and unrecognized proprietary extensions. It supports several ways of putting the data into the tables: batched prepared inserts or loading from an intermediate tab separated text file. Our previous approach involved loading GTFS CSV tables into Java objects and then using an object-relational mapping to put those objects into a database. In that case a fixed number of fields are represented. If the GTFS feed contains extra proprietary fields, they are lost immediately on import. The Java model objects must contain fields for all GTFS columns that will ever be retrieved, and memory and database space are required to represent all those fields even when they are not present in a particular feed. The same would be true of a direct-to-database approach if multiple feeds were loaded into the same tables: a "lowest common denominator" set of fields would need to be selected. However, we create one set of tables per GTFS feed loaded into the same database and namespace them with what the SQL spec calls "schemas". The structure of the CSV file and the data it contains are validated line by line during the load process. This allows us to handle error recovery ourselves, recording detailed messages about the largest possible number of errors rather than failing at the first error. This is important because of the generally long turnaround for GTFS publication, repair, and validation. If a newly submitted feed fails to import because of a missing file, we don't want to report that single error to the feed producer, only to discover and report additional errors when the repaired feed is re-submitted weeks later. The fact that existing libraries would abort a GTFS import upon encountering very common, recoverable errors was the original motivation for creating gtfs-lib. Error recovery is all the more important when bulk-loading data into database systems - the Postgres 'copy' import is particularly brittle and does not provide error messages that would help the feed producer repair their feed. The validation performed during CSV loading includes: - columns are present for all required fields - all rows have the same number of fields as there are headers - fields do not contain problematic characters - field contents can be converted to the target data types and are in range - referential integrity

Field Summary

Fields

Modifier and Type	Field and Description
static long	INSERT_BATCH_SIZE
protected ZipFile	zip

Constructor Summary

Constructors

Constructor and Description
JdbcGtfsLoader(String gtfsFilePath, DataSource dataSource)

Method Summary

Modifier and Type	Method and Description
static void	copyFromFile(Connection connection, File file, String targetTableName) Method that uses the PostgreSQL-specific copy from file command to load csv data into a table on the provided connection.
FeedLoadResult	loadTables()
static String	sanitize(String string, SQLErrorStorage errorStorage) Protect against SQL injection.
void	setValueForField(Table table, int fieldIndex, int lineNumber, Field field, String string, boolean postgresText, String[] transformedStrings) Set value for a field either as a prepared statement parameter or (if using postgres text-loading) in the transformed strings array provided.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

INSERT_BATCH_SIZE

public static final long INSERT_BATCH_SIZE

See Also:

Constant Field Values

zip

protected ZipFile zip

Constructor Detail

JdbcGtfsLoader

public JdbcGtfsLoader(String gtfsFilePath,
                      DataSource dataSource)

Method Detail

loadTables

public FeedLoadResult loadTables()

copyFromFile

public static void copyFromFile(Connection connection,
                                File file,
                                String targetTableName)
                         throws IOException,
                                SQLException

Method that uses the PostgreSQL-specific copy from file command to load csv data into a table on the provided connection. NOTE: This method does not commit the transaction or close the connection.

Throws:

IOException

SQLException

setValueForField

public void setValueForField(Table table,
                             int fieldIndex,
                             int lineNumber,
                             Field field,
                             String string,
                             boolean postgresText,
                             String[] transformedStrings)

Set value for a field either as a prepared statement parameter or (if using postgres text-loading) in the transformed strings array provided. This also handles the case where the string is empty (i.e., field is null) and when an exception is encountered while setting the field value (usually due to a bad data type), in which case the field is set to null.

sanitize

public static String sanitize(String string,
                              SQLErrorStorage errorStorage)

Protect against SQL injection. The only place we include arbitrary input in SQL is the column names of tables. Implicitly (looking at all existing table names) these should consist entirely of lowercase letters and underscores. TODO add a test including SQL injection text (quote and semicolon)