Package com.marklogic.client.ext.file

Class GenericFileLoader

  • All Implemented Interfaces:
    FileLoader
    Direct Known Subclasses:
    AssetFileLoader, DefaultSchemasLoader
    public class GenericFileLoader
extends LoggingObject
implements FileLoader
    Generic implementation of FileLoader. Delegates to a DocumentFileReader for reading from a set of file paths, and delegates to a BatchWriter for writing to MarkLogic (where that BatchWriter could use XCC, the REST API, or the Data Movement SDK in ML9).

    The batchSize property defaults to null, which means all files are written in one call via the BatchWriter. Setting this means that the List of DocumentFile objects read from the DocumentFileReader will be written in batches, each the size of the batchSize property, except for the final one that may be less than this size.

    • Constructor Detail

      • GenericFileLoader

        public GenericFileLoader​(com.marklogic.client.DatabaseClient client)
        The given DatabaseClient is used to construct a BatchWriter that writes to MarkLogic via the REST API. The expectation is that the client will then either call setDocumentFileReader or will rely on the default one that's created by this class if one has not yet been set.
        Parameters:
        client -

      • GenericFileLoader

        public GenericFileLoader​(BatchWriter batchWriter)
        Clients should use this when they already have a BatchWriter ready to go.
        Parameters:
        batchWriter -

    • Method Detail

      • loadFiles

        public java.util.List<DocumentFile> loadFiles​(java.lang.String... paths)
        Load files from each of the given paths, using the underlying DocumentFileReader.

        If a DocumentFileReader has not been set yet, then one will be constructed before any files are read.

        Specified by:
        loadFiles in interface FileLoader
        Parameters:
        paths -
        Returns:
        list of files from the given paths

      • getDocumentFiles

        protected final java.util.List<DocumentFile> getDocumentFiles​(java.lang.String... paths)

      • writeDocumentFiles

        protected final void writeDocumentFiles​(java.util.List<DocumentFile> documentFiles)

      • writeBatchOfDocuments

        protected void writeBatchOfDocuments​(java.util.List<DocumentFile> documentFiles,
                                     int startPosition)
        If batchSize is not set, then this method will load all the documents in one call to the BatchWriter. Otherwise, this will divide up the list of documentFiles into batches matching the value of batchSize, with the last batch possibly being less than batchSize.
        Parameters:
        documentFiles -
        startPosition -

      • initializeDocumentFileReader

        public void initializeDocumentFileReader()
        If no DocumentFileReader is set, this will construct a DefaultDocumentFileReader, which is then configured based on several properties of this class.

      • prepareAbstractDocumentFileReader

        public void prepareAbstractDocumentFileReader​(AbstractDocumentFileReader reader)
        This was initially part of building a DefaultDocumentFileReader. But in the event that a client sets a custom DocumentFileReader on this class that extends AbstractDocumentFileReader, it's useful to reuse this code on that custom DocumentFileReader. Thus, it's public.
        Parameters:
        reader -

      • applyTokenReplacerOnKnownDocumentProcessors

        protected void applyTokenReplacerOnKnownDocumentProcessors​(AbstractDocumentFileReader reader)
        If this is an instance of DefaultDocumentFileReader and a TokenReplacer has been set on the instance of this class, then pass the TokenReplacer along to the known processors in the reader so that those processors can replace token occurrences in property values.
        Parameters:
        reader -

      • buildDocumentFileProcessors

        protected java.util.List<DocumentFileProcessor> buildDocumentFileProcessors()
        Returns:
        a set of DocumentFileProcessor objects based on how this class has been configured.

      • addFileFilter

        public void addFileFilter​(java.io.FileFilter fileFilter)

      • setWaitForCompletion

        public void setWaitForCompletion​(boolean waitForCompletion)

      • setDocumentFileReader

        public void setDocumentFileReader​(DocumentFileReader documentFileReader)

      • setPermissions

        public void setPermissions​(java.lang.String permissions)

      • setCollections

        public void setCollections​(java.lang.String... collections)

      • setTokenReplacer

        public void setTokenReplacer​(TokenReplacer tokenReplacer)

      • setFileFilters

        public void setFileFilters​(java.util.List<java.io.FileFilter> fileFilters)

      • setDocumentFileProcessors

        public void setDocumentFileProcessors​(java.util.List<DocumentFileProcessor> documentFileProcessors)

      • setAdditionalBinaryExtensions

        public void setAdditionalBinaryExtensions​(java.lang.String... additionalBinaryExtensions)

      • isLogFileUris

        public boolean isLogFileUris()

      • setLogFileUris

        public void setLogFileUris​(boolean logFileUris)

      • getFileFilters

        public java.util.List<java.io.FileFilter> getFileFilters()

      • setBatchSize

        public void setBatchSize​(java.lang.Integer batchSize)

      • setBatchWriter

        public void setBatchWriter​(BatchWriter batchWriter)