Package org.apache.pinot.spi.filesystem

Class NoClosePinotFS

    • Field Summary

      Fields 
      Modifier and Type Field Description
      protected PinotFS _delegate  

    • Constructor Summary

      Constructors 
      Modifier Constructor Description
      protected NoClosePinotFS​(PinotFS delegate)  

    • Method Summary

      Modifier and Type Method Description
      boolean copy​(URI srcUri, URI dstUri)
      Copies the file from the src to dst.
      boolean copyDir​(URI srcUri, URI dstUri)
      Copies the file or directory from the src to dst.
      void copyFromLocalDir​(File srcFile, URI dstUri)  
      void copyFromLocalFile​(File srcFile, URI dstUri)
      The src file is on the local disk.
      void copyToLocalFile​(URI srcUri, File dstFile)
      Copies a file from a remote filesystem to the local one.
      boolean delete​(URI segmentUri, boolean forceDelete)
      Deletes the file at the location provided.
      boolean exists​(URI fileUri)
      Checks whether the file or directory at the provided location exists.
      void init​(PinotConfiguration config)
      Initializes the configurations specific to that filesystem.
      boolean isDirectory​(URI uri)
      Allows us the ability to determine whether the uri is a directory.
      long lastModified​(URI uri)
      Returns the age of the file
      long length​(URI fileUri)
      Returns the length of the file at the provided location.
      String[] listFiles​(URI fileUri, boolean recursive)
      Lists all the files and directories at the location provided.
      List<FileMetadata> listFilesWithMetadata​(URI fileUri, boolean recursive)
      Like listFiles but also return some file metadata so no need to call length(), isDirectory(), and lastModified() separately, which can be slow and costly for remote file system.
      boolean mkdir​(URI uri)
      Creates a new directory.
      boolean move​(URI srcUri, URI dstUri, boolean overwrite)
      Moves the file or directory from the src to dst.
      InputStream open​(URI uri)
      Opens a file in the underlying filesystem and returns an InputStream to read it.
      boolean touch​(URI uri)
      Updates the last modified time of an existing file or directory to be current time.

      • Methods inherited from interface org.apache.pinot.spi.filesystem.PinotFS

        close

    • Field Detail

      • _delegate

        protected final PinotFS _delegate

    • Constructor Detail

      • NoClosePinotFS

        protected NoClosePinotFS​(PinotFS delegate)

    • Method Detail

      • init

        public void init​(PinotConfiguration config)
        Description copied from interface: PinotFS
        Initializes the configurations specific to that filesystem. For instance, any security related parameters can be initialized here and will not be logged.
        Specified by:
        init in interface PinotFS

      • mkdir

        public boolean mkdir​(URI uri)
              throws IOException
        Description copied from interface: PinotFS
        Creates a new directory. If parent directories are not created, it will create them. If the directory exists, it will return true without doing anything.
        Specified by:
        mkdir in interface PinotFS
        Returns:
        true if mkdir is successful
        Throws:
        IOException - on IO failure

      • delete

        public boolean delete​(URI segmentUri,
                      boolean forceDelete)
               throws IOException
        Description copied from interface: PinotFS
        Deletes the file at the location provided. If the segmentUri is a directory, it will delete the entire directory.
        Specified by:
        delete in interface PinotFS
        Parameters:
        segmentUri - URI of the segment
        forceDelete - true if we want the uri and any sub-uris to always be deleted, false if we want delete to fail when we want to delete a directory and that directory is not empty
        Returns:
        true if delete is successful else false
        Throws:
        IOException - on IO failure, e.g Uri is not present or not valid

      • move

        public boolean move​(URI srcUri,
                    URI dstUri,
                    boolean overwrite)
             throws IOException
        Description copied from interface: PinotFS
        Moves the file or directory from the src to dst. Does not keep the original file. If the dst has parent directories that haven't been created, this method will create all the necessary parent directories. Note: In Pinot we recommend the full paths of both src and dst be specified. For example, if a file /a/b/c is moved to a file /x/y/z, in the case of overwrite, the directory /a/b still exists, but will not contain the file 'c'. Instead, /x/y/z will contain the contents of 'c'. If src is a directory /a/b which contains two files /a/b/c and /a/b/d, and the dst is /x/y, the result would be that the directory /a/b under /a gets removed and dst directory contains two files which is /x/y/c and /x/y/d. If src is a directory /a/b needs to be moved under another directory /x/y, please specify the dst to /x/y/b.
        Specified by:
        move in interface PinotFS
        Parameters:
        srcUri - URI of the original file
        dstUri - URI of the final file location
        overwrite - true if we want to overwrite the dstURI, false otherwise
        Returns:
        true if move is successful
        Throws:
        IOException - on IO failure

      • copy

        public boolean copy​(URI srcUri,
                    URI dstUri)
             throws IOException
        Description copied from interface: PinotFS
        Copies the file from the src to dst. The original file is retained. If the dst has parent directories that haven't been created, this method will create all the necessary parent directories. If dst already exists, this will overwrite the existing file in the path. Note: In Pinot we recommend the full paths of both src and dst be specified. For example, if a file /a/b/c is copied to a file /x/y/z, the directory /a/b still exists containing the file 'c'. The dst file /x/y/z will contain the contents of 'c'.
        Specified by:
        copy in interface PinotFS
        Parameters:
        srcUri - URI of the original file
        dstUri - URI of the final file location
        Returns:
        true if copy is successful
        Throws:
        IOException - on IO failure

      • copyDir

        public boolean copyDir​(URI srcUri,
                       URI dstUri)
                throws IOException
        Description copied from interface: PinotFS
        Copies the file or directory from the src to dst. The original file is retained. If the dst has parent directories that haven't been created, this method will create all the necessary parent directories. If dst already exists, this will overwrite the existing file/directory in the path. Note: In Pinot we recommend the full paths of both src and dst be specified. For example, if a file /a/b/c is copied to a file /x/y/z, the directory /a/b still exists containing the file 'c'. The dst file /x/y/z will contain the contents of 'c'. If a directory /a/b is copied to another directory /x/y, the directory /x/y will contain the content of /a/b. If a directory /a/b is copied under the directory /x/y, the dst needs to be specify as /x/y/b.
        Specified by:
        copyDir in interface PinotFS
        Parameters:
        srcUri - URI of the original file
        dstUri - URI of the final file location
        Returns:
        true if copy is successful
        Throws:
        IOException - on IO failure

      • exists

        public boolean exists​(URI fileUri)
               throws IOException
        Description copied from interface: PinotFS
        Checks whether the file or directory at the provided location exists.
        Specified by:
        exists in interface PinotFS
        Parameters:
        fileUri - URI of file
        Returns:
        true if path exists
        Throws:
        IOException - on IO failure

      • length

        public long length​(URI fileUri)
            throws IOException
        Description copied from interface: PinotFS
        Returns the length of the file at the provided location.
        Specified by:
        length in interface PinotFS
        Parameters:
        fileUri - location of file
        Returns:
        the number of bytes
        Throws:
        IOException - on IO failure, e.g if it's a directory.

      • listFiles

        public String[] listFiles​(URI fileUri,
                          boolean recursive)
                   throws IOException
        Description copied from interface: PinotFS
        Lists all the files and directories at the location provided. Lists recursively if recursive is set to true. Throws IOException if this abstract pathname is not valid, or if an I/O error occurs.
        Specified by:
        listFiles in interface PinotFS
        Parameters:
        fileUri - location of file
        recursive - if we want to list files recursively
        Returns:
        an array of strings that contains file paths
        Throws:
        IOException - on IO failure. See specific implementation

      • listFilesWithMetadata

        public List<FileMetadata> listFilesWithMetadata​(URI fileUri,
                                                boolean recursive)
                                         throws IOException
        Description copied from interface: PinotFS
        Like listFiles but also return some file metadata so no need to call length(), isDirectory(), and lastModified() separately, which can be slow and costly for remote file system.
        Specified by:
        listFilesWithMetadata in interface PinotFS
        Parameters:
        fileUri - location of file
        recursive - if we want to list files recursively
        Returns:
        a list of FileMetadata that contains file path, and a few file metadata.
        Throws:
        IOException - on IO failure. See specific implementation

      • copyToLocalFile

        public void copyToLocalFile​(URI srcUri,
                            File dstFile)
                     throws Exception
        Description copied from interface: PinotFS
        Copies a file from a remote filesystem to the local one. Keeps the original file.
        Specified by:
        copyToLocalFile in interface PinotFS
        Parameters:
        srcUri - location of current file on remote filesystem (must not be a directory)
        dstFile - location of destination on local filesystem
        Throws:
        Exception - if srcUri is not valid or not present, or timeout when downloading file to local

      • copyFromLocalDir

        public void copyFromLocalDir​(File srcFile,
                             URI dstUri)
                      throws Exception
        Specified by:
        copyFromLocalDir in interface PinotFS
        Parameters:
        srcFile - location of src file on local disk (must be a directory)
        dstUri - location of dst on remote filesystem
        Throws:
        Exception - if fileUri is not valid or not present, or timeout when uploading file from local

      • copyFromLocalFile

        public void copyFromLocalFile​(File srcFile,
                              URI dstUri)
                       throws Exception
        Description copied from interface: PinotFS
        The src file is on the local disk. Add it to filesystem at the given dst name and the source is kept intact afterwards.
        Specified by:
        copyFromLocalFile in interface PinotFS
        Parameters:
        srcFile - location of src file on local disk (must not be a directory)
        dstUri - location of dst on remote filesystem
        Throws:
        Exception - if fileUri is not valid or not present, or timeout when uploading file from local

      • isDirectory

        public boolean isDirectory​(URI uri)
                    throws IOException
        Description copied from interface: PinotFS
        Allows us the ability to determine whether the uri is a directory.
        Specified by:
        isDirectory in interface PinotFS
        Parameters:
        uri - location of file or directory
        Returns:
        true if uri is a directory, false otherwise.
        Throws:
        IOException - on IO failure, e.g uri is not valid or not present

      • lastModified

        public long lastModified​(URI uri)
                  throws IOException
        Description copied from interface: PinotFS
        Returns the age of the file
        Specified by:
        lastModified in interface PinotFS
        Parameters:
        uri - location of file or directory
        Returns:
        A long value representing the time the file was last modified, measured in milliseconds since epoch (00:00:00 GMT, January 1, 1970) or 0L if the file does not exist or if an I/O error occurs
        Throws:
        IOException - if uri is not valid or not present

      • touch

        public boolean touch​(URI uri)
              throws IOException
        Description copied from interface: PinotFS
        Updates the last modified time of an existing file or directory to be current time. If the file system object does not exist, creates an empty file.
        Specified by:
        touch in interface PinotFS
        Parameters:
        uri - location of file or directory
        Throws:
        IOException - if the parent directory doesn't exist

      • open

        public InputStream open​(URI uri)
                 throws IOException
        Description copied from interface: PinotFS
        Opens a file in the underlying filesystem and returns an InputStream to read it. Note that the caller can invoke close on this inputstream. Some implementations can choose to copy the original file to local temp file and return the inputstream. In this case, the implementation it should delete the temp file when close is invoked.
        Specified by:
        open in interface PinotFS
        Parameters:
        uri - location of the file to open
        Returns:
        a new InputStream
        Throws:
        IOException - on any IO error - missing file, not a file etc