Package org.apache.pinot.spi.filesystem

Class LocalPinotFS

    • Constructor Summary

      Constructors 
      Constructor Description
      LocalPinotFS()  

    • Method Summary

      Modifier and Type Method Description
      boolean copyDir​(URI srcUri, URI dstUri)
      Copies the file or directory from the src to dst.
      void copyFromLocalDir​(File srcFile, URI dstUri)  
      void copyFromLocalFile​(File srcFile, URI dstUri)
      The src file is on the local disk.
      void copyToLocalFile​(URI srcUri, File dstFile)
      Copies a file from a remote filesystem to the local one.
      boolean delete​(URI segmentUri, boolean forceDelete)
      Deletes the file at the location provided.
      boolean doMove​(URI srcUri, URI dstUri)
      Actual move implementation for each PinotFS.
      boolean exists​(URI fileUri)
      Checks whether the file or directory at the provided location exists.
      void init​(PinotConfiguration configuration)
      Initializes the configurations specific to that filesystem.
      boolean isDirectory​(URI uri)
      Allows us the ability to determine whether the uri is a directory.
      long lastModified​(URI uri)
      Returns the age of the file
      long length​(URI fileUri)
      Returns the length of the file at the provided location.
      String[] listFiles​(URI fileUri, boolean recursive)
      Lists all the files and directories at the location provided.
      List<FileMetadata> listFilesWithMetadata​(URI fileUri, boolean recursive)
      Like listFiles but also return some file metadata so no need to call length(), isDirectory(), and lastModified() separately, which can be slow and costly for remote file system.
      boolean mkdir​(URI uri)
      Creates a new directory.
      InputStream open​(URI uri)
      Opens a file in the underlying filesystem and returns an InputStream to read it.
      boolean touch​(URI uri)
      Updates the last modified time of an existing file or directory to be current time.

      • Methods inherited from class org.apache.pinot.spi.filesystem.BasePinotFS

        move

      • Methods inherited from interface org.apache.pinot.spi.filesystem.PinotFS

        close, copy

    • Constructor Detail

      • LocalPinotFS

        public LocalPinotFS()

    • Method Detail

      • init

        public void init​(PinotConfiguration configuration)
        Description copied from interface: PinotFS
        Initializes the configurations specific to that filesystem. For instance, any security related parameters can be initialized here and will not be logged.

      • mkdir

        public boolean mkdir​(URI uri)
              throws IOException
        Description copied from interface: PinotFS
        Creates a new directory. If parent directories are not created, it will create them. If the directory exists, it will return true without doing anything.
        Returns:
        true if mkdir is successful
        Throws:
        IOException - on IO failure

      • delete

        public boolean delete​(URI segmentUri,
                      boolean forceDelete)
               throws IOException
        Description copied from interface: PinotFS
        Deletes the file at the location provided. If the segmentUri is a directory, it will delete the entire directory.
        Parameters:
        segmentUri - URI of the segment
        forceDelete - true if we want the uri and any sub-uris to always be deleted, false if we want delete to fail when we want to delete a directory and that directory is not empty
        Returns:
        true if delete is successful else false
        Throws:
        IOException - on IO failure, e.g Uri is not present or not valid

      • copyDir

        public boolean copyDir​(URI srcUri,
                       URI dstUri)
                throws IOException
        Description copied from interface: PinotFS
        Copies the file or directory from the src to dst. The original file is retained. If the dst has parent directories that haven't been created, this method will create all the necessary parent directories. If dst already exists, this will overwrite the existing file/directory in the path. Note: In Pinot we recommend the full paths of both src and dst be specified. For example, if a file /a/b/c is copied to a file /x/y/z, the directory /a/b still exists containing the file 'c'. The dst file /x/y/z will contain the contents of 'c'. If a directory /a/b is copied to another directory /x/y, the directory /x/y will contain the content of /a/b. If a directory /a/b is copied under the directory /x/y, the dst needs to be specify as /x/y/b.
        Parameters:
        srcUri - URI of the original file
        dstUri - URI of the final file location
        Returns:
        true if copy is successful
        Throws:
        IOException - on IO failure

      • exists

        public boolean exists​(URI fileUri)
        Description copied from interface: PinotFS
        Checks whether the file or directory at the provided location exists.
        Parameters:
        fileUri - URI of file
        Returns:
        true if path exists

      • length

        public long length​(URI fileUri)
        Description copied from interface: PinotFS
        Returns the length of the file at the provided location.
        Parameters:
        fileUri - location of file
        Returns:
        the number of bytes

      • listFiles

        public String[] listFiles​(URI fileUri,
                          boolean recursive)
                   throws IOException
        Description copied from interface: PinotFS
        Lists all the files and directories at the location provided. Lists recursively if recursive is set to true. Throws IOException if this abstract pathname is not valid, or if an I/O error occurs.
        Parameters:
        fileUri - location of file
        recursive - if we want to list files recursively
        Returns:
        an array of strings that contains file paths
        Throws:
        IOException - on IO failure. See specific implementation

      • listFilesWithMetadata

        public List<FileMetadata> listFilesWithMetadata​(URI fileUri,
                                                boolean recursive)
                                         throws IOException
        Description copied from interface: PinotFS
        Like listFiles but also return some file metadata so no need to call length(), isDirectory(), and lastModified() separately, which can be slow and costly for remote file system.
        Parameters:
        fileUri - location of file
        recursive - if we want to list files recursively
        Returns:
        a list of FileMetadata that contains file path, and a few file metadata.
        Throws:
        IOException - on IO failure. See specific implementation

      • copyToLocalFile

        public void copyToLocalFile​(URI srcUri,
                            File dstFile)
                     throws Exception
        Description copied from interface: PinotFS
        Copies a file from a remote filesystem to the local one. Keeps the original file.
        Parameters:
        srcUri - location of current file on remote filesystem (must not be a directory)
        dstFile - location of destination on local filesystem
        Throws:
        Exception - if srcUri is not valid or not present, or timeout when downloading file to local

      • copyFromLocalFile

        public void copyFromLocalFile​(File srcFile,
                              URI dstUri)
                       throws Exception
        Description copied from interface: PinotFS
        The src file is on the local disk. Add it to filesystem at the given dst name and the source is kept intact afterwards.
        Parameters:
        srcFile - location of src file on local disk (must not be a directory)
        dstUri - location of dst on remote filesystem
        Throws:
        Exception - if fileUri is not valid or not present, or timeout when uploading file from local

      • copyFromLocalDir

        public void copyFromLocalDir​(File srcFile,
                             URI dstUri)
                      throws Exception
        Parameters:
        srcFile - location of src file on local disk (must be a directory)
        dstUri - location of dst on remote filesystem
        Throws:
        Exception - if fileUri is not valid or not present, or timeout when uploading file from local

      • isDirectory

        public boolean isDirectory​(URI uri)
        Description copied from interface: PinotFS
        Allows us the ability to determine whether the uri is a directory.
        Parameters:
        uri - location of file or directory
        Returns:
        true if uri is a directory, false otherwise.

      • lastModified

        public long lastModified​(URI uri)
        Description copied from interface: PinotFS
        Returns the age of the file
        Parameters:
        uri - location of file or directory
        Returns:
        A long value representing the time the file was last modified, measured in milliseconds since epoch (00:00:00 GMT, January 1, 1970) or 0L if the file does not exist or if an I/O error occurs

      • touch

        public boolean touch​(URI uri)
              throws IOException
        Description copied from interface: PinotFS
        Updates the last modified time of an existing file or directory to be current time. If the file system object does not exist, creates an empty file.
        Parameters:
        uri - location of file or directory
        Throws:
        IOException - if the parent directory doesn't exist

      • open

        public InputStream open​(URI uri)
                 throws IOException
        Description copied from interface: PinotFS
        Opens a file in the underlying filesystem and returns an InputStream to read it. Note that the caller can invoke close on this inputstream. Some implementations can choose to copy the original file to local temp file and return the inputstream. In this case, the implementation it should delete the temp file when close is invoked.
        Parameters:
        uri - location of the file to open
        Returns:
        a new InputStream
        Throws:
        IOException - on any IO error - missing file, not a file etc