Package org.jesterj.ingest.processors

Class SendToSolrCloudProcessor

    • Constructor Detail

      • SendToSolrCloudProcessor

        protected SendToSolrCloudProcessor()

    • Method Detail

      • log

        public org.apache.logging.log4j.Logger log()

      • perDocFailLogging

        protected void perDocFailLogging​(java.lang.Exception e,
                                 Document doc)

      • isPotent

        public boolean isPotent()
        Description copied from interface: DocumentProcessor
        Indicates a processor for which repeated invocations have cumulative external side effects. "Potent" is a term coined for use in JesterJ meant to be faster to type and easier to think about than "non-idempotent". Potent processors are the key processors that fault tolerance must avoid repeating, and thus adding potent processors increases the load on the internal cassandra instance.
        Specified by:
        isPotent in interface DocumentProcessor
        Returns:
        true if the repeated execution of this processor with the same inputs results in cumulative or otherwise inconstant external state.

      • individualFallbackOperation

        protected void individualFallbackOperation​(org.apache.cassandra.utils.ConcurrentBiMap<Document,​org.apache.solr.common.SolrInputDocument> oldBatch,
                                           java.lang.Exception e)
        If the bulk request fails it might be just one document that's causing a problem, try each document individually
        Parameters:
        oldBatch - The batch for which to handle failures. This will have been detached from this object and will become eligible for garbage collection after this method returns, so do not add objects to it.
        e - the exception reported with the failure

      • batchOperation

        protected void batchOperation​(org.apache.cassandra.utils.ConcurrentBiMap<Document,​org.apache.solr.common.SolrInputDocument> oldBatch)
                       throws org.apache.solr.client.solrj.SolrServerException,
                              java.io.IOException
        Throws:
        org.apache.solr.client.solrj.SolrServerException
        java.io.IOException

      • exceptionIndicatesDocumentIssue

        protected boolean exceptionIndicatesDocumentIssue​(java.lang.Exception e)

      • convertDoc

        protected org.apache.solr.common.SolrInputDocument convertDoc​(Document document)

      • getName

        public java.lang.String getName()
        Description copied from interface: Configurable
        A name for this object to distinguish it from other objects. This value is generally supplied by the plan author. Every object in a plan must have a unique name, begin with a letter and only contain letters, digits, underscores and periods.
        Specified by:
        getName in interface Configurable
        Returns:
        The user supplied name for this step

      • processDocument

        public Document[] processDocument​(Document document)
        Description copied from interface: DocumentProcessor
        Mutate, validate or transmit a document. Implementations must not throw any * Throwable that is not a JVM Error and should be written expecting the possibility that the code might be interrupted at any point. Practically this means Document processors should perform no more than one persistent or externally visible actions and that action should be transactional. Large complex processors that write to disk, DB, or elsewhere multiple times run the risk of partial completion. Similarly, since JesterJ is a long-running system it will often cease operation due to unexpected outages (power cord, etc.), so it is not a good idea to hold resources that require an explicit release or "return". "Check then write" is of course a performance anti-pattern with respect to external networked or disk resources since network and disk io are typically slow to access. Processors should feel free to set the status of a document and add a status message via Document.setStatus(Status, String, java.io.Serializable...) however the easiest way to communicate a failure (for which all further processing is in error) is to simply throw a runtime exception. The document processor has no need to add the document to the next step in the plan as this will be handled by the infrastructure in StepImpl based on the status of the document so long as the document is emitted via the return value of this method. If the document enters via the parameters and is not emitted for any reason the processor MUST set an appropriate status before the end of this method, though it is preferable to just set the status and emit it.
        Specified by:
        processDocument in interface DocumentProcessor
        Parameters:
        document - the item to process
        Returns:
        The documents that result from the processing in this step. Documents with status of Status.PROCESSING will be processed by subsequent steps, and documents with any other status will have their status recorded and will not be processed by subsequent steps.

      • perDocumentFailure

        protected void perDocumentFailure​(org.apache.cassandra.utils.ConcurrentBiMap<Document,​?> oldBatch,
                                  java.lang.Exception e)