com.twitter.twittertext

Class Extractor

java.lang.Object

com.twitter.twittertext.Extractor

public class Extractor
extends Object

A class to extract usernames, lists, hashtags and URLs from Tweet text.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Extractor.Entity

Field Summary

Fields

Modifier and Type	Field and Description
protected boolean	extractURLWithoutProtocol
static int	MAX_TCO_SLUG_LENGTH The maximum t.co path length that the Twitter backend supports.
static int	MAX_URL_LENGTH The maximum url length that the Twitter backend supports.

Constructor Summary

Constructors

Constructor and Description
Extractor() Create a new extractor.

Method Summary

Modifier and Type	Method and Description
List<String>	extractCashtags(String text) Extract $cashtag references from Tweet text.
List<Extractor.Entity>	extractCashtagsWithIndices(String text) Extract $cashtag references from Tweet text.
List<Extractor.Entity>	extractEntitiesWithIndices(String text) Extract URLs, @mentions, lists and #hashtag from a given text/tweet.
List<String>	extractHashtags(String text) Extract #hashtag references from Tweet text.
List<Extractor.Entity>	extractHashtagsWithIndices(String text) Extract #hashtag references from Tweet text.
List<String>	extractMentionedScreennames(String text) Extract @username references from Tweet text.
List<Extractor.Entity>	extractMentionedScreennamesWithIndices(String text) Extract @username references from Tweet text.
List<Extractor.Entity>	extractMentionsOrListsWithIndices(String text) Extract @username and an optional list reference from Tweet text.
String	extractReplyScreenname(String text) Extract a @username reference from the beginning of Tweet text.
List<String>	extractURLs(String text) Extract URL references from Tweet text.
List<Extractor.Entity>	extractURLsWithIndices(String text) Extract URL references from Tweet text.
boolean	isExtractURLWithoutProtocol()
static boolean	isValidHostAndLength(int originalUrlLength, String protocol, String originalHost) Verifies that the host name adheres to RFC 3490 and 1035 Also, verifies that the entire url (including protocol) doesn't exceed MAX_URL_LENGTH
void	modifyIndicesFromUnicodeToUTF16(String text, List<Extractor.Entity> entities) Modify Unicode-based indices of the entities to UTF-16 based indices.
void	modifyIndicesFromUTF16ToUnicode(String text, List<Extractor.Entity> entities) Modify UTF-16-based indices of the entities to Unicode-based indices.
void	setExtractURLWithoutProtocol(boolean extractURLWithoutProtocol)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_URL_LENGTH

public static final int MAX_URL_LENGTH

The maximum url length that the Twitter backend supports.

See Also:

Constant Field Values

MAX_TCO_SLUG_LENGTH

public static final int MAX_TCO_SLUG_LENGTH

The maximum t.co path length that the Twitter backend supports.

See Also:

Constant Field Values

extractURLWithoutProtocol

protected boolean extractURLWithoutProtocol

Constructor Detail

Extractor

public Extractor()

Create a new extractor.

Method Detail

extractEntitiesWithIndices

public List<Extractor.Entity> extractEntitiesWithIndices(String text)

Extract URLs, @mentions, lists and #hashtag from a given text/tweet.

Parameters:

text - text of tweet

Returns:

list of extracted entities

extractMentionedScreennames

public List<String> extractMentionedScreennames(String text)

Extract @username references from Tweet text. A mention is an occurrence of @username anywhere in a Tweet.

Parameters:

text - of the tweet from which to extract usernames

Returns:

List of usernames referenced (without the leading @ sign)

extractMentionedScreennamesWithIndices

public List<Extractor.Entity> extractMentionedScreennamesWithIndices(String text)

Extract @username references from Tweet text. A mention is an occurrence of @username anywhere in a Tweet.

Parameters:

text - of the tweet from which to extract usernames

Returns:

List of usernames referenced (without the leading @ sign)

extractMentionsOrListsWithIndices

public List<Extractor.Entity> extractMentionsOrListsWithIndices(String text)

Extract @username and an optional list reference from Tweet text. A mention is an occurrence of @username anywhere in a Tweet. A mention with a list is a @username/list.

Parameters:

text - of the tweet from which to extract usernames

Returns:

List of usernames (without the leading @ sign) and an optional lists referenced

extractReplyScreenname

public String extractReplyScreenname(String text)

Extract a @username reference from the beginning of Tweet text. A reply is an occurrence of @username at the beginning of a Tweet, preceded by 0 or more spaces.

Parameters:

text - of the tweet from which to extract the replied to username

Returns:

username referenced, if any (without the leading @ sign). Returns null if this is not a reply.

extractURLs

@Nonnull
public List<String> extractURLs(@Nullable
                                         String text)

Extract URL references from Tweet text.

Parameters:

text - of the tweet from which to extract URLs

Returns:

List of URLs referenced.

extractURLsWithIndices

@Nonnull
public List<Extractor.Entity> extractURLsWithIndices(@Nullable
                                                              String text)

Extract URL references from Tweet text.

Parameters:

text - of the tweet from which to extract URLs

Returns:

List of URLs referenced.

isValidHostAndLength

public static boolean isValidHostAndLength(int originalUrlLength,
                                           @Nullable
                                           String protocol,
                                           @Nullable
                                           String originalHost)

Verifies that the host name adheres to RFC 3490 and 1035 Also, verifies that the entire url (including protocol) doesn't exceed MAX_URL_LENGTH

Parameters:

originalUrlLength - The length of the entire URL, including protocol if any

protocol - The protocol used

originalHost - The hostname to check validity of

Returns:

true if the host is valid

extractHashtags

public List<String> extractHashtags(String text)

Extract #hashtag references from Tweet text.

Parameters:

text - of the tweet from which to extract hashtags

Returns:

List of hashtags referenced (without the leading # sign)

extractHashtagsWithIndices

public List<Extractor.Entity> extractHashtagsWithIndices(String text)

Extract #hashtag references from Tweet text.

Parameters:

text - of the tweet from which to extract hashtags

Returns:

List of hashtags referenced (without the leading # sign)

extractCashtags

public List<String> extractCashtags(String text)

Extract $cashtag references from Tweet text.

Parameters:

text - of the tweet from which to extract cashtags

Returns:

List of cashtags referenced (without the leading $ sign)

extractCashtagsWithIndices

public List<Extractor.Entity> extractCashtagsWithIndices(String text)

Extract $cashtag references from Tweet text.

Parameters:

text - of the tweet from which to extract cashtags

Returns:

List of cashtags referenced (without the leading $ sign)

setExtractURLWithoutProtocol

public void setExtractURLWithoutProtocol(boolean extractURLWithoutProtocol)

isExtractURLWithoutProtocol

public boolean isExtractURLWithoutProtocol()

modifyIndicesFromUnicodeToUTF16

public void modifyIndicesFromUnicodeToUTF16(String text,
                                            List<Extractor.Entity> entities)

Modify Unicode-based indices of the entities to UTF-16 based indices.

In UTF-16 based indices, Unicode supplementary characters are counted as two characters.

This method requires that the list of entities be in ascending order by start index.

Parameters:

text - original text

entities - entities with Unicode based indices

modifyIndicesFromUTF16ToUnicode

public void modifyIndicesFromUTF16ToUnicode(String text,
                                            List<Extractor.Entity> entities)

Modify UTF-16-based indices of the entities to Unicode-based indices.

In Unicode-based indices, Unicode supplementary characters are counted as single characters.

This method requires that the list of entities be in ascending order by start index.

Parameters:

text - original text

entities - entities with UTF-16 based indices