com.cloudhopper.commons.charset

Class ModifiedUTF8Charset

java.lang.Object

com.cloudhopper.commons.charset.BaseCharset

com.cloudhopper.commons.charset.ModifiedUTF8Charset

All Implemented Interfaces:

Charset

public class ModifiedUTF8Charset
extends BaseCharset

Charset representing "Modified UTF-8". Java originally used 2 byte char primitives to store characters in its Strings. These were originally encoded in UCS2 -- which let Java natively support ~65K characters in Unicode. In Java 5, UCS2 is no longer used -- UTF-16 is now used. This let's Java natively support the entire range of Unicode characters which can be > 65K. For higher range UTF-16 characters with a Java char value of (> 0x7FFF), this charset does NOT correctly encode these values to the correct UTF-8 byte sequence.

Its usually quite uncommon in most situations to actually use a character value > 0x7FFF. This is why this charset exists -- it takes advantage of this property to speed up UTF-8 encoding/decoding of byte arrays. If you decide to solely use this charset for serialization, you also don't risk any issues with encoding/decoding since the resulting Java String will always be the same as if you actually used UTF-8.

This charset turns out to be very useful for directly encoding/decoding from byte arrays (especially if the byte array is already allocated), where the default Java classes would force you to create a new byte array. It also is ~30% faster than Java at decoding/encoding in most cases. In some cases it's a little slower. On average though it usually matches Java and has a good chance at being much faster during decoding.

This charset is originally based on much of the work in DataOuputStream.java and DataInputStream.java with a few notable tweaks:

• 0x0000 is encoded as 0x00 rather than DataOutputStreams 2 byte version. This is identical to how UTF-8 is actually supposed to work
• CharSequenceAccessor utility class uses reflection to access the private fields in String.java to reduce copying of char arrays.
• CharSequenceAccessor utility class uses reflection to access the private fields in String.java to directly access the underlying char array vs. ever calling charAt.

Author:

joelauer (twitter: @jjlauer or http://twitter.com/jjlauer)

Constructor Summary

Constructors

Constructor and Description
ModifiedUTF8Charset()

Method Summary

Methods

Modifier and Type	Method and Description
static int	calculateByteLength(CharSequence charSeq) Highly efficient method for calculating the byte length of a String if it was encoded as modified UTF-8 bytes.
static int	calculateByteLength(CharSequence charSeq, char[] charBuffer, int charOffset, int charLength) Highly efficient method for calculating the byte length of a String if it was encoded as modified UTF-8 bytes.
String	decode(byte[] bytes) Default implementation that simply returns a String by creating a new StringBuffer, appending to it, and then returning a new String.
String	decode(byte[] bytes, int offset, int length)
void	decode(byte[] bytes, StringBuilder buffer) Decode the byte array to a Java string that is appended to the buffer.
static int	decodeToCharArray(byte[] byteBuffer, int byteOffset, int byteLength, char[] charBuffer, int charOffset)
byte[]	encode(CharSequence charSeq) Encode the Java string into a byte array.
static int	encodeToByteArray(CharSequence charSeq, char[] charBuffer, int charOffset, int charLength, byte[] byteBuffer, int byteOffset) Encode the string to an array of UTF-8 bytes.
int	estimateDecodeCharLength(byte[] bytes)
int	estimateEncodeByteLength(CharSequence str0)

Methods inherited from class com.cloudhopper.commons.charset.BaseCharset

normalize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ModifiedUTF8Charset

public ModifiedUTF8Charset()

Method Detail

estimateEncodeByteLength

public int estimateEncodeByteLength(CharSequence str0)

estimateDecodeCharLength

public int estimateDecodeCharLength(byte[] bytes)

encode

public byte[] encode(CharSequence charSeq)

Description copied from interface: Charset

Encode the Java string into a byte array.

Parameters:

charSeq - The Java string to convert into a byte array

Returns:

A new byte array

decode

public void decode(byte[] bytes,
          StringBuilder buffer)

Description copied from interface: Charset

Decode the byte array to a Java string that is appended to the buffer. Implementations of this method will not change any of the byte values contained in the byte array.

Parameters:

bytes - The array of bytes to decode

buffer - The String buffer to append chars to

decode

public String decode(byte[] bytes)

Description copied from class: BaseCharset

Default implementation that simply returns a String by creating a new StringBuffer, appending to it, and then returning a new String. NOTE: This method is NOT efficient since it requires a double copy of a new String. Some charsets will override this default implementation to provide a more efficient impl.

Specified by:

decode in interface Charset

Overrides:

decode in class BaseCharset

Parameters:

bytes - The array of bytes to decode

Returns:

A new String with characters decoded from the byte array in the given charset.

decode

public String decode(byte[] bytes,
            int offset,
            int length)

calculateByteLength

public static int calculateByteLength(CharSequence charSeq)

Highly efficient method for calculating the byte length of a String if it was encoded as modified UTF-8 bytes. Since no byte array is allocated just for calculating the byte length, this method can speed up checks by 90% vs. something like s.getBytes("UTF8").length. This method is adapted from JDK source code for DataOutputStream.java.

Parameters:

charSeq - The character sequence to use for encoding.

Returns:

The number of bytes required to represent the String as modified UTF-8 encoded bytes.

calculateByteLength

public static int calculateByteLength(CharSequence charSeq,
                      char[] charBuffer,
                      int charOffset,
                      int charLength)

Highly efficient method for calculating the byte length of a String if it was encoded as modified UTF-8 bytes. Since no byte array is allocated just for calculating the byte length, this method can speed up checks by 90% vs. something like s.getBytes("UTF8").length. This method is adapted from JDK source code for DataOutputStream.java.

Parameters:

charSeq - The optional character sequence to use for encoding rather than the provided character buffer. It is always higher performance to supply a char array vs. use a CharSequence. Set to null if the character array is supplied.

charBuffer - The source char array to encode

charOffset - The offset in the source char array to start encode from

charLength - The length from the offset in the source char array to encode

Returns:

The number of bytes required to represent the String as modified UTF-8 encoded bytes.

encodeToByteArray

public static int encodeToByteArray(CharSequence charSeq,
                    char[] charBuffer,
                    int charOffset,
                    int charLength,
                    byte[] byteBuffer,
                    int byteOffset)

Encode the string to an array of UTF-8 bytes. The buffer must be pre-allocated and have enough space to hold the encoded string.

Parameters:

charSeq - The optional character sequence to use for encoding rather than the provided character buffer. It is always higher performance to supply a char array vs. use a CharSequence. Set to null if the character array is supplied.

charBuffer - The source char array to encode

charOffset - The offset in the source char array to start encode from

charLength - The length from the offset in the source char array to encode

byteBuffer - The destination byte array to encode to

byteOffset - The offset in the destination byte array to start encode to

Returns:

The number of bytes written to the destination byte array

See Also:

calculateByteLength(java.lang.CharSequence)

decodeToCharArray

public static int decodeToCharArray(byte[] byteBuffer,
                    int byteOffset,
                    int byteLength,
                    char[] charBuffer,
                    int charOffset)