Class WhitespaceTokenizer

java.lang.Object
opennlp.tools.tokenize.WhitespaceTokenizer
All Implemented Interfaces:
Tokenizer

public class WhitespaceTokenizer extends Object
A basic Tokenizer implementation which performs tokenization using white spaces.

To obtain an instance of this tokenizer use the static final INSTANCE field. Use newInstance(boolean) for a tokenizer with a non-default setKeepNewLines(boolean) setting.

  • Field Details

    • INSTANCE

      public static final WhitespaceTokenizer INSTANCE
      Use this static reference to retrieve an instance of the WhitespaceTokenizer.

      This instance is shared by every caller in the process, so it must not be reconfigured: calling setKeepNewLines(boolean) on it changes the output of all other users and is not safely published to other threads. Use newInstance(boolean) to obtain a tokenizer with a different setting.

  • Method Details

    • newInstance

      public static WhitespaceTokenizer newInstance(boolean keepNewLines)
      Creates a tokenizer that is not shared with INSTANCE, for a caller that sets setKeepNewLines(boolean) without touching other users of the shared instance.
      Parameters:
      keepNewLines - true if line separators are returned as tokens.
      Returns:
      A new WhitespaceTokenizer with that setting.
    • tokenizePos

      public Span[] tokenizePos(String d)
      Description copied from interface: Tokenizer
      Finds the boundaries of atomic parts in a string.
      Parameters:
      d - The string to be tokenized.
      Returns:
      The spans (offsets into s) for each token as the individuals array elements.
    • tokenize

      public String[] tokenize(String s)
      Description copied from interface: Tokenizer
      Splits a string into its atomic parts.
      Specified by:
      tokenize in interface Tokenizer
      Parameters:
      s - The string to be tokenized.
      Returns:
      The String[] with the individual tokens as the array elements.
    • setKeepNewLines

      public void setKeepNewLines(boolean keepNewLines)
      Switches whether to keep new lines or not.
      Parameters:
      keepNewLines - True if new lines are kept, false otherwise.