Class OrderTokenizer


  • public class OrderTokenizer
    extends java.lang.Object
    Splits a string into OrderToken objects. The tokenizer recognizes quoted strings and comments and marks the generated tokens as such. Note that the tokenizer is not intended to detect keywords, identifiers and numbers, since such a classification can only be made with syntactical context. For proper handling of escaped new lines this class wraps the underlying stream in a MergeLineReader. This implies that the start and end attributes of the produced OrderToken objects reflect the actual position and length of the token on the underlying stream, including escape line breaks.
    • Constructor Summary

      Constructors 
      Constructor Description
      OrderTokenizer​(java.io.Reader r)
      Creates a new OrderTokenizer object which will perform its read operations on the specified stream.
    • Method Summary

      All Methods Instance Methods Concrete Methods 
      Modifier and Type Method Description
      protected void eatWhiteSpace()
      Consumes the stream up the next non-whitespace character.
      OrderToken getNextToken()
      Reads the next chunk of text from the underlying stream.
      protected boolean isQuote​(int c)  
      protected boolean isSpace​(int c)  
      protected OrderToken readQuote​(int quote)
      Reads from the underlying stream up to the next quotation mark or line break.
      protected OrderToken readSCComment()
      Reads a one line comment beginning with a semicolon up to the next line break.
      protected OrderToken readSSComment​(OrderToken retVal2)
      Reads a one line comment beginning with a double slash up to the next line break.
      protected OrderToken readWord()
      Reads one word from the underlying stream.
      void setQuotes​(char[] quotes)  
      • Methods inherited from class java.lang.Object

        clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
    • Constructor Detail

      • OrderTokenizer

        public OrderTokenizer​(java.io.Reader r)
        Creates a new OrderTokenizer object which will perform its read operations on the specified stream.
        Parameters:
        r - the stream this OrderTokenizer reads from.
    • Method Detail

      • getNextToken

        public OrderToken getNextToken()
        Reads the next chunk of text from the underlying stream. The token types are only recognized and set partially by this tokenizer. Apart from comments and quoted strings this has to be done in a semantical context.
      • readQuote

        protected OrderToken readQuote​(int quote)
                                throws java.io.IOException
        Reads from the underlying stream up to the next quotation mark or line break.
        Returns:
        a OrderToken object of type TT_STRING containing the quoted string.
        Throws:
        java.io.IOException - DOCUMENT-ME
      • readSCComment

        protected OrderToken readSCComment()
                                    throws java.io.IOException
        Reads a one line comment beginning with a semicolon up to the next line break.
        Returns:
        a OrderToken object of type TT_COMMENT containing the comment.
        Throws:
        java.io.IOException - DOCUMENT-ME
      • readSSComment

        protected OrderToken readSSComment​(OrderToken retVal2)
                                    throws java.io.IOException
        Reads a one line comment beginning with a double slash up to the next line break.
        Parameters:
        retVal2 -
        Returns:
        a OrderToken object of type TT_COMMENT containing the comment.
        Throws:
        java.io.IOException - DOCUMENT-ME
      • readWord

        protected OrderToken readWord()
                               throws java.io.IOException
        Reads one word from the underlying stream.
        Returns:
        a OrderToken object of type TT_UNDEF containing the word.
        Throws:
        java.io.IOException - DOCUMENT-ME
      • eatWhiteSpace

        protected void eatWhiteSpace()
                              throws java.io.IOException
        Consumes the stream up the next non-whitespace character.
        Throws:
        java.io.IOException - DOCUMENT-ME
      • isQuote

        protected boolean isQuote​(int c)
      • setQuotes

        public void setQuotes​(char[] quotes)
      • isSpace

        protected boolean isSpace​(int c)