Thank you so much for your comments, Gregory. My notes point by point:

>
>Antonio,
>
>It's a good starting point
>(1) Ensure you take the longest match
>eg
>
>

>=
>==
>!
>!=
><
><=
>>
>>=
>

>
>You can do this by testing all patterns and take the longest match - or combining all the patterns into one

Good point - and catch! Forgot about logical operators.

Revised pattern for operators:

m.loTokenizer.AddTokenPattern("(==|!=|<=|>=|<>|!|=|>|<|\+|-|\*|\/|%|,|\)|\(|\$)")

>
>(2) When you trim, there are more white space chars than the space char, there is eg the TAB char which you do not trim
>To catch those, I would let the patterns start with ^ to make sure you start matching at the beginning of the string
>
>Also the ^ at the beginning of a pattern ensures you don't match somewhere in the middle

Changed trimmers to catch all [accepted] white space:

m.lcExpression = ALLTRIM(m.tcExpression,0," ",CHR(13),CHR(10),CHR(9))

and

m.lcExpression = LTRIM(SUBSTR(m.lcExpression,LEN(m.lcToken) + 1),0," ",CHR(13),CHR(10),CHR(9))

All patterns are already set to the beginning of the expression. The AddTokenPattern() method makes sure of that by appending ^ at the start of pattern, if it is not already included.

>
>(3) You can easily add a pattern for a string

Sorry, I don't understand this remark. There is already a pattern for strings, isn't it?

>
>Update
>(4) You are not splitting into 'real' tokens
>A token also tells which token class it is, eg number, string, word, operator
>
>To do that you need one or more patterns per token class
>
>eg
>

>NUMBER:("(\+|-)?(\.\d+|\d+(\.\d+)?)((e|E)(\+|-)?\d+)*")
>

>
>And take the longest match

Will do that, it will ease the work of the post-tokenizer (for instance, just check on "FUNCTION" tokens for allowed functions).