TOKEN

Copyright © Cloud Software Group, Inc. All Rights Reserved

TOKEN

Parses an input string and returns the first token and the string with the token removed. (F)

Invocation

token_string = TOKEN(string)

token_string

The token returned. Its syntax is V (variable length character).

string

The character string to parse. On return, it contains string with the token removed. Its syntax must be C (fixed-length character), UN (Unicode), or V (variable-length character) and its semantic type must be S (string).

Usage Notes

•

The input string for TOKEN must be a local variable or a field of a table.

•

TOKEN parses string until it encounters a space (' '), a special character, a literal notation (r', R', u', U', x', or X'), or one of the following operators: ¬, &, |, *, /, **, +, -, =, ||, <, >, <=, >=, or ¬=. It then returns the first token and the string with the token removed. If string begins with an operator, the operator is the token returned.

•

Valid tokens are: identifiers, positive numbers, quoted strings, Unicode literals, raw data literals, hexadecimal literals, operators, and special characters. For additional information about these valid tokens, refer to TIBCO Object Service Broker Parameters.

•

In quoted strings, two single quotation marks are converted to a single quotation mark.

•

If string is of syntax UN and the token to be returned contains any character that cannot be coerced to syntax V, token_string is a Unicode literal representing the value of the token.

•

For more complex manipulation of tokens, refer to the PARSE tool.

Exceptions

MISMATCHEDQUOTES

Raised if there are unmatched quotation marks in the argument string.

Examples

The following rule prints the returned token and truncated string to the message log.

RULE EDITOR ===> SCROLL: P

TOKEN_EXAMPLE(INPUT);

_ LOCAL TOK, STRING;

_ ---------------------------------------------------------------------------

_ ------------------------------------------------------------+--------------

_ STRING = INPUT; | 1

_ TOK = TOKEN(STRING); | 2

_ CALL MSGLOG('FIRST TOKEN IS ' || TOK); | 3

_ CALL MSGLOG('RETURNED STRING IS ' || STRING); | 4

Resulting Output

If it is executed with the argument '10-12-2000', the following message log is produced:

------------------------- INFORMATIONAL MESSAGE LOG --------------------------

COMMAND ===> SCROLL ===> P

FIRST TOKEN IS 10

RETURNED STRING IS -12-2000

Example that Completely Parses a String

The following rules completely parse a string into tokens:

RULE EDITOR ===> SCROLL: P

TOKEN_EXAMPLE2(INPUT);

_ LOCAL TOK, STRING;

_ ---------------------------------------------------------------------------

_ ------------------------------------------------------------+--------------

_ STRING = INPUT; | 1

_ TOK = TOKEN(STRING); | 2

_ CALL MSGLOG('TOKEN IS ' || TOK); | 3

_ CALL TOKEN_EXAMPLE3(STRING); | 4

_ ---------------------------------------------------------------------------

RULE EDITOR ===> SCROLL: P

TOKEN_EXAMPLE3(STRING);

_

_ ---------------------------------------------------------------------------

_ STRING = ''; | Y N

_ ------------------------------------------------------------+--------------

_ CALL TOKEN_EXAMPLE2(STRING); | 1

_ ---------------------------------------------------------------------------

Resulting Output

If TOKEN_EXAMPLE2 is run with the argument 10-12-2000, the following message log is produced:

------------------------- INFORMATIONAL MESSAGE LOG --------------------------

COMMAND ===> SCROLL ===> P

TOKEN IS 10

TOKEN IS -

TOKEN IS 12

TOKEN IS -

TOKEN IS 2000

Example With Processing of Tokens

The following rules retrieve the keywords for a table, separate the keywords, and insert the name of the table associated with the keyword in a table instance for the keyword.

TABLE_KEYWORD Table

The first table, which contains the keywords for table descriptions, is TABLE_KEYWORD:

BROWSING TABLE: TABLE_KEYWORD

COMMAND ==>

TABLE KEYWORDS

    _ ---------------- ---------------------------------------------------

    _ EMPLOYEE PERSONNEL,NAME,NUMBER,DEPARTMENT,MANAGER,SALARY

    _ DEPARTMENT DEPARTMENT,NAME,NUMBER

    _ MANAGER MANAGER,NAME,NUMBER

TKEYWORDINDEX Table

The second table, TKEYWORDINDEX, is parameterized by KEYWORD. Each table instance lists the table names that have the parameter value as a keyword. An example is shown here.

BROWSING TABLE: TKEYWORDINDEX(NAME)

COMMAND ==>

TABLE

_ ----------------

_ EMPLOYEE

_ DEPARTMENT

_ MANAGER

KEYWORD_PARSE Rule

The parent rule is KEYWORD_PARSE. It retrieves each occurrence of the TABLE_KEYWORD table, and calls the KEYWORD_INDEX rule. The arguments for KEYWORD_INDEX are:

•

The name of the table that is stored in the TABLE field of the TABLE_KEYWORD table.

•

The first token (that is, the first keyword) for the table, which is extracted with TOKEN:

KEYWORD_PARSE;

_ LOCAL STR;

   _ ------------------------------------------------------------------------

_ ------------------------------------------------------------+-----------

_ FORALL TABLE_KEYWORD: | 1

_ STR = TABLE_KEYWORD.KEYWORDS; |

_ CALL KEYWORD_INDEX(TABLE_KEYWORD.TABLE,TOKEN(STR)); |

_ END; |

_ ------------------------------------------------------------------------

KEYWORD_INDEX Rule

KEYWORD_INDEX checks for valid keywords, and then inserts the table name into the TKEYWORDINDEX table. It also uses TOKEN to extract the next keyword.

KEYWORD_INDEX(NAME, TOK);

     _

     _ ------------------------------------------------------------------------

     _ TOK = ','; | Y N N N

_ TOK = ''; | Y N N

_ STR = ''; | Y N

_ ----------------------------------------------------------+-------------

_ TKEYWORDINDEX.TABLE = NAME; | 1 1

_ INSERT TKEYWORDINDEX(TOK); | 2 2

_ CALL KEYWORD_INDEX(NAME, TOKEN(STR)); | 1 3

_ ------------------------------------------------------------------------

Resulting Output

The result of running KEYWORD_PARSE against the table TABLE_KEYWORD is a TKEYWORDINDEX table with table instances such as DEPARTMENT, MANAGER, and NAME. If you browse the NAME table instance of the TKEYWORDINDEX table, you see the following:

BROWSING TABLE: TKEYWORDINDEX(NAME)

COMMAND ==>

TABLE

_ ----------------

_ EMPLOYEE

_ DEPARTMENT

_ MANAGER

token_string	The token returned. Its syntax is V (variable length character).
string	The character string to parse. On return, it contains string with the token removed. Its syntax must be C (fixed-length character), UN (Unicode), or V (variable-length character) and its semantic type must be S (string).