Tokenizer

The Tokenizer transforms raw script text into a structured sequence of tokens.

Key features:

• Correctly identifies command names and arguments

• Handles parentheses and nested structures

• Preserves parameter order and hierarchy

Tokenizer breaks designer-friendly text into fundamental elements that the parser can understand.

Example: Script with Nested Structure

Consider this script:

If(StrEq(Get("difficulty"), "Hard"), {
    Set("playerHealth", 50),
    Log("Difficulty set to Hard")
})

The Tokenizer would break it down into:

• Main command: If

  • Argument 1: function call

    • StrEq

      • Arg 1: function call

        • Get

          • Arg: "difficulty"

      • Arg 2: "Hard"

  • Argument 2: block (curly braces)

    • Sequence of commands:

      • Set

        • Arg 1: "playerHealth"

        • Arg 2: 50

      • Log

        • Arg: "Difficulty set to Hard"

This clearly shows:

• The If condition argument itself is a call to Eq

• Eq has a nested call to Get as its first argument

• The second argument is a code block containing multiple sequential commands