The Xtal Language

You can also type Xtal commands directly in interactive mode using the Xtal shell which accepts both Xtal and Tcl syntax. This is described in The Xtal shell.

In this document, for ease of exposition we will simply show the relevant Xtal fragments without any reference to how they are executed.

Top ↑

Like Tcl, a Xtal script is a sequence of statements separated by either a newline character or a semicolon.

A statement may be an assignment as above, or simply an expression:

Tcl and Xtal code can be intermixed. To embed Xtal within Tcl, use the xtal::xtal command as

where SCRIPT is the Xtal fragment which may be spread across multiple lines.

Conversely, Tcl can be embedded with Xtal by placing the Tcl script within < and > delimiters.

Multi-line scripts are permitted.

The terminating > must be followed by a Xtal statement terminator such as a newline or ;, optionally with intervening whitespace.

Comments begin with # and extend to the end of the line.

Top ↑

As seen above, variables are syntactically different from Tcl:

The variables themselves are standard Tcl variables and follow the same name resolution rules.

Numeric literals are as in Tcl and have the same syntax as in Tcl. For example,

String literals are delimited by single or double quotes and follow the backslash substitution rules as in Tcl. The difference between the two is that double quoted strings undergo variable and command substition as in Tcl whereas single quoted strings do not.

Unlike in Tcl, a plain unquoted string is treated as an identifier and not a string literal.

Braces in Xtal are used to construct lists, similar to the list command in Tcl.

The list elements are separated using commas and an element may be a literal, variable, function call or general expression. The list elements may be spread across multiple lines but must still be separated by commas.

As an aside, we could also have invoked the Tcl list command as a function to construct a list.

Invocation of functions/commands takes a form similar to function calls in C. Function parameters are wrapped in parenthesis and separated by commas.

The above calls Tcl’s built-in format command. There is no difference between invoking commands implemented in Tcl or in Xtal.

As for list elements, the parameters for a function call may be spread across multiple lines.

The function called need not be an identifier; it can be supplied as any expression, for example a function call that returns a function name.

There are two ways to define functions that implemented in Xtal:

Both offer the same functionality, differing only because of Tcl and Xtal syntax.

The Xtal return statement returns control to the caller from a function or procedure.

EXPRESSION is any valid Xtal expression.

The if statement has the general form

There may be zero or more elseif clauses and the else clause is optional. If present, the else and elseif keywords must be on the same logical line as the preceding statement block.

EXPRESSION can be any Xtal expression and does not need to be enclosed in parenthesis.

The while statement has the syntax

EXPRESSION can be any Xtal expression and does not need to be enclosed in parenthesis.

The for statement is used to iterate over a range of integers with a suitable increment.

The loop variable IDENTIFIER is initially assigned a value of LOWEXPR. Thereafter, STATEMENTBLOCK is executed as long as the the value of the loop variable is less than or equal to HIGHEXPR. After each iteration, the loop variable is incremented by INCREMENT which defaults to 1 if unspecified. The lower limit of the range is computed only once, before the loop is iterated. The upper limit is computed on every iteration.

If HIGHEXPR is unspecified, the loop will not terminate unless you break out of it with a return, break or similar. Some examples of the various forms:

This statement behaves similar to the Tcl foreach statement where the loop variable takes on values from a collection. The syntax takes one of the forms

where COLLECTION is any Xtal expression.

In the first form, SCRIPT is executed for each element in COLLECTION. If COLLECTION is a column, the variable VAR takes the value of each element of the column. If COLLECTION is a table, VAR is assigned each row of the table in list form. Otherwise, COLLECTION is treated as a Tcl list and VAR takes on the value of each element of the list.

The second form is similar in that again SCRIPT is executed for each element of COLLECTION. However in addition to VAR being assigned the value of the element, the variable named INDEXVAR is assigned the index of the element.

The break statement behaves like the Tcl break command. It causes the innermost loop to stop execution and continue execution at the following statement.

The continue statement behaves like the Tcl continue command. It aborts the current iteration of the innermost loop containing it without aborting the loop itself which then proceeds with its next iteration.

The try statement closely resembles Tcl’s try command differing only in that the syntax is a little different and the clauses are in Xtal.

The general syntax of the command is

Multiple handlers may be specified. Each HANDLER may be one of two forms:

The statement results in the execution of BODY. The exception code resulting from the execution is matched against each handler in turn. The HANDLERBODY corresponding to the first matching handler is executed and propagation of the exception stops in that case. Further handlers are not checked. If no handler matches, the exception is propagated. In all cases, irrespective of the exception code and whether any handler matched, the FINALBODY script is run if present.

In the case of the on handler, EXCEPTIONCODE is an integer value or one of the strings ok, error, continue, break or return. The handler will match if this matches the exception code resulting from the execution of BODY.

The trap handler only matches if the exception code is error. Additionally the appropriate number of leading values from the -errorcode entry in the interpreter status dictionary must match the values specified in ERRORPREFIX which must be of the form used for constructing lists.

In both types of handlers, the handler script body may be preceded by zero or more variable names. If present, the first of these receives the result of the execution of BODY. The second variable, if present, receives the status dictionary.

The following example will help clarify the working.

A more bare-bones version of the above, with a different error this time, would be

The throw statement raises a Tcl error in similar manner to Tcl’s own throw and error statement. Its general syntax is

If a single argument is given, it is taken to be the equivalent of the message argument to a Tcl error or throw command. If multiple arguments are given, the last argument is interpreted as the message. Moreover, all arguments are gathered into a list which becomes the value of the error code.

Columns are created using the built-in constructors @boolean, @byte, @int, @uint, @wide, @double, @string and @any corresponding to the different column types. The syntax of the constructor is

Here TYPE is one of the tarray column types, SIZE provides a hint of initial sizing of the column and INITIALIZER is an optional initializer.

In the simplest case,

creates an empty column of type int and assigns it to A.

We can optionally specify a sizing hint for the initial allocation.

Except in the case of random initializers discussed below, this is only used as a hint as to how large the column is expected to grow. Notice that the column is still empty. Intervening whitespace is allowed.

If INITIALIZER is specified, it is used to initialize the elements of the created column. It may take several forms.

The first of these is a list literal. The statement

creates a column of type any with 3 elements. An optional size specifier may be present.

This will create a column with four elements, expected to grow to 10. Again, the size specifiers is only a hint to the Xtal storage allocator. It does not impact any language semantics. The number of items in the column can exceed the size specified, even at initialization time. For example,

The second form of an initializer specifies a series. The command then look like

This creates a column of the specified type containing elements from START (inclusive) to STOP (excluded) at intervals of STEP which default to 1 if unspecified.

The SIZE specifier may be included here as well.

The third initializer form is an expression surrounded by parenthesis, again with an optional size specifier.

Again, here TYPE is one of the tarray column types. EXPR, if specified, is an expression that will be used to initialize the column. Thus this form can also be used to create a column of a different type provided the values are compatible. For example, we can create an int column from a double column.

Notice the size specifier and whitespace are both optional.

The final form initializes the created column with random values of the appropriate type. Random initializers cannot be used for columns of type string and any.

Here the size specifier is more than a storage hint. It is the actual number of elements, each of which is initialized with a random value.

Note the similarity with the series initializer syntax earlier.

The optional LOWERBOUND (inclusive) and UPPERBOUND (exclusive) arguments specify the range of values within which the random values must lie. By default, all values for the specific column type are allowed.

Whitespace is optional and if the upper bound is not specified it defaults to the highest value for that type.

Note that constructors are expressions and can be used as such.

Tables are defined and created using the @table built-in operator. The syntax for the command is

where COLNAME is the name of the column and TYPE is one of the tarray column types. TABLEROWS may be specified to initialize the table and should be a list of rows as for the tarray::table create command. Here is the table definition from our Quick Tour section.

Top ↑

Columns from a table can be selected using the . operator.

The column specification COLSPEC may be the actual name of the column or specified through a variable via the dereferencing operator.

These operators can be used in two contexts:

We will see examples later when we discuss indexing operators.

Top ↑

The table slicing operators .() operator return a table containing specific columns from a table.

Each column is specified using its name or by derefencing a variable containing its name.

Like the table column selection operators, the table slicing operator can also be used in two contexts:

Again, we will see examples later when we discuss indexing operators.

Columns and tables can be converted to lists and dictionaries with the @list and @dict functions respectively. In the case of the latter, the key for the dictionary is the index of the value in the column or table.

Any expression that results in a column or table may be supplied as the parameter to these functions. If the expression does not result in a column or table, an error is generated.

The relational operators include the ==, !=, <, <=, >, >= operators that are present in Tcl.

In addition, they include the operators related to string comparisons that are shown in Xtal relational operators not in Tcl.

Note that for the regular expression operators, the order of the operands is significant - the first indicates the string being matched and the second is the regular expression.

The boolean columns returned in the last two cases can be used as index column. For example,

We will discuss this in Indexing and search operators.

Top ↑

The logical operators include && and || as in Tcl. However, unlike Tcl these operators do not short-circuit the evaluation of their operands.

If any column operand is of type any or string, an error is raised.

Top ↑

The operator . looks up keys in a dictionary or column.

In the case of a dictionary, the operators retrieve the value associated with the key.

The key operand is treated as a literal and returns the corresponding element from the dictionary. If the key follows the syntax for an identifier, it does not have to be quoted.

To look up a key that is stored in a variable use the $ dereferencing operator.

In the case of a column the lookup returns the index of the specified value.

In the prior example, note the overloading of the . operator When applied to tables, it acts as a column selector. When applied to columns (selected from the table in this case) it acts as a lookup operator.

In all cases, an error is raised if the value being looked up does not exist in the dictionary or column.

These operators can also be used on the left hand side of an assignment to set new values.

Top ↑

Much of the convenience of Xtal in dealing with columns and tables stems from its variety of indexing options. Indices may be simple numerics, ranges, lists or selectors based on boolean expressions.

The flexibility of index selectors also means that they are the primary means of searching tables and columns for elements matching desired criteria. Many indexing and search operators can also be used with lists.

The @delete deletes elements from a column or table. It has one of two forms:

The first form deletes all elements in the index range from FIRST to LAST. The second form deletes the elements whose indices are specified by INDICES which may be a single index, an index column or a list of indices.

So for instance we could get rid of all our high-priced employees.

The @fill fills elements from a column or table with a fixed value. It has one of two forms:

The first form sets all elements in the index range from FIRST to LAST to VALUE. The second form fills the elements whose indices are specified by INDICES which may be a single index, an index column or a list of indices.

We can raise all low paid employees to a minimum salary level.

The function can also be used to initialize a column or table with a fixed value.

The assignment operators in Xtal cannot be used to insert elements between two elements in a column or rows in a table. The @inject and @insert functions are provided instead for that purpose. The @inject command inserts the contents of a list or column (table) into another and returns the result. The syntax is

where SOURCE is a list or column (table) to be inserted at position START in COLUMN_OR_TABLE. For example, we can add a couple of low priced lackeys to replace the ones we layed off.

We supplied a list of rows to be added above but we could also have specified a compatible table instead.

The @insert function is similar to @inject but inserts a single repeated value or row into a column or table. It has the syntax

The function inserts COUNT instances of VALUE at the index position START. COUNT defaults to 1 if unspecified.

The function returns the index of an element in a column which must be of type string. It has the syntax

The command returns the index of an element in STRINGCOLUMN that exactly matches LOOKUPKEY or -1 if not found. If KEY is not specified, command builds an internal dictionary (see below) and the return value is an empty string.

Unlike the @search command, the returned index is not necessarily that of the first occurence in cases where KEY occurs multiple times in the column.

The primary purpose of this function is to provide fast access to columns that are used as an index. For more details, see the documentation for the column lookup command.

The @reverse function reverses columns and tables.

The @search command searches a specified column and returns matching elements or the corresponding indices. Unlike most of the other built-ins, @search is called as a command and does not use the standard function call notation. It can take one of two forms. In the first form, the search target is specified directly:

In this form, COLUMNEXPR can be any expression that results in a column value. RELOP is any relational operator. Without any options specified, the command returns the index of the first element of the column that matches EXPRESSION. SEARCHOPTIONS is a space-separated list of options that modify this behaviour. The possible option values are shown in @search options.

The following examples illustrate the various combinations.

The second form that @search takes is

In this case, INDEXCOLUMN is expected to contain indices into the column COLUMNEXPR and the search only examines the corresponding elements for a match. In all other respects, this form of the command behaves the same as the first form. See the next section for an example.

Given that selectors perform a similar function to @search, when might one be preferred to the other?

The selector form is more succint, convenient and not limited in its expressive power related to conditions. However, the search command in the current implemenation has some performance advantages illustrated in the example below.

Suppose we want to list the salaries of all employees in Boston who make more than 45000/year. We could write this using selectors.

Alternatively, using @search,

In the first case, Xtal searches the appropriate columns for salaries greater than the specified amount and then again for the specified location. Finally, it returns the intersection of the two and retrieves those names from the Name column of the table.

In the second case, the list of indices where the salary is greater than the specified amount is retrieved. Then only those elements are searched for the location. Consequently, this method is significantly faster for large data sets.

The @sort command provides a flexible mechanism for sorting columns. Like @search and unlike most of the other built-ins, @sort is called as a command and does not use the standard function call notation. It can take one of two forms. In the first form, the sort target is specified directly:

In this form, COLUMNEXPR can be any expression that results in a column value. The command then returns a column of the same type sorted based on the specified options. SORTOPTIONS is a space-separated list of options that control the sorting operations. The possible option values are shown in @sort options.

So we might want to get the rainfall in sorted order.

But what is probably more interesting is the order of months so we might instead do the following (this time choosing a decreasing sort order).

This returns a column of type int containing the indices of the sorted values.

The sorted indices are useful in many circumstances. For example, we might display the employee database in multiple windows, one sorted by age, the other by salary. Instead of keeping two copies of the table sorted differently, it is cheaper in terms of memory to keep indices sorted differently. For example,

The second form that @sort takes is

Here the sorting is done indirectly. INDEXCOLUMN is an integer index column as the ones we computed above. This is treated as containing indices into the column COLUMNEXPR. The return value is INDEXCOLUMN sorted by comparing the corresponding values from the column COLUMNEXPR.

There are a couple of scenarios where this is useful. One is in keeping a stable sort order when sorting successively on multiple keys. This is discussed in detail in Nested sorts and sort stability in the Programmer’s guide.

The other example is when only a subset of a column or table is to be sorted. If we wanted to list salaries in sorted order but only for employees older than 35, we could do the following:

Or if you want to save on lines,

This method is a lot cheaper than working directly with the table.

The @sum function returns the sum of the elements in a column.

So to get our annual salary expenditure,