1. Overview

As is true for scripting languages in general, Tcl incurs significant overhead in terms of memory as well as performance when operating on large amounts of data. The set of packages described here provide a more performant and convenient means of working with large data sets. The three packages are

  • tarray — implements the core data types and operations

  • xtal — provides succint, convenient syntax for working with data sets

  • tarray_ui — contains Tk widgets that are useful for display and user interaction

The packages come with comprehensive documentation, an extensive test suite and are covered by a liberal open source license.

Note The packages require Tcl version 8.6 or later.

2. The tarray package

The tarray (typed array) extension implements two new Tcl collection data types — columns and tables. Typed arrays store elements of a specific data type in native format and provide some features that are not built into lists and dictionaries

  • vector operations on whole columns

  • more powerful search and ordering facilities

  • flexible indexing and modification commands

  • grouping, joins, summarization and histograms

  • utilities like random numbers and sampling

In addition, typed arrays offer considerable benefits in performance in both memory consumption and execution time through native representation and parallelization on multiple cores.

3. The xtal package

The related extension xtal implements Xtal, a language geared towards vector operations on lists, columns and tables. Xtal can be embedded within Tcl in a similar fashion to the expr command or used to write entire procedures and scripts. Xtal adds a succint syntactic layer on top of the tarray extension. For example, the following Xtal script to retrieve employees in Boston within a specific salary range would replace several lines of Tcl.

Emps[Emps.Salary > 30000 && Emps.Salary < 50000 && Emps.Location == "Boston"]

Xtal can be used with standard Tcl structures like lists and dictionaries as well.

% set L [list 1.01 10.1 .101 101]
→ 1.01 10.1 .101 101
% xtal::xtal { L[L > 1 && L < 100] }
→ 1.01 10.1

See the Xtal Quick Tour for a short overview of the language.

You can also use Xtal in interactive mode, either at the tclsh prompt, the wish console or tkcon. See The Xtal shell for details.

4. The tarray_ui package

The tarray_ui package implements widgets that useful for displaying and interacting with typed arrays. This includes widgets for graphs using the RBC package, displaying table data with sorting/filtering capabilities via the tktreectrl package, and for importing CSV files.

RBC chart example
Using visuals and formatting

5. Documentation guide

6. License

The packages are primarily covered by the liberal open source BSD license. See the file license.terms in the source directories for details. Some third-party components used in the software may have their own license.