Introduction

Main Content ▼

1. Overview

As is true for scripting languages in general, Tcl incurs significant overhead in terms of memory as well as performance when operating on large amounts of data. The set of packages described here provide a more performant and convenient means of working with large data sets. The three packages are

tarray — implements the core data types and operations
xtal — provides succint, convenient syntax for working with data sets
tarray_ui — contains Tk widgets that are useful for display and user interaction

The packages come with comprehensive documentation, an extensive test suite and are covered by a liberal open source license.

The packages require Tcl version 8.6 or later.

2. The tarray package

The tarray (typed array) extension implements two new Tcl collection data types — columns and tables. Typed arrays store elements of a specific data type in native format and provide some features that are not built into lists and dictionaries

vector operations on whole columns
more powerful search and ordering facilities
flexible indexing and modification commands
grouping, joins, summarization and histograms
utilities like random numbers and sampling

In addition, typed arrays offer considerable benefits in performance in both memory consumption and execution time through native representation and parallelization on multiple cores.

3. The xtal package

The related extension xtal implements Xtal, a language geared towards vector operations on lists, columns and tables. Xtal can be embedded within Tcl in a similar fashion to the expr command or used to write entire procedures and scripts. Xtal adds a succint syntactic layer on top of the tarray extension. For example, the following Xtal script to retrieve employees in Boston within a specific salary range would replace several lines of Tcl.

Emps[Emps.Salary > 30000 && Emps.Salary < 50000 && Emps.Location == "Boston"]

Xtal can be used with standard Tcl structures like lists and dictionaries as well.

% set L [list 1.01 10.1 .101 101]
→ 1.01 10.1 .101 101
% xtal::xtal { L[L > 1 && L < 100] }
→ 1.01 10.1

See the Xtal Quick Tour for a short overview of the language.

You can also use Xtal in interactive mode, either at the tclsh prompt, the wish console or tkcon. See The Xtal shell for details.

4. The tarray_ui package

The tarray_ui package implements widgets that useful for displaying and interacting with typed arrays. This includes widgets for graphs using the RBC package, displaying table data with sorting/filtering capabilities via the tktreectrl package, and for importing CSV files.

5. Documentation guide

Installation and build documents download and installation procedures.
Programmer’s guide is an introduction to programming using the tarray extension.
Tarray commands documents general commands in the tarray package.
Column commands documents commands specific to working with columns.
Table commands documents commands specific to working with tables.
Xtal commands contains reference documentation for Tcl commands related to the Xtal language.
The Xtal Language serves as both an introduction to Xtal as well as a reference documentation for the language itself.
The Xtal shell describes the use of the Xtal shell for interactive use.
Tk widgets documents the tarray_ui package.

6. License

The packages are primarily covered by the liberal open source BSD license. See the file license.terms in the source directories for details. Some third-party components used in the software may have their own license.