# Working With Data

## Supported File Formats

Currently, Kst supports ASCII text files, BOOMERANG frame files, and BLAST dirfile files as data sources, as well as PIOLib and FITS files with the appropriate libraries. This section describes basic data source concepts common to all file types, and specifically details the ASCII and BLAST dirfile file types.

### Data Source Concepts

A data source in &kst; is simply a supported data file. The following concepts are important in understanding how &kst; works with different data sources. Some terminology is also introduced in this section.

#### Samples

A sample is considered to be the fundamental unit with respect to data files. Each sample consists of one data value in the file. Note, however, that one sample may not correspond to one value in a data vector in &kst;, due to the concept of frames.

#### Fields

A field usually corresponds to one vector in &kst;. For example, a column in an ASCII data file is considered to be one field. Each field can have an explicit or implicit name. Datasource readers provide functions for reading and obtaining fields and field names.

#### Frames

A frame corresponds to a set number of samples, and each field in a data file can have its own ratio of samples per frame. The size of a data file is measured by the number of frames it contains. For ASCII data files, the number of samples per frame is 1, but for some data files, there may be multiple samples per frame. In the below illustration, the first 3 frames of an imaginary data file are shown. In this particular data file, Field 1 has a ratio of 5 samples per frame, Field 2 has a ratio of 2 samples per frame, Field 3 has a ratio of 3 samples per frame, and Field 4 has a ratio of 1 sample per frame.

Depending on the specific data vector settings in &kst;, data from files may be read as frames instead of samples, with either the first sample in a frame or the mean of all the samples in a frame representing the value of the frame.

#### INDEX Field

Some data files may not have a field that represents the x axis of a plot. However, &kst; implicitly creates an INDEX field for all data sources. The INDEX field simply contains integers from 0 to N-1, where N is the number of frames in the data file.

### ASCII Input Files

The simplest input file format is the ASCII text file. These files are usually human-readable and can be easily created by hand or simple scripts if desired. The following is an example of an ASCII input file.

    112.5 3776 428
    187.5 5380 429
    262.5 5245 345
    337.5 2942 184
    412.5 1861 119
    487.5 2424 138
    567.5 2520 162
    637.5 1868 144
    712.5 1736 211
    787.5 1736 211
    862.5 2172 292
    937.5 1174 377
    1000.5 499 623

Each column of this file represents a field, while each row represents one frame. Columns are separated by tabs or spaces, and rows are separated by carriage returns. Note that due to their structure, ASCII files cannot have multiple samples per frame. Also, since the columns do not have labels, field names are assigned by &kst; based on the order of the columns (the leftmost column has a field name of `1`).

Commented lines in ASCII files start with one of the characters in the set `{#, !, /, ;, c}`. All commented lines and empty lines are ignored by &kst;. Valid numbers include those with decimals, negative signs, or `e`, to indicate scientific notation. Invalid numbers (such as English words) are replaced with 0 by &kst;.

### BLAST Dirfiles

BLAST dirfile data sources are actually directories of files. Each directory represents one data source, and each file in the directory, with the exception of a file called `format`, represents a single field. The file named `format` lists each field and its properties. Below is example of such a file:

    scount RAW f 1
    fcount RAW f 20
    sine RAW f 20
    ssine RAW f 1
    cos RAW f 20

In this example, `scount`, `fcount`, `sine`, `ssine`, and `cos` are field names. `RAW` indicates the file is written in raw format, and the last number in each row is the number of samples per frame.

When selecting a BLAST dirfile for use in &kst;, the directory containing the field files should be selected. &kst; will when automatically look for a `format` file, if it exists, to determine the fields and their properties.

## The Data Wizard

The Data Wizard provides a quick and easy way of creating vectors, curves, power spectra, and plots in &kst; from data files. To launch the wizard, select Data Wizard... from the Data menu. You will be prompted to select a data source. Browse to a valid data file (or enter in the path to a data file) and click Next. The following window will be displayed.

Data Wizard Screen 2
Data Wizard Screen 2

Select the fields you wish to import into &kst;. You may filter the list of fields by entering a string to match (wildcards such as `*` are supported) into the text box above the list.

The Data Range section is used to specify the range of data to read from the selected vectors in the input file. The following discussion assumes knowledge of frames. For a detailed description of this concept, see File Formats.

Starting frame, Count from end, Number of frames, and Read to end

  Using these four settings, the lower and upper boundaries of the data range can be set. For example, to read from frame 10 to frame 900, enter `10` for Starting frame and `890` for Number of frames. To read from frame 500 to the end of the file, enter `500` for Starting frame and select the Read to end option. To read only the last 450 frames from the file, select the Count from end option and enter `450` for Number of frames. The combinations used in the previous two examples are often useful for reading data from a file that is being updated in real time. Subsequent additions to the file are read, causing associated vectors to be updated as well.

Read 1 sample per N frames and Boxcar filter first

  In addition to the lower and upper range boundaries, the samples to read from the selected range can be set. If Read 1 sample per N frames (where N is the entered value) is not selected, all samples in the selected range will be read. Alternatively, frames in the data file can be skipped by selecting Read 1 sample per N frames The value of the 1 sample that is used to represent a frame depends on whether or not Boxcar filter first is selected. If Boxcar filter first is not selected, the value is the same as the value of the 1st sample in the frame. If Boxcar filter first is selected, the value is the mean (average) of all the samples in that particular frame.

Power Spectrum and X axis settings can be specified within the Plot Types section. These settings are described below.

XY, Power Spectrum, and XY and Power Spectrum

  Select whether to plot the power spectrum only, data (XY) only, or both. If the power spectrum is selected for plotting, additional settings in this section become available.

X Axis Vector

  The vector to be used as the independent vector for the plots. You may select a field from your data file, or the INDEX vector. The INDEX vector is simply a vector containing elements from 0 to N-1, where N is the number of frames in the data file.

The FFT Options subsection in the Plot Types section is available only if a power spectrum is to be plotted. The below definitions assume basic knowledge of power spectra—for further details, refer to Numerical Recipes in C: The Art of Scientific Computing, published by Cambridge University Press.

Interleaved average and FFT Length

  The power spectrum is defined as the square root of the absolute value of the mean of the interleaved Fast Fourier Transforms of length `2^x`, where x is the value entered in the FFT Length selection box. Selecting Interleaved average allows the length of the interleaved Fast Fourier Transforms to be specified. If Interleaved average is unchecked, &kst; will determine the length based on the length of the vector.

Data units and Rate units

  The units specified in these textboxes are used for the purpose of auto-generating axes labels for the plots.

Sample rate

  The sample rate is used to generate the X axis of power spectrum plots.

Apodize

  If this option is selected, the data is apodized using a Hanning window, to reduce bin-to-bin leakage.

Remove Mean

  Select this option to remove the mean from the selected data (i.e. translate the data so that the mean is zero).

Once you are satisfied with all the settings, click Next to advance to the next window. Select any filters you wish to apply to the data, and click Next to advance to the final window.

Data Wizard Screen 3
Data Wizard Screen 3

From here, you can change general plotting settings. Most of the settings are self-explanatory. A brief overview of each section is provided below.

Curve Style

  Select whether to plot data points only, lines connecting the data points only, or both. By default, lines are continuous with weight 0, and data points are marked using crosses. Line and data point colours are chosen so that curves with identical colours are minimized. Note that the curve style settings apply to both Power Spectra and XY plots.

Curve Placement

  Select the plots to place the new curves on. Cycle through distributes the curves on the plots by cycling through the plots. Note that the curve placement settings apply to both Power Spectra and XY plots.

Label Generation

  Select the desired labels and legends to be placed on the plots.

Plot Placement

  Select the desired window(s) to place the new plots on. New windows can be created for the plots by selecting In new window.

Once you are satisfied with all the settings, click Finish and the plots will be generated.

## The Data Manager

The Data Manager provides a central location for adding, deleting, and modifying all the data objects used in &kst;. It can be accessed by selecting Data Manager from the Data menu.

### Creating New Data Objects

To create a new data object from the Data Manager, click on one of the buttons listed under New:. A new window will be displayed, allowing options and settings for the data object to be set.

You can also create new curves by right-clicking on a vector and choosing Make Curve...

Since you are creating a new data object, you may enter a unique name to identify the object. The default name can be used as well (if it is unique).

The settings for all plottable data objects except images have two common sections—Appearance and Placement (images only have the Placement section). These sections are described below. For data-specific settings, see Data Types.

#### Appearance

The Appearance section allows you to change the appearance of the data object when it is plotted.

  Clicking this button displays a colour chooser dialog box, which can be used to change the colour of both the lines and points.

Show points

  Selecting this checkbox shows the data points used to plot the data object. The drop-down list to the right allows the shape of the points to be changed.

Show lines

  Selecting this checkbox shows the lines joining the data points for the data object. The thickness of the line as well as the line style can be changed.

#### Placement

The Placement section specifies the plotting location of new plottable data objects.

Plot window

  Specify the window in which to plot the data object. You may create a new window by clicking the New... button.

Place in existing plot, Place in new plot and Do not place in any plot

  You can choose whether or not to immediately plot the data object on a new or existing plot. If Place in new plot is selected, you can also choose to re-arrange the layout of the plots by selecting re-grid and entering the number of columns to use for the new layout grid.

Once all the desired settings for the new data object have been set, click the OK button to create the new data object.

### Editing and Deleting Existing Data Objects

To edit an existing data object, simply highlight it in the Data Manager window and click the Edit button. The settings window for the selected object will be displayed. This window is identical to the one displayed when creating new data objects (with the exception that the Placement section may be absent), so refer to Creating New Data Objects for more information on the settings and options.

To delete a data object, highlight it in the Data Manager window and click the Delete button. Note that the entry in the # Used column for an object must be 0 before the object can be deleted. The # Used column indicates the number of times that a particular data object and its children (if any) are used by either other data objects or by plots. Listed below are some consequences of this restriction to keep in mind when attempting to delete data objects.

- All plottable objects (curves, equations, histograms, power spectra, and images) must be removed from plots before they can be deleted. An object can be removed from a plot by right-clicking on it in the Data Manager window and selecting the desired plot from the Remove From Plot submenu.

- All data objects that use a particular data vector must be deleted before the data vector can be deleted.

- All children of a parent data object must be unused before the parent data object can be deleted.

After a sequence of deletions and removals of plottable data objects from plots, you may find that there are numerous unused data objects displayed in the Data Manager. To quickly remove these objects, you can click the Purge button.

## Data Types

There are nine main types of data objects in &kst;. Data objects can contain other data objects, as represented by the tree structure view in the Data Manager. The following diagram illustrates the relationships between the different data types.

&kst; Data Types

As can be seen from the above diagram, the curve, equation, histogram, power spectrum and image data objects are the only data objects that are plottable. All data objects (other than vectors) have the capability of using vectors, while equations, power spectra, events, and plugins all contain slave vectors.

Descriptions of each data type are provided below, along with overviews of the settings and options available when creating or editing each type of data object. Settings common to almost all data types have been omitted—see Creating New Data Objects for more information.

### Vectors

Vectors are one of the most often-used data objects in &kst;. As their name implies, vectors are simply ordered lists of numbers. Most often they contain the x or y coordinates of a set of data points.

As vectors can be potentially quite large, it is a good idea to be aware of the amount of memory &kst; has allocated for use. The current available memory is displayed in the lower right corner of the status bar of the main &kst; window.

If the status bar is not available, ensure that Show Statusbar is checked in the Settings menu.

The following is a screenshot of the window displayed when creating or editing vectors. Explanations of the settings are listed below.

Edit Vectors
Edit Vectors

#### Data Vector Parameters

The source file and field to read can be set using the following options.

File name

  The path to the desired data file. Clicking the button to the right displays a file browser window that can be used to graphically browse to the file.

Field or column

  The field or column to create a vector from.

#### Data Range

This section specifies the range of data to use from the selected field for the data vector.

Starting frame, Count from end, Number of frames, and Read to end

  Using these four settings, the lower and upper boundaries of the data range can be set. For example, to read from frame 10 to frame 900, enter `10` for Starting frame and `890` for Number of frames. To read from frame 500 to the end of the file, enter `500` for Starting frame and select the Read to end option. To read only the last 450 frames from the file, select the Count from end option and enter `450` for Number of frames. The combinations used in the previous two examples are often useful for reading data from a file that is being updated in real time. Subsequent additions to the file are read, causing associated vectors to be updated as well.

Read 1 sample per and Boxcar filter first

  In addition to the lower and upper range boundaries, the samples to read from the selected range can be set. If Read 1 sample per is not selected, all samples in the selected range will be read. If Read 1 sample per is selected, only 1 sample per `N` frames will be read, where `N` is the number entered in selection box to the right. The value of the 1 sample that is used to represent a frame depends on whether or not Boxcar filter first is selected. If Boxcar filter first is not selected, the value is the same as the value of the 1st sample in the frame. If Boxcar filter first is selected, the value is the mean (average) of all the samples in that particular frame.

### Curves

Curves are primarily used to create plottable objects from vectors. Curves are created from two vectors - an X axis vector and a Y axis vector, that presumably provide a set of data points. Thus, a curve can be thought of as a set of data points and the lines that connect them (even though the points or lines may not be visible on plots).

The following is a screenshot of the window displayed when creating or editing curves. Explanations of the settings are listed below.

Edit Curves
Edit Curves

#### Curve Contents

The contents of the curve can be set from this section.

X axis vector

  The vector to use for the independent (horizontal) axis.

Y axis vector

  The vector to use for the dependent (vertical) axis.

+X error bar and +Y error bar

  The vectors containing error values corresponding to the X axis vector and Y axis vector, respectively. The vectors should contain the sizes of the error bars in the same order as the data points.

Use +X/Y error bar for -X/Y

  If this item is checked, the error bars are drawn symmetrically about the data point. To draw asymmetric error bars in the x or y direction, uncheck the box, and choose a vector for the length of the error bar below the data point in the -X/Y error bar combo box.

-X/Y error bar

  The vector to use for the error bar below the data point. See the previous entry for more detail on using asymmetric error bars.

### Equations

An equation data object consists of a mathematical expression and an independent variable. The expression is built using a combination of scalars, vectors, and operators, and usually represents the values of the dependent variable. The independent variable can be an existing vector or it can be generated when the equation object is created or edited. Since an equation ultimately consists of a set of data points, an equation object is plottable.

The following is a screenshot of the window displayed when creating or editing equations. Explanations of the settings are listed below.

Edit Equation
Edit Equation

#### Equation

The mathematical expression representing the dependent variable can be modified from this section.

Operators

  A list of available operators. Choosing an operator from the list inserts the operator at the current insertion point in the Equation text box.

Vectors

  A list of the current vector objects in &kst;. Choosing a vector from the list inserts the vector at the current insertion point in the Equation text box.

Scalars

  A list of the available scalar values. This list is composed of both the scalar values in the current &kst; session as well as some built-in constants. Choosing a scalar from the list inserts the scalar at the current insertion point in the Equation text box.

Equation

  The mathematical expression representing the independent variable. You may manually type in this text box or you may select items to insert using the above drop-down lists.

#### Independent Variable

This section is used to specify the source of the values for the independent variable.

Use

  Select this option to use an existing vector as the independent variable. Select a vector from the drop-down list, or quickly create a new one by clicking the button to the right.

Interpolate to highest resolution vector

  Selecting this option interpolates the selected vector to the greatest number of samples possible.

Generate

  Select this to generate a set of values to use as the independent variable. Specify the lowest value to generate in the From field, and the highest value to generate in the to field. Set the value for Number of samples to be the number of equally spaced values to generate between the lowest value and the highest value (inclusive).

### Histograms

A histogram data object simply represents a histogram of a particular vector. Histogram objects are plottable.

The following is a screenshot of the window displayed when creating or editing histograms. Explanations of the settings are listed below.

Edit Equation
Edit Equation

#### Histogram Properties

The source vector, as well as basic histogram properties, can be modified from this section. You can either manually specify settings or use the Auto Bin button.

Data vector

  The data vector to create the histogram from. Although a vector is needed to create a histogram, the vector is treated as an unordered set for the purposes of creating a histogram.

From and to

  The From field contains the left bound for the leftmost bin in the histogram. The to field contains the right bound for the rightmost bin in the histogram.

Num bins

  Enter the total number of bins to use in the Num bins field.

Auto Bin

  Instead of specifying values for From, to, and Num bins, you can click Auto Bin to automatically generate values for all three fields based based on the lowest and highest values, as well as the number of elements found in the source vector.

#### Y Axis Normalization

This section is used to specify the type of normalization used for the y axis of the histogram.

Number in bin

  Selecting this option causes the y axis to represent the number of elements in each bin.

Fraction in bin

  Selecting this option causes the y axis to represent the fraction of elements in each bin out of the total number of elements.

Percent in bin

  Selecting this option causes the y axis to represent the percentage of elements (out of the total number of elements) in each bin.

Peak bin = 1.0

  Selecting this option causes the y axis to represent the number of elements in each bin divided by the number of elements in the largest bin (the bin with the greatest number of elements). In other words, the y axis is normalized to 1.0.

### Power Spectra

A power spectrum data object represents the power spectrum of a vector, defined as the square root of the absolute value of the mean of the interleaved Fast Fourier Transforms of length `2^x` of the vector, where x is the value entered in the FFT Length selection box. The below definitions assume basic knowledge of power spectra—for further details, refer to Numerical Recipes in C: The Art of Scientific Computing, published by Cambridge University Press.

Power Spectra Window
Power Spectra Window

#### Curve Contents

The source vector, as well as basic power spectrum properties, can be modified from this section.

Data vector

  The data vector to create a power spectrum from.

Interleaved average and FFT Length

  Selecting Interleaved average allows the length of the interleaved Fast Fourier Transforms to be specified. The length is specified as a power of 2. If Interleaved average is unchecked, &kst; will determine the length based on the length of the vector.

Data units and Rate units

  The units specified in these textboxes are used for the purpose of auto-generating axes labels for the plots.

Sample rate

  The sample rate is used to generate the X axis of power spectrum plots.

Apodize

  If this option is selected, the data is apodized using a Hanning window, to reduce bin-to-bin leakage.

Remove Mean

  Select this option to remove the mean from the selected data (i.e. translate the data so that the mean is zero).

### Plugins

A plugin data object represents a &kst; plugin. All plugins have a common format, and show up as type Plugin in the Data Manager. For more information about plugins, please see Plugins and Filters

### Event Monitors

An event monitor data object represents an instance of an event monitor. For more information on monitoring of events, see Event Monitoring.

### Matrices

A matrix represents a set of three-dimensional data points (x, y, z) arranged in a two-dimensional grid. The z values of the matrix are obtained from a vector, and the grid structure is manually specified using the Edit or New Matrix dialog. The descriptions below refer to the following diagram depicting &kst;'s matrix structure.

A screenshot and explanation of the matrix dialog follow.

Matrix Window
Matrix Window

#### Vector Selection

Z Vector

  The data vector to obtain the z values from.

#### Grid Parameters

X Minimum and Y Minimum

  The origin of the matrix is specified by the coordinates (X Minimum, Y Minimum). The location of the origin is represented by a red circle in the above diagram.

X Step Size and Y Step Size

  These two values specify the dimensions of each rectangular cell in the matrix grid. All cells in the matrix have identical dimensions.

Dimensions and Use maximum x length

  Enter the number of steps for the x dimension of the matrix, followed by the number of steps for the y dimension of the matrix. If Use maximum x length is checked, the x dimension of the matrix will be determined based on the length of the vector and the entered y dimension. If this option is checked and the length of the vector subsequently changes, the x dimension of the matrix will be updated accordingly.

  Note that the minimum allowed y dimension is 1, while the minimum allowed x dimension is 0.

### Images

Images are graphical representations of Matrices. Images can be plotted as color maps, contour maps, or both.

Image Window
Image Window

#### Matrix Selection

Matrix

  Select the matrix to use for this image. New matrices can be created by clicking on the button to the right.

#### Image Type

Color Map, Contour Map, and Color Map and Contour Map

  Select the type of image to be plotted. Changing this selection enables or disables sections of the image dialog as appropriate.

#### Color Map Parameters

A color map represents the z value of each cell in the matrix using a color. This section is only available if Color Map or Color Map and Contour Map is selected under Image Type.

Color palette

  Select the color palette to use for the color map. By default, two palettes suitable for color maps are installed by &kst;—Kst Grayscale 256 (a 256 color grayscale palette) and Kst Spectrum 1021 (a 1021 color rainbow spectrum that ranges from blue to red). Additional palettes can be installed by simply copying GIMP compatible palette files to the `colors` subdirectory of the user configuration directory (for example, `/usr/share/config/colors/`). Note that saved images using a non-default palette may not be viewable by other users of &kst; if they do not have the required palette. In such cases, a default grayscale palette is used instead.

Lower Z threshold, Upper Z threshold and Auto Threshold

  Enter the lower and upper thresholds to use for the mapping of colors. Palette colors are evenly distributed between Lower Z threshold and Upper Z threshold. Any cells in the selected matrix with z values less than Lower Z threshold are mapped to the first color in the palette. Any cells in the selected matrix with z values greater than Upper Z threshold are mapped to the last color in the palette. Clicking Auto Threshold will fill in the lower and upper threshold values using the least and greatest z values found in the selected matrix.

#### Contour Map Parameters

A contour map draws a set of lines, with each line representing a certain z value. This section is only available if Contour Map or Color Map and Contour Map is selected under Image Type.

Number of contour levels

  Select the number of contour levels to use. The contour levels will be distributed evenly between the lowest and highest z values found in the matrix.

Color

  Select the color to use for the contour lines. Clicking this button displays a standard &kde; color chooser dialog.

Weight and Use variable line weight

  Select the weight, or thickness of the contour lines. If Use variable line weight is selected, contour lines representing higher elevations are drawn thicker than those representing lower elevations. Use discretion when selecting this option, as images with high contour line densities may become unreadable.

## The Data Menu

The Data menu provides quick access to many features related to data objects in &kst;. Most of the menu functions duplicate functions found elsewhere, so only brief descriptions are provided below.

Reload

  Reloads all data vectors from their source files.

Data Wizard...

  Displays the Data Wizard.

Data Manager

  Displays the Data Manager.

New [data object]

  Displays the corresponding dialog for creating the data object. Refer to Data Types for descriptions of each dialog.

View Scalars

  Displays a dialog from which the values of all the scalars in the current &kst; session can be viewed. The dialog is updated dynamically if the values change.

View Vectors

  Displays a dialog from which the values in all the current vectors can be viewed. Select a vector to view using the drop-down list. The dialog is updated dynamically if the values change.

View Fit Results

  Displays a dialog which shows all the resulting values from performed fits. Select a fit result to view using the drop-down list. The dialog is updated dynamically if the values change.

Change Data File

  Displays a dialog for quickly changing data files that vectors are associated with. Select the vectors to change, and then browse to a different data file. Click Apply to save the changes.