Home Introduction Capabilities News Software Examples FAQ Standard Licence Contact

Fielded Text:

Introduction

What is the Fielded Text standard all about?

The ultimate aim of the Fielded Text Standard is to make it a lot easier to use text files to store or distribute tables of values.

Currently files such as Comma Separated Value (CSV) text files are often used to transfer this type of data.  Their advantage is that they are easy to understand, can be viewed with a plain text editor, and programmers can access the data without any special run-times.  Their disadvantage is that everyone uses different structures and formats for storing their data in text files.  Hence programmers invariably have to write a new parser whenever they want to extract data from a text file from a new supplier.

Fielded Text takes a new approach when working with text files containing tables of values!

Fielded Text allows you to associate a Meta File with a text file.  This Meta file is a small XML file which describes the structure and format of data within the text file.  More importantly, it allows you to access the data in the file in the same way you access records in a database table.  It combines the simplicity of text files with the convenience of database access.

A full description of how the Fielded Text standard enables this can be found in the Standard page on this website.  However if you are looking for a concise description, read on.

Capabilities

The Fielded Text standard aims to be compatible with all text files today that contain tables of values.  In support of this aim, it has a set of capabilities which should cover nearly all structures and formats used in existing text files.  These capabilities are summarised here.

Creating Meta Files

There are several ways of creating a Meta file for a FieldedText file:

1. Use a text editor.  Meta files are XML files so, provided you know the tags and schema, you can use any text editor (or XML file editor) to create it.
2. Use a Fielded Text Editor.  This is the easy way.  A Fielded Text Editor is a specialised text editor for working with Fielded Text files.  With a Fielded Text Editor you can load a sample text file and then visually set the Meta properties for it.  The editor will interactively show you any parsing errors arising from incorrect properties.  Once the properties are correctly set, you can export them into a Meta File.
   
   A list of Fielded Text editors is here - including at least one free one.
3. Programatically.  Fielded Text software components can also be used to generate Meta files programatically.  This method will be used in specialised Fielded Text applications (for example, Fielded Text Editors).

Meta files typically have a file extension of "ftm".

Declared and Undeclared Fielded Text Files

Fielded Text files can be either declared or not declared (undeclared).

A declared Fielded Text file has 2 special lines at the start of the file.  These 2 lines are called the file declaration.  The declaration contains a marker which identifies the file as a Fielded Text file and it specifies what version of the Fielded Text Standard the file conforms to.

In addition, the declaration can also specify the Meta file which this text file is associated with.  It can specify the Meta with:

• A URL pointing to the Meta file.
• The file path of the Meta file.
• A flag indicating that the Meta is embedded in the text file.

Declared Fielded Text files remove the need for end users to match a text file with the Meta.  This makes them more reliable as users are less likely to parse or interpret the data incorrectly.  For example, organisations publishing data, can place a copy of the Meta on their website and have the downloadable data text files reference it.  End users can then simply download the text file and the parser will automatically know how to obtain the correct Meta.

Nearly all existing text files containing tables of values will be 'Undeclared' Fielded Text files.  They can be handled exactly the same way as Declared Fielded Text files however the text file will need to be explicitly associated with its Meta.

Basic Example

Below is a basic CSV file. It has 2 heading lines and 4 data lines. The lines contain 7 fields of various types.

The following Fielded Text Meta file specifies the structure and layout (schema) of the above text file.

Following is a Declared Fielded Text file which contains the above CSV text together with the its meta embedded as comments. The ~ character specifies a comment line.

Fielded Text file Structure

A Fielded Text file consists of 2 main parts: Header and Body.  The Body contains the lines which hold the data (the records).  The Headers consist of all the lines prior to the Body (including heading lines).

At a more detail level, the header part of the file can be split into the following sections:

• Declaration.  The first 2 lines of a Declared Fielded Text file.  The first line of the Declaration begins with the comment character (can be any character) followed by the marker: "|!Fielded Text^|" (or "|>Fielded Text<|" for backwards compatibility).  The presence of this marker denotes that this file is a Declared Fielded Text file.
• Comments.  This section can be present after the declaration (if it is present) or at the start of the header part.  It consists of a block of comment lines.  A comment line is a line beginning with the "Line Comment" character.  It is ignored when parsing the file.  The line comment character is either the first character in a Declared Fielded Text file or it is specified by the Meta.
• Embedded Meta.  If the Delaration section specifies that the Meta is embedded in the file, then it will be included as a block of comment lines in the header. The Meta itself is a XML data stream.  In order to embed it, each line in the data stream is prefixed with the line comment character and some (optional) white space characters. The data stream can then be embedded in the header as a block of comment lines.
• Comments.  After the Embedded Meta section, it is possible to have another block of comment lines.
• Headings.  This section will only occur if the Meta specifies that a text file contains heading lines.  If heading lines are included, then the first line which is not part of the declaration and does not begin with a line comment character, will be the first heading line.  The number of heading lines is specified in the Meta.  Note that comment lines cannot be included in the headings section (ie lines beginning with the line comment character will not be treated as comment lines).  Also note that blank lines are not ignored in this section.

The body part of the text file begins either:

• after the last heading line; or
• if there are no heading lines, then after the last comment line in the header; or
• if there are no comment lines as well, then after the declaration section; or
• if there is not declaration section as well, then at the beginning of the file.

The record part can contain record lines and comment line (and possibly ignored blank lines).  A record line contains the actual data - ie. a row of values.  Each record line consists of a sequence of field values.  The format of these field values is specified by the Meta.  The Meta also specifies the structure of the record lines, including how field values are separated.

It is possible for a record to span multiple lines in the text file.  This will occur when a field value contains an "End of Line" character(s).  If a record does span multiple lines, then any line in that record will not be treated as comment line or an ignored blank line.  Accordingly, it is possible for lines in the body part to begin with a line comment character but not be treated as a comment line.

Fielded Text Meta file Structure

The Meta contains the following groups of information:

Main Section which specifies properties applying to the whole text file.  In the above example Meta file, the attributes in the <FieldedText> element apply to the whole text file and make up the main section.  The Main Section can contain the following properties/attributes:

Field Sections which specify the properties of each field of data used within the text file.  In the above example Meta file, the attributes in a <Field> element apply to the respective field and make up a field section.  A Field Section can contain the following properties/attributes:

Fields can have a DataType of: String, Boolean, Integer, Float, Decimal (similar to Float but better suited for financial calculations) or DateTime.  Some of the attributes listed above are not applicable to all field DataTypes and some use different values in different DataTypes.

Substitution Sections specify which substitutions are used within the text file. Substitutions are similar to Escape Sequences used in some CSV files (eg \n).  A Substitution Section can contain the following properties/attributes:

Each Sequence has a series of <Item> elements which specify the fields included in the sequence.  An <Item> element can contain the following properties/attributes:

An <Item> element can also contain a series of <Redirect> elements.  The <Redirect> elements determine which sequence should be invoked if a field contains specified values.  A <Redirect> element can contain the following properties/attributes:

Sequence Sections.  A Fielded Text file can have lines with different sets of fields depending on the value of a key field(s).  The Sequence Sections in the Meta File specify the sequence of fields which can follow a key field.  A Sequence Section can contain the following properties/attributes: