3.2. Creating A Pipeline

In DREAM3D-NX, the user processes their data by creating what is known as a Pipeline which is constructed out of a series of Filters. By chaining together a series of Filters, the underlying data structure is processed and modified in quick succession. The user should be aware that all processing on the data is done in place, i.e., there is only a single copy of the data resident in memory at any one time. While this reduces the overall memory footprint, the user will have to be aware that their data may be modified from its original form. To help the user avoid accidentally overwriting data, all objects at a given hierarchy level in the Data Structure are not allowed to share the same name. For example, a data structure can have an Attribute Matrix named foo that holds a Data Array named bar, but if the user tries to create a new Attribute Matrix named foo, an error will be presented in the Pipeline Error Table. The user is allowed to create another Data Array named bar, however, as long as that Data Array belongs to a different Attribute Matrix. In other words, no two children of a given data structure object can share the same name!

Building a Pipeline

The interface mechanics of building a Pipeline are discussed in the Pipeline View section. The concepts behind creating a Pipeline are presented here. In an abstract sense, a Pipeline can be thought of as a series of tubes, where the individual pieces of pipe are the Filters. The fluid traversing the Pipeline is the underlying data structure. The Filters can change the overall direction of the data structure, compress it, morph it, modify it, and add to it, until finally the data structure comes out the other side. Each individual Filter will require certain kinds of inputs, which means that some Filter further up in the Pipeline must, at some point, create those necessary pieces of the data structure. DREAM3D-NX alerts the user if any of these pieces are missing by executing a process called Preflight. Think of Preflight as compiling a computer program: any compiler errors are populated in the Pipeline Issues View table. Using this feedback, a user can modify the parameters of existing Filters and add new Filters to the Pipeline until all errors are silenced. It is then safe to run the Pipeline by clicking Run! While the Pipeline is running, you can click on Filters in the Pipeline View and inspect the items in the Filter Parameters View. Also, the Run button will change to a Cancel button when a Pipeline is running. Clicking Cancel will stop the Pipeline execution. Note that some Filters may not check for this Cancel click at any point during their operation. If this is the case, the Pipeline may wait to quit until after the current Filter has finished running.

But how do you know what kind of Pipeline to create? What are the primary ingredients of a useful Pipeline? First, it is important to become familiar with the DREAM3D-NX ontology. The best place to start is by reading and understanding the DREAM3D-NX Data Structure section. The terms presented to the user on Filters will use the same language as explained in the DREAM3D-NX Data Structure section. Once you are familiar with the DREAM3D-NX language, understanding the functions of Filters becomes much easier. The filter documentation is also useful for getting to know specific Filters.

When constructing a Pipeline, try to think about your data in terms of the DREAM3D-NX language. What kind of Geometry are you working? Is it a 2D picture made up of pixels (an Image Geometry)? A triangulated surface mesh (a Triangle Geometry)? What are the unit elements of your Geometry? More importantly, where on your Geometry does your data lie? The dimensional topology of the Elements the data lie on determines the type of Attribute Matrix to which that data will ultimately belong. For example, a 3D stack of images with grayscale data at every voxel would have a grayscale Data Array located in a Cell Attribute Matrix.

Once you’ve translated your data into the DREAM3D-NX ontology, you can start thinking about constructing a Pipeline to do something interesting. All Pipelines will need to start by creating an instance of the DREAM3D-NX Data Structure in some fashion. This is usually accomplished by reading in some data from a file. Next, you’ll want to string together Filters to accomplish an analysis. This is the interesting part about building a Pipeline. You can begin analyzing the underlying data that lies on your Geometry, or think about grouping Geometry elements to form Features. For example, in the 3D grayscale image stack example, you may wish to group all elements into two classes by applying a threshold. This would group the Geometry elements into 2 Features. There are a host of Filters used to analyze your newly created Features, calculating information like their sizes and shapes (and much more!). At all steps in creating a Pipeline, try to think about your data using the abstract DREAM3D-NX ontology. This will not only help you in understanding what Filters do, but also allow you to remain flexible. Be imaginative!

After you’ve constructed the analysis portion of a Pipeline, you’ll probably want to export your new data structure to your disk. In particular, the native .dream3d file format is very useful, since it was designed to handle the abstract data structure. Also, the file is based on open source HDF5, meaning anyone can read the data!