Chapter III.3. Defining a data processing workflow and working with files

Table of Contents

III.3.1. A custom workspace
III.3.2. Preparing the HDF5 file
III.3.3. Testing the workspace to display our data

In the previous chapter, we have built a small bunch of paths and nodes, and used them to process some data. In practical situations, things can become much more complex, and it is desirable to separate an application into several entities, each of them responsible for a specific task, and therefore having its own bunch of transforms defined. These entities are indeed called "bunches", but most of the times it is better to work with the derived class "workspace" which has some additional abilities like retrieving parameters from the command line.

III.3.1. A custom workspace

In this section we want to design a custom workspace to do the following tasks :

  • load a 2D array from an HDF5 file
  • display a colour rendering of the array

A node always belongs to a bunch. By default, nodes are attributed to the "root" workspace, which is managed internally by the library, and can be accessed through the workspace::root() static member function. New workspaces are commonly created by extending the workspace class. Here is the declaration for our brave renderer, to be put in hdf5_renderer.hpp:

#include "transform/io/hdf5_reader.hpp"
#include "coefficients/coefficients.hpp"
#include "arrays/cartesian/local_geometry.hpp"
class hdf5_renderer : public spectral::workspace
{
public :
	typedef spectral::coefficients<spectral::array<spectral::cartesian::local_geometry<2> > > Coefs_t;
	typedef spectral::hdf5_reader<Coefs_t> Reader_t;
	typedef Reader_t::Source_t HDF5_Group_t;
	
	/** Constructor with parameters passed directly to base class */
	hdf5_renderer(Params_t params = Params_t(), workspace* parent = 0, std::string base_name = "hdf5_renderer");
	
	/** Late initialization method */
	void initialize();
	
	/** Call this to update the rendered image. */
	void display();

	/** Returns a reference to the HDF5 group */
	inline HDF5_Group_t& group() const { return m_group; }

private :
	HDF5_Group_t m_group;
};		
	

Here, we had to include the typedefs for Reader_t and HDF5_Group_t in the class declaration, in order to declare the member variable m_group that we will need later on. We chose to make those typedefs public so that the outside world has a convenient way to know what kind of data we are expecting.

We now turn to the definitions, to be put in hdf5_renderer.cpp Let us give right away the constructor's:

hdf5_renderer::hdf5_renderer(Params_t params, workspace* parent, std::string base_name)
	: workspace(params, parent, base_name), m_group(this, "source_file")
{
	typedef spectral::draw2d<Coefs_t,spectral::image_renderer> Draw_t;
	
	Draw_t().dig(Reader_t().dig(m_group));
}
	

We see that the three parameters are simply passed to the base class. We need not worry too much about them for now. The null default value for the second argument means that the parent is the root workspace. What is more interesting is the initialization of the m_group member by calling a constructor with two arguments. The first argument is the bunch who owns the node: ourself. The second argument is a descriptive name that we want to give to this node.

In the constructor's body, we build our little bunch of transforms, as in chapter 1, but here we prefer to call the overloaded version of transform::dig that only takes one argument: the source node. That makes it is very easy to build a chain of several transforms without bothering too much about the intermediate states. These will be stored internally as nodes in the hdf5_renderer workspace, exactly like m_group, but we simply do not need an explicit variable pointing to them.

The call to Reader_t::dig is always licit because the node m_group is of type HDF5_Group_t, which was precisely defined as the source node type for Reader_t. But what does this node represent ? The group in the HDF5 file that contains our data. You can read more about our conventions for organizing HDF5 files here.

It is possible to set the color palette and rendering resolution by passing arguments to the Draw_t constructor, as explained in the VTK interface documentation.

hdf5_renderer::initialize()
{
	typedef HDF5_Group_t::parameter_type param;
	typedef spectral::make_hdf5_file<engine_traits<Coefs_t>::geometry_type>::type File_t;
	
	param.get<0>() = this->get<File_t>().file_id();
	param.get<1>() = "test_image";
	
	m_group.initialize(param);
	m_group.propagate();
}
	

As before, to initialize a node we first retrieve its parameter type, and we pass an object of this type to the "initialize" member function. It turns out that the parameter type for HDF5 groups is a boost::tuple with 3 elements, of which we only need to set 2 here.

The first is the HDF5 identifier of the file where the group is located. It should be a properly opened HDF5 file. Here, we use the workspace's slot manager to avoid having to handle the opening and closing of the file ourself. See this page for more details.

The second parameter element is the name of the HDF5 group. If no group exists with the specified name, one will be created on initialization. When calling propagate, the size of the HDF5 dataset will automatically be retrieved from the file, memory will be allocated to read it, and a VTK render window will pop-up, ready for rendering. The instruction triggering the actual rendering belongs to the hdf5_renderer::display member function:

hdf5_renderer::display()
{
	m_file.inject();
}
	

III.3.2. Preparing the HDF5 file

We now need a way to write some data to an HDF5 file so that we can test our little renderer workspace. We are going to do it from a different program, so let's open another cpp file and write:

#include "hdf5_renderer.hpp"
#include "transform/io/hdf5_writer.hpp"
#include "coefficients/algebra.hpp"
int main(int argc, char** argv)
{
	using namespace spectral;
	workspace::initialize(argc,argv);
	
	typedef hdf5_writer<hdf5_renderer::Coefs_t> Writer_t;
	hdf5_renderer::Coefs_t data("test_image");
	typedef engine_traits<hdf5_renderer::Coefs_t>::geometry_type geometry_type;
	geometry_type::global_domain_type domain;
	domain[0] = 128;
	domain[1] = 128;
	
	// initialize
	data.initialize(geometry_type(param));

	// define a constant field with the value "2"
	data = 2.0;
		
	// prepare
	Writer_t().dig(data);
	
	// write
	data.push();
	
	workspace::finalize();
	return 0;
}		
	

When you compile and run this program, it should create a file named root.0.l.hdf. You can check with hdfview that this file indeed contains a single group named "test_image", which in turn contains a single dataset of the right size and datatype.

III.3.3. Testing the workspace to display our data

From yet another cpp file, we can now use the workspace we created earlier to read data from the HDF5 file and display it on the screen.

#include "hdf5_renderer.hpp"
int main(int argc, char** argv)
{
	using namespace spectral;
	workspace::initialize(argc,argv);
	
	hdf5_renderer r;
	r.initialize();
	r.display();
	
	workspace::finalize();
	return 0;
}		
	

That's it ! However, before running this program, we must be careful about renaming root.0.l.hdf with the correct name. Since we are reading from the default file associated with the workspace hdf5_renderer, and since hdf5_render is the first workspace we create in the program, the filename will be hdf5_renderer.1.l.hdf. If everything goes smoothly you should see a window pop-up with a large red square occupying most of its area, and a red colorbar at the top of it.