Blobs

Motivation

A few application developers have requested the capability to store an arbitrarily-sized object in an exodus file. The size of the object is not related to any of the existing exodus entities -- nodes, elements, sides, edges, or faces. An example of a blob would be an arbitrary vector of floating-point values; the coefficients of a solution matrix.

The blob concept would allow one or more arbitrarily sized objects to be defined. The object would have:

Size
The type would correspond to the native floating point type specified for the exodus file (float or double)
Unique id (required, positive, unique among blobs)
Unique name (required, unique among all blobs/blocks/sets/assemblies)
Zero or more named attributes.
- Attribute is per blob, not a value for each entry in the blob
- attribute can be of type -- integer, char* double
- if type is integer or double, then there can be multiple values per attribute.
Transient data -- zero or more named floating point values which are output on each timestep.
- Value for each item in the blob for each variable
- Names should be unique across all transient reduction variable names in the model.
Transient Reduction Data -- zero or more named floating point values which are output on each timestep.
- Single value per timestep per blob for each variable
- Names should be unique across all transient reduction variable names in the model.
Zero or more "named maps" can be specified
- The map will have a size equal to the size of the blob
- The map entries will be 32- or 64-bit integers (based on exodus integer size)
- The value of the entries in the map(s) is arbitrary and it is the responsibility of the writer/reader to agree on the map conventions and the meaning of the values.

Parallel Considerations

Client will be responsible for handling all parallel issues.
Assume that blobs will only be used in the "1->N->1" mode -- parallel application reads and writes from/to a single file on a parallel filesystem.
Will need to provide partial read/write functions for variables and maps so client can efficiently read/write blob data in a parallel execution.

C API

Determine number of blobs on an exodus file:

 int num_blob = ex_inquire_int(exoid, EX_INQ_BLOB);

Define number of blobs which will be defined in an exodus file:

The number of blobs does not need to be specified prior to defining an blob.

Get / Put names of all blobs or an individual blob:

Blob names are set and queried via the ex_put_blobs and ex_get_blobs functions. They are somewhat different from other object types in that the names are not defined via the ex_put_name calls. However, the names can be queried using ex_get_name() although it is recommended to use the ex_get_blob() function.

 ex_get_name(exoid, EX_BLOB, blob_id, name);
 ex_get_names(exoid, EX_BLOB,  names[]);

Define and output one or more blob(s)

typedef struct ex_blob
{
  int64_t        id;
  char           *name;
  int64_t        size;
} ex_blob;

ex_put_blob(exoid, ex_blob blob);
ex_put_blobs(exoid, num_blob, ex_blob *blobs);

Defines and outputs an blob with id id and name name of size size entries.

NOTE: Efficiency -- it is more efficient to define multiple blobs via a call to ex_put_blobs() then to call ex_put_blob() multiple times.

   /* conceptual; not real C code */
   ex_blob blob[3];
   blob.id = {11, 22, 3};
   blob.name = {"Fred", "Wilma", "BamBam"};
   blob.size = {200000, 100000, 2000000000};
   ex_put_blobs(exoid, 3, blob);

Read all blobs

   int num_blob = ex_inquire_int(exoid, EX_INQ_BLOB);
   int64_t ids[num_blob];
   ex_get_ids(exoid, EX_BLOB, ids);

   int name_len = ex_inquire_int(exoid, EX_INQ_DB_MAX_USED_NAME_LENGTH);
   struct ex_blob blobs[num_blob];
   for (int i=0; i < num_blob; i++) {
      blobs[i].name = malloc(name_len + 1);
   }

   /* Get parameters for all blobs */
   ex_get_blobs(exoid, blobs);

Define/Query blob attributes and attribute names

An blob attribute is similar to an IOSS property consisting of a name, a type, and a value or values. It is not a value per entity in the blob, but a value for the blob. For now, they types will be limited to text, integer, and double to provide capability without the complexity of supporting the many types available in NetCDF-4 including user-defined types. Note that an attribute can have multiple values, for example if the attribute is a range, a single attributed could have two double values -- {1.0, 100.0}

NOTE: This type of attribute (value on entity instead of value per entities members, for example nodes in a nodeset) has also be added to the other entity types (blocks and sets).

NOTE: Need a better name or way of distinguishing from the attributes which are currently supported in Exodus.

define and output an attribute

   ex_attribute attribute = {EX_BLOB, 100, "Offset", EX_DOUBLE, 3, {1.1, 2.2, 3.3}};
   ex_put_attribute(exoid, attribute);

define and output multiple attributes

   ex_attribute attributes[10];
   /* ... Initialize `attributes` */
   ex_put_attributes(exoid, 10, attributes);

define and output a double attribute

   ex_put_double_attribute(exoid, EX_BLOB, id, name, num_values, values);

define and output an integer attribute

   ex_put_integer_attribute(exoid, EX_BLOB, id, name, num_values, values);

[The size of the integers used in values is based on int-size of the database]
define and output a text attribute....

   ex_put_text_attribute(exoid, EX_BLOB, id, name, char *values);

Query number of attributes on an blob

   int num_attr = ex_get_attribute_count(exoid, EX_BLOB, id);

Query names, types, size/length, values of all attributes on an blob

   int num_attr = ex_get_attribute_count(exoid, EX_BLOB, id);
   ex_attribute attributes[num_attr];
   ex_get_attributes(exoid, EX_BLOB, id, attributes);

Get values of all attributes on the specified entity (BLOB 100)

   int count = ex_get_attribute_count(exoid, EX_BLOB, 100)
   ex_attribute attributes[count];
   memset(attributes, 0, sizeof(ex_attribute)*count);
   ex_get_attribute_param(exoid, EX_BLOB, 100, attributes);
   /* Get attribute values for all attributes listed in `attributes`
      `ex_get_attributes()` will allocate memory for `attributes[i].values` 
      which must be freed by caller.
    */
   ex_get_attributes(exoid, count, attributes);

Get value of specific attribute on the specified entity (BLOB 100)

   int count = ex_get_attribute_count(exoid, EX_BLOB, 100)
   ex_attribute attributes[count];
   ex_get_attribute_param(exoid, EX_BLOB, 100, attributes);
   /* ... allocate space on specific attribute to store values. If
    *     not allocated, it will be allocated by `ex_get_attribute()`
    */
   attributes[3].values = calloc(attributes[3].value_count, sizeof(double));
   /* Get attribute values for attribute at index 3 */
   ex_get_attribute(exoid, attributes[3]);

The ex_attribute argument is the struct:

typedef struct ex_attribute
{
  ex_entity_type entity_type,
  int64_t   entity_id,
  char  name[EX_MAX_NAME];
  ex_type type; /* EX_INTEGER, EX_DOUBLE, EX_CHAR */
  size_t   value_count;
  void* values; /* not accessed if NULL */
} ex_attribute;

NOTE: is there benefit to getting all attribute names/types/size in single call, or can we simplify API and just provide the query of individual attribute. Can do this in single call if only fill in non-NULL arguments

Define/Query maps and map data

[ ] TODO: Add api functions

Define/Query number of transient variables on an blob

 ex_put_variable_param(exoid, EX_BLOB, num_variables);
 ex_get_variable_param(exoid, EX_BLOB, &num_variables);

[UNDECIDED] Can also define and query the blob truth table which will specify which variables are defined on which blobs.

Define/Query number of transient reduction variables on an blob

 ex_put_reduction_variable_param(exoid, EX_BLOB, num_variables);
 ex_get_reduction_variable_param(exoid, EX_BLOB, &num_variables);

Note that there will be a value written to and read from the database for all variables on all blobs, but only the values corresponding to a defined variable will be valid values.

Define / Query names of variables on an blob

 ex_put_variable_names(exoid, EX_BLOB, num_variables, names);
 ex_get_variable_names(exoid, EX_BLOB, num_variables, names);
 ex_put_variable_name(exoid, EX_BLOB, index, name);
 ex_get_variable_name(exoid, EX_BLOB, index, name);

Define / Query names of reduction variables on an blob

 ex_put_reduction_variable_names(exoid, EX_BLOB, num_variables, names);
 ex_get_reduction_variable_names(exoid, EX_BLOB, num_variables, names);
 ex_put_reduction_variable_name(exoid, EX_BLOB, index, name);
 ex_get_reduction_variable_name(exoid, EX_BLOB, index, name);

Write / Read data for blob variable(s) at a time step

All variables for a single blob (with id id):

  ex_get_vars(exoid, step, EX_BLOB, id, num_assem_vars, values);
  ex_put_vars(exoid, step, EX_BLOB, id, num_assem_vars, values);

[ ] Need partial get/put functions.

Write / Read data for blob reduction variable(s) at a time step