MetaKit Tips and Tricks December 1999

This page list some ideas and background information related to the use of MetaKit. If you have suggestions, please send them to info@equi4.com ...

Streaming		Some details about streaming and on-demand loading
Faster		What to do and what to avoid for maximum speed
Smaller		How to manage data in the most compact format

---

Streaming - Some details about streaming and on-demand loading

When saving data, MetaKit uses different formats, depending on whether the data was saved with c4_Storage::Commit or with c4_Storage::SaveToStream. This has a number of consequences on how and when data is loaded back in:

Saved using	Loaded as	Comments
Commit Commit SaveToStream SaveToStream	File Stream File Stream	The normal case, loads data on-demand This combination is not allowed! Ok, will load all data during open Loads all data serially right away

As you can see in this overview, random-access and streaming can be mixed in three out of four ways. Once loaded, you can again save using either c4_Storage::Commit or c4_Storage::SaveToStream.

When random-access and streaming are mixed on a storage object, the following will happen:

• Files are only modified (conceptually) during a call to c4_Storage::Commit.
• c4_Storage::SaveToStream makes a snapshot of the current (uncommitted) state.

It is therefore possible to open a file, apply a few changes, save the changed contents to a stream, and then close the file without committing those changes. Streaming does not interfere with the commit/rollback mechanism.

---

Faster - What to do and what to avoid for maximum speed

The data structures used in MetaKit can have surprising effects on performance, both positively and negatively. Here are a few suggestions which should help in most cases:

Preallocate rows - if the number or rows to add is known in advance, use c4_View::SetSize to set the view size accordingly. Fixed-size properties will need to do fewer internal reallocations if pre-allocated.

When inserting several rows, consider inserting them all with a dummy value first, and then using a loop to adjust each of the inserted values.

Limit the use of c4_Row objects - The construction of c4_Row objects is relatively slow, and can often easily be avoided. Here are three ways to accomplish the same thing:

Slow	for (int i = 0; i < n; ++i) {     c4_Row temp;     property (temp) = ...;     view[i] = temp; }
Faster	c4_Row temp; for (int i = 0; i < n; ++i) {     property (temp) = ...;     view[i] = temp; }
Fastest	for (int i = 0; i < n; ++i) {     c4_RowRef row = view[i];     property (row) = ...; }

The last two statements could also have been combined as "property (view[i]) = ...;"

Note that "prop1 [value1] + prop2 [value2]" also creates several temporary c4_Row objects, and should be avoided when top performance is critical.

Remove views before storage objects are destroyed - If a view still exists when its underlying storage object is destroyed, MetaKit has to create a full copy of the data in memory before releasing the storage object, to avoid even more confusing side-effects. The following example illustrates a common source of suprises in MetaKit:

Very slow (on close)	{     c4_View view;     c4_Storage storage (...);     view = storage.View(...);     ... }
Instant	{     c4_Storage storage (...);     c4_View view = storage.View(...);     ... }

In the first case, the view still exists as the storage object goes out of scope. There is a simple way to detach views from their storage objects, since they act merely as (reference-counted) pointers: just set "view = c4_View ();" before the storage object is destroyed.

---

Smaller - How to manage data in the most compact format

MetaKit implements automatic "adaptive integer sizing" and will often create very compact data files as a result. Strings and binary properties also use a variable format, and require no "field width" setting. Some tips to allow you to make optimal use of these features:

Attached views are more compact - You can trade off size versus speed by selecting either views attached to a storage object (smallest), or unattached views (fastest). Note that "storage" does not imply the need to store data on file in this context. It is perfectly valid to use the following code to manage temporary data in memory without ever saving it to file:

c4_Storage storage; c4_View view = storage.GetAt("temp[prop1:I,prop2:I]"); ... prop1 (view[i]) = ...; ...

In this example, the view would use "adaptive integer sizing" even though its data is valid only for the duration of each run. Unattached views always use 32 bits to store integers.

Integer formats only increase in size - The representation used to store integers in attached views (and on file) is based on the largest value (in magnitude) ever stored in each property and view. Storing large values in one subview will not affect the storage used by that same property in another subview.

For integers, the number of bits used to store values is determined as follows:

Value range	Bits used	Comments
0 0 .. 1 0 .. 3 0 .. 15 -128 .. 127 -32768 .. 32727 all others	0 1 2 4 8 16 32	If all rows are 0, no storage space is used Booleans are stored as 8 rows per byte Tiny positive codes A hex nibble One byte A short integer A long integer