[Metakit] Maximum practical size of Metakit databse?

[Jean-Claude Wippler]

Davis Adrian wrote:
> What factors govern the maximum practical size for a Metakit database?

(This email was pending in a queue I rarely check nowadays due to the  
levels of spam flooding it, please consider subscribing to the  
mailing list to avoid getting in there)

There's a hard limit at 2 Gb due to the way signed 32-bit ints are  
used in MK and due to the limitations of a 32-bit address space.

You'll be able to get close to that if there are not too many  
subviews, you don't modify large amounts of data (modifications use 4  
Kb memory buffer chunks).  Reading is usually ok, it's usually the  
creation side that causes trouble first.  I'd expect a blocked view  
with only numerical data to get furthest, all the way up to that 2 Gb  
barrier in fact.

So with MK 2.4.9.3, I'd say that generally speaking 1 Gb is roughly  
the end of it.  If you have large amounts of data being small ints,  
these sizes cannot easily be compared with other database solutions,  
due to MK's use of adaptive int vectors which can be substantially  
more compact.

Btw, on a 64-bit architecture you can actually sneak around these  
limits by using multiple datafiles.  Note that all view operators can  
be used across datafiles.

The current codebase won't go beyond these limits even on a 64-bit  
architecture.  In the lab I've been growing a new "strain" of MK  
which overcomes this, as a recent test with an 8 Gb MK datafile  
proved (the next-generation size limit will end up in the Tb range).   
IOW, the file format can handle larger datasets - it's just the code  
which runs out of steam.

-jcw