DN636731243

Subject: Re: Large structured array memory use - DN [1]

Paul Duffin <pduffin@hursley.ibm.com> - 20 Jun 2000 - comp.lang.tcl

 Stefan Weichselbaum wrote:
 >
 > Hi all!
 >
 > I am planning on an itcl class capable of handling huge amounts of data
 > and trying to establish a worst case estimation on memory use. I want to
 > find out, whether I can refrain from storing temporary files.
 >
 > Is there a way to predict the memory required for a structured (three
 > dimensional) array like:
 >
 > set arr(IndexA,IndexB, Index C) value,
 >
 > where IndexA is an integer lying anywhere between 1 and 450000, IndexB
 > is a 32 byte string, but exists only once for every value of IndexA,
 > IndexC is an integer between 1 and 50, and value may range from 0 exact
 > to a 43 byte string.
 >
 > All that should happen on a WinNT 4.0 platform with 256MB of physical
 > RAM.
 >
 > The combined values of IndexA (strings of varied length like the
 > integers they represent) sum up to 2588895 bytes.
 > 450000 32 byte integer strings sum up to 14400000 bytes.
 > Calculating values for IndexC results in 91 bytes (that may be
 > important, see below).
 >
 > A complete set of 50 values as they are stored in my application
 > requires up to 220 bytes.
 >
 > The questions I have:
 >
 > 1) How much memory overhead is used to store the data?
 > 2) Are index values stored reduntantly, i.e. are complete index
 > combinations stored for each value, or not?
 >

 For every value you store in the array a seperate copy of the
 index / key is kept.

 The cost per entry is approximately
     sizeof (Var) + strlen (index) + (sizeof (Tcl_Obj) + sizeof (value))

 The value's can be shared between entries as they are Tcl_Obj *.