DN482508145

Subject: Re: Fastest way to shove data from file to tcl-array - DN [1]

Andreas Kupries <a.kupries@westend.com> - 26 May 1999 - comp.lang.tcl

 Jonas Beskow <beskow@fuzzy.ucsc.edu> writes:

  > Greetings,

  > We have a large dictionary (3MB, 130K lines), that is to be read
  > into an array (one line per element) When read line by line, the
  > whole thing takes about 70 seconds (on a 200MHz laptop). I also
  > tried using the zipchan extension, to read a zipped version of the

 Can you post an url for this one ? I would like to compare it against
 my Trf.

  > dictionary (840 KB) directly. This shaved off 10 seconds.  Now, if
  > it would speed things up, it would be OK to do a little
  > pre-processing of the dictinary file, i.e. it could be formatted and
  > stored as tcl-code, something like:

  > array set a {
  >  key1 val1
  >  key2 val2
  >  ...
  > }

  > or
  > set a(key1) val1
  > set a(key2) val2
  > ...

 One advice at

     http://purl.org/thecliff/tcl/wiki/TclPerformance

 is to 'time' it. So, convert your file to the format to test, then do

     puts [time {source yourFile} 1]

 to check out the speed.

 Remark: One trick to remember for reading large files is to determine
 their size beforehand and then to tell 'read' that number, allowing
 the command to do a better allocation of channel buffers. So:

     set sz   [file size yourFile]
     set f    [open yourFile r]
     set data [read $f $sz]
     close $f

     foreach {key value} [split $data \n] {
         set yourArray($key) $value
     }
     set data {} ; unset data ; # one of the two commands should
     # free the associated memory, but I don't remember which one

 --
 Sincerely,
     Andreas Kupries <a.kupries@westend.com>
             <http://www.westend.com/~kupries/>;
 -------------------------------------------------------------------------------