[Metakit] [Fwd: memory leaks in python interface [was Hash view questions]]

[Ralf Schmitt]

[sorry if this has been posted twice]

zooko at zooko.com wrote:
>> Your sample code does show increasing memory use on my setup as  
>> well.  Is there a way to find out what the content or at least sizes/ 
>> counts of the leaked memory areas is in Python?
> 
> Yes!  A beautiful tool known as valgrind [1].  It tells me that you are right
> and the C++ metakit code is exonerated.
> 
> This is because the stack traces of the code that allocated the leaking memory
> do not have any metakit functions in them.  Hm... the biggest one has
> posix_read() in it.  The other two are both involving allocation of Python
> Ints...
> 
> Regards,
> 
> Zooko
> 

Using a debug build of python (i.e. configure --with-pydebug), one
notices a massive amount references not being decref'ed. I'll have a 
look at it.

- Ralf


$ cat t.py
#! /usr/bin/env python

import sys
import os
import metakit
import gc

def doit(num):
    store = metakit.storage("leakcheck", 1)
    db = store.getas("test_db[k:B]")

    for i in xrange(num):
        k = os.urandom(20)
        db.append(k=k)
        db.delete(db.find(k=k))
#        if i % 100 == 0:
#            store.commit()


def main():
    if len(sys.argv)==1:
        for x in [0, 100, 1000, 2000]:
            os.system("./t.py %s" % x)
    else:
        print
        num = int(sys.argv[1])
        doit(num)
        gc.collect()
        print "done. %s iterations." % num

if __name__=='__main__':
    main()

ralf at schrotti:~$ ./t.py

done. 0 iterations.
[8211 refs]

done. 100 iterations.
[9711 refs]

done. 1000 iterations.
[23211 refs]

done. 2000 iterations.
[38211 refs]
[8208 refs]