[Metakit] [Fwd: memory leaks in python interface [was Hash view questions]]

[Ralf Schmitt]

zooko at zooko.com wrote:
>  Ralf Schmitt wrote:
>
>   
>> Using a debug build of python (i.e. configure --with-pydebug), one
>> notices a massive amount references not being decref'ed. I'll have a 
>> look at it.
>>     
>
> Ralf: Great!
>
> Please keep us informed.
>
>   

Here's what I have found so far. Note that some of the 'lost references'
are from a LoseRef method, which does decrease the refcount without
releasing the object if refcount drops to zero (and which circumvents 
pythons refcounting debugging...)

The test program isn't loosing references anymore:

ralf at schrotti:~$ python t.py

done. 0 iterations.
[8207 refs]

done. 100 iterations.
[8207 refs]

done. 1000 iterations.
[8207 refs]

done. 2000 iterations.
[8207 refs]
[8209 refs]

I can use a longer value to write with 'k = os.urandom(2000)'
and the program's memory  usage stays around 5MB.
With the unpatched version memory usage goes up to around 200MB. (100000 
iterations).

I'm pretty much convinced that the python wrapper code contains even 
more leaks...
Maybe I'll do some more tests...



> By the way, the valgrind results that I posted were with current SVN head of
> Python built with pydebug on.  valgrind is an excellent tool, and if you are
> working on Linux then I recommend that you spend a couple of minutes learning
> how to use it.
>
>
>   
I've been using it without having really having read the docs, but this 
is on my list of things todo.



-------------- next part --------------
=== PWOBase.h
==================================================================
--- PWOBase.h	(revision 21935)
+++ PWOBase.h	(local)
@@ -9,6 +9,7 @@
 
 #include <Python.h>
 #include <limits.h>
+#include <assert.h>
 
 // Dummy, minimal exception thrown when a Python exception is generated
 // (after PyErr_Format, PyErr_SetString, etc.)
@@ -30,9 +31,19 @@
     // incref new owner, decref old owner, and adjust to new owner
   void GrabRef(PyObject* newObj);
     // decrease reference count without destroying the object
-  static PyObject* LoseRef(PyObject* o)
-    { if (o != 0) --(o->ob_refcnt); return o; }
+  static PyObject* LoseRef(PyObject* o) { 
+	  if (o != 0) {
+		  if (o->ob_refcnt == 1) {
+			  o->ob_refcnt = 0;
+		  } else {
+			  assert (o->ob_refcnt > 1);
 
+			  Py_DECREF(o);
+		  }
+	  }
+	  return o; 
+  }
+
 private:
   PyObject* _own; // set to _obj if we "own" a reference to _obj, else zero
 
=== PWOImp.cpp
==================================================================
--- PWOImp.cpp	(revision 21935)
+++ PWOImp.cpp	(local)
@@ -15,7 +15,11 @@
 void PWOBase::GrabRef(PyObject* newObj)
 {
     // be careful to incref before decref if old is same as new
-  Py_XINCREF(newObj);
+  if (newObj && newObj->ob_refcnt==0) {
+	  newObj->ob_refcnt = 1;
+  } else {
+	  Py_XINCREF(newObj);
+  }
   Py_XDECREF(_own);
   _own = _obj = newObj;
 }
=== PWOMapping.h
==================================================================
--- PWOMapping.h	(revision 21935)
+++ PWOMapping.h	(local)
@@ -59,7 +59,9 @@
     if (rslt == NULL)
       PyErr_Clear();
     PWOString _key(key);
-    return PWOMappingMmbr(rslt, *this, _key);
+    PWOMappingMmbr res = PWOMappingMmbr(rslt, *this, _key);
+    Py_XDECREF(rslt);
+    return res;
   };
   //PyDict_GetItem
   PWOMappingMmbr operator [] (PyObject* key) {
=== PWONumber.h
==================================================================
--- PWONumber.h	(revision 21935)
+++ PWONumber.h	(local)
@@ -100,6 +100,7 @@
     PyObject* Int = PyNumber_Int(_obj);
     if (Int == NULL)
       Fail(PyExc_TypeError, "can't convert to int");
+    Py_DECREF(Int);
     long r = PyInt_AsLong(_obj);
     if (r == -1) FailIfPyErr();
     return r;