[Metakit] Corrupted database over network - WinNT

[Jean-Claude Wippler]

Thanks, Pat Thoyts for your detailed analysis and description of the  
workaround you found.

Let me summarize as follows:

> Metakit maps the file as
> a read-only memory mapping and when it saves it re-serializes into the
> underlying file. It then drops the current mapping and re-maps the
> file. What I observed was that when it re-mapped it got the original
> data and did not in fact re-map the newly written data at all.

In other words, Windows does not guarantee a consistent view of the  
data it buffers for networked file systems: write something to a  
file, map it to memory *afterwards*, and apparently you can get stale  
data.

If MK were to use r/w maps and push its data through there, then  
presumably it would work.  That will not work in the case when the  
file has to be extended, though.  Keep in mind that MK can write data  
to file for which partially comes from the r/o map.  Furthermore, MK  
maps r/o deliberately, because it prevents stray pointer writes from  
damaging a MK datafile.

So what we have today, is a design which works on Windows local  
disks, and on Unix/MacOSX both locally and on remote file systems  
(both NFS and SMB afaik).

[From: <Hector at coollector.com>]
> 1/ You confirm that metakit files have a serious design flaw that  
> puts at risk our data when accessed through a network.

With all due respect, the "serious design flaw" is in Windows.   
Getting that fixed is beyond our control.  We can only hope that XP  
or Vista get it right.

> 2/ You give indications on the nature of the corruption, and how to  
> prevent it.

Yes, and for that I am very grateful.  In addition to Pat's most  
recent post, more information can be found at http://www.equi4.com/ 
pipermail/metakit/2005-August/002112.html

So the solution is to derive a c4_Strategy class which opens the file  
in exclusive mode.  Pat has most generously handed everyone the  
solution on a silver platter.  If you don't want to subclass  
c4_Strategy, then open the file yourself and pass it to MK.  If you  
never need concurrent readers, use CreateFile and always open with  
dwAccessMode set to 0, then turn it into a FILE* with freopen and  
pass it to MK.

I for one never use MK in r/w mode on remote file systems.  Not just  
due to the above risks, but because locking also leads to all sorts  
of "Heisenberg" trouble.  In my book, shared data use should use  
client/server solutions.  That need only take a few dozen lines of  
code, depending on your tool choices.

-jcw