Belt and suspenders

Not long ago I received an email in the context of Vlerq, with pointers to research I had not been aware of before. As it turns out, that work is highly relevant to what I’ve been attempting to do over the past several years, and could possibly lead to a breakthrough that has been evading me for some time.

It’s hard to predict breakthroughs of course - especially when they appear to be around the next corner though you haven’t actually passed that point. Then again, I’ve come to rely a lot on my instincts when it comes to Vlerq. Hunches and gut feelings can be surprisingly effective when deciding how to explore avenues to solutions.

Here’s what I’m up against while working on Vlerq:

Lots of ways to take a wrong turn. Worse: several breakthroughs have to be made to achieve real progress - each one hiding the next so I can’t even imagine the end result yet.

Normal software development is a walk in the park compared to that. You just turn the crank a couple of times until your code reaches escape velocity:

Research is an erratic process, even the home-grown kind I’m practicing. It’s hard to get from A to B when you don’t even know what B is really going to be like. That’s probably also what distinguishes this sort of work: in IT, this is not about understanding an existing world, nor of constructing theories about an abstract but nevertheless external world. It’s more an engineering task really: given what we have built so far, can we also build X? As well as: what would happen if we try to take approach Y further in some way? And: can we generalize existing approaches so things become simpler?

Vlerq is a bit of all this. Given the column-wise design of Metakit and its virtual views, can some known performance bottlenecks be removed? Given how Metakit has been used as extension for Tcl and Python, can these binding be improved to reduce the remaining impedance mismatch further? Can the mechanism of tying Metakit to languages be extended to many more languages, and with less effort? Now that the basic approach has been validated, can it be generalized way further, to support a full-scale dataflow model with automatic change propagation inside an application as well as between multiple applications?

After nearly a decade of tackling these issues from a huge range of angles, my position is a resounding “yes” - definitely, but a few more breakthroughs are still needed. Alas, breakthroughs have this nasty habit of not showing up when you want them too. They tend to happen when you’re not looking, or when you’ve started giving up. Or simply after a nice day doing something completely unrelated. That’s why I hate breakthroughs - they are never there when you need them.

Which brings me back to the title of this article. It has happened on several occasions that - out of the blue - someone sends an email and comments on Vlerq. Have you heard about X? Do you know what Y is working on? How does this differ from Z? Etc. The people network you operate in may well be more important than the work you currently do. Because what you do is the easy part (it’s doable after all), whereas real progress depends on overcoming obstacles and solving the open-ended problems. Thinking outside the box and creativity certainly helps - but that requires seeing the bigger picture. Which in turn requires moving up in abstraction. That “up” is where outside influences are not just useful, but might in fact be crucial. On most days, I’m so totally clueless that I couldn’t tell up from down! A single comment or URL can change that.

I’m starting to think that it might help me if more people read my blog and through it, hopefully, also a bit about Vlerq. The more diverse its readership is, the more likely some valuable cross-connections & associations will be made by those readers and hopefully I’ll get to find out about it eventually. Because the biggest risk I see when working on Vlerq, is that my horizon narrows to the point where I won’t be able to find the right paths to explore. Reading more is not an option (there’s already too much), and searching doesn’t work when you don’t know what you’re looking for.

Could the blog and these articles be a way to stay on track and to get in touch with others with the same interest - i.e. my belt and suspenders? I think they are. So is the set of other blogs I read, and so are the occasional emails that get exchanged with comments or pointers. I don’t track massively active blogs or news sites. Instead, I try to rely on a few people who track and comment on them in their blogs & emails. This works in both directions: occasionally, I’ll reciprocate and forward or comment on an article. Or blog about it. Am always surprised to see how often people appreciate it.

Note that this is completely different from discussion and debate, whether in private or in public. This is about value-free association, not about being right or wrong: Did you know about this? No? I think you should have a look at it. Then you can decide for yourself how relevant and useful it is. Remarks like these are easy but nevertheless valuable - they have gone through a filtering process. Most importantly, that process is different from your own - so the choices, perspectives, and horizons are slightly disruptive - yet relevant. Which is great, since breakthroughs are always caused by disruptions.

Heh... let’s call this the semantic web: people emailing and blogging.

jcw @ equi4.com : 2007-05-04