Model View Controller

In my book, MVC rules.

From Wikipedia:

Model-view-controller (MVC) is an architectural pattern used in software engineering. In complex computer applications that present lots of data to the user, one often wishes to separate data (model) and user interface (view) concerns, so that changes to the user interface do not affect the data handling, and that the data can be reorganized without changing the user interface. The model-view-controller solves this problem by decoupling data access and business logic from data presentation and user interaction, by introducing an intermediate component: the controller.

While exploring and designing Vlerq, it often struck me how MVC is not just nice - MVC is actually a perfect match for the mental model I have of how views are meant to be used.

Let me explain, by describing some key concepts behind MetaChrom - a special-purpose data processing and visualization application I built and sold in the early 90’s. This was the product for which I founded my company, back in 1987.

Our goal was to process hourly measurement data from automated air quality monitoring stations placed in the field. I won’t go into the details of Gas Chromatography (GC), other than to point out that these are incredibly sensitive instruments, which require very sophisticated calibration and constant re-calibration to deliver accurate results. Until then, GC’s had never been deployed in the field. Instead, people would collect canisters of air and bring them to the lab, for analysis by trained operators. As you can imagine, you better make sure each of these flasks is inert and very clean if you’re going to suck up a few liters of outside air, drive around the country with them, and expect to find anything useful in there!

So instead of collecting hundreds of canisters, my partner and I were figuring out how to set up the infrastructure needed to measure everything in the field. Instead of bringing the mountain to Mohammed, we’d go the other way. After all, bits travel easily, quickly, and most importantly: they don’t change. In our approach each GC was running on location, unattended, and in more or less continuous mode. It takes a good part of an hour for a GC to sniff out all the Volatile Organic Compounds (VOC’s), so we sure kept ‘em analyzers busy.

Bear me with me. This is a gorgeous example of MVC, as you will see.

My role in this was to get all those incoming bits turned into meaningful results. And that’s where it gets tricky, very tricky. Remember, we’re automating something that takes skilled operators. Why? Well, a GC is a finicky beast. It’s sensitive beyond belief, but the problem is that it doesn’t have a clue what it’s measuring. It couldn’t tell a toxic substance like benzene from a smelly whiff of perfectly harmless and natural methane even if you immersed the whole instrument in it. The trick is calibration. You inject some known quantities of known substances, so that you can interpret the measured time-series afterwards. And you get to play all sorts of clever tricks, since this is about measuring environmentally “clean” air, so some compounds are almost always present and others usually aren’t. GC lab assistants tend to become very good at this sort of thing.

Except that we didn’t have assistants. We were accumulating orders of magnitude more data than under normal lab conditions. Imagine you’re on CSI for a moment: think of a fire, and you’re called to the scene to take a few air samples so you can figure out whether the fire was caused by an arsonist. Fuel tends so leave lots of traces. So you get to spend a few days to push those samples through your GC. We didn’t have that luxury, nor the funds to maintain an army of GC operators. Besides, who’d want to analyze normal outside air over and over again? So we automated everything. We achieved three orders of magnitude performance increases over traditional GC lab station software. A 1990-vintage PC (a Mac, actually) could easily keep up with incoming data from a mesh of several dozen stations across the Netherlands. Operated by a single person.

Note that this is not about number crunching. The end result for each air sample is a set of up to perhaps a hundred <substance,concentration> values. Say 1 Kb per result. With 25 instruments sniffing 24 hours a day, that’s still way under 1 Mb per day. Even back then, this was a piece of cake to deal with. After processing and picking out only the substances that were important, even less remained. A demo we sent around back then came on two floppies: the app and a compressed demo dataset.

The problem we faced was one of quality, not quantity. To get those results, you had to get all the calibrations for all the instruments just right. And THAT was what MetaChrom was about. We essentially created a special-purpose dataflow engine, whereby you could take a large set of measurements from a single GC, and turn a number of carefully chosen dials - so to speak. As you make adjustments, you see how substance identification “snaps” into place, and how sensible concentration trends emerge for each of them over time. The environment is a big chemistry kit, but a lot of the effects of sunlight and wind dispersion and the main chemical reactions are well understood.

Some people got frighteningly good at finding their way around in N-dimensional parameter space. And once the results snap into a pattern, they were often able to turn the whole thing around and deduce operating details about the instrument itself. For example, from the way parameter settings varied over time you could “sense” the effect of daily temperature cycles on the instruments, as the cabins where they were located heated up during the day and cooled down during the night. Also, these GC’s use liquid nitrogen for part of their operating cycle, so there were huge tanks to support a week or so of unattended use. At some point, one site was showing a lack of sensitivity near the end of each measurement cycle. And just by looking at the results, the operator was able to deduce that that particular site had been running out of liquid nitrogen sooner than expected. And indeed - a larger supply tank fixed the problem (maybe it got more sunlight than the other locations).

All it took was a spreadsheet-like visualization program, which worked with a sufficient amount of data and was interactive enough, to enable people to get a completely new feel for the datasets they were dealing with. We called this “Exploratory Chromatography” because you could explore hypotheses and develop an understanding of the measurement data, by simply “turning” a few knobs (well, using a mouse, actually).

Ok, enough. This story is about Model-View-Controller, after all.

In MetaChrom, the model was the input dataset plus a few parameters (“knob settings”). The view is literally what you see on the screen, a rather sophisticated combination of numbers and markers with drill-down matrices to inspect specific details. And the controller was the mouse (the “knobs”).

It doesn’t get any more MVC than this really: the software combines the dataset + parameters to present information on the screen. The user explores the information, possibly requesting more specific details, and then chooses to adjust parameters via mouse and keyboard. This triggers some pretty elaborate recalculations to update the information shown on the screen, and then the process repeats. All the heavy lifting happens automatically under the hood - there is no “go” button.

Here’s a light-hearted way to look at MVC: the model represents your accumulated time and energy (don’t lose it!), the view is where the CPU gets a serious workout (who cares?), the controller transforms the blood, sweat, and tears you’re investing in manipulating the mouse and keyboard (it better be effective...).

There are several important observations to make about all this.

First of all, “M” in MVC is really a synonym for “state” or “database”: when you quit the application, all that stays behind and needs to persist is the (unaltered!) input dataset, and a fairly small set of parameter settings. The view and controller cease to exist. While MetaChrom did cache a substantial amount of data, storing all those intermediate results was merely an optimization. In a nutshell:

The Model lives on disk. Both the View and the Controller live in RAM.

Second, “V” in MVC is really about side-effect free functional programming: given the input data set, the parameters, and the user’s choice of what needs to be shown, all computation from stored state to visualized representations on the screen can be treated as being side-effect free. Showing something has no effect on the data. Once we define the processing needed to visualize results, changes to the input simply lead to re-processing. Since there are no side effects, all sorts of caching & memoization tricks can be used to perform this re-processing step in the most efficient possible way. In a nutshell:

The View is essentially a transformation from input data to a visual form.

And lastly, “C” in MVC is about turning knobs: all user interaction leads to state changes. Some will be transient, such as a scroll position or a window size, others will lead to changes which need to be saved on disk to reflect choices and decisions made, or to collect additional input data. Unlike the functional view aspect of MVC, controlling is about making state changes and as such a very procedural mechanism. Views are just innocent bystanders which tirelessly show the consequences of changes over and over again. Controllers on the other hand are actors, they are disruptive and cause new things to happen (or to revert to previous conditions). In a nutshell:

The Controller acts on state, by adding/altering/removing some part of it.

Now step back to see what Vlerq is about: persistence, dataflow, and scripting.

Got it?

jcw @ equi4.com : 2007-04-21