How to deploy an application with TclKit A simple way of getting an application into the hands of your users and clients consists of giving them two files: A Scripted Document - your code, which may or may not be portable TclKit - the application-independent, but platform-dependent part What is a "Scripted Document" (SD)? This is a name coined for a single file which can contain both scripts and data, and which can be set up to run from the command line (or by double-clicking in a graphical environment). As of Tcl 8.4a the Tcl core has support for launching such files (which start with pure text, but are followed by unparsable binary data). SD's are similar to ZIP archives in that they contain a collection of files, compressed and packed into a single file. They differ however, in that they use the MetaKit database system, which has its own format, and which supports making changes in place. SD's can be used to store just application scripts, or just data, or a combination of both. As such, they are not more than a packaging scheme plus some trickery to make them runnable/executable. But due to VFS, the story does become more interesting. What is VFS? VFS stands for "Virtual File System". There are several uses of this term, but in the context of Tcl, it is meant as a way to make things look like a file system at the Tcl level, even if they are not. There are VFS "drivers" for SD's, zip archives, ftp servers, and more. The concept was introduced in Tcl by Matt Newman, who created a nearly pure-Tcl emulation layer to fake Tcl commands such as open, file, and even source and load. The idea is has been taken further by Vince Darley, as a C-based VFS layer now part of the Tcl 8.4 core. In TclKit, VFS is used to "mount" scripted documents. This means that the file "tclkit" (which is a scripted document too!) is made to look like a *directory* as far as Tcl is concerned, in which there are more files and sub-directories. The illusion is more or less complete, in that you can do a "cd tclkit; puts [glob lib/*]" and there will be the whole Tcl runtime lib as if it really is a file system. But it's all trickery which breaks down the moment you do things like running "ls". Versions and compatibility issues TclKit has been involved in all this from the very start, which is why there are a few incompatible binaries and two types of SD's. To start with SD's: the first version pre-dates VFS and uses a MetaKit- specific structure or storing scripts and datafiles. This is no longer in use, but some early SD's, most notably WiKit (the Tcl'ers Wiki) do still use this format. The new VFS-based format has been in active use for quite some time, without changes - and is considered final. TclKit itself has gone from being a very special build, pulling a range of tricks such as using Jan Nijtman's Plus patches and Wrap, to being an almost completely standard static Tcl/Tk build with a couple of extensions thrown in (MetaKit database, zlib compression, MD5 message digests, and "rchan" - which is used to emulate the "memchan" package). Right now, December 2001, TclKit is still in the transition stage, but nearing completion - in the sense that it is about to be packaged and made separately available as (relatively easy to build) distribution. SDX: a utility for Scripted Documents SDX lets you create and disect scripted documents, among other things, which will be ignored for now. In this context, "SD" is always based on MetaKit, even though it should be possible to create an implementation working off ZIP archives, etc. The choice of MK is convenience since TclKit already includes it as a general database system, and since it has transaction-commit support. SDX itself is also a scripted document, and is nothing but a collection of various utility scripts. The ones of relevance here are: fs2sd - pack a dir tree ("file system") into a scripted document sd2fs - unpack a scripted document to a directory tree Note that TclKit too is a scripted document, albeit one with a pretty large preamble (the exexcutable itself). So with some care, you can also manipulate (a copy of) the TclKit executable with SDX. How to build a scripted document, quickly FIRST: Set up working versions of TclKit and SDX - Get the proper TclKit binary for your platform, from: http://www.equi4.com/pub/tk/ - Get the SDX utility, i.e. the file "sdx.bin" from here: http://www.equi4.com/pub/tk/examples/ - On Windows, you should end up with files "tclkit.exe" + "sdx.bin", then create a batch file, called "sdx.bat", containing: @tclkit sdx.bin %1 %2 %3 %4 %5 %6 %7 %8 %9 - On Unix, unzip/rename as needed to end up with "tclkit" and "sdx", then do a "chmod +x tclkit sdx" to make them both runnable. - That's it, test that it works by launching "sdx": it should report a few lines of usage summary SECOND: Prepare your scripts to work in a certain structure - supposing your app is called "myapp", create these dirs: myapp.vfs/ myapp.vfs/bin/ - place your main startup script in bin/, as: myapp.vfs/bin/main.tcl - make sure you don't make assumptions on what the currnt dir is, so if you have lines of the form: source utils.tcl then *CHANGE* them to be: source [file join [file dirname [info script]] utils.tcl] - if you use packages, copy each of them in their own dir inside: myapp.vfs/lib/ THIRD: Test that things work in unpacked form, then wrap it up - launch your app as follows: $ tclkit % lappend auto_path myapp.vfs/lib % source myapp.vfs/bin/main.tcl - if things are ok, your app should now be running properly now: congratulations, you're almost there! - run the following command: sdx fs2sd myapp This takes the myapp.vfs/ dir, packages it into a scripted doc called "myapp", and makes the file executable (SDX also creates a "myapp.bat" if run on Windows). - that's it, you have a file called "myapp", with your app inside What if the above doesn't work as advertised? TclKit + SDX are only slowly getting to the point of being a positive "out of the box" experience. Tinkering is still needed, usually. First of all, the above was coded with Unix in mind. On Windows, you cannot as easily use tclkit.exe as command line tool (it's a GUI app, not a console app, a most unfortunate distinction in the Win* world). There are two workarounds: - get "tclkitsh.exe" and use it to run SDX, instead of tclit.exe - use tclkit in TkCon mode, so instead of doing "sdx blah ...", do: tclkit sdx.bin tkcon sdx.bin blah ... It ain't gonna win a beauty contest, but it works. Second, you need to keep in mind that tclkit starts up as tclsh under Unix, and that doing "package require Tk" is needed to work as wish. Solution: always add the "package require Tk" to your GUI app scripts. Third, VFS takes some getting used to. The 8.4-26 release of TclKit uses a fair bit of Tcl trickery to make things like "open" work with VFS. This shows in that the emulation of the "file" command is not perfect, but most notably in that things like "image -file ..." do not work without some help. That help can be found in a file called "compat.tcl", it's available at http://www.equi4.com/previews/ - you need to source this file before doing "image" requests (and place it somewhere inside your SD!). The good news is that the most recent builds of TclKit rely on a new 8.4a4 Tcl core, which has built-in C-level support for VFS. Because of this, "compat.tcl" will no longer be needed in newer releases. Another area which needs to be carefully dealt with is the loading of dynamic extensions. This is supported, but with a trick: "load" will create a temporary copy of the shared library before loading it - the reason being that an OS is not able to link from inside SD's. If you want to support multi-platform apps with compiled extensions, then you need to create a mechanism whereby your app picks the right shared lib, from a set of them - all stored inside the scripted doc. An example of this is the "dok" files, which uses Tkhtml builds for Windows, Linux, and Solaris and hence runs as is on all three: http://www.equi4.com/pub/dok/ If there are any other stumbling blocks, please report them, either on the comp.lang.tcl newsgroup, or by email to me - little will improve unless you tell people about it and help fix it. What about shipping everything as ONE file? TclKit + SD's are different from tools such as ProWrap and FreeWrap, which end up as a single self-contained executable. There are various reasons for this, the most important one being that TclKit also tries to address the issue of distributing *portable* applications. If an SD contains only Tcl scripts and data, then it is in fact also machine-independent. To use it on a platform, you simply need to have a suitable build of TclKit and the combination will work. Even if an SD needs compiled extensions, one can make them portable to a certain extent by including a set of pre-compiled extensions, one of which then gets selected at run-time. Yet more flexibility can be had by also providing pure-Tcl "fallback" code, which might be slower or less funtional, or which simply terminates with a clear explanation. So the original goal of TclKit was to try to create a very clear split between a machine dependent, but totally generic "application runtime" called TclKit, and the rest - which is wrapped up a single-file SD. Still, what about single-file deployment? There have been some experiments to extend the above model, given that in some scenarios the point of having exactly *one* file is essential. The idea is to support "custom-extended versions" of TclKit. Which is a specially modified version of TclKit containing all necessary files, and which starts up using the merged scripts. The special property of every custom-extended TclKit, will be that it contains all information needed to reconstruct a 100% standard version of TclKit, i.e. without the modifications. There will be support for easily extracting that, as well as for easily upgrading applications by re-constructing them to work with a newer version of TclKit. But it goes much further than that, because one will be able to take such a custom application built for platform X, and transform it into a modified version which runs on platform X - in an "app-neutral" way. The reason for doing this, is to make sure that deploying all sorts of customized builds will not lead to a maintenance nightmare. Inside, things continue to consist of two distinct parts (in concept): tclkit, i.e. the generic runtime, and everything else - scripts, images, dlls. The one thing a custom-extended TclKit cannot do is to modify itself, this is technically impossible (an exe can only be opened R/O). So, athough there are good reasons sometimes to go for single file - the benefits of modifiable scripted docs and of robust self-updating apps are not going to be available with a custom-extended TclKit.