DN432495024

Subject: ANNOUNCE: Stubs: A dynamic linking mechanism for Tcl, Tk and extensions. - DN [1]

Paul Duffin <pduffin@mailserver.hursley.ibm.com> - 14 Jan 1999 - comp.lang.tcl

 Stubs: A dynamic linking mechanism for Tcl, Tk and extensions.
 ==============================================================

 What does it do ?
 -----------------

 1)  Adds support for backlinking to all platforms, thus making it possible for
     static executables to dynamically load extensions on any platform.

 2)  Eliminates all the operating system specific problems associated with
     dynamically linking an extension to Tcl/Tk and any other library which
     exports a stub interface. This may sound like a wild claim but it is true
     simply because the operating system does not have to do anything.

 3)  Extensions and libraries built with different compilers will work together
     even if it is not possible to link them normally. The only requirement for
     them to be able to work together is that function calls are compatible.

 4)  As extensions do not have a hard coded reference to the library it is
     possible to use them with any library which is compatible. e.g. an extension
     which was originally built for Tcl 8.0 should work with Tcl 8.1, and an
     extension built for Tcl 8.1 could work with Tcl 8.0.

 How does it do it ?
 -------------------

 Basically the stub mechanism provides a cross platform dynamic linking
 mechanism using tables of function pointers.

  o  A library which wishes to provide a stub interface populates a function
     table with the addresses of its interface functions.

  o  A pointer to that table is then made available to any extension which needs
     to access that library's interface.

  o  Once an extension has obtained the pointer it simply uses it to get the
     address of the function it wants to call and calls it.

  o  At no time in this process does the operating system become involved.

 To see the differences take a look at the following which describes what happens
 when Tcl loads two different extensions.

  o  Loading an extension which is linked directly to Tcl.

     1)  Tcl asks the operating system to load the extension.
     2)  The operating system loads the extension and then tries to resolve any
     undefined symbols. This process is very operating system dependent but
     involves one or more of the following steps.
     1)  Resolve any symbols which are defined in the current process
         context. (Backlinking).
     2)  Find and load any libraries that the extension is dependent on.
         This involves searching paths defined through a variety of
         operating system dependent methods.
     3)  Resolve any symbols which are defined in the new libraries.
     3)  Tcl then calls the extension's initialisation entry point.

  o  Loading an extension which uses Tcl's stub interface.

     1)  Tcl asks the operating system to load the extension.
     2)  The operating system does so, resolving any non Tcl symbols.
     3)  Tcl calls the extension's initialisation entry point.
     4)  The extension obtains the pointer(s) to Tcl's stub table(s) and uses
     that to call Tcl.

 Where is it ?
 -------------

 ftp://www.neosoft.com/pub/tcl/sorted/packages-8.0/devel/StubPatch804.tar.gz

 How to apply it ?
 -----------------

 The StubPatch804.tar.gz file contains the following files.
     Tcl_StubPatch.diff    The patch for Tcl.
     Tk_StubPatch.diff    The patch for Tk.
     README            This file.

 Unpack them as usual. e.g.
     gunzip -c StubPatch804.tar.gz | tar -xvf -

 These patches should only be applied to vanilla 8.0.4.

 Move into the Tcl directory which contains the generic directory and apply the
 patch making sure to specify -p1. e.g.
     cd Tcl8.0.4
     patch -p1 < ../Tcl_StubPatch.diff

 And do the same for Tk. e.g.
     cd Tk8.0.4
     patch -p1 < ../Tk_StubPatch.diff

 You are now free to build it.

 How to build it ?
 -----------------

  o  Unix

      o  Tcl

     There is one new configure option for Tcl.

     --enable-stub
         If this option is disabled (default) then you will simply get a
         normal tcl installation plus a few enhancements.

         If it is enabled then Tcl will provide a stub interface for
         extensions to use. It will also build an archive library called
         libtclstub*.a which is used to initialise and access Tcl's stub
         interface.

         You should delete the following files (if they exist) if you have
         changed the value of this option.
         tclBasic.o

      o  Tk

     There are two new configure options for Tk; both of the require that Tcl
     is built with stub support.

     --enable-tcl-stub
         If this option is disabled then Tk will link to Tcl in the normal
         way.

         If it is enabled then Tk will use Tcl's stub interface.

         You should delete the following files (if they exist) if you have
         changed the value of this option.
         tkMain.o tkConsole.o

     --enable-stub
         If this option is disabled then Tk will not provide a stub
         interface.

         If it is enabled then Tk will provide a stub interface for
         extensions to use. It will also build an archive library called
         libtkstub*.a which is used to initialise and access Tk's stub
         interface.

         You should delete the following files (if they exist) if you have
         changed the value of this option.
         tkWindow.o

  o  Windows

      o  Tcl

     Two new makefiles have been provided, one for Borland and one for
     Visual C++.

     makefile.vc.stub    (The Visual C++ version)
     makefile.bc.stub    (The Borland version)

         These makefiles build a version of Tcl which provides a stub
         interface for extensions to use and will also build a static
         library tclstub80.lib which is used to initialise and access Tcl's
         stub interface.

         You should delete the following files if they exist when building
         with these makefiles for the first time.
         tclBasic.obj

      o  Tk

     Four new makefiles have been provide, two for Borland and two for
     Visual C++.

     makefile.vc.stub    (The Visual C++ version)
     makefile.bc.stub    (The Borland version)

         These makefiles build a version of Tk which uses Tcl's stub
         interface.

         You should delete the following files if they exist when building
         with these makefiles for the first time.
         tkMain.obj tkConsole.obj
         (Visual C++ users will need to delete all .obj files)

     makefile.vc.stub2    (The Visual C++ version)
     makefile.bc.stub2    (The Borland version)

         These makefiles build a version of Tk which uses Tcl's stub
         interface and also provides one of its own, it will build a static
         library tkstub80.lib which is used to initialise and access Tk's
         stub interface.

         You should delete the following files if they exist when building
         with these makefiles for the first time.
         tkMain.obj tkConsole.obj tkWindow.obj

  o  Macintosh

     At the moment there is no 'makefile' available to build a Macintosh version.
     However the code has been designed to work with the Macintosh and therefore
     it should not be too difficult to create one. If anyone would like to try
     then I will be happy to provide them with some more details about what needs
     to be done.

 Compatability of the stub interfaces with the standard interfaces ?
 -------------------------------------------------------------------

 The stub interfaces of Tcl and Tk provide access to almost all of the external
 and internal functions both generic and platform dependent. The exceptions to
 these rules are ones which it makes no sense to make available through the stub
 interface. These include all Tcl command functions plus the following functions:
     Tcl_Main
     Tcl_AppInit
     Tk_Main

 Neither Tcl nor Tk provides access to the global variables which are available
 with the standard interface. This is for two reasons, the first being that it
 is bad practice to access global variables and the second being that they could
 only really be accessed using macros which can be very dangerous, especially in
 C++.

 On Windows and Macintosh the emulated X calls are included as part of the Tk
 stub interface.

 How to enable an extension to use a stub interface ?
 ----------------------------------------------------

 Using Tcl's stub interface.

 1)  Before the extension calls any Tcl function is must call Tcl_Required. This
     is essentially the same as Tcl_PkgRequire except that the package name is
     implicitly "Tcl". As this function is supported by both the standard and the
     stub interface the call to it does not need to be conditional. The stub
     interface version checks that stubs are supported and initialises the stub
     table pointers.

 2)  The makefile needs to be modified to use the stub library (libtclstub*.a)
     instead of the standard library, and the compile flags need to be changed
     to define ACCESS_TCL_THROUGH_STUB.

 3)  An inline versions of the stub interface is available, it uses macros in C
     and inline functions in C++ to access Tcl's interface. These will be used
     if INLINE_TCL_STUB is defined before including tcl.h, alternatively you
     can include tclStubInls.h yourself. The stub library is still needed to
     ensure that the stub initialisation is done correctly.

 Using Tk's stub interface.

 1)  Before the extension calls any Tk function it must call either Tk_Required
     or Tk_Present. These are essentially the same as Tcl_PkgRequire and
     Tcl_PkgPresent respectively except that the package name is implicitly "Tk".
     These functions are implemented as a macro in the standard version of the
     interface because Tk may not be present when they are called. The stub
     interface version checks that stubs are supported and initialises the stub
     table pointers.

 2)  The makefile needs to be modified to use the stub library (libtkstub*.a)
     instead of the standard library, and the compile flags need to be changed
     to defined ACCESS_TK_THROUGH_STUB.

 Inline versions of the stub interfaces are available, they uses macros in C
 and inline functions in C++ to access the library's interface. The Tcl inline
 interface is used if INLINE_TCL_STUB is defined before including tcl.h, or you
 can include tclStubInls.h directly. Similarly for Tk. The stub library is still
 needed to ensure that the stub initialisation is done correctly.

 Unix users should take a look at the tclConfig.sh and tkConfig.sh file to see
 what variables are available to help configuring stubs.

 How to create an extension with a different compiler ?
 ------------------------------------------------------

 All that is needed to do this is to build the stub library with your compiler.
 e.g. assuming that you have a Borland compiler and want to build an extension
 which works with a Visual C++ built core you need to do the following.

 1)  Get a patched version of the source and make tclstub80.lib. e.g.
     make -f makefile.vc.stub tclstub80.lib
 2)  Build your extension making sure that you use the Borland version of the
     stub library.

 Although it is possible to link objects built with different compilers together
 in most cases, each combination of compilers requires a different process to do
 it and starts to get very complicated when you have to link an extension to two
 or more libraries which have been built with a combination of compilers and
 compiler versions.

 This method should work for all combinations of compilers because it only
 requires that function calls are compatible; if they are not then the compilers
 cannot be used together anyway.

 Changes
 -------

 The following lists the changes made to Tcl.

  o  Improved the support for AIX shared libraries. By making use of
     export/import files I was able to create Tcl and Tk libraries which are
     shared objects rather than archives containing shared objects. This makes
     it possible to dynamically load Tk on AIX now. Extensions which are
     correctly configured should not need to make any changes, extensions which
     are also libraries will need to create an export file and use that instead
     of the library when linking. See the Tk configure.in file to see how it
     does it.

  o  Extended the package mechanism to support stubs, and added the ability to
     determine whether a package has been loaded without causing it to be loaded.
     See the PkgRequire.3 man page for more details.

  o  Added ByteArray type. There is a major problem when porting extensions which
     wish to handle binary data from 8.0 to 8.1. In 8.0 they can use string
     objects to hold the binary data, unfortunately this will not work on 8.1
     because the internal representation of strings is now UTF. Binary data in
     8.1 is handled using ByteArray objects. Therefore I have added them to
     improve the upward compatibility.

  o  For each function which takes a variable number of arguments I have added
     a function which takes a va_list instead. This is so that I can create
     wrapper functions around them. These new functions names are the original
     function names appended with "VA"

 The following lists the changes made to Tk.

  o  Modified Tk_Main and TkConsoleCreate to work properly with stubs. The
     problem is that those functions and the other Tk functions that they use
     should not actually be in the standard Tk library because they are only
     really needed by applications. The problem that they cause with stubs is
     that they are called before the Tk initialisation function is called and
     therefore the pointers to the Tcl stub tables have not been properly set up
     in the Tk library. The solution was to make them take a Tcl_Interp parameter
     (which is used to get the Tcl stub table pointers) and to get them to
     initialise the pointers as well. The change to Tk_Main is both binary
     compatible and source compatible but the change to TkCreateConsole is not
     because no interface is provided for those functions.

 Future
 ------

 There is one more thing that I have left to do which will make this mechanism
 even more useful; that is to make the libraries use their stub interface to
 access their own functions. This will allow dynamic extensions to change the
 behavior of the libraries, either to fix a problem or to add some function.

 Plus Patch
 ----------

 The plus patches also have a stub interface available for them although it only
 supports a subset of Tcl functions and one Tk function. They are available from
     <http://home.wxs.nl/~nijtmans/plus.html#files80>;

 Acknowledgements
 ----------------

 Jean-Claude Wippler <jcw@equi4.com>
     For the initial seed idea from which this all came from and also for
     his continued support and feedback during development.

 Jan Nitjmans <Jan.Nijtmans@wxs.nl>
     For his ideas and feedback which improved the final result and made
     me think.

 Contact
 -------

 Paul Duffin <pduffin@mailserver.hursley.ibm.com>

 --
 Paul Duffin
 DT/6000 Development    Email: pduffin@hursley.ibm.com
 IBM UK Laboratories Ltd., Hursley Park nr. Winchester
 Internal: 7-246880    International: +44 1962-816880