Blocklayout version 2
=====================

Other places of information:
    * http://wiki.xaraya.com/index.php?title=Blocklayout_Version_2
    * RFC-0054 (which should receive most of the specs written here)

Concept change:
---------------

Instead of grabbing each character in the input and parsing out the templates
we implement a 'pass through, unless' type XSL transformation. This means that
each tag we want to do something special to the output has one or more XSL
templates which match it in context and transform the input to the desired
output. The specifics of the transformation are determined by context and the
value signature of the attributes. This whole process is XML based, from
beginning to (just before) the end. All rules XML implies, apply to this
process too.


Issues
------

- [STARTED] how do we create a suiteable test suite (make a compilation of the
  core templates?) Testsuite in theme XClassic
  modules/base/user-main-compilertest.xt]
- can we make a stub inserting some random values for the template vars,
  so we can compare somehow
- how do we handle #$var# constructs?
  - ideally i want to handle it through separation of the template in two sections, data and presentation, both in the xml domain:
  - one way of doing that is to make #$var# ~ &var; but this is a pain to    handle for XSLT, since it assumes entities to be known/declared at    transform time, which is clearly not the case
  - another separation mechanism is to create a "data" section (xml fragment) to go with the template

  Example::

        <tpldata>
          <vars>
            <var name="var">value</var>
              ...
          </vars>
        </tpldata>

    or something like that, generated dynamically, From then on we can
    reach each var by using XPath expressions like

      /tpldata/vars/var[@name='varname']

    which sounds sort of attractive because it is almost exactly like the
    array stuff, but then XML compliant. It also means that we would need
    to translate each and every template to this syntax.
- the xarBLCompiler.php does some processing here and there, which of
  these need to stay in php, which of them can be done by xsl? We can take
  them on a case by case basis, since php functions can be called
  reasonably easy from within the transform, but each case is a weakness
  in portability so we try to avoid it.
- it really doesnt make sense anymore now to go through the hoops of
  registering custom tags etc. One xsl snippet for a custom tag,
  generating the right code is a lot easier. Note: this would also
  invalidate the whole GUI where tags are shown on screen and can be
  manually entered into the database, which is of questionable use anyway,
  apart from a debugging perspective.
- go over all xar: tags attributes and decide how resolving should be
  handled, dont add a resolvement unless you have a usecase.
- our # delimiter conflicts with the use of generic entities in not-so-rare
  cases.

  Example 1::

      #$modinfo['adminurltitle']#&#160;#$modinfo['displayname']#
       \________first expr_____/ | \__/  \_________text_______/\_ lonely #
                                 |   \_ second expr '160;'
                                 |_ single & 'text'`

  Example 2::

    `##$variable##`

    Could be:

        [empty expression][text: $variable][empty expression]  OR
        [text: #][content of $variable][text: #]

    depending on how it's looked upon

  Example 3:

    Inline CSS:

        <style>
        #someID  { color: red; }
        #someID2 { color: blue;}
        </style>

    As seen through compiler eyes: [expression: 'someID { color: ref; }\n'][text: 'someID2 { color: blue;}']
- we are now in a postion to create a DTD, let's do so.
- instead of organising by tag, it seems to make more sense to organize by
  'group' (flow, comment, blocks etc.) as those tags often interrelate and use
  similar constructs in their handlers.
- 'Friendliness' options:
    * if a variable is not set, boom! Why not wrap it with an isset() for
      example? Or give it a predefined !!UNKOWN!! value?
    * can we separate the parse and render stages? during parse errors,
      the rendering bombs out in ugliness, where it possibly doesnt need to
      do that.
- put the new tags locations into the dirlayout proposal
- we omit the xml declaration now, because it doesnt work, but i dont remember
  why anymore. Either make it work (preferably) or document why it can't.
  This has to do with <?xml conflicting with <?php when producing code.
- declare xml specials somewhere:
    - <!ATTLIST element xml:space (default|preserve) 'preserve'>
        for elements which may choose
    - <!ATTLIST element xml:space (preserve) #FIXED 'preserve'>
        for elements which should have space preserving.
- [DONE] is merging with other output namespaces just a question of copying
  output (xhtml in our case) (A: yes)
- [DONE] how to cross the border? i.e. how do parameters from the module
  get passed into the xslt processor (see parameter section below and
  xsltransformer.php )
- [DONE]the construct <div style="clear: left|right|both"><div> appears to 
    render improperly.
    See:
        - http://www.stylusstudio.com/xsllist/200209/post10350.html
        - mtn rev: 6143d31238f3aaa6309a390f16fca034b9d1cda2
- [DONE]javascript construct:
    <script type="text/javascript">Calendar.setup({#$js_options#});</script>
  (from the dhtml calendar at dynarch.com) appears to render improperly.
    This turned out to be a `<script />` empty tag issue, as above with the div.

- [DONE] we now add a root tag to module templates artificially. This leads to a
  number of things we can not do now: 
  - as we 'pre-read' the templatefile to add this tag, the filename of the
    template gets lost when an exception is raised. We really should add the root
    tag to the templates themselves. (this can be automated)
  - individual templates are not well-formed (interoperability) (all core 
    templates have had their root tags added in)
- [DONE] an empty template with just <xar:template xmlns...> </xar:template> gets an
  Error: 0: FileNotFoundException The file "modules/base/xartemplates/includes/.xd
  could not be found. Workaround, add a comment inside the tags.
  (Cause: we strip redundant spacing, so from the pov of the compiler its a closed tag,
  thus processing it as the 'include' variety. Fixed by explicitly testing on the
  file attribute. This keeps the error when file="", which is what we want.)

Refactoring needs
=================

While fixing the errors along the way, some redesign issues have surfaced.

* generating XML can confuse HTML renderers. The empty div and script issues
  are not really compiler/blocklayout issues, but many browsers just dont
  deal well with empty tags. There is a solution (see xar2php.xsl) but it really
  should only be deployed when the output type *is* actually (X)HTML. We can
  deploy the fix for all XML varieties, as it doesnt change semantics, but its
  not very nice. We plan to separate the *format* in the process anyway, so lets
  keep this in mind when we actually will.
* We generate a xml + php document from (now well formed) xml, thus losing
  some of our power (the compiled templates are not necessarily well formed)
  This means we get into trouble with the DOCTYPE generation, and using the
  xsl:output attributes for this, which would be the correct way, will lead to
  some changes in the xar: language, as we can not set the dtd in the blocklayout
  tag anymore then. (which makes sense, it was an emergency exit anyway). Nevertheless,
  that means we have to go to the drawing board on how to do this.
* language changes (see section further on)


Language change proposals:
==========================


Expression resolving
====================

Before going into the issues at hand, a couple of general statements on
templating engines, or templating languages.

There are a lot of them! By far, the most are sloppy, ill designed, slow or
just grew out of a hack. A few of them are real gems to use though and well
thought out. Without exception, they all have some sort of mechanism to
control flow, set and use variables etc. in an extension to their 'hosting
language' Sometime built as 'escape mechanisms' into the programming language
in which the template engine was written, sometimes defining its own
mini-language.

To further narrow the templating languages we consider interesting; the XML
native ones. By this, I mean templating languages which, considered as such,
are a valid XML document. There are other means to construct XML 'based'
templating engines, meaning they look like XML, but dont use much more than
some tags to describe the templating constructs and escape to other mechanisms
for defining structures or operations. We wont consider those here.

So, what techniques do we have to make up a native XML templating language
with sufficient power?

Technique 1. namespaced taglib to define operations. : <xar:sometag />
----------------------------------------------------------------------

The obvious one, is to define a tag lib in a namespace specifically for the
templating domain. Blocklayout is no exception. We define the xar: namespace
as ours and all tags in that namespace are handled by BL.

What you can do with a tag is pretty limited. It has a name, is empty or open,
and may define attributes which have values. Combining all of those leads to
lots of different options, but it's all still static, so that is why most TPLs
come up with an expression language to bring variables into the picture. More
on that later on. Xaraya uses this techniques extensively, but often in inconsistent
ways, or even invalid ways.

Technique 2: namespaced attribute lib to extend sibling taglibs. : <div xar:extra="test"/>
------------------------------------------------------------------------------------------

What only a handful template engines use is the namespaced attribute
technique. Similarly to the namespaced tags, you can employ namespaced
attributes to be handled by your engine. BL doesnt use this technique yet
either. The key addition this brings to the table is that you can hook your
namespace into others. Say you combine html and blocklayout in a template a
div tag, belonging to the html (default) namespace can be (legally) extended
by: 

    <div style="border:thin" xar:extra="somevalue">
        <img src="myPicture.png"/>
    </div>

Assuming the namespaces are properly tagged onto the document, this is
perfectly valid XML. This allows a template engine to extend another namespace
with extended behaviour. Still, the same XML rules apply and all information
is still static.

The above is really what XML has to offer. Through clever combinations of both
techniques it may seem there is more, but there really isn't :-)
Defining multiple namespaces (one for structure, one for data for example) and
defining ways in which they should interoperate is one such method which seems
to add additional things.

Technique 3: define dynamic behaviour through an expression language
--------------------------------------------------------------------

Here's where the problem starts. XML, as such, unless implemented explicitly
in one of the two previous techniques doesnt define dynamic behaviour in
itself. There is however a staggering amount of ways to give it one: 

    1. XPath expression language to point to XML things and execute some 
       functions on them
    2. Extensions of XPath like XPointer and XLink for example
    3. XQuery to search into XML
    4. An event model layered into what is known as a DOM (into 3 levels) which
introduces an indirect way to manipulate XML documents by modelling them as an
hierarchical object. It's not entirely clear where the OnEvent like attributes
come into play. (that is, to which doctype they actually belong)
    5. Choose a specific doctype like XHTML and you get another set of dynamics,
       specific to what the dialect was designed for (xul+javascript for example)
    6. Databinding specifications like XForms and RDF

and last but not least the custom expression languages which every template
engine i ever came across has.

Looking at the future for Blocklayout, technique 1 will continue to dominate
how people perceive the language. One thing which sets apart BL from many
other template languages, is that most of its operations are encoded in the
tag namespace and not as a specific expression language. The possibilities of
using namespaced attributes we havent begun to explore yet. A weakness which
BL has had from the beginning, perhaps due to the intial focus to encode the
operations as XML tags, is the definition of its expression language, or the
lack thereof.

In short, we escape to PHP through a # character and try to evaluate and pass
back whatever the PHP delivers in the expression. This has worked reasonably
well because we compile the BL tags to PHP and it makes perfect sense to
inject the expressions in PHP too.

We allow expressions in pretty much an ad-hoc set of places. They are almost
always allowed in XML text nodes, sometimes allowed in attributes, but not
everywhere and there are also two ways of interpreting them. Some tags require
a direct-PHP like syntax (like the xar:data* tags), others adhere to the #..#
enclosed syntax while quite a few of them dont support expressions at all. All
of these grew on us over time.

