Framework directory layout (outwebroot scenario)
================================================

Having literally everything below the webroot is limiting deployment choices.
Below are some thoughts on how we could organize the files and directories in 
a way where things which do not need to be below the webroot, can be placed 
outside of it. 

Before we outline the directory layout, lets list some definitions and assumptions 
so we can make informed decision on the directory layout.

Definitions
-----------
    'install':
        An 'install' is one copy of the framework code, identified by a
        version (number, hash or otherwise) at a certain location.
    'instance':
        An 'instance' is the result of using one install to create one or more sites.
    'database': 
        A 'database' is a storage container which holds the data for exactly one instance. 
    'site':
        A 'site' is a collection of data within one instance belonging to 
        one website.

Assumptions
-----------

1. Each 'install' can have one or more 'instances'.
2. Each 'instance' has a default 'site', in absence of extra information, data belongs to the default 'site'
3. Each 'instance' can have many 'sites', but always has at least one.(master-site?)
4. Each 'site' belongs to exactly one 'instance'.
5. One server can support many 'installs', possibly differing in version.

Consequences 
------------

* the changed directory layout satisfying the above assumptions (that was the
  aim)
* introduction of a 'site' concept/object
* introduction of a 'instance' concept/object


Directory layout
----------------
The above suggests:

* 2 main locations, one for the install and one for (each) instance
* we need at least one webroot per instance, it makes sense to leave it at
  that too and let sites within one instance share their webroot. This
  effectively links one webroot to one database, which is easy to remember.
* the default site is implicit and sites can be managed from within the
  instance so we dont need to explicitly create directories for it. This makes
  a multi site instance look the same as a single site instance from a
  files/dirs perspective.
* its obvious that the 'install' area needs a place to hold code and libraries
  => lib This area will be the main include area for the program. At the same
  time we want to have the option to transparently include and use code
  specific to one instance, so we mirror that lib directory in the instance
  location.
* to distinguish libraries from us and third parties below lib there is the
  'namespace' xaraya which can hold our libraries. Third party libraries also
  get their own namespace (directory)
* to bootstrap an instance and tie it to its database, a file containing the
  configuration for that is needed (current config.system.php I guess) To hold
  this and possible other configurations => conf (xml/txt/php tbd) The global
  environment also has a conf location to provide sensible defaults
* each instance may/will generate variable data related only to its specific
  configuration => var. This has no global equivalent in the install.
* locale information is typically global (note: this is not the same as
  translations!) so we designate a global location for that => locales We can
  look up the most convenient way to store locale information, i.e. one file
  per locale or a directory with multiple, doesnt matter much.
* modules will definitely have to go outside the webroot, as we want to have
  them both globally installed (for all instances) and at the specific
  instance levels.

Global environment:
-------------------

The global install environment should be viewed upon as something like 'xar.exe'
it's an immutable collection of things which are used to generate new instances
(i.e. a new 'default site'), and provide defaults for the sites using this particular install.
"Running xar.exe" could be done through a web interface or a cmdline, whatever
we have time to create. (ideally both).
Translation from our current situation, it could be viewed upon as a 'consolidated installer'
or 'server admin part'; one level higher than what our current UI exposes. 
The 'global environment' needs to be configured in the webserver specifically::

     /path/to/install
         /bin                            [cmd line interface for the managing of instances]
             ...                         
         /conf
             xaraya.conf                 [global configuration defaults]
             /skeleton                   [entry point for instance skeletons, used during creation of new instances]
                 ?
         /html                           [web interface of the install]
             .htaccess
             default.php                 [named it default.php to indicate its different than index.php in the instance]
             /themes                     [themes globally available for instances]
                 /default
                     package.xml         [theme meta information]
                     /style              [styling files]
                     /scripts            [client scripts]
                     /images
                     /pages              [page structure templates]
                         default.xt      (why do we use .xt vs .xd again?)
                 ...    
         /lib
             /xaraya                     [ xaraya library, 'core']
                 ...
             /zend                       [ 
                 ...                       third party libraries like Zend, propel,
             ...                           binarycloud, PEAR, etc.                             
             /greatlib                   ]    
         /modules                        [modules globally available for instances]
             /globalmodname
                 package.xml             [module meta information]
                 /api                    [offering]
                     ?
                 /gui                    [presentation]
                     ?
                 /lib                    [implementation]
                     ...
                 /data
                     schema.xml          [storage container definitions]
                     data.xml            [default data for initial populate]
                 /templates
                     ?
             ...
         /locales
             /en_US
                 locale.xml              [ locale specification ]
             /nl_NL
                 locale.xml

Instance environment:
---------------------
The instance environment is created through the 'xar instance manager'. The effect of
using the instance manager is to 'generate' the necessary structure for starting a 
(collection of) websites. The necessary directories are created, the proper config is
put into place etc.

A minimal instance might look something like this::

    /path/to/instance
        /conf
            default.conf                
        /html
            .htaccess
            index.php
        /var
            /default
                /cache
                    /output
                    /templates
                /logs
            

An instance structure showing all possible options might look something like this::

    /path/to/instance
        /conf
            default.conf                [default site config, plus defaults for all sites]
            siteX.conf                  [siteX specific config]
            siteY.conf
        /html                           [webroot for all sites in the instance]
            .htaccess                   [webserver specific config file]
            index.php                   [entry point]
            /themes
                /themename
                    package.xml         [theme meta information]
                    /style              [styling files]
                    /scripts            [client scripts]
                    /images
                    /pages              [page structure templates]
                        default.xt      (why do we use .xt vs .xd again?)
        /lib
            /specialstuff
            ...   
        /modules
            /localmodname
                package.xml             [module meta information]
                /api                    [offering]
                /conf                   [additional configuration]
                /gui                    [presentation]
                /lib                    [implementation]
                /data
                    schema.xml          [storage container definitions]
                    data.xml            [default data for initial populate]
                /templates
            ...
        /var
            /default
                /cache
                    /output
                    /templates
                /logs
            /siteX
                /cache
                    /output
                    /templates
                /logs
            /siteY
                /cache
                    /output
                    /templates
                /logs
            /modules
                /modname                [module modname's data area]
                    ...

Analysis
--------

* As there is a global and instance environment now, we basically also create
  a new override point, which we may or may not want to exploit. An example is
  to override a module template in the instance area. The 'meaning' of this 
  override would be: "i want this to apply for this instance, regardless of
  theme, unless the theme overrides it specifically."
* 
          
Question and Ansers
-------------------

    Q:    how do we deal with themes and modules? themes typically want to have 
    access to webroot accessible stuff like styles and images, modules too, but they 
    shouldn't :-) i have a tendency to place modules outside the webroot, 
    but that may be really uncomfortable.
    A: Themes go below the webroot of an instance. Modules go outside the webroot which
    means that modules need to be offered strong tools to access files in their own
    parts. 

    Q: how do we define module data which is file based? (in terms of what is 
    data and how, for example does it relate to things inside xardata like 
    directories) Self contained modules are quite comfortable.
    A:

    Q: where do client libraries like js frameworks go?
    A: either they go directly below the 'scripts' directory in the theme or are
    made available in the lib section globally or in an instance. 

    Q: why are locales needed on fs? 
    A: 

    Q: how much webserver specific configuration like aliases etc. would we 
    need for this
    A: - the install needs to have a configuration in the webserver probably, the
    instances are generated from that install, so we can make sure things are properly
    generated for each instance without having a need for a specific configuration
    in the webserver.

    Q: is there a description why all this is better than just put stuff below the
    webroot?
    A: yes, see section below

    Q: do we want (parts of) in lib or have (core) as the main 'Application' 
    (pro/con?)
    A: lib/xaraya would be the old 'core' in the proposal.

    Q: if there is such a thing needed as cgi-bin, where does it go in this 
    picture? if at all
    A: nowhere by default.

    Q: this assume a one tree deployment, what if we want to repeat this for some
    reason. Should we take an 'order' into account, i.e. first look in the site 
    tree, then look in the global tree? (cfr. trac organisation)
    (aka: where do i put my custom code/content/data?)
    (aka: where do i put my php snippets?)
    A: This question was from an earlier proposal, the formulation of the definitions
    and the assumption led to a split into a global environment and an instance
    environment which more than caters for the problem outlined in the question.

    Q: we probably want an easy-access taglib location, so it's easy to create 
    custom tag definitions (where would that go?)
    A:

    Q: how often can the include_path on servers *not* be changed? That is, how
    common is it to disallow calls to call ini_set()?
    A:

    Q: how does multisites affect the above in the doc_root piece? (1 or many? both?)
    A: one instance serves a multisite setup from one doc root.

    Q: what impact does  1 codebase multisite/multiapp on this layout? (db or storage 
    is following the application config, so multi-db is less of an issue in this area)
    A: see the description at the top of this file.

    Q: what is a good way to assess the magnitude of work on this? That it is a 
    lot we know, but a metric would be nice.
    A:

    Q: somehow modules seem painful in this layout, 'too big' comes to mind, 
    'too loose' too. Meant in the way that modules do so much in our system, it's 
    hard to give them a proper place. Where do we put them?
    A: see above, outside the webroot at the instance level.

    Q: where would we put framework wide shared stuff (the equiv of 
    /usr/share/some_app/ in unices) things like a standardized iconset (which 
    themes can override obviously) Stuff like that. (data, I guess)
    A: in the install global env at least.

    Q: wont this scare more users away as it becomes more complex?
    A1: There is the option of distributing with the new configuration
    parameters set to those values which would lead to the exact same setup we
    have now, thus having the same complexity on an initial install. We can discuss 
    this when the time comes to distribute the package and see what is desired. As
    the proposed extension to have locations configurable can be made backwards
    compatible, the current directory layout is _one_ possible configuration.
    However, it is my expectation we would want at least some things to change, but
    we can deal with those when the time comes.
    A2: with the above description, A1 is not true anymore, its impossible to mimick
    the exact same setup, as currently we have no split between global and instance. 

Why is the above better than all below webroot?
===============================================

Reorganizing the directory layout is intrusive, very intrusive. Why is it 
that we are even considering to do this? This section hopes to clarify some
of the reasons.

Security argument
-----------------

This is a simple argument. Anything below the webroot is in theory accessible 
by the webserver. There are special actions needed to prevent certain access. 
Putting in a special config directive in the webserver configuration, putting 
empty index.html files in folders, checking wether files are accessed directly, 
where they should not have been etc. etc.

By putting files outside the webroot of which we know upfront the webserver 
doesnt need direct/URL based access to (a php program running on that server may 
have/need access to them however) we make that access impossible, so the amount
of measures needed is less, and if files contain code which would be risky to
put in an URL accessible file it is protected from such access by default.
In short, this makes the people who admin a server sleep better.

Deployment argument
-------------------

Xaraya is often used as a hub to manage more than one, of even a lot of sites.
Currently, if not using special deployment tricks like symlinks or bind mounting,
the code has to be put into each Xaraya install. Typically, the code used in
all those installs is for the most part identical except for the app/site specific
parts the site has to have.
Allowing to put common parts in a shared location can save a considerable amount
of diskspace (esp. in php, since all php files are usually bincached too) and
provide extra flexibility to run different core versions.
Specifically for 'xaraya hosting' this is interesting since the shared install
can be pre-made for the clients, where they only have to worry about their specifics.

Integration argument
--------------------

Being a framework, xaraya often integrates other libraries and is integrated 
often in larger scope setups _as_ a library. Other packages all have their own
rules and peculiarities on how they deal with files and where to put them. 
Taking a library approach provides more options and can make integration
activities a lot easier when there are configuration settings provided on how
to deal with file locations, include paths etc. 


Deployment 'before and after' stories
=====================================

    here's where we would list known deployment scenarios in the current
    situation and compare it with how the new situation would look.
    We can then compare it in terms of disk size, backup strategy, site mgmt.
    coding efficiency and other metric we deem important.


CURRENT 1.x Xaraya SETUPS
-------------------------

A. One Install, One (or more) Instances, Multiple Sites

Generalized actual example, draft, needs tidying yet::

    /path/to/instance
        /html
            .htaccess
            index.php
            /modules
            /db middleware (adodb/creole)
            /var
                 config.system.php           [may be used sometimes for 'default' site]
                 .key.php
                /cache
                /locales
                /logs
                /messaging
                /otherglobal module dir usage [eg config dirs, dirs requiring write access or global access]
            /themes
            /mythemes        [have to keep theme dirs in webroot atm, so this places some restrictions
                             I keep a separate global theme dir for easier upgrade/maintenance]
            /shared func/mod/library stuff [sometimes used, keeps custom code etc together]
              /shared custom modules  -symlinked to main mod dir

    /path/to/sites
         /site1
            config.system.php
            /var
                /locales
                /cache
                /otherlocal module dir usage
           /themes [only possible atm if the path/to/sites is in webroot. Required if sites need own theme dir]

         /site2
            config.system.php
            /var
                /locales
                /cache
                /otherlocal module dir usage
 
            /site library files  [keep them in specific site dir for easy maintenance backup -
                                 symlinked where/if necessary to main install or theme dir]
               /functions
               /modules
               /themes [would be good to be able to point to this but can't unless path/to/dir is under web root]
               /whatever

Path to the sites are, if possible, kept ouf of the webroot.
However in some 'instances' if theme directories per site are required, and need to be isolated, 
this path-to-site needs to be moved into a path in the web root (eg each site has their own 
isolated theme dir in their site config). 


COPY from older wiki to be consolidated here:
=============================================
* Modules: do they need to be completely self-contained in a single directory branch? There are some advantages and disadvantages in doing so. I think it would be easier for the application if the module functions were kept in one place, and the templates (for the various output types) kept on another place. That way, all the default templates for all modules, for any particular output type, are kept in one place, where they can be easily managed (e.g. there could be a "default XHTML theme" - assuming that isn't an over-simplification - that templates would fall back to, rather than falling back to special templates in the module area). The problem arises during development, where having a module split over several branches would be more difficult to manage.

!!!Alternative file structure – Dedicated user space

This is just an alternative to the current xaraya file structure but contributes a lot of change to the xaraya design towards a more “framework” structure. Some of the idea’s here are borrowed from frameworks including, apache struts, eclipse and prado.
The main directories are mostly the same as the current setup except for framework and applications. The structure is based on the Model View Controller design pattern.

<pre>
<framework>
<modules>
<xaradobdb>
<applications>
Index.php
Upgrade.php
Etc.php
</pre>

-Framework:
Possible alternative to the current “includes” directory and will contain the core files to xaraya. Elements that are considered essential to xaraya will be thrown in here and called here.

-Applications:
Here is where the big difference is.  The core and modules will team up to construct functionality which is implemented by applications. Similar to how eclipse has the “features” and “plugins” folders. The separation of functionality and implementation creates a space dedicated to the application developer or “the user space”.

The application folder replaces Themes. Templates will be overwriten by applications and there specific views (pages). Modules and applications have a one to many relationships in the sense that several applications can implement the functionality of a single module. This emulates the itemtype concept currently being used in xaraya.

The application tree could look like the following
<pre>
<applications>
  <blog>
    <models>
    <views>
    <controllers>
    Blog.spec
  <forum>
    <models>
      Thread.xml
      Poll.xml
    <views>
      Showthread.xt
      Edithread.xt
      Editthread-overwrite.xt
    <controllers>
      Editthread.php
    Forum.spec
</pre>

Spec will include application definitions for url encoding/decoding, data sources, security masks, and act similarly to the init.php file inside modules.

This kind of file structure has a lot of advantages. Packages will be easy to download and configure. Instead of just providing the core with modules, applications allow specific solutions to be packaged together. Users can simply download the applications they want with less or potentially no additional configuration. Putting the configuration of applications into files instead of database makes it easier to describe and accomplish common tasks.

The clear disadvantage to this is the elimination of themes all together. The theme behavior is still there its just slightly less accessible. This is a tradeoff. It pushes xaraya in the direction of being a framework more the CMS. This isn’t terribly serious. It is possible still, under this structure, to implement a content management application that has a mechanism (in the controller) to allow themes.
