ILearnable .Net

Using T4 to generate multiple versions of xml config files with inheritance

(NB: baked into this sample is a method for generating multiple output files from T4, it is stolen shamelessly from Damien Guard (thanks Damien!), except I couldn’t get his deleting of files no longer needed to work (it deleted needed files as well) so I commented out that bit and it worked like a charm.
For a full description of that part of the script go to http://damieng.com/blog/2009/11/06/multiple-outputs-from-t4-made-easy-revisited and read Damiens excellent blog)

quite a while back I got really irritated with how .Net does configuration files. My main gripe is that there is no easy way to
relate the configuration for the local dev environment, build server, test server(s), staging server(s) and production server(s).

let me give you an example: In a standard Web Application Project the web.config file is a single file which is used to define all the configuration
data for the entire site:
- WCF web services exposed
- WCF web services used
- logging
- App settings
- etc.

Now i do know that you can split the file up where each section can have its own file, and use different files for the different environments,
but that doesn’t really solve the problem as
- you end up with HEAPS of files if you want fine grained control
- you cannot really use that strategy for having differences within a section (appSettings for instance)
- you need to do a lot of file copying magic in order to get the correct sections together for a workable web.config for a given environment
- even if you only need one tiny difference between say testserver1 and testserver2 the entire section must be duplicated into two different files
(major DRY violation)

I have spent a lot of time pondering how to do this better (WAY too much time actually, but then I’m not really that smart ) and have come up with
a way to define configuration files using a sort of inheritance scheme. That way you can have a setup like this:
- web.config.root ( – file that defines 90% of the configuration, everything that is common throughout all environments)
- web.config.build ( – contains specifics for the build server )
- web.config.dev ( – contains common specifics for local dev instances )
- web.config.dev.local ( one per dev, not checked into SCM – contains specifics for each dev, local testing ground)
- web.config.test ( – defines common specifics for all test servers )
- web.config.test.testserver1 ( – defines specifics for a given test machine )
- etc.

and then have some engine process these source files in a controlled order. For instance:
- local web.config (used for local development) = web.config.root + web.config.dev + web.config.dev.local
- buildserver web.config = web.config.root + web.config.build
- testserver1 web.config = web.config.root + web.config.test + web.config.test.testserver1

so that for the testserver1 config file the web.config.root would first be applied, then settings in web.config.test would be applied, adding to
and overriding the settings from the root file. and lastly the settings in web.config.test.testserver1 would be applied, adding to and overriding
the settings from the resulting merge between root and test. (phew!)

There are two main problems with such a setup that must be solved:
A) how to identify which setting to override when there are multiple possibilities
B) how to define a setting in a base config file, then delete that setting in an overriding config file

for solving A) I have two rules:
- if the element is an only child (or at least the only child with that element name) then that element can be
overridden / expanded by subsequent input files.
- if an element has siblings with the same element name then it must be identified by an identificator attribute.
For standard .net configuration they are (in prioritized order) “id”, “name”, “key”, “path”.
- if there are equal siblings without any of these identificators the whole process throws up and informs that an ambiguity exists.

for solving B) I introduce the magic keyword DELETEME. I’ll give two examples:

1)

<configuration>
	<location path="Documents" DELETEME="true" />
</configuration>

2)

<configuration>
	<system.web>
      <httpRuntime maxRequestLength="DELETEME" />
	</system.web>
</configuration>

the config in 1) will delete the entire configuration for location path=”documents”
the config in 2) will delete the attribute maxRequestLength but otherwise leave the httpRuntime element alone

so, on to the code. To get the files generated automatically I’m using T4. I use two files, the first is a .ttinclude file which defines both the
xml merging logic and logic for generating multiple output files:

first, here’s the ConfigurationMerger.ttinclude file

<#@ assembly name="System.Core"#>
<#@ assembly name="System.Data.Linq" #>
<#@ assembly name="EnvDTE"#>
<#@ assembly name="System.Xml"#>
<#@ assembly name="System.Xml.Linq"#>
<#@ import namespace="System"#>
<#@ import namespace="System.CodeDom"#>
<#@ import namespace="System.CodeDom.Compiler"#>
<#@ import namespace="System.Collections.Generic"#>
<#@ import namespace="System.Data.Linq"#>
<#@ import namespace="System.Data.Linq.Mapping"#>
<#@ import namespace="System.IO"#>
<#@ import namespace="System.Linq"#>
<#@ import namespace="System.Reflection"#>
<#@ import namespace="System.Text"#>
<#@ import namespace="System.Xml.Linq"#>
<#@ import namespace="Microsoft.VisualStudio.TextTemplating"#>

<#+ 
   
Dictionary<string, string> MergeConfigFiles(string inputFilesFolder, string outputFilesFolder, Dictionary<string, string[]> setup){
	var ret = new Dictionary<string, string>();
	var merger = new XmlConfigMerge("name", "id", "key");
   Dictionary<string, string> configFiles = new Dictionary<string, string>();
    DirectoryInfo di = new DirectoryInfo(System.IO.Path.GetDirectoryName(this.Host.TemplateFile)+inputFilesFolder);
	
    foreach(var f in di.GetFiles())
    {
        configFiles[f.Name] =File.ReadAllText(f.FullName);
    }

    foreach (var outputFileName in setup.Keys)
    {
        List<XDocument> xdocs = new List<XDocument>();
        foreach (var inputFileName in setup[outputFileName])
        {
            if (configFiles.ContainsKey(inputFileName))
            {
                xdocs.Add(XDocument.Parse(configFiles[inputFileName]));
            }
        }
        string result = merger.MergeConfigs(xdocs[0], xdocs.Skip(1).ToList()).ToString();
     	ret[outputFilesFolder+outputFileName] = result;
    }
	return ret;
}
	

public class XmlConfigMerge
    {
        public string DeleteKeyword = "DELETEME";

        public XmlConfigMerge(params string[] unique)
        {
            UniqueAttributes = unique.ToList();
        }

        public List<string> UniqueAttributes { get; set; }

        private XElement FindByAttr(IEnumerable<XElement> elements, string attribute, string value)
        {
            return elements.FirstOrDefault(e => HasAttribute(e, attribute, value));
        }

        private bool HasAttribute(XElement elem, string attributeName)
        {
            var attr = elem.Attribute(attributeName);
            if (attr != null
                && !string.IsNullOrEmpty(attr.Value)
                )
            {
                return true;
            }
            return false;
        }

        private bool HasAttribute(XElement elem, string attributeName, string attributeValue)
        {
            if (HasAttribute(elem, attributeName))
            {
                if (0 ==
                    string.Compare(elem.Attribute(attributeName).Value, attributeValue,
                                   StringComparison.InvariantCultureIgnoreCase))
                {
                    return true;
                }
            }
            return false;
        }


        public XDocument MergeConfigs(XDocument template, List<XDocument> dataFiles)
        {
            if (dataFiles == null || dataFiles.Count == 0) return template;
            var temp = template;
            foreach (var data in dataFiles)
            {
                temp = MergeConfigs(temp, data);
            }
            return temp;
        }

        private XDocument MergeConfigs(XDocument template, XDocument data)
        {
            var rootNodeTemplate = template.Root;
            var rootNodeData = data.Root;
            MergeConfigs(rootNodeTemplate, rootNodeData);
            return template;
        }

        private void MergeConfigs(XElement template, XElement data)
        {
            bool shouldDeleteNode = ShouldDeleteNode(data);
            if (shouldDeleteNode)
            {
                template.Remove();
                return;
            }

            MergeAttributes(template, data);

            if (!data.HasElements)
            {
                if (data.Value == DeleteKeyword) template.Value = string.Empty;
                else if (!string.IsNullOrEmpty(data.Value)) template.Value = data.Value;
                return;
            }


            var templateElements = template.Elements();
            var dataElements = data.Elements();


            foreach (var dataNode in dataElements)
            {
                KeyValue uniquey = GetUniqueKey(dataNode);

                var matchingNodes = templateElements.Where(x => x.Name == dataNode.Name);
                if (matchingNodes.Count() == 0)
                {
                    template.Add(dataNode);
                    continue;
                }

                if (uniquey == null && matchingNodes.Count() > 1)
                {
                    throw new ApplicationException("Cannot merge into file containing multiple undelimited equal siblings, needs attribute with name: " + string.Join(", ", UniqueAttributes.ToArray()));
                }
                if (uniquey == null && matchingNodes.Count() == 1)
                {
                    var templateNode = templateElements.Where(x => x.Name == dataNode.Name).First();
                    MergeConfigs(templateNode, dataNode);
                    continue;
                }

                if (uniquey != null)
                {
                    var ambigous =
                        templateElements.Count(
                            x =>
                            x.Attribute(uniquey.Key) != null &&
                            x.Attribute(uniquey.Key).Value.ToLower() == uniquey.Value.ToLower()) > 1;
                    if (ambigous)
                    {
                        throw new ApplicationException("Cannot merge into file containing multiple equal siblings with same identifier, needs unique value for attribute with name: " + string.Join(", ", UniqueAttributes.ToArray()));
                    }
                    var templateNode = (from x in templateElements
                                        where
                                            x.Attribute(uniquey.Key) != null
                                            && x.Attribute(uniquey.Key).Value.ToLower() == uniquey.Value.ToLower()
                                        select x).FirstOrDefault();
                    if (templateNode == null)
                    {
                        template.Add(dataNode);
                        continue;
                    }
                    else
                    {
                        MergeConfigs(templateNode, dataNode);
                        continue;
                    }
                }

            }
            return;

        }

        private void MergeAttributes(XElement template, XElement data)
        {
            foreach (var attrib in data.Attributes())
            {
                var value = attrib.Value;
                if (value == DeleteKeyword)
                {
                    value = null;
                }
                template.SetAttributeValue(attrib.Name, value);
            }
        }

        private KeyValue GetUniqueKey(XElement element)
        {
            foreach (var key in UniqueAttributes)
            {
                var elementsUnique = element.Attributes(key);
                if (elementsUnique.Count() == 1)
                {
                    return new KeyValue { Key = key, Value = elementsUnique.First().Value };
                }
            }
            return null;
        }

        private bool ShouldDeleteNode(XElement data)
        {
            var deleteattrib = data.Attributes(DeleteKeyword);
            if (deleteattrib.Count() == 1 && bool.Parse(deleteattrib.First().Value))
            {
                return true;
            }
            return false;
        }
    }

    public class KeyValue
    {
        public string Key { get; set; }
        public string Value { get; set; }
    }
	
	
// CodegenManager class records the various blocks so it can split them up
class CodegenManager {
    private class Block {
        public String Name;
        public int Start, Length;
    }

    private Block currentBlock;
    private List<Block> files = new List<Block>();
    private Block footer = new Block();
    private Block header = new Block();
    private ITextTemplatingEngineHost host;
    private StringBuilder template;
    protected List<String> generatedFileNames = new List<String>();

    public static CodegenManager Create(ITextTemplatingEngineHost host, StringBuilder template) {
        return (host is IServiceProvider) ? new VSCodegenManager(host, template) : new CodegenManager(host, template);
    }

    public void StartNewFile(String name) {
        if (name == null)
            throw new ArgumentNullException("name");
        CurrentBlock = new Block { Name = name };
    }

    public void StartFooter() {
        CurrentBlock = footer;
    }

    public void StartHeader() {
        CurrentBlock = header;
    }

    public void EndBlock() {
        if (CurrentBlock == null)
            return;
        CurrentBlock.Length = template.Length - CurrentBlock.Start;
        if (CurrentBlock != header && CurrentBlock != footer)
            files.Add(CurrentBlock);
        currentBlock = null;
    }

    public virtual void Process(bool split) {
        if (split) {
            EndBlock();
            String headerText = template.ToString(header.Start, header.Length);
            String footerText = template.ToString(footer.Start, footer.Length);
            String outputPath = Path.GetDirectoryName(host.TemplateFile);
            files.Reverse();
            foreach(Block block in files) {
                String fileName = Path.Combine(outputPath, block.Name);
                String content = headerText + template.ToString(block.Start, block.Length) + footerText;
                generatedFileNames.Add(fileName);
                CreateFile(fileName, content);
                template.Remove(block.Start, block.Length);
            }
        }
    }

    protected virtual void CreateFile(String fileName, String content) {
        if (IsFileContentDifferent(fileName, content))
            File.WriteAllText(fileName, content);
    }

    public virtual String GetCustomToolNamespace(String fileName) {
        return null;
    }

    public virtual String DefaultProjectNamespace {
        get { return null; }
    }

    protected bool IsFileContentDifferent(String fileName, String newContent) {
        return !(File.Exists(fileName) && File.ReadAllText(fileName) == newContent);
    }

    private CodegenManager(ITextTemplatingEngineHost host, StringBuilder template) {
        this.host = host;
        this.template = template;
    }

    private Block CurrentBlock {
        get { return currentBlock; }
        set {
            if (CurrentBlock != null)
                EndBlock();
            if (value != null)
                value.Start = template.Length;
            currentBlock = value;
        }
    }

    private class VSCodegenManager: CodegenManager {
        private EnvDTE.ProjectItem templateProjectItem;
        private EnvDTE.DTE dte;
        private Action<String> checkOutAction;
        private Action<IEnumerable<String>> projectSyncAction;

        public override String DefaultProjectNamespace {
            get {
                return templateProjectItem.ContainingProject.Properties.Item("DefaultNamespace").Value.ToString();
            }
        }

        public override String GetCustomToolNamespace(string fileName) {
            return dte.Solution.FindProjectItem(fileName).Properties.Item("CustomToolNamespace").Value.ToString();
        }

        public override void Process(bool split) {
            if (templateProjectItem.ProjectItems == null)
                return;
            base.Process(split);
            projectSyncAction.EndInvoke(projectSyncAction.BeginInvoke(generatedFileNames, null, null));
        }

        protected override void CreateFile(String fileName, String content) {
            if (IsFileContentDifferent(fileName, content)) {
                CheckoutFileIfRequired(fileName);
                File.WriteAllText(fileName, content);
            }
        }

        internal VSCodegenManager(ITextTemplatingEngineHost host, StringBuilder template)
            : base(host, template) {
            var hostServiceProvider = (IServiceProvider) host;
            if (hostServiceProvider == null)
                throw new ArgumentNullException("Could not obtain IServiceProvider");
            dte = (EnvDTE.DTE) hostServiceProvider.GetService(typeof(EnvDTE.DTE));
            if (dte == null)
                throw new ArgumentNullException("Could not obtain DTE from host");
            templateProjectItem = dte.Solution.FindProjectItem(host.TemplateFile);
            checkOutAction = (String fileName) => dte.SourceControl.CheckOutItem(fileName);
            projectSyncAction = (IEnumerable<String> keepFileNames) => ProjectSync(templateProjectItem, keepFileNames);
        }

        private static void ProjectSync(EnvDTE.ProjectItem templateProjectItem, IEnumerable<String> keepFileNames) {
            var keepFileNameSet = new HashSet<String>(keepFileNames);
            var projectFiles = new Dictionary<String, EnvDTE.ProjectItem>();
            var originalFilePrefix = Path.GetFileNameWithoutExtension(templateProjectItem.get_FileNames(0)) + ".";
            foreach(EnvDTE.ProjectItem projectItem in templateProjectItem.ProjectItems)
                projectFiles[projectItem.get_FileNames(0)] = projectItem;

            // Remove unused items from the project
			foreach(var pair in projectFiles)
                if (!keepFileNames.Contains(pair.Key) && !(Path.GetFileNameWithoutExtension(pair.Key) + ".").StartsWith(originalFilePrefix))
                    //pair.Value.Delete();

            // Add missing files to the project
            foreach(String fileName in keepFileNameSet)
                if (!projectFiles.ContainsKey(fileName))
                    templateProjectItem.ProjectItems.AddFromFile(fileName);
        }

        private void CheckoutFileIfRequired(String fileName) {
            var sc = dte.SourceControl;
            if (sc != null && sc.IsItemUnderSCC(fileName) && !sc.IsItemCheckedOut(fileName))
                checkOutAction.EndInvoke(checkOutAction.BeginInvoke(fileName, null, null));
        }
    }
}
#>

This file can be used as is with no modifications. Just put it into the root directory of the project

and then there is the actual T4 file which will cause stuff to be generated: configuration.tt
(you can call it anything you like)

<#@ template debug="true" hostSpecific="true" #>
<#@ output extension=".log" #>
<#@ include file="ConfigurationMerger.ttinclude"  #>
<#   
var setup = new Dictionary<string, string[]>();
string inputFilesRelativeRootPath = @"\";
string outputFilesRelativeRootPath = @"\";
//////////////////////////////////////////////////////////////	
//Configure the configuration generation process like this: 
//setup[<outputfilename>] = new []{<input1>,<input2>,<input3>,..};
//
// configs are merged in the order given (input 3 overrides input 2, which overrides input1)
//
// make your modifications below here
//////////////////////////////////////////////////////////////
inputFilesRelativeRootPath = @"\config\input";
setup[@"web.config"] = new []{"web.config.template","web.config.dev","web.config.dev.local"};
setup[@"config\output\web.config.test1"] = new []{"web.config.template","web.config.test","web.config.test.testserver1"};
setup[@"config\output\web.config.test2"] = new []{"web.config.template","web.config.test","web.config.test.testserver2"};
setup[@"config\output\web.config.build"] = new []{"web.config.template","web.config.build"};







////////////////////////////////////////////////////////////////
//End of custom configuration, the rest is just static stuff:
////////////////////////////////////////////////////////////////


var configs = MergeConfigFiles(inputFilesRelativeRootPath, outputFilesRelativeRootPath, setup);
var manager = CodegenManager.Create(Host, GenerationEnvironment);
foreach(var filepath in configs.Keys){
	#> 
	-------------- <#= System.IO.Path.GetDirectoryName(this.Host.TemplateFile)+filepath #> --------
	<#= configs[filepath] #>
	------------------------------------------------------------
	<#
	 manager.StartNewFile(System.IO.Path.GetDirectoryName(this.Host.TemplateFile)+filepath);
	#><#= configs[filepath] #><#
	manager.EndBlock();
}
manager.Process(true);
#>

This file should be tweaked to fit your project (you only need to tweak within the top part as indicated).
For instance: in this example the output files are all placed in the same directory with
different file endings, but that can be customized any way you like. (for instance having all the files called web.config but be in separate directories)

In addition to the output files, a separate log file is written called Configuration.log. This file contains all the resulting config files for easy
reference.

Normally the code(config) generation will only be triggered by a modification to the .tt file, but there is an easy way to get T4 to fire on every build.

I would suggest not checking the generated files into source control but rather have the CI server ( you DO use continuous integration, right?) generate the files when it performs the build