Introduction
I am writing BenchmarkSuite the framework which helps to write benchmark tests in the same way like NUnit unit tests. You write a method which do the benchmark of you code, then mark it with [Bench] attribute and run the bench-console application. It search all the methods marked with [Bench] in the the assembly runs them several times, measures the metrics, calculates Mean, Standard Deviation and other statistical variables and outputs the results to the console and XML file. For testing the usage of the BenchmarkSuite library I've decided to benchmark various binary serializers for some common operations.
I've created SerializersBenchmarks project on github, got some well-known and some unknown binary serializers and start to write benchmarks for them. At the start that was funny, I've written a benchmark ran the console and immediately saw its results. But when number of binary serializers became more than 3, and number of test types grew it became a pain to add almost identical lines of code (which differentiate only by name of serializer and method to serialize/deserialize data) to every project. At this stage I decided to automatize the process and use the code generation
Transforming the code to T4
Look at the benchmark code
[Bench]
[Iterations(10000)]
public void SerializeByteArray64KStream()
{
var ser = SerializationContext.Default.GetSerializer<ByteArray64K> ();
var arr = ByteArray64K.Create();
var b = Benchmark.StartNew ();
using (MemoryStream ms = new MemoryStream ()) {
for (int i = 0; i < 10000; i++) {
ms.Position = 0;
ser.Pack(ms,arr);
}
}
b.Stop ();
}
It creates serializer, then creates object of the serialized type and calls 10000 operation of serializing the type to a MemoryStream.
For every serializer and type it differs only few lines of code:
- function name
- creation of serializer
- creation of type
- calling method to serialize the type to a stream
To reuse common code I decided to write one base T4 precompiled template, derive from it in every benchmarking project and customize derived template for serializer needs. I used "Inheritance Pattern: Text in Base Body" from MSDN library to create base template.The code for template
<#
foreach (BenchTypeInfo typeInfo in SerializedTypes) {
#>
[Bench]
[Iterations(<#=typeInfo.Iterations#>)]
public void Serialize<#=typeInfo.Name#>Stream()
{
<# InstantiateSerializer("ser",typeInfo.Name); #>
var arr = <#=typeInfo.Name#>.Create();
var b = Benchmark.StartNew ();
using (MemoryStream ms = new MemoryStream ()) {
for (int i = 0; i < <#=typeInfo.Iterations#>; i++) {
ms.Position = 0;
<# Serialize("ser","arr",typeInfo.Name,"ms"); #>
}
}
b.Stop ();
}
<#
}
#>
<#+
public virtual BenchTypeInfo[] SerializedTypes {
get {
return new BenchTypeInfo[] {
new BenchTypeInfo(typeof(ByteArray64K),10000),
new BenchTypeInfo(typeof(PrimitiveType),1000000)
};
}
}
public virtual void InstantiateSerializer(string name, string type){}
public virtual void Serialize(string serName, string objName,
string objType, string streamName){}
#>
What do this template do? For every type added in SerializedType array it creates text of the Serializing function, which serializes type to the memory stream. In the placeholders where Serializer should be created it calls the virtual method InstantiateSerializer(), which must be overrided in derived class and must write the code text which instantiates serializer. Then it calls virtual method Serialize() to fill the placeholder with Serialization code.
The next step was to create derived template, which would fill up all placeholders in base template. This was a little tricky. At the first, we must to say that template is ihnerited from base template with
<#@ template language="C#" inherits="BenchArrayBase" #>
Then we need to reference the base template assembly and base template namespaces in derived templates. To do it there are commands
<#@assembly name="absolute_path_to_assembly" #>
<#@import namespace="namespace_name" #>
Unfortunately, in most cases you don't know the absolute path to the referenced assembly, because you can place your project everywhere. Putting assemblies into GAC is not the good option to avoid this issue. But there is a solution. You can use project macros like ${ProjectDir} or ${TargetDir} in the assembly name and they will be evaluated into the absolute path. I added project 'SerializersBenchmarks' which contains base template as a reference to benchmarking projects and used
<#@assembly name="${TargetDir}/SerializersBenchmarks.dll" #>
as a reference in T4 template. If your base template located in the same assembly as the derived template you can use
<#@assembly name="${TargetPath}" #>
construction.
So I've added these lines at the top of the template
<#@ template language="C#" inherits="BenchArrayBase" #>
<#@ assembly name="${TargetDir}/SerializersBenchmarks.dll" #>
<#@ assembly name="System.Core" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Text" #>
<#@ import namespace="System.Collections.Generic" #>
<#@ import namespace="SerializersBenchmarks.Templates" #>
<# base.TransformText(); #>
base.TransformText(); calls all transformations in the base template and outputs generated code. During the transformation it calls the overrided methods from the derived template, so I have to define these methods at the end of the file
<#+
public override void InstantiateSerializer(string name, string type)
{
base.WriteLine("var {0} = SerializationContext.Default.GetSerializer<{1}> ();",name, type);
}
public override void Serialize(string name, string objname, string objtype, string stream)
{
//ser.Serialize(ms,arr);
base.WriteLine("{0}.Pack({1},{2});",name,stream,objname);
}
#>
As you guessed base.WriteLine() writes the line into the output which is generated by base template. That's all! Only few lines of code and we can add new serializer to our benchmarks without copy-pasting and replacing bunch of strings.
You can look into the sources in the real project:
Base Template
Derived Template
Generated CS File
Issues with Monodevelop
When I start to use inherited T4 templates in Monodevelop I've found that Monodevelop did not support them, it threw an exceptions on trying to use derived templates. So I made a patch with fixes for Monodevelop, which was accepted in version 5.8. Also I made a patch which allow to regenerate all T4 templates in the project or solution (very useful when you made some changes in base template and need to update generated code in the whole solution). To do it right click on the Solution and Project and go to "Tools/Generate T4 Templates" option menu. If you read the article when monodevelop 5.8 is not released yet and want to use T4 inheritance, you can use latest dev monodevelop snapshot. To do it, add the Xamarin dev repository to you repos (see the instruction), and then do the following commands from the command line:
sudo apt-get update
sudo apt-get install monodevelop-snapshot-latest
. mono-snapshot monodevelop
monodevelop
