Monodevelop project macros

In Visual Studio you can use macros like $(SolutionName) in *.csproj file. The full list of macros you can find in MSDN. Monodevelop also has got the macros, but the list differs from MSDN. Here is the list of the project macros I've found in Monodevelop sources. It's actual for Monodevelop 5.8, in future version the list can be changed.

• ProjectFile
• ProjectConfig
• ProjectConfigName
• ProjectConfigPlat
• TargetPath
• TargetFile
• TargetName
• TargetDir
• TargetExt
• ProjectName
• ProjectDir
• AuthorName
• AuthorEmail
• AuthorCopyright
• AuthorCompany
• AuthorTrademark
• SolutionFile
• SolutionName
• SolutionDir

Two words about continuous integration for mono projects

Github has a great continuous integration system called Travis.CI. I use it for HyperFastCgi server to check that solution correctly builds after commit and I am going to use it for unit-tests in future. Travis.CI has a very simple configuration syntax, for example HyperFastCgi travis.yml file looks like that

language: c

before_install:
    #add badgerpots ppa key
    - wget http://badgerports.org/directhex.ppa.asc
    - sudo apt-key add directhex.ppa.asc
    #add bagderport repository
    - sudo apt-get install python-software-properties
    - sudo add-apt-repository "deb http://badgerports.org $(lsb_release -sc) main"
    - sudo apt-get update
    #install mono
    - sudo apt-get install mono-devel

script:
    - ./autogen.sh --prefix=/usr
    - make
    - sudo make install

Yesterday I've found, that drone (analogue of Travis.CI) made support for GitLab (analogue of Github) two month ago. So now you can run Github-like version control with Travis-like continuous integration for your private projects without using github and travis. I did not try to install drone to my gitlab server yet, but if it works without serious issues it's a really, really cool!

Running ASP.NET vNext on mono/linux

This is a quick starting guide to run "Hello, world" ASP.NET vNext app on mono/linux.

Installing mono 3.4.1

At first, you need to compile the latest mono version from sources. Sources are located at http://github.com/mono/mono. You can follow the docs on the main page, but BEWARE of using --prefix=/usr/local as option of autogen.sh file! Before doing it check where is your system mono installed. You can check it with which command.

$ which mono
/usr/bin/mono

If mono is located in /usr/bin (for example Ubuntu holds it there) then you should change prefix to --prefix=/usr otherwise you'll get two different mono installation and could run into the issues "where is the proper library located?". If you use Ubuntu, you can run this script. It'll install mono, xsp (mono web server) and monodevelop IDE.

Installing ASP.NET vNext

Run the following commands:

wget https://raw.githubusercontent.com/graemechristie/Home/KvmShellImplementation/kvmsetup.sh 
chmod a+x kvmsetup.sh
./kvmsetup.sh
source ~/.kre/kvm/kvm.sh
kvm upgrade

Running "Hello, world!" application

git clone https://github.com/davidfowl/HelloWorldVNext
cd HelloWorldVNext
git submodule update --init
kpm restore
cd src/helloworldweb
k web-firefly

It will start the web application at localhost:3001. To change host and port edit the file firefly/src/main/Firefly/ServerFactory.cs at line 30. Put there your host and port. No need to compile, just run k web-firefly again

You can also try to run Nowin host with the command k web, but due to the issue with sockets, you can run only ~1000 requests to your web server

Mono unmanaged calls performance

Let's imagine that you implemented some great algorithm using C# and think about improving performance of it. You might suggest to rewrite some bottleneck part in native C/C++ and call it from managed code using PInvoke. That may look like a good idea because native code is generally faster than managed but even moon has its own dark side. In the case it will be the cost of PInvoke calls to unmanaged functions. In this post you can find speed comparison of various approaches and choose the best fitting to your needs.

PInvoke call

For example, let's take a function, which calculates the sum of char codes in the string. Something like this: (Note: all source code you can find at github)

public static int ManagedCount(string s)
{
    int sum = 0;

    for (int j = 0; j < s.Length; j++) {
        sum+=(int)s[j];
    }
    return sum;
}

To add some complexity we will pass array of strings and index in the array to the function.

public static int ManagedCount(string[] arr,int i)
{
    int sum = 0;

    for (int j = 0; j < arr [i].Length; j++) {
        sum+=(int)arr[i][j];
    }
 
    return sum;
}

OK, that's the managed function we will work with. Now, translate this function to native code.

int
unmanagedCount (guint16 **arr,int index)
{
    int sum=0;
    guint16 *str=arr[index];

    while(*str)
    {
        sum+=*str;
        str++;
    }

    return sum;
}

string in CLR has two-byte representation, so we use guint16** pointer to access array of strings. Also we have to add some declaration in *.cs file

[DllImport ("libperf.so",EntryPoint="unmanagedCount")]
public static extern int UnmanagedCount(
    [MarshalAs(UnmanagedType.LPArray, ArraySubType=UnmanagedType.LPWStr)] 
    string[] arr, 
    int i
    );

[DllImport] attribute tells which native library to use and the name of the function in the library (EntryPoint), [MarshalAs] attribute says that PInvoke must pass first parameter as array of two-bytes strings

If you're unfamiliar with PInvoke, you should know one thing: at every call PInvoke converts parameters from managed type to unmanaged and then convert it back on return value. The attribute [MarshalAs] of parameters tells CLR how they should be converted. Such conversions consume additional time and affects to performance as well.

Now, we can create array of strings, and call these functions ten million times to check the time execution.

Managed: 3 939 ms
PInvoke: 11 616 ms

You can see, that PInvoke is three times slower than managed function and mostly because of these managed to unmanaged conversions, so you can't improve performance with PInvoke to unmanaged function if it is called very often.

Internal call

Mono has a hidden feature which is not well-known yet. It's called Internal Calls. Primary purpose of Internal Calls are mostly provide the way to implement in native code some critical methods of corlib library (memory allocation, copiing of objects, interaction with sockets and so on). Secondary it allows native application which embeds mono calls native functions of the application. With some magic I found a way to use Internal Calls in common mono application without embedding mono or changing corlib assembly.

At first, declare InternalCount method in *.cs file

[DllImport ("libperf.so",EntryPoint="internalCount")]
[MethodImpl(MethodImplOptions.InternalCall, MethodCodeType = MethodCodeType.Runtime)]
public static extern int InternalCount(string[] arr, int i);

Difference between platform invoke method declaration and internal calls is that you place the attribute [MethodImpl(MethodImplOptions.InternalCall)] over the method. MethodCodeType field is optional and may be omitted. Also, there is another difference you don't need to specify how parameters will be marshaled, because Internal Calls don't convert parameters of method to unmanaged types and place parameter to the stack as is.

Then we have to write registration function for our internal call. Add the declaration to cs file.

[DllImport ("libperf.so",EntryPoint="init")]
public static extern void InitInternals();

And add the code to c file

#include <mono/metadata/loader.h>

void
init()
{
    mono_add_internal_call (
         "PInvokePerf.PerformanceTest::InternalCount(string[],int)",
         internalCount
         );
}

You see that init() function calls mono_add_internal_call. This function is defined in mono runtime, and you have to add header <loader.h>, add to compiler options include search path and link with mono library. To know headers include path, run from command line

pkg-config --cflags mono-2

To find library name and path, run

pkg-config --libs mono-2

An example of Makefile

Function mono_add_internal_call has two parameters: CLI method name (with optional signature) and pointer to a native function, which will be called when CLR calls the declared method. The name of the method is constucted as "Namespace.ClassName::MethodName" and may be optionally added with method signature (this is usefull, when you have got overloaded methods)

Now we are ready to the final part: implementation of internalCount function. Let see at the function body

int
internalCount (MonoArray *arr,int index)
{
    MonoString* el = mono_array_get(arr,MonoString *,index);
    int len = mono_string_length(el);
    gint32 sum=0;
    guint16 *str = mono_string_chars(el);
    int i;

    for(i = 0; i < len; i++)
    {
        sum += str[i];
    }

    return sum;
}

You may notice that the function has MonoArray type in the signature which represents string[] type in csharp. That is the most important difference versus standard PInvoke: Internal Calls works directly with managed types and you have to use Mono API to access parameters and return values. Header files of Mono API you can find at pkg-config --cflags mono-2 directory mentioned above.

Some comments about the code:

MonoString* el = mono_array_get(arr,MonoString *,index);
returns element from array arr with elements of type MonoString * at the index location

mono_string_length(el)
returns string length

mono_string_chars(el)
returns pointer to the internal char array of the managed string

Now all is done and we can run our InternalCount function. When I did it for the first time, I did it like that

public static void Main (string[] args)
{
    //We must call InitInternals to initialize internal calls 
    PerformanceTest.InitInternals();
    PerformanceTest.InternalCount(arr,0);
}

But surprisingly for me it worked as expected only in mono AOT mode, when I run this program in normal mode I got MissingMethodException. I have to spent some time with debugger and found the interesting thing

When mono starts to execute method 'Main' JIT compiler compiles 'Main' at first and recursively all the methods which are called from the 'Main'. As 'Main' is referenced to 'InternalCount' method, JIT starts to compile 'InternalCount' method too. In compilation it searches the method name in registered internal calls, because the method has [MethodImplOption.InternalCall] attribute. But it could not find it, because 'InitInternals' function is not yet run! In this case JIT generates 'throw new MissingMethodException' in IL and all subsequent calls are throwing that exception, even when we register proper internal call later.

To avoid such behaviour I hide the method InternalCount from JIT. To do this I placed all the meaningful code out of the 'Main' function, and in 'Main' function created delegate to my function and called it. JIT compiles delegate only when it starts to run and 'MissingMethodException' goes away! The code now looks like this

delegate void HideFromJit();

public static void Main (string[] args)
{
    //Create array
    InitArray ();
    //Register internal calls
    PerformanceTest.InitInternals ();
    Console.WriteLine ("Performance measuring starting");

    //You can call it directly in AOT mode
    //Performance ();

    HideFromJit d=Performance;
    d ();
}

public static void Performance()
{
    //All the code is here
    for(int i = 0; i < 1000000; i++)
        PerformanceTest.InternalCount(arr,i%100);
}

Finally performance comparison for these methods

Method	Mono no optimizations	Mono --optimize=unsafe
Managed	3 939 ms	2 784 ms
PInvoke	11 616 ms	11 804 ms
Internal Call	872 ms	855 ms

Internal calls is a total winner, when the PInvoke is outsider with no chances to beat even managed code. PInvoke to Internal Call performance differs more to ten times!

And here are the results for byte buffer xoring algorithm

Method	Mono no optimizations	Mono --optimize=unsafe
Managed	2 068 ms	1 507 ms
PInvoke	4 387 ms	3 707 ms
Internal Call	1 372 ms	1 381 ms

You can see that with 'unsafe' optimization managed code executes close to Internal Calls, but without optimizations it's 50% slower. I choose 'unsafe' optimization, because it shows maximal speed boost for code working with arrays (unsafe optimization removes bounds checks). PInvoke again at the last place

Opened questions:

• GC movements. Should we pin managed data or do something another to be sure, that the data is not moving by GC when we are in the internal call?

Conclusion

Mono is a powerful framework and allows you to do great things with native code as well as managed. If you want to increase performance of you managed code don't use PInvoke to unmanaged as it defeats performance, but instead you might look onto Internal Calls. But you should be aware that the internal call mechanism is platform depended and you could not run you great app on .NET if you use it. By the way you always can add conditional #ifdef and compile your app with managed method for .NET and internal for Mono

References

• Source codes
• Embedding mono
• Mono: interop with native libraries

Unexpected unloading of mono web application

After several bugs in mono gc were fixed, I was able to run benchmarks for aspx page in apache2+mod-mono server. I used mono from master branch, mono --version says: "Mono Runtime Engine version 3.2.7 (master/01b7a50 Sat Dec 14 01:48:49 NOVT 2013)". Crashes with SIGSEGV went away but unfortunately I can't say that serving aspx with apache2 are stable now. Two times during benchmarks I've got something similar to deadlock: mono stopped to process requests and stuck at consuming 100% of CPU. Don't know what was that, my try to debug mono process with GDB did not bring an answer (unlike the other cases when GDB help me to find cause of deadlocks/SIGSEGV or at least the place of suspicious code and send this info to mono team). Also there are memory leaks. And a bad thing exists, that the server stops responding after processing ~160 000 requests, but there is workaround for it.

Mono .aspx 160K requests limit

If you run ab -n 200000 http://yoursite/hello.aspx where hello.aspx is a simple aspx page which do nothing, and site is served under apache mod-mono, after ~160K request you'll get deny of service. This error caused by several reasons I'll try to explain, what is going on and how to avoid this

When request comes to aspx page web server creates new session. Than the session saves to internal web cache. When the second request comes, the server tries to read session cookies and, if not found, creates and saves new session to the cache again. So every request without cookies creates new session object in the cache. This could provide huge memory leaks, when the number of sessions grow unstoppable, to prevent this web server has the maximal limit of objects, which internal web cache can store. This limit is defined as constant in Cache.cs and hardcoded to 15000

When the number of objects in internal cache hits 15000, web server starts to aggressively delete all objects from the cache using LRU strategy. So if user got the session 5 minutes ago and works with site by clicking the page every minute his session will be removed from cache (and lost all the data inside the session) in opposite to some hazardous script (without session cookies was set) which gets 15K requests to the page during last minute and creates 15K empty sessions. But this is not all.

Internal cache is also used for storing some important server objects, for example all dynamically compiled assemblies are stored there. And there is no preference for server objects when deleting from cache all objects are equal. So if some server object was not accessed too long it will be removed. And this is the cause of second error

Here the code of GetCompiledAssembly() method. It's called every time, when the page is accessed

   string vpabsolute = virtualPath.Absolute;
   if (is_precompiled) {
    Type type = GetPrecompiledType (vpabsolute);
    if (type != null)
     return type.Assembly;
   }
   BuildManagerCacheItem bmci = GetCachedItem (vpabsolute);
   if (bmci != null)
    return bmci.BuiltAssembly;

   Build (virtualPath);
   bmci = GetCachedItem (vpabsolute);
   if (bmci != null)
    return bmci.BuiltAssembly;
   
   return null;

Let's look. When .aspx page is accessed for the first time it tries to check if it was precompiled. If did it run process method. If not, it tries to find the compiled page in the internal cache and if not found there it compiles the page and stores compiled type into the cache (inside the Build() function). The schema looking good, but not in our case. When the internal cache overgrows 15K limit compiled type is removed from the cache even it was accessed right now! I think there is some bug in LRU implementation or maybe object are got from LRU only once and saved into some temp variable, so LRU object does not update last access time.

You may ask: "So what? Compiled type was deleted from the cache, but won't it be there on the next page get? Algorithm checks existence of the type in the cache, and if not found it compiles it again and places to cache. It could reduce performance, but could not be a reason of denial of service". And you'll be right. This is not exactly the reason of DoS. But if you look inside of page compilation, you'll find that it has a limit of recompilation times. And if this limit is reached it starts to unload AppDomain with the whole application! And at the last mod-mono somehow does not control AppDomain unloading, don't know why it should, but after 160K request the page is stopped responding.

try {
 BuildInner (vp, cs != null ? cs.Debug : false);
 if (entryExists && recursionDepth <= 1)
  // We count only update builds - first time a file
  // (or a batch) is built doesn't count.
  buildCount++;
} finally {
 // See http://support.microsoft.com/kb/319947
 if (buildCount > cs.NumRecompilesBeforeAppRestart)
  HttpRuntime.UnloadAppDomain ();
 recursionDepth--;
}

How can this be workarounded?
I know only one way - always use precompiled web site. At first look I had a hope, that constants LOW_WATERMARK and HIGH_WATERMARK for cache can be changed by setting appropriate environment variable, but, unfortunately it's not. In my opinion cache usage should be rewritten - user sessions and web server internal objects should have different storage places and must not affect each other. Also session should not be created at first page access, if the page doesn't asks for session object, it can be created later, when it really needed for processing the page

ServiceStack performance on mono part4

Today I again tried to increase performance of ServiceStack on the Mono. In the first part I noted that profiler showed large amount of calls and execution time of Hashtable:GetHash(), SimpleCollator:CompareInternal() and Char:ToLower() methods. To understand why these methods works slow I checked the call stack and found that most of the calls are maden from HttpHeadersCollection class. When I looked inside the source and saw that HttpHeadersCollection uses InvariantCultureIgnoreCase string comparison instead of OrdinalIgnoreCase which is more suitable when comparing names of headers (because they do not need be linguistic equivalent) and should be more performant

To be sure of Hashtable and Dictionary performance with various StringComparing options I wrote simple benchmark. It adds 100 000 strings and than tries to get them one by one for every StringComparing options. The original idea of test code I get from here. My test is slightly modified.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Collections;

namespace DictPerfomanceTest
{
 class ComparerInfo
 {
  public string Name { get; set;}

  public StringComparer Comparer { get; set;}

  public ComparerInfo(string name, StringComparer comparer)
  {
   Name = name;
   Comparer = comparer;
  }
 }

 class MainClass
 {
  const int nCount=100000;
  const string prefix = "SomeSomeString";

  static readonly ComparerInfo[] Comparers=new ComparerInfo[]
  {
   new ComparerInfo("CurrentCulture",StringComparer.CurrentCulture),
   new ComparerInfo("CurrentCultureIgnoreCase",StringComparer.CurrentCultureIgnoreCase),
   new ComparerInfo("InvariantCulture",StringComparer.InvariantCulture),
   new ComparerInfo("InvariantCultureIgnoreCase",StringComparer.InvariantCultureIgnoreCase),
   new ComparerInfo("Ordinal",StringComparer.Ordinal),
   new ComparerInfo("OrdinalIgnoreCase",StringComparer.OrdinalIgnoreCase)
  } ;

  public static void Main (string[] args)
  {
   foreach(var ci in Comparers)
   {
    Console.WriteLine ("Hashtable: {0}", ci.Name);
    Run (new Hashtable (ci.Comparer));
   }

   foreach(var ci in Comparers)
   {
    Console.WriteLine ("Dictionary: {0}", ci.Name);
    Run (new Dictionary<string,string> (ci.Comparer));
   }
  }

  private static void Run(Hashtable hashtable)
  {
   for(int i = 0; i < nCount; i++)
   {
    hashtable.Add(prefix+i.ToString(), i.ToString());
   }

   Stopwatch sw = new Stopwatch();
   sw.Start();
   for (int i = 0; i < nCount; i++)
   {
    string a = (string)hashtable[prefix+i.ToString()];
   }
   sw.Stop();
   Console.WriteLine("Time: {0} ms", sw.ElapsedMilliseconds);
  }

  private static void Run(Dictionary<string, string> dictionary)
  {
   for(int i = 0; i < nCount; i++)
   {
    dictionary.Add(prefix+i.ToString(), i.ToString());
   }

   Stopwatch sw = new Stopwatch();
   sw.Start();
   for (int i = 0; i < nCount; i++)
   {
    string a = dictionary[prefix+i.ToString()];
   }
   sw.Stop();
   Console.WriteLine("Time: {0} ms", sw.ElapsedMilliseconds);
  }

 }
}

Comparison Option	Hashtable time (ms)	Dictionary time (ms)
CurrentCulture	19 131	16 030
CurrentCultureIgnoreCase	20 458	16 587
InvariantCulture	18 359	15 161
InvariantCultureIgnoreCase	21 128	16 192
Ordinal	58	46
OrdinalIgnoreCase	73	73

What can I say? Don't use InvariantCulture or Culture-depended comparison in mono if you don't need it really! In most cases when you use string as dictionary key you can safely use Ordinal or OrdinalIgnoreCase string comparing options. For example names of caching keys in Redis, paths, names of configuration elements in xml are good candidates for Ordinal comparison. By default Dictionary uses Ordinal and Hashtable uses OrdinalIgnoreCase comparison for strings, but don't forget to pass these options to String.Compare(), String.StartWith(), String.EndWith() methods if you want to run you software fast and more predictable

Very good explanation about differencies about InvariantCulture and Ordinal comparison you can read here. In two lines of code it's looking like this:

Console.WriteLine(String.Equals("æ", "ae", StringComparison.Ordinal)); // Prints false  
Console.WriteLine(String.Equals("æ", "ae", StringComparison.InvariantCulture)); // Prints true

I changed HttpHeadersCollection in the commit and made a pull request to mono. Hope it will be reviewed and approved. Also I am going to change hashing functions for HttpRequest headers, first tests shows 3x to 6x performance improvement of ordinal case insensitive hash function without any changes of hashing algorithm

Links:

ServiceStack performance in mono. Part 1
ServiceStack performance in mono. Part 2
ServiceStack performance in mono. Part 3

ServiceStack performance in mono part 3

In previous post I benchmarked various HTTP mono backends in linux and found that Nginx+mono-server-fastcgi pair is very slow in comparison with others. There was several times difference in number of served requests per second! So two questions were raised: the first is "Why is so slow?" and second "What can be done to improve performance?". In this post I'll try to answer to both questions

Why is so slow?

Let's profile fastcgi mono server. You should remember that profiling can be enabled by setting appropriate MONO_OPTIONS environment variable. If you don't you can read about web servers profiling options in the first part

After running profile I've got the results

Total(ms) Self(ms)      Calls Method name
  243637        4       1002 (wrapper remoting-invoke-with-check) Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (Mono.WebServer.FastCgi.Responder)
  140963        4        591 (wrapper runtime-invoke) :runtime_invoke_void__this___object (object,intptr,intptr,intptr)
  140863       60        501 Mono.FastCgi.Server:OnAccept (System.IAsyncResult)
  140570       25        501 Mono.FastCgi.Connection:Run ()
  129977        3        501 Mono.FastCgi.Request:AddInputData (Mono.FastCgi.Record)
  129971        5        501 Mono.FastCgi.ResponderRequest:OnInputDataReceived (Mono.FastCgi.Request,Mono.FastCgi.DataReceivedArgs)
  129964        0        501 Mono.FastCgi.ResponderRequest:Worker (object)
  129963        1        501 Mono.WebServer.FastCgi.Responder:Process ()
  129959       34        501 (wrapper xdomain-invoke) Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (Mono.WebServer.FastCgi.Responder)
  122777        3        501 (wrapper xdomain-dispatch) Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (object,byte[]&,byte[]&)
  113673        3        501 Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (Mono.WebServer.FastCgi.Responder)
  112227       14        501 Mono.WebServer.BaseApplicationHost:ProcessRequest (Mono.WebServer.MonoWorkerRequest)
  112205        2        501 Mono.WebServer.MonoWorkerRequest:ProcessRequest ()
  111942        2        501 System.Web.HttpRuntime:ProcessRequest (System.Web.HttpWorkerRequest)
  111761        3        501 System.Web.HttpRuntime:RealProcessRequest (object)
  111745       11        501 System.Web.HttpRuntime:Process (System.Web.HttpWorkerRequest)
  110814        7        501 System.Web.HttpApplication:System.Web.IHttpHandler.ProcessRequest (System.Web.HttpContext)
  110785        7        501 System.Web.HttpApplication:Start (object)
  110148       14        501 System.Web.HttpApplication:Tick ()
  110133      346        501 System.Web.HttpApplication/c__Iterator1:MoveNext ()
   73347       92       6012 System.Web.HttpApplication/c__Iterator0:MoveNext ()
   64025       32        501 System.Web.Security.FormsAuthenticationModule:OnAuthenticateRequest (object,System.EventArgs)
   62704      141      21042 Mono.WebServer.FastCgi.WorkerRequest:GetKnownRequestHeader (int)
   62550      250      45647 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadObject (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,long&,object&,System.Runtime.Serialization.SerializationInfo&)
   62273        5       1002 System.Web.HttpRequest:get_Cookies ()
   62203      134      20040 Mono.WebServer.FastCgi.WorkerRequest:GetUnknownRequestHeaders ()
   56381        6       1002 (wrapper remoting-invoke-with-check) Mono.WebServer.FastCgi.Responder:GetParameters ()
   56373       34        501 (wrapper xdomain-invoke) Mono.WebServer.FastCgi.Responder:GetParameters ()
   54634      368      44653 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteObjectInstance (System.IO.BinaryWriter,object,bool)
   51554       16       1514 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:Deserialize (System.IO.Stream)
   51537       47       1514 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:NoCheckDeserialize (System.IO.Stream,System.Runtime.Remoting.Messaging.HeaderHandler)
   51531       34      12007 System.Runtime.Remoting.RemotingServices:DeserializeCallData (byte[])
   50521       19       1514 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadObjectGraph (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,bool,object&,System.Runtime.Remoting.Messaging.Header[]&)
   48246       46       7536 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadNextObject (System.IO.BinaryReader)
   47020      999      54096 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadValue (System.IO.BinaryReader,object,long,System.Runtime.Serialization.SerializationInfo,System.Type,string,System.Reflection.MemberInfo,int[])
   35051      143      22013 System.Runtime.Remoting.RemotingServices:SerializeCallData (object)
   34198        7       1516 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:Serialize (System.IO.Stream,object)
   34190       15       1516 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:Serialize (System.IO.Stream,object,System.Runtime.Remoting.Messaging.Header[])
   33354       28       1516 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteObjectGraph (System.IO.BinaryWriter,object,System.Runtime.Remoting.Messaging.Header[])
   33253       78       1516 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteQueuedObjects (System.IO.BinaryWriter)
   29792      539      16549 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteObject (System.IO.BinaryWriter,long,object)
   28486      656      49652 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteValue (System.IO.BinaryWriter,System.Type,object)
   26041      101        501 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadGenericArray (System.IO.BinaryReader,long&,object&)
   24552       16        501 System.Web.HttpApplication:PipelineDone ()
   23851       58        501 System.Web.HttpApplication:OutputPage ()
   23782       20        501 System.Web.HttpResponse:Flush (bool)
   23079      598      16539 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadObjectContent (System.IO.BinaryReader,System.Runtime.Serialization.Formatters.Binary.ObjectReader/TypeMetadata,long,object&,System.Runtime.Serialization.SerializationInfo&)
   22542       24        501 (wrapper xdomain-dispatch) Mono.WebServer.FastCgi.Responder:GetParameters (object,byte[]&,byte[]&)
   19536       39       3030 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteArray (System.IO.BinaryWriter,long,System.Array)
   18377      105        501 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteGenericArray (System.IO.BinaryWriter,long,System.Array)

In profile you can see there are alot of binary serialization calls which take most of the processing time. But if you look into the mono fastcgi code, you don't find any explicit calls of BinarySerializer. What is going on? I hope you've already guessed what caused such overhead in serialization calling in other case let's look on to the picture:

New FastCGI request handler is created for every request from Nginx, than request looks for corresponding web application by HTTP_HOST server variable and after application have found creates new HttpWorkerRequest inside of it, and calls Process method to process it. While processing web application communicates with FastCGI request handler (asks for HTTP headers, returns HTTP response and so on). Because FastCGI request handler and web application are located in different domains all calls between them goes through remoting. Remoting calls binary serialization for objects are passed and this makes application slow. I'd rather say remoting makes application VERY VERY VERY SLOW if you pass complex types between endpoints. It's a prime evil of distributed applications which need to be performant. Don't use remoting if you have another choice to communicate between your apps.

OK, we found, that fastcgi server actively uses remoting inside of it and this can reduce performance. But is the remoting only one thing which dramatically reduces the performance? Maybe FastCGI protocol itself is a very slow and we couldn't use fast and reliable mono web server with nginx?

To check this I decided to write simple application based on mono-server-fastcgi source code. The application should instantly return "Hello, world!" http response for every http request without using remoting. If I could write such app and it would be more performant, I would proved that more reliable web server could be created.

Proof of concept

I took FastCGI server sources and wrote my own network server based on async sockets. From the old sources I only got FastCGI record parser, all other I rid off. After the simple app has been completed, I made a benchmarks

Before publishing results, let's remember benchmarks of mono-server-fastcgi were maden in previous post.

Configuration	requests/sec	Standart deviation	std dev %	Comments
Nginx+fastcgi-server+ServiceStack	571.36	8.81	1.54	Memory Leaks
Nginx+fastcgi-server hello.html	409.48	9.14	2.23	Memory Leaks
Nginx+fastcgi-server hello.aspx	458.55	9.89	2.16	Memory Leaks, Crashes
Nginx+proxy xsp4+ServiceStack	1402.33	45.42	3.24	Unstable Results, Errors

This benchmarks were maden with Apache ab tool using 10 concurrent requests. You can see, that fastcgi mono server performs 400-500 requests per second. In new benchmarks I additionally variate number of concurrent requests to see influence on the results. The command was
ab -n 100000 -c <concurency> http://testurl

Nginx configuration:

 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass 127.0.0.1:9000;
                 include /etc/nginx/fastcgi_params;
         }
}

Benchmark results:

Nginx fastcgi settings	Concurency	Requests/Sec	Standart deviation	std dev %
TCP sockets	10	2619.56	49.95	1.83
TCP sockets	20	2673.198	19.43	0.72
TCP sockets	30	2681.166	15.83	0.59

Significant difference isn't it? These results give us a hope, that we can increase throughoutput of fastcgi server if we change the architecture and remove remoting communication from it. By the way there is a room to increase performance. Are you ready to go further?

Faster higher stronger

Next step I've done I switched connumication between nginx and server from TCP sockets to Unix sockets. Config and results

 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass unix:/tmp/fastcgi.socket;
                 include /etc/nginx/fastcgi_params;
         }
}

Results

Nginx fastcgi settings	Concurency	Requests/Sec	Standart deviation	std dev %
Unix sockets	10	2743.622	40.91	1.49
Unix sockets	20	2952.244	67.86	2.29
Unix sockets	30	2949.118	86.19	2.92

It gained up to 5-10%. Not so bad but I want to increase performance more better, because when we'll change simple http response from fastcgi request handler to real ASP.NET process method we will loose a lot of performance points.

One of the questions, answer to it could help to increase performance: is there a way to keep connection between nginx and fastcgi server instead of create it for every request? In above configurations nginx requires to close connection from fastcgi server to approve end of processing request. By the way FastCGI protocol has EndRequest command and keeping connection and using EndRequest command instead of closing connection could save huge amount of time in processing small requests. Fortunately, nginx has support of such feature, it's called keepalive. I enabled keepalive and set minimal number of open connections to 32 between nginx and my server. I choosen this number, because it was higher than the maximum number of concurrent requests I did with ab.

 upstream fastcgi_backend {  
#   server 127.0.0.1:9000;
    server unix:/tmp/fastcgi.socket;
    keepalive 32;
}
 
 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_keep_conn on;
                 fastcgi_pass fastcgi_backend;
                 include /etc/nginx/fastcgi_params;
         }
}

Nginx fastcgi settings	Concurency	Requests/Sec	Standart deviation	std dev %
TCP sockets. KeepAlive	10	3720.23	49.36	1.33
TCP sockets. KeepAlive	30	3907.85	80.48	2.06
Unix sockets. KeepAlive	10	4024.678	122.33	3.04
Unix sockets. KeepAlive	20	4458.714	72.87	1.63
Unix sockets. KeepAlive	30	4482.648	19.40	0.43

Wow! That is a huge performance gains! Up to 50% compared with previous results! So I thought this is enough for proof of concept and I could start to create more faster fastcgi mono web server. To proove my thought I made simple .NET web server (without nginx), which always returns "Hello, world!" http response and test it with ab. It shows me ~5000 reqs/sec and this is close to my fastcgi proof of concept server

HyperFastCGI server

The target is clear now. I am going to create fast and reliable fastcgi server for mono, which can serve in second as much requests as possible and be stable. Unfortunatly it cannot be maden as just performance tweaking of current mono fastcgi server. The architecture needs to be changed to avoid cross-domain calls while processing requests.

What I did:

• I wrote my own connection handling using async sockets. It should also decrease processor usage, but I did not compare servers by this parameter.
• I totally rewrote FastCGI packets parsing, trying to decrease number of operations needed to handle them.
• I changed the architecture by moving FastCGI packet handling to the same domain, where web application is located.
• Currently there are no known memory leaks when processing requests.

This helped to improve performance of the server, here are the benchmarks:

Url	Nginx fastcgi settings/Concurency	Requests/Sec	Standart deviation	std dev %
/hello.aspx	TCP keepalive/10	1404.174	24.93	1.78
/servicestack/json	TCP keepalive/10	1671.15	21.40	1.28
/servicestack/json	TCP keepalive/20	1718.158	41.46	2.41
/servicestack/json	TCP keepalive/30	1752.69	34.56	1.97
/servicestack/json	Unix sockets keepalive/10	1755.55	40.30	2.30
/servicestack/json	Unix sockets keepalive/20	1817.488	39.30	2.16
/servicestack/json	Unix sockets keepalive/30	1822.984	36.48	2.00

The performance compared to original mono fastcgi server raised up serveral times! But this is not enough. While testing I found that threads created and destroyed very often. Creation of threads is expensive operation and I decided to increase minimal number of threads in threadpool. I added new option /minthreads to the server and set it to /minthreads=20,8 which means that there will be at least 20 running working threads in threadpool and 8 IO threads (for async sockets communications).

/minthreads=20,8 benchmarks:

Url	Nginx fastcgi settings/Concurency	Requests/Sec	Standart deviation	std dev %
/servicestack/json	TCP keepalive/10	2041.246	23.18	1.14
/servicestack/json	TCP keepalive/20	2070.08	10.95	0.53
/servicestack/json	TCP keepalive/30	2093.526	24.27	1.16
/servicestack/json	Unix sockets keepalive/10	2156.754	37.74	1.75
/servicestack/json	Unix sockets keepalive/20	2182.774	42.96	1.97
/servicestack/json	Unix sockets keepalive/30	2268.676	28.39	1.25

Such easy thing gives performance boost up to 20%!

Finally, I place all nginx configurations benchmarks in one chart

At the end I say that HyperFactCgi server can be found at github. Currently it's not well tested, so use it at your own risk. But at least all ServiceStack(v3) WebHosts.Integration tests which passed with XSP passed with HyperFastCgi too. To install HyperFastCgi simply do:

git clone https://github.com/xplicit/HyperFastCgi.git
cd HyperFastCgi
./autogen.sh --prefix=/usr && make
sudo make install

configuration options are the same as mono-server-fastcgi plus few new parameters:
/minthreads=nw,nio - minimal number of working and iothreads
/maxthreads=nw,nio - maximal number of working and iothreads
/keepalive=<true|false> - use keepalive feature or not. Default is true
/usethreadpool=<true|false> - use threadpool for processing requests. Default is true

If HyperFastCgi server be interesting to others for using it in production I am going to improve it. What can be improved:

• Support several virtual paths in one server.Currently only one web application is supported
• Write unit tests to be sure, that the server is working properly
• Catch and properly handle UnloadDomain() command from ASP.NET. This command is raised when web.config is changed or under some health checking by web-server. (Edit: already done)
• Add management and monitoring application which shows server statistics (number of requests serverd and so on) and recommends performance tweaks
• Additional performance improvements

Links:
HyperFastCgi server source code
ServiceStack performance in mono. Part 1
ServiceStack performance in mono. Part 2

ServiceStack performance in mono. Part 4