ServiceStack performance on mono part4

Today I again tried to increase performance of ServiceStack on the Mono. In the first part I noted that profiler showed large amount of calls and execution time of Hashtable:GetHash(), SimpleCollator:CompareInternal() and Char:ToLower() methods. To understand why these methods works slow I checked the call stack and found that most of the calls are maden from HttpHeadersCollection class. When I looked inside the source and saw that HttpHeadersCollection uses InvariantCultureIgnoreCase string comparison instead of OrdinalIgnoreCase which is more suitable when comparing names of headers (because they do not need be linguistic equivalent) and should be more performant

To be sure of Hashtable and Dictionary performance with various StringComparing options I wrote simple benchmark. It adds 100 000 strings and than tries to get them one by one for every StringComparing options. The original idea of test code I get from here. My test is slightly modified.

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Collections;

namespace DictPerfomanceTest
{
 class ComparerInfo
 {
  public string Name { get; set;}

  public StringComparer Comparer { get; set;}

  public ComparerInfo(string name, StringComparer comparer)
  {
   Name = name;
   Comparer = comparer;
  }
 }

 class MainClass
 {
  const int nCount=100000;
  const string prefix = "SomeSomeString";

  static readonly ComparerInfo[] Comparers=new ComparerInfo[]
  {
   new ComparerInfo("CurrentCulture",StringComparer.CurrentCulture),
   new ComparerInfo("CurrentCultureIgnoreCase",StringComparer.CurrentCultureIgnoreCase),
   new ComparerInfo("InvariantCulture",StringComparer.InvariantCulture),
   new ComparerInfo("InvariantCultureIgnoreCase",StringComparer.InvariantCultureIgnoreCase),
   new ComparerInfo("Ordinal",StringComparer.Ordinal),
   new ComparerInfo("OrdinalIgnoreCase",StringComparer.OrdinalIgnoreCase)
  } ;

  public static void Main (string[] args)
  {
   foreach(var ci in Comparers)
   {
    Console.WriteLine ("Hashtable: {0}", ci.Name);
    Run (new Hashtable (ci.Comparer));
   }

   foreach(var ci in Comparers)
   {
    Console.WriteLine ("Dictionary: {0}", ci.Name);
    Run (new Dictionary<string,string> (ci.Comparer));
   }
  }

  private static void Run(Hashtable hashtable)
  {
   for(int i = 0; i < nCount; i++)
   {
    hashtable.Add(prefix+i.ToString(), i.ToString());
   }

   Stopwatch sw = new Stopwatch();
   sw.Start();
   for (int i = 0; i < nCount; i++)
   {
    string a = (string)hashtable[prefix+i.ToString()];
   }
   sw.Stop();
   Console.WriteLine("Time: {0} ms", sw.ElapsedMilliseconds);
  }

  private static void Run(Dictionary<string, string> dictionary)
  {
   for(int i = 0; i < nCount; i++)
   {
    dictionary.Add(prefix+i.ToString(), i.ToString());
   }

   Stopwatch sw = new Stopwatch();
   sw.Start();
   for (int i = 0; i < nCount; i++)
   {
    string a = dictionary[prefix+i.ToString()];
   }
   sw.Stop();
   Console.WriteLine("Time: {0} ms", sw.ElapsedMilliseconds);
  }

 }
}

Comparison Option	Hashtable time (ms)	Dictionary time (ms)
CurrentCulture	19 131	16 030
CurrentCultureIgnoreCase	20 458	16 587
InvariantCulture	18 359	15 161
InvariantCultureIgnoreCase	21 128	16 192
Ordinal	58	46
OrdinalIgnoreCase	73	73

What can I say? Don't use InvariantCulture or Culture-depended comparison in mono if you don't need it really! In most cases when you use string as dictionary key you can safely use Ordinal or OrdinalIgnoreCase string comparing options. For example names of caching keys in Redis, paths, names of configuration elements in xml are good candidates for Ordinal comparison. By default Dictionary uses Ordinal and Hashtable uses OrdinalIgnoreCase comparison for strings, but don't forget to pass these options to String.Compare(), String.StartWith(), String.EndWith() methods if you want to run you software fast and more predictable

Very good explanation about differencies about InvariantCulture and Ordinal comparison you can read here. In two lines of code it's looking like this:

Console.WriteLine(String.Equals("æ", "ae", StringComparison.Ordinal)); // Prints false  
Console.WriteLine(String.Equals("æ", "ae", StringComparison.InvariantCulture)); // Prints true

I changed HttpHeadersCollection in the commit and made a pull request to mono. Hope it will be reviewed and approved. Also I am going to change hashing functions for HttpRequest headers, first tests shows 3x to 6x performance improvement of ordinal case insensitive hash function without any changes of hashing algorithm

Links:

ServiceStack performance in mono. Part 1
ServiceStack performance in mono. Part 2
ServiceStack performance in mono. Part 3

ServiceStack performance in mono part 3

In previous post I benchmarked various HTTP mono backends in linux and found that Nginx+mono-server-fastcgi pair is very slow in comparison with others. There was several times difference in number of served requests per second! So two questions were raised: the first is "Why is so slow?" and second "What can be done to improve performance?". In this post I'll try to answer to both questions

Why is so slow?

Let's profile fastcgi mono server. You should remember that profiling can be enabled by setting appropriate MONO_OPTIONS environment variable. If you don't you can read about web servers profiling options in the first part

After running profile I've got the results

Total(ms) Self(ms)      Calls Method name
  243637        4       1002 (wrapper remoting-invoke-with-check) Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (Mono.WebServer.FastCgi.Responder)
  140963        4        591 (wrapper runtime-invoke) :runtime_invoke_void__this___object (object,intptr,intptr,intptr)
  140863       60        501 Mono.FastCgi.Server:OnAccept (System.IAsyncResult)
  140570       25        501 Mono.FastCgi.Connection:Run ()
  129977        3        501 Mono.FastCgi.Request:AddInputData (Mono.FastCgi.Record)
  129971        5        501 Mono.FastCgi.ResponderRequest:OnInputDataReceived (Mono.FastCgi.Request,Mono.FastCgi.DataReceivedArgs)
  129964        0        501 Mono.FastCgi.ResponderRequest:Worker (object)
  129963        1        501 Mono.WebServer.FastCgi.Responder:Process ()
  129959       34        501 (wrapper xdomain-invoke) Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (Mono.WebServer.FastCgi.Responder)
  122777        3        501 (wrapper xdomain-dispatch) Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (object,byte[]&,byte[]&)
  113673        3        501 Mono.WebServer.FastCgi.ApplicationHost:ProcessRequest (Mono.WebServer.FastCgi.Responder)
  112227       14        501 Mono.WebServer.BaseApplicationHost:ProcessRequest (Mono.WebServer.MonoWorkerRequest)
  112205        2        501 Mono.WebServer.MonoWorkerRequest:ProcessRequest ()
  111942        2        501 System.Web.HttpRuntime:ProcessRequest (System.Web.HttpWorkerRequest)
  111761        3        501 System.Web.HttpRuntime:RealProcessRequest (object)
  111745       11        501 System.Web.HttpRuntime:Process (System.Web.HttpWorkerRequest)
  110814        7        501 System.Web.HttpApplication:System.Web.IHttpHandler.ProcessRequest (System.Web.HttpContext)
  110785        7        501 System.Web.HttpApplication:Start (object)
  110148       14        501 System.Web.HttpApplication:Tick ()
  110133      346        501 System.Web.HttpApplication/c__Iterator1:MoveNext ()
   73347       92       6012 System.Web.HttpApplication/c__Iterator0:MoveNext ()
   64025       32        501 System.Web.Security.FormsAuthenticationModule:OnAuthenticateRequest (object,System.EventArgs)
   62704      141      21042 Mono.WebServer.FastCgi.WorkerRequest:GetKnownRequestHeader (int)
   62550      250      45647 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadObject (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,long&,object&,System.Runtime.Serialization.SerializationInfo&)
   62273        5       1002 System.Web.HttpRequest:get_Cookies ()
   62203      134      20040 Mono.WebServer.FastCgi.WorkerRequest:GetUnknownRequestHeaders ()
   56381        6       1002 (wrapper remoting-invoke-with-check) Mono.WebServer.FastCgi.Responder:GetParameters ()
   56373       34        501 (wrapper xdomain-invoke) Mono.WebServer.FastCgi.Responder:GetParameters ()
   54634      368      44653 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteObjectInstance (System.IO.BinaryWriter,object,bool)
   51554       16       1514 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:Deserialize (System.IO.Stream)
   51537       47       1514 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:NoCheckDeserialize (System.IO.Stream,System.Runtime.Remoting.Messaging.HeaderHandler)
   51531       34      12007 System.Runtime.Remoting.RemotingServices:DeserializeCallData (byte[])
   50521       19       1514 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadObjectGraph (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,bool,object&,System.Runtime.Remoting.Messaging.Header[]&)
   48246       46       7536 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadNextObject (System.IO.BinaryReader)
   47020      999      54096 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadValue (System.IO.BinaryReader,object,long,System.Runtime.Serialization.SerializationInfo,System.Type,string,System.Reflection.MemberInfo,int[])
   35051      143      22013 System.Runtime.Remoting.RemotingServices:SerializeCallData (object)
   34198        7       1516 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:Serialize (System.IO.Stream,object)
   34190       15       1516 System.Runtime.Serialization.Formatters.Binary.BinaryFormatter:Serialize (System.IO.Stream,object,System.Runtime.Remoting.Messaging.Header[])
   33354       28       1516 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteObjectGraph (System.IO.BinaryWriter,object,System.Runtime.Remoting.Messaging.Header[])
   33253       78       1516 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteQueuedObjects (System.IO.BinaryWriter)
   29792      539      16549 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteObject (System.IO.BinaryWriter,long,object)
   28486      656      49652 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteValue (System.IO.BinaryWriter,System.Type,object)
   26041      101        501 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadGenericArray (System.IO.BinaryReader,long&,object&)
   24552       16        501 System.Web.HttpApplication:PipelineDone ()
   23851       58        501 System.Web.HttpApplication:OutputPage ()
   23782       20        501 System.Web.HttpResponse:Flush (bool)
   23079      598      16539 System.Runtime.Serialization.Formatters.Binary.ObjectReader:ReadObjectContent (System.IO.BinaryReader,System.Runtime.Serialization.Formatters.Binary.ObjectReader/TypeMetadata,long,object&,System.Runtime.Serialization.SerializationInfo&)
   22542       24        501 (wrapper xdomain-dispatch) Mono.WebServer.FastCgi.Responder:GetParameters (object,byte[]&,byte[]&)
   19536       39       3030 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteArray (System.IO.BinaryWriter,long,System.Array)
   18377      105        501 System.Runtime.Serialization.Formatters.Binary.ObjectWriter:WriteGenericArray (System.IO.BinaryWriter,long,System.Array)

In profile you can see there are alot of binary serialization calls which take most of the processing time. But if you look into the mono fastcgi code, you don't find any explicit calls of BinarySerializer. What is going on? I hope you've already guessed what caused such overhead in serialization calling in other case let's look on to the picture:

New FastCGI request handler is created for every request from Nginx, than request looks for corresponding web application by HTTP_HOST server variable and after application have found creates new HttpWorkerRequest inside of it, and calls Process method to process it. While processing web application communicates with FastCGI request handler (asks for HTTP headers, returns HTTP response and so on). Because FastCGI request handler and web application are located in different domains all calls between them goes through remoting. Remoting calls binary serialization for objects are passed and this makes application slow. I'd rather say remoting makes application VERY VERY VERY SLOW if you pass complex types between endpoints. It's a prime evil of distributed applications which need to be performant. Don't use remoting if you have another choice to communicate between your apps.

OK, we found, that fastcgi server actively uses remoting inside of it and this can reduce performance. But is the remoting only one thing which dramatically reduces the performance? Maybe FastCGI protocol itself is a very slow and we couldn't use fast and reliable mono web server with nginx?

To check this I decided to write simple application based on mono-server-fastcgi source code. The application should instantly return "Hello, world!" http response for every http request without using remoting. If I could write such app and it would be more performant, I would proved that more reliable web server could be created.

Proof of concept

I took FastCGI server sources and wrote my own network server based on async sockets. From the old sources I only got FastCGI record parser, all other I rid off. After the simple app has been completed, I made a benchmarks

Before publishing results, let's remember benchmarks of mono-server-fastcgi were maden in previous post.

Configuration	requests/sec	Standart deviation	std dev %	Comments
Nginx+fastcgi-server+ServiceStack	571.36	8.81	1.54	Memory Leaks
Nginx+fastcgi-server hello.html	409.48	9.14	2.23	Memory Leaks
Nginx+fastcgi-server hello.aspx	458.55	9.89	2.16	Memory Leaks, Crashes
Nginx+proxy xsp4+ServiceStack	1402.33	45.42	3.24	Unstable Results, Errors

This benchmarks were maden with Apache ab tool using 10 concurrent requests. You can see, that fastcgi mono server performs 400-500 requests per second. In new benchmarks I additionally variate number of concurrent requests to see influence on the results. The command was
ab -n 100000 -c <concurency> http://testurl

Nginx configuration:

 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass 127.0.0.1:9000;
                 include /etc/nginx/fastcgi_params;
         }
}

Benchmark results:

Nginx fastcgi settings	Concurency	Requests/Sec	Standart deviation	std dev %
TCP sockets	10	2619.56	49.95	1.83
TCP sockets	20	2673.198	19.43	0.72
TCP sockets	30	2681.166	15.83	0.59

Significant difference isn't it? These results give us a hope, that we can increase throughoutput of fastcgi server if we change the architecture and remove remoting communication from it. By the way there is a room to increase performance. Are you ready to go further?

Faster higher stronger

Next step I've done I switched connumication between nginx and server from TCP sockets to Unix sockets. Config and results

 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass unix:/tmp/fastcgi.socket;
                 include /etc/nginx/fastcgi_params;
         }
}

Results

Nginx fastcgi settings	Concurency	Requests/Sec	Standart deviation	std dev %
Unix sockets	10	2743.622	40.91	1.49
Unix sockets	20	2952.244	67.86	2.29
Unix sockets	30	2949.118	86.19	2.92

It gained up to 5-10%. Not so bad but I want to increase performance more better, because when we'll change simple http response from fastcgi request handler to real ASP.NET process method we will loose a lot of performance points.

One of the questions, answer to it could help to increase performance: is there a way to keep connection between nginx and fastcgi server instead of create it for every request? In above configurations nginx requires to close connection from fastcgi server to approve end of processing request. By the way FastCGI protocol has EndRequest command and keeping connection and using EndRequest command instead of closing connection could save huge amount of time in processing small requests. Fortunately, nginx has support of such feature, it's called keepalive. I enabled keepalive and set minimal number of open connections to 32 between nginx and my server. I choosen this number, because it was higher than the maximum number of concurrent requests I did with ab.

 upstream fastcgi_backend {  
#   server 127.0.0.1:9000;
    server unix:/tmp/fastcgi.socket;
    keepalive 32;
}
 
 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_keep_conn on;
                 fastcgi_pass fastcgi_backend;
                 include /etc/nginx/fastcgi_params;
         }
}

Nginx fastcgi settings	Concurency	Requests/Sec	Standart deviation	std dev %
TCP sockets. KeepAlive	10	3720.23	49.36	1.33
TCP sockets. KeepAlive	30	3907.85	80.48	2.06
Unix sockets. KeepAlive	10	4024.678	122.33	3.04
Unix sockets. KeepAlive	20	4458.714	72.87	1.63
Unix sockets. KeepAlive	30	4482.648	19.40	0.43

Wow! That is a huge performance gains! Up to 50% compared with previous results! So I thought this is enough for proof of concept and I could start to create more faster fastcgi mono web server. To proove my thought I made simple .NET web server (without nginx), which always returns "Hello, world!" http response and test it with ab. It shows me ~5000 reqs/sec and this is close to my fastcgi proof of concept server

HyperFastCGI server

The target is clear now. I am going to create fast and reliable fastcgi server for mono, which can serve in second as much requests as possible and be stable. Unfortunatly it cannot be maden as just performance tweaking of current mono fastcgi server. The architecture needs to be changed to avoid cross-domain calls while processing requests.

What I did:

• I wrote my own connection handling using async sockets. It should also decrease processor usage, but I did not compare servers by this parameter.
• I totally rewrote FastCGI packets parsing, trying to decrease number of operations needed to handle them.
• I changed the architecture by moving FastCGI packet handling to the same domain, where web application is located.
• Currently there are no known memory leaks when processing requests.

This helped to improve performance of the server, here are the benchmarks:

Url	Nginx fastcgi settings/Concurency	Requests/Sec	Standart deviation	std dev %
/hello.aspx	TCP keepalive/10	1404.174	24.93	1.78
/servicestack/json	TCP keepalive/10	1671.15	21.40	1.28
/servicestack/json	TCP keepalive/20	1718.158	41.46	2.41
/servicestack/json	TCP keepalive/30	1752.69	34.56	1.97
/servicestack/json	Unix sockets keepalive/10	1755.55	40.30	2.30
/servicestack/json	Unix sockets keepalive/20	1817.488	39.30	2.16
/servicestack/json	Unix sockets keepalive/30	1822.984	36.48	2.00

The performance compared to original mono fastcgi server raised up serveral times! But this is not enough. While testing I found that threads created and destroyed very often. Creation of threads is expensive operation and I decided to increase minimal number of threads in threadpool. I added new option /minthreads to the server and set it to /minthreads=20,8 which means that there will be at least 20 running working threads in threadpool and 8 IO threads (for async sockets communications).

/minthreads=20,8 benchmarks:

Url	Nginx fastcgi settings/Concurency	Requests/Sec	Standart deviation	std dev %
/servicestack/json	TCP keepalive/10	2041.246	23.18	1.14
/servicestack/json	TCP keepalive/20	2070.08	10.95	0.53
/servicestack/json	TCP keepalive/30	2093.526	24.27	1.16
/servicestack/json	Unix sockets keepalive/10	2156.754	37.74	1.75
/servicestack/json	Unix sockets keepalive/20	2182.774	42.96	1.97
/servicestack/json	Unix sockets keepalive/30	2268.676	28.39	1.25

Such easy thing gives performance boost up to 20%!

Finally, I place all nginx configurations benchmarks in one chart

At the end I say that HyperFactCgi server can be found at github. Currently it's not well tested, so use it at your own risk. But at least all ServiceStack(v3) WebHosts.Integration tests which passed with XSP passed with HyperFastCgi too. To install HyperFastCgi simply do:

git clone https://github.com/xplicit/HyperFastCgi.git
cd HyperFastCgi
./autogen.sh --prefix=/usr && make
sudo make install

configuration options are the same as mono-server-fastcgi plus few new parameters:
/minthreads=nw,nio - minimal number of working and iothreads
/maxthreads=nw,nio - maximal number of working and iothreads
/keepalive=<true|false> - use keepalive feature or not. Default is true
/usethreadpool=<true|false> - use threadpool for processing requests. Default is true

If HyperFastCgi server be interesting to others for using it in production I am going to improve it. What can be improved:

• Support several virtual paths in one server.Currently only one web application is supported
• Write unit tests to be sure, that the server is working properly
• Catch and properly handle UnloadDomain() command from ASP.NET. This command is raised when web.config is changed or under some health checking by web-server. (Edit: already done)
• Add management and monitoring application which shows server statistics (number of requests serverd and so on) and recommends performance tweaks
• Additional performance improvements

Links:
HyperFastCgi server source code
ServiceStack performance in mono. Part 1
ServiceStack performance in mono. Part 2

ServiceStack performance in mono. Part 4

ServiceStack performance in mono part2

In previous part I told about some performance enhancements which could be used with ServiceStack running over mono XSP web server. But nobody uses XSP in production environment, the most common use cases are nginx+mono-fastcgi and apache+mod_mono. But what is the performance in such environment? Will see it.

Configuration

Apache

If you want to use mono with apache, you have to install mod_mono for apache and configure it according to this article. To install mod_mono in Ubuntu you can type

  sudo apt-get install libapache2-mod-mono

after that you have to reinstall mono web server was compiled from sources. Change the directory to xsp source code and run sudo make install from it. I am going to benchmark mono under apache in following configurations:

• 1. Direct access to static html file from apache without mono.
• 2. Get ServiceStack "Hello, World!" service throught apache2-mod-mono
• 3. Get static html file and "Hello, World!" aspx page throught apache2-mod-mono without ServiceStack.

To manage this I use following config in /etc/apache2/http.conf. For direct static file access I placed hello.html in the web server root (/var/www)

NameVirtualHost ssbench3:80
NameVirtualHost ssbench2:80

<VirtualHost ssbench3:80>
    ServerName ssbench3
    DocumentRoot /var/www/ssbench3
#    MonoPath default "/usr/bin/mono/2.0"
    MonoServerPath ssbench3 /usr/bin/mod-mono-server4
    AddMonoApplications ssbench3 "ssbench3:/:/var/www/ssbench3"
        
    <location />
 MonoSetServerAlias ssbench3
 Allow from all
 Order allow,deny
 SetHandler mono
    </location>
</VirtualHost>

<VirtualHost ssbench2:80>
    ServerName ssbench2
    DocumentRoot /var/www/ssbench2
#    MonoPath default "/usr/bin/mono/2.0"
    MonoServerPath ssbench2 /usr/bin/mod-mono-server4
    AddMonoApplications ssbench2 "ssbench2:/:/var/www/ssbench2"
        
    <location />
 MonoSetServerAlias ssbench2
 Allow from all
 Order allow,deny
 SetHandler mono
    </location>
</VirtualHost>

Nginx

Configuration of Nginx is similar to Apache, differences are only in transport between mono and front-end web server. Apache uses mod-mono-server while nginx uses fastcgi-mono-server. Also, you may note that I added one additional configuration: nginx as proxy to xsp4.

To configure nginx I followed this guide. I added following lines to /etc/nginx/fastcgi_params

fastcgi_param HTTP_HOST $host;
fastcgi_param  PATH_INFO          "";
fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;

And added virtual hosts to /etc/nginx/sites-enabled/default

 server {
         listen   81;
         server_name  ssbench1;
         access_log   /var/log/nginx/ssbench1.log;
 
         location / {
     proxy_pass http://127.0.0.1:8080/;
     proxy_set_header   X-Real-IP $remote_addr;
     proxy_set_header   Host $http_host;
     proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
         }
 }

 server {
         listen   81;
         server_name  ssbench2;
         access_log   /var/log/nginx/ssbench2.log;
 
         location / {
                 root /var/www/ssbench2/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass 127.0.0.1:9000;
                 include /etc/nginx/fastcgi_params;
             }

 }
 
 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass 127.0.0.1:9000;
                 include /etc/nginx/fastcgi_params;
         }

 }

after that, I ran the command:

fastcgi-mono-server4 /applications=ssbench2:/:/var/www/ssbench2/,ssbench3:/:/var/www/ssbench3 /socket=tcp:127.0.0.1:9000

Also, I ran xsp4 server hosted ServiceStack on port 8080

EDIT: after the post was written, I additionaly benchmarked several configuration were not mentioned in the first version, they are:

• Nginx as frontend proxy to apache server with mod-mono
• Self-hosted ServiceStack instance based on two classes: AppHostHttpListenerBase and AppHostHttpListenerLongRunningBase. How to create self-hosted ServiceStack you can read in ServiceStack wiki. Also, you can look in test source code to get additional details
• Nginx as frontend proxy to self-hosted ServiceStack.
• Nginx plus HyperFastCgi (is a new fastcgi server I written. Replacement of mono-webserver-fastcgi)

Benchmark results

Before I'll print the results I want to say a couple words about my expectations. What did I expect? At first, I predicted that nginx be winner of serving static html pages. It was obvious. Secondly, I thought that nginx+ServiceStack get slightly better results versus Apache+ServiceStack and maybe XSP+ServiceStack due to nginx async behaviour and lower processor usage. Also, I thought that performance difference between Apache+ServiceStack and XSP+ServiceStack should be minimal. They are both use the same threading model and what could I expect a little overhead in apache<->mod-mono communications. But... Here are the results

Configuration	requests/sec	Standart deviation	std dev %	Comments
Apache2 direct file	7129.95	217.57	3.05
Apache2+mod_mono+ServiceStack	1314.30	22.40	1.70
Apache2+mod_mono hello.html	924.02	12.82	1.39
Apache2+mod_mono hello.aspx	----	----	---	Memory Leaks, Crashes
Nginx direct file	10458.71	147.28	1.41
Nginx+fastcgi-server+ServiceStack	571.36	8.81	1.54	Memory Leaks
Nginx+fastcgi-server hello.html	409.48	9.14	2.23	Memory Leaks
Nginx+fastcgi-server hello.aspx	458.55	9.89	2.16	Memory Leaks, Crashes
Nginx+proxy to Apache2+mod-mono+ServiceStack	1143.82	8.49	0.74
Nginx+proxy to self-hosted ServiceStack (AppHost HttpListenerBase)	1993.82	17.62	0.88
Nginx+proxy to self-hosted ServiceStack (AppHost HttpListenerLongRunningBase)	1664.94	27.45	1.65
Nginx+HyperFastCgi (tcp keepalive)+ServiceStack	2041.25	23.18	1.14	See more info
Nginx+proxy to xsp4+ServiceStack	1402.33	45.42	3.24	Unstable Results, Errors
xsp4+ServiceStack	2246.51	21.31	0.94
Self-hosted ServiceStack (AppHost HttpListenerBase)	2697.1	30.1	1.12
Self-hosted ServiceStack (AppHost HttpListenerLongRunningBase)	2313.11	33.14	1.43

What can we see? First place in serving ServiceStack takes xsp4. Then goes Apache+mod_mono and the last one is Nginx+fastcgi-server which is four times worse then the winner. I did not mentioned here Nginx+proxy xsp4 configuration because during test execution in half of test runs I get errors when receive json data. There were not so many errors (~1500 on 100 000 requests), but they were exist and this was the reason to drop away nginx+xsp4 configuration from competition. By the way performance result for the configuration slightly better than apache+mod_mono and much better than Nginx+fastgi-server.

Also I did not include HyperFastCgi server in the chart which shows good performance, because it was created after these benchmarks have done. Benchmarks of the Nginx+HyperFastCgi server you can find in next part

As serving static html files the first place takes Nginx as expected, second by Apache and after them goes all other configuration: xsp4 (you can see test results for static xsp4 html serving in previous post), Apache+mod_mono, Nginx+fastcgi. They all are really very slow comparing with Nginx or Apache.

For .aspx page I could not get reliable results. At first, there are memory leaks in mono web server during processing the aspx pages and they are possible a reason of crashes I've got. I could only get ~20000 requests with Nginx+fastcgi and several thouthands request with Apache+mod_mono before mono hanged or got SIGSEGV. I suspect that the reason of these faults are changes of hadling and spawning threads and changes performed in mono GC. Hope that this instablity will be fixed in next mono release.

Also I've mentioned that fastcgi-mono-server produced a huge memory leaks during runs. After processing 100 000 requests it was used about 600M of memory! With such configuration you cannot serve large amount or requests without regular restarting of the server. Also performance of fastcgi-mono-server is extremely slow compared to mod-mono-apache. What is going on in the server? I am going to look inside it in the next posts

Links:

• ServiceStack performance in mono part 1
• ServiceStack performance in mono part 3
• ServiceStack performance in mono part 4

Servicestack performance in mono

When I read ServiceStack channel on Google+ I found an benchmark which said that ServiceStack serialization under mono is very slow. That is discouraged me because I thought that SS demonstrated very good json serialization performance versus other .net json serialization frameworks. Maybe testers used wrong configuration or bad test case? The questions were opened for me and I decided to check it by myself.

Preparing environment and measurement metrics

My environment:
CPU: Intel(R) Core(TM)2 Duo CPU E4600 @ 2.40GHz
OS: Ubuntu 12.04 32 bit
Mono Runtime Engine version 3.2.5 (master/6a9c585 Fri Oct 25 01:56:00 NOVT 2013)

I have built mono from the github sources as described here. As measurement tool I am going to use ab from apache2-utils package. If you want to install ab, you can write apt-get install apache2-utils. I am going to run ab 5 times, performing 100000 url gets each time and get the result mean. Every run I will use 10 threads to run request in parallel.

The command looks like this: ab -n 100000 -c 10 http://host:port/url

ServiceStack was compiled from github v3 branch in mono release build for Mono/.NET 4.0 platform

As soon as environment is prepared I have to create test case. I choose to create very simple ServiceStack service similar to benchmarks which returns "Hello, world!" message. You can find source code at github. Also I would like to get some metrics for comparison. I choose to create simple ASP.NET application with "Hello, world" .aspx and .html files and benchmark them.

Start benchmarking

All tests I made from localhost. This reduces overhead for network traffic, but takes processor resources what penalties to absolute results. But difference is not so much for mono benchmarks, so I decide to choose more stable results rather than higher absolute values (which could be more higher when run at faster processor unit)

Url	Web server	requests/sec	Standart deviation	std dev %
hello.aspx	xsp4	1659.238	79.39	4.78
hello.html	xsp4	1004.428	34.47	3.43
hello.html	apache2	7129.956	76.80	1.08
Servicestack	xsp4	1913.746	34.84	1.82

Amazing results. You can see, that serving static html page in apache2 has the better performance than do it with xsp4, what was predictable, but not seven-times difference! Also, apsx page serves 1.6x faster than static html. Do you expect this? I did not.

Also, when I ran these benchmarks, I found that xsp4 grew in memory very fast when serving apsx pages, and after some limit (~265m) killed threads and produced deny of service error. Seems there is some memory leak in mono web server

But our goal is ServiceStack. You can see, that ServiceStack runs faster than aspx page or static html page in xsp4, but not so fast as apache2 static html. Why is so slow? Can we improve the performance? Answers to these questions you will find in next chapters

Looking inside ServiceStack runs

Why ServiceStack runs on mono not so fast as we can expect? To find answers to the question I turned up profile mode for xsp4 and look into generated profiles. To do it, before running xsp4 execute following command in shell:

export MONO_OPTIONS="--profile=log:noalloc,output=../output.mlpd"

log:noalloc means that we don't want to gather info about allocated objects. We are interested only in method calls timing
output=../output.mlpd sets the name of file for profiling information be gathered. Please note that we set parent directory instead of current for output file. Web server watches for changes in current directory and if we set it web server will get a lot of notification messages that the directory has changed and it draws back on the performance.

After that run the commands:

ab -n 500 -c 10 http://host:port/url
mprof-report output.mlpd > profile.txt

500 method calls is enough for getting profiling information, mprof-report produces human-readable form for the info.

Method call summary
Total(ms) Self(ms)      Calls Method name
   56244        8       1581 (wrapper runtime-invoke) :runtime_invoke_void__this___object (object,intptr,intptr,intptr)
   54344        3        500 Mono.WebServer.XSPWorker:RunInternal (object)
   54240        3        500 (wrapper remoting-invoke-with-check) Mono.WebServer.XSPApplicationHost:ProcessRequest (int,System.Net.IPEndPoint,System.Net.IPEndPoint,string,string,string,string,byte[],string,intptr,Mono.WebServer.SslInformation)
   54237        5        500 Mono.WebServer.XSPApplicationHost:ProcessRequest (int,System.Net.IPEndPoint,System.Net.IPEndPoint,string,string,string,string,byte[],string,intptr,Mono.WebServer.SslInformation)
   53513        4        500 Mono.WebServer.BaseApplicationHost:ProcessRequest (Mono.WebServer.MonoWorkerRequest)
   53390        1        500 Mono.WebServer.MonoWorkerRequest:ProcessRequest ()
   53226        5        500 System.Web.HttpRuntime:ProcessRequest (System.Web.HttpWorkerRequest)
   53173        4        500 System.Web.HttpRuntime:RealProcessRequest (object)
   53157       14        500 System.Web.HttpRuntime:Process (System.Web.HttpWorkerRequest)
   44442       14        500 System.Web.HttpApplication:System.Web.IHttpHandler.ProcessRequest (System.Web.HttpContext)
   44403       18        500 System.Web.HttpApplication:Start (object)
   41356      416        500 System.Web.HttpApplication:Tick ()
   40940      148        500 System.Web.HttpApplication/c__Iterator1:MoveNext ()
   17158       10        500 ServiceStack.WebHost.Endpoints.Support.EndpointHandlerBase:ProcessRequest (System.Web.HttpContext)
   17136       39        500 ServiceStack.WebHost.Endpoints.RestHandler:ProcessRequest (ServiceStack.ServiceHost.IHttpRequest,ServiceStack.ServiceHost.IHttpResponse,string)
   11422       25        500 System.Web.HttpApplication:PipelineDone ()
   11047       12        500 System.Web.HttpApplication:OutputPage ()
   11033       53        500 System.Web.HttpResponse:Flush (bool)
   10996      74       2108 System.Web.Configuration.WebConfigurationManager:GetSection (string,string,System.Web.HttpContext)
   10646       5       1004 System.Configuration.Configuration:GetSectionInstance (System.Configuration.SectionInfo,bool)
    9401     569     130811 System.Collections.Hashtable:GetHash (object)
    9252     596     106531 System.Collections.Hashtable:get_Item (object)
    8405       11        500 System.Web.HttpApplicationFactory:GetApplication (System.Web.HttpContext)
    8082        1        500 System.Web.HttpApplication:GetHandler (System.Web.HttpContext,string)
    8081        9        500 System.Web.HttpApplication:GetHandler (System.Web.HttpContext,string,bool)
    6861    1760      25111 Mono.Globalization.Unicode.SimpleCollator:CompareInternal (string,int,int,string,int,int,bool&,bool&,bool,bool,Mono.Globalization.Unicode.SimpleCollator/Context&)
    6707        8      2500 ServiceStack.WebHost.Endpoints.Extensions.HttpRequestWrapper:get_HttpMethod ()
    6699       13       500 ServiceStack.WebHost.Endpoints.Extensions.HttpRequestWrapper:Param (string)

I bold suspicious methods with both long execution time and large number of calls. As you can see only one is from ServiceStack code it is a property HttpRequestWrapper.HttpMethod. So what can we do, how can we increase performance, when most of long executing calls are related to mono and mono web server?

Lets have a look what methods call long-executing methods. To get info about backtraces, you should run command

mprof-report --traces ../output.mlpd > profile-traces.txt

   10996       74       2108 System.Web.Configuration.WebConfigurationManager:GetSection (string,string,System.Web.HttpContext)
 500 calls from:
  System.Web.HttpApplication:Start (object)
  System.Web.HttpApplication:Tick ()
  System.Web.HttpApplication/c__Iterator1:MoveNext ()
  System.Web.HttpApplication:GetHandler (System.Web.HttpContext,string)
  System.Web.HttpApplication:GetHandler (System.Web.HttpContext,string,bool)
  System.Web.HttpApplication:LocateHandler (System.Web.HttpRequest,string,string)
 500 calls from:
  System.Web.HttpRuntime:RealProcessRequest (object)
  System.Web.HttpRuntime:Process (System.Web.HttpWorkerRequest)
  System.Web.HttpApplication:System.Web.IHttpHandler.ProcessRequest (System.Web.HttpContext)
  System.Web.HttpApplication:Start (object)
  System.Web.HttpApplication:PreStart ()
  System.Web.Configuration.WebConfigurationManager:GetSection (string)
 500 calls from:
  Mono.WebServer.XSPWorkerRequest:SendHeaders ()
  Mono.WebServer.XSPWorkerRequest:GetHeaders ()
  Mono.WebServer.MonoWorkerRequest:get_HeaderEncoding ()
  System.Web.HttpResponse:get_HeaderEncoding ()
  System.Web.Configuration.WebConfigurationManager:SafeGetSection (string,System.Type)
  System.Web.Configuration.WebConfigurationManager:GetSection (string)
 500 calls from:
  System.Web.HttpApplication:System.Web.IHttpHandler.ProcessRequest (System.Web.HttpContext)
  System.Web.HttpApplication:Start (object)
  System.Web.HttpApplication:Tick ()
  System.Web.HttpApplication/c__Iterator1:MoveNext ()
  System.Web.HttpApplication/c__Iterator0:MoveNext ()
  System.Web.Security.UrlAuthorizationModule:OnAuthorizeRequest (object,System.EventArgs)

Look at the first backtrace. Don't you think that locating handler in web.config for every request looking strange? I think, all info about handlers should be loaded only once at application start and then reused for each request. If you look into mono code you will see that handlers are cached by mono, but why is ServiceStack handler is not cached?

The answer in these lines of code:

HttpHandlersSection httpHandlersSection = WebConfigurationManager.GetSection ("system.web/httpHandlers", req.Path, req.Context) as HttpHandlersSection;
ret = httpHandlersSection.LocateHandler (verb, url, out allowCache);

IHttpHandler handler = ret as IHttpHandler;
if (allowCache && handler != null && handler.IsReusable)
        cache [id] = ret;

To be cachable ServiceStack factory handler must implement IHttpHandler interface has IsReusable property set to 'true' and be allowed to cache. In mono source code you can find that allowCache means handler path in configuration section must not be "*" but it allowed to be "servicestack*" for example. So I changed httpHandlers section in web.config by changing attribute path="*" to path="servicestack*" and added implementation of IHttpHandler interface to ServiceStackHttpHandlerFactory

  #region IHttpHandler implementation

  void IHttpHandler.ProcessRequest(HttpContext context)
  {
   throw new NotImplementedException();
  }

  bool IHttpHandler.IsReusable
  {
   get
   {
    return true;
   }
  }

  #endregion

Then I recompiled ServiceStack and performed new benchmarks

Url	Web server	requests/sec	Standart deviation	std dev %
Servicestack	xsp4	1913.746	34.84	1.82
Servicestack reusable handler factory	xsp4	2003.238	35.39	1.77

Performance is increased by 4.68%. Not so much, but this just a start

In profiler we see that GetSection now called 1624 times instead of 2108

   14158       33       1624 System.Web.Configuration.WebConfigurationManager:GetSection (string,string,System.Web.HttpContext)

Now we will try to remove another overheads of GetSection calling. We can see that this method is called from HttpApplication.PreStart method and HttpResponse.HeaderEncoding property. Looking into source code brings a solution: get globalization section only once and than reuse it. This can be done only by changing mono sources. I did it and get results:

Url	Web server	requests/sec	Standart deviation	std dev %
Servicestack (mono 9eda1b4)	xsp4	1958.37	21.54	1.10
Servicestack (patched mono 9eda1b4)	xsp4	2025.316	21.56	1.06

Performance additionally gained 3.46%. Unfortunately before the patch I have had to update mono to revision 9eda1b4 and this dropped performance by 50 points from previous results

Now profiler shows 611 calls of GetSection and 7500ms what is much better

    7516       13        611 System.Web.Configuration.WebConfigurationManager:GetSection (string,string,System.Web.HttpContext)

Please note that this hack will work only if you don't use different globalization sections in web.config files are located in subdirectories of your site. If you site requires to use own globalizations for each path, don't use this hack

Now lets look to the HashTable:GetHash method. This method is fast, but it called too much times. It is not simply to reduce number of calls, but some hints could help. For example: add key in appSetting section of web.config file and you will reduce several thousands of GetHash calls but you should know this does not boost performance to any significant value

  <add key="MonoAspnetInhibitSettingsMap" value="true"/>

This key is used by mono to map some config sections to another one. If you do not use RoleMembership functionality or SqlServerCache you can disable mappings by adding the key. For more information you can read an article http://www.mono-project.com/ASP.NET_Settings_Mapping

..To be continued

• ServiceStack performance in mono part 2
• ServiceStack performance in mono part 3

ServiceStack десериализация inherited объектов

Есть абстрактный класс Item и выведенный из него DressItem

public abstract class Item
{
   public int ItemId {get; set;}

   public string Name {get; set;}
}  

public class DressItem : Item
{
}  

Класс Item сделан абстрактным, для того, чтобы ServiceStack умел сериализовать Item с указанием типа, т .е. если у нас есть конструкция:

Item item=new DressItem(){ItemId=1,Name="Пальто"};

то при сериализации в ServiceStack данная конструкция преобразуется в json:

{"__type": "MyNameSpace.DressItem, MyAssembly","ItemId":1,"Name":"Пальто"}

и при десериализации из такого json в Item у нас создастся DressItem. Например, если сделаем сервис редактирования Items

public class ItemRequest
{
  public Item Item {get; set;}
}

public class ItemEditService : Service
{
  public object Put(ItemRequest request)
  {
    DataContext.UpdateItem(request.Item);
    return true;
  }
}

То при вызове метода Put с вышеприведеным json у нас в request.Item окажется объект типа DressItem. Но вот что, интересно, если json немного изменить, сделав его таким:

{"ItemId":1, "__type": "MyNameSpace.DressItem, MyAssembly", "Name":"Пальто"}

т. е. просто поставив ItemId вперед, то в request.Item мы получим null! Получается, при передаче данных сервису "__type" всегда должен идти первым, иначе десериализация не сработает. А ведь это не всегда возможно, ведь мы не можем знать, каким образом будут вести себя сериализаторы объектов... Довольно странное поведение сервисстэка, возможно это ошибка.

ServiceStack и сериализация struct

Столкнулся с такой проблемой: при сериализации ServiceStack'ом .NET структур, вместо значений свойств в Json формате после сериализации получал название типа данных данной структуры.

 public struct Point
 {
  public int X { get; set;}
  public int Y { get; set;}
 }

 Point p = new Point (){X=10,Y=20};
 JsonSerializer.SerializeToString<Point> (p);

получал "MyNameSpace.Point" вместо "{X=10,Y=20}". В описании сериализатора ServiceStack есть заметка, что структуры сериализуются с помощью метода ToString(), но писать для каждой структуры свой сериализатор в ToString() - это чересчур. Но, к счастью, нашлось решение. Оказалось, что достаточно добавить строчку

 JsConfig<Point>.TreatValueAsRefType = true;

После чего сериализация стала работать как ожидается.