ServiceStack performance in mono part2

In previous part I told about some performance enhancements which could be used with ServiceStack running over mono XSP web server. But nobody uses XSP in production environment, the most common use cases are nginx+mono-fastcgi and apache+mod_mono. But what is the performance in such environment? Will see it.

Configuration

Apache

If you want to use mono with apache, you have to install mod_mono for apache and configure it according to this article. To install mod_mono in Ubuntu you can type

  sudo apt-get install libapache2-mod-mono

after that you have to reinstall mono web server was compiled from sources. Change the directory to xsp source code and run sudo make install from it. I am going to benchmark mono under apache in following configurations:

• 1. Direct access to static html file from apache without mono.
• 2. Get ServiceStack "Hello, World!" service throught apache2-mod-mono
• 3. Get static html file and "Hello, World!" aspx page throught apache2-mod-mono without ServiceStack.

To manage this I use following config in /etc/apache2/http.conf. For direct static file access I placed hello.html in the web server root (/var/www)

NameVirtualHost ssbench3:80
NameVirtualHost ssbench2:80

<VirtualHost ssbench3:80>
    ServerName ssbench3
    DocumentRoot /var/www/ssbench3
#    MonoPath default "/usr/bin/mono/2.0"
    MonoServerPath ssbench3 /usr/bin/mod-mono-server4
    AddMonoApplications ssbench3 "ssbench3:/:/var/www/ssbench3"
        
    <location />
 MonoSetServerAlias ssbench3
 Allow from all
 Order allow,deny
 SetHandler mono
    </location>
</VirtualHost>

<VirtualHost ssbench2:80>
    ServerName ssbench2
    DocumentRoot /var/www/ssbench2
#    MonoPath default "/usr/bin/mono/2.0"
    MonoServerPath ssbench2 /usr/bin/mod-mono-server4
    AddMonoApplications ssbench2 "ssbench2:/:/var/www/ssbench2"
        
    <location />
 MonoSetServerAlias ssbench2
 Allow from all
 Order allow,deny
 SetHandler mono
    </location>
</VirtualHost>

Nginx

Configuration of Nginx is similar to Apache, differences are only in transport between mono and front-end web server. Apache uses mod-mono-server while nginx uses fastcgi-mono-server. Also, you may note that I added one additional configuration: nginx as proxy to xsp4.

To configure nginx I followed this guide. I added following lines to /etc/nginx/fastcgi_params

fastcgi_param HTTP_HOST $host;
fastcgi_param  PATH_INFO          "";
fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;

And added virtual hosts to /etc/nginx/sites-enabled/default

 server {
         listen   81;
         server_name  ssbench1;
         access_log   /var/log/nginx/ssbench1.log;
 
         location / {
     proxy_pass http://127.0.0.1:8080/;
     proxy_set_header   X-Real-IP $remote_addr;
     proxy_set_header   Host $http_host;
     proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
         }
 }

 server {
         listen   81;
         server_name  ssbench2;
         access_log   /var/log/nginx/ssbench2.log;
 
         location / {
                 root /var/www/ssbench2/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass 127.0.0.1:9000;
                 include /etc/nginx/fastcgi_params;
             }

 }
 
 server {
         listen   81;
         server_name  ssbench3;
         access_log   /var/log/nginx/ssbench3.log;
 
         location / {
                 root /var/www/ssbench3/;
                 index index.html index.htm default.aspx Default.aspx;
                 fastcgi_index Default.aspx;
                 fastcgi_pass 127.0.0.1:9000;
                 include /etc/nginx/fastcgi_params;
         }

 }

after that, I ran the command:

fastcgi-mono-server4 /applications=ssbench2:/:/var/www/ssbench2/,ssbench3:/:/var/www/ssbench3 /socket=tcp:127.0.0.1:9000

Also, I ran xsp4 server hosted ServiceStack on port 8080

EDIT: after the post was written, I additionaly benchmarked several configuration were not mentioned in the first version, they are:

• Nginx as frontend proxy to apache server with mod-mono
• Self-hosted ServiceStack instance based on two classes: AppHostHttpListenerBase and AppHostHttpListenerLongRunningBase. How to create self-hosted ServiceStack you can read in ServiceStack wiki. Also, you can look in test source code to get additional details
• Nginx as frontend proxy to self-hosted ServiceStack.
• Nginx plus HyperFastCgi (is a new fastcgi server I written. Replacement of mono-webserver-fastcgi)

Benchmark results

Before I'll print the results I want to say a couple words about my expectations. What did I expect? At first, I predicted that nginx be winner of serving static html pages. It was obvious. Secondly, I thought that nginx+ServiceStack get slightly better results versus Apache+ServiceStack and maybe XSP+ServiceStack due to nginx async behaviour and lower processor usage. Also, I thought that performance difference between Apache+ServiceStack and XSP+ServiceStack should be minimal. They are both use the same threading model and what could I expect a little overhead in apache<->mod-mono communications. But... Here are the results

Configuration	requests/sec	Standart deviation	std dev %	Comments
Apache2 direct file	7129.95	217.57	3.05
Apache2+mod_mono+ServiceStack	1314.30	22.40	1.70
Apache2+mod_mono hello.html	924.02	12.82	1.39
Apache2+mod_mono hello.aspx	----	----	---	Memory Leaks, Crashes
Nginx direct file	10458.71	147.28	1.41
Nginx+fastcgi-server+ServiceStack	571.36	8.81	1.54	Memory Leaks
Nginx+fastcgi-server hello.html	409.48	9.14	2.23	Memory Leaks
Nginx+fastcgi-server hello.aspx	458.55	9.89	2.16	Memory Leaks, Crashes
Nginx+proxy to Apache2+mod-mono+ServiceStack	1143.82	8.49	0.74
Nginx+proxy to self-hosted ServiceStack (AppHost HttpListenerBase)	1993.82	17.62	0.88
Nginx+proxy to self-hosted ServiceStack (AppHost HttpListenerLongRunningBase)	1664.94	27.45	1.65
Nginx+HyperFastCgi (tcp keepalive)+ServiceStack	2041.25	23.18	1.14	See more info
Nginx+proxy to xsp4+ServiceStack	1402.33	45.42	3.24	Unstable Results, Errors
xsp4+ServiceStack	2246.51	21.31	0.94
Self-hosted ServiceStack (AppHost HttpListenerBase)	2697.1	30.1	1.12
Self-hosted ServiceStack (AppHost HttpListenerLongRunningBase)	2313.11	33.14	1.43

What can we see? First place in serving ServiceStack takes xsp4. Then goes Apache+mod_mono and the last one is Nginx+fastcgi-server which is four times worse then the winner. I did not mentioned here Nginx+proxy xsp4 configuration because during test execution in half of test runs I get errors when receive json data. There were not so many errors (~1500 on 100 000 requests), but they were exist and this was the reason to drop away nginx+xsp4 configuration from competition. By the way performance result for the configuration slightly better than apache+mod_mono and much better than Nginx+fastgi-server.

Also I did not include HyperFastCgi server in the chart which shows good performance, because it was created after these benchmarks have done. Benchmarks of the Nginx+HyperFastCgi server you can find in next part

As serving static html files the first place takes Nginx as expected, second by Apache and after them goes all other configuration: xsp4 (you can see test results for static xsp4 html serving in previous post), Apache+mod_mono, Nginx+fastcgi. They all are really very slow comparing with Nginx or Apache.

For .aspx page I could not get reliable results. At first, there are memory leaks in mono web server during processing the aspx pages and they are possible a reason of crashes I've got. I could only get ~20000 requests with Nginx+fastcgi and several thouthands request with Apache+mod_mono before mono hanged or got SIGSEGV. I suspect that the reason of these faults are changes of hadling and spawning threads and changes performed in mono GC. Hope that this instablity will be fixed in next mono release.

Also I've mentioned that fastcgi-mono-server produced a huge memory leaks during runs. After processing 100 000 requests it was used about 600M of memory! With such configuration you cannot serve large amount or requests without regular restarting of the server. Also performance of fastcgi-mono-server is extremely slow compared to mod-mono-apache. What is going on in the server? I am going to look inside it in the next posts

Links:

• ServiceStack performance in mono part 1
• ServiceStack performance in mono part 3
• ServiceStack performance in mono part 4