Setting up a NodeJS platform for multiplayer games/realtime application (Ubuntu+Nodejs+Varnish+Monit)

La version FR de cet article sera disponible bientôt

If you are reading this you already know that nodejs is a powerful engine for your realtime application or multiplayer game.
But it’s not easy to setup an up and running platform, which is self monitored.
by self monitored here I mean that the platform will automatically detect a nodejs crash/stop and will restart the service.
The platform should also support load, and for this we’ll use Varnish to optimize static content delivery.

Prerequisites

This tutorial suppose that you are already familiar with linux (I’ll use Ubuntu here) : know how to install/remove package from package manager and sources, and how to edit configuration files.
obviously, some programming and networking skills are preferable, but not required.

Used tools

nodejs
Varnish cache
Monit
upstart

Notes

I’ll suppose that you already own a dedicated server/VM for your node application
That no service is running on port 80 if you want your node server to run there.

that helloapp.nodeserver.com point to the server ip
And that you want your node application to run on helloapp.nodeserver.com

Let’s start

Setting up the environment

the first step is to install the tools we’ll need to build our platform

Nodejs

nodejs can be installed using the package manager (apt-get, yum …etc) but I’d recommande to install it from sources to get the latest version.

here is how you setup nodejs from sources :

$ sudo apt-get install python g++ make
$ wget http://nodejs.org/dist/node-latest.tar.gz
$ tar xvfvz node-latest.tar.gz
$ cd node-v0.xx.xx (replace a version with your own)
$ ./configure
$ make
$ sudo make install

$ sudo apt-get install python g++ make

$ wget http://nodejs.org/dist/node-latest.tar.gz

$ tar xvfvz node-latest.tar.gz

$ cd node-v0.xx.xx (replace a version with your own)

$ ./configure

$ make

$ sudo make install

Monit

monit is a little daemon which monitor services and restart them when a configured condition is met.
to setup monit

$ apt-get install monit

1	$ apt-get install monit

Varnish cache

Varnish is an HTTP cache proxy, it’ll accelerate the delivery of static content by caching them while reducing nodejs server load.
installing vernish is straightforward, here are the instruction from official web site :

$ apt-get install apt-transport-https
$ curl https://repo.varnish-cache.org/ubuntu/GPG-key.txt | apt-key add -
$ echo "deb https://repo.varnish-cache.org/ubuntu/ precise varnish-4.0" >> /etc/apt/sources.list.d/varnish-cache.list
$ apt-get update
$ apt-get install varnish

$ apt-get install apt-transport-https

$ curl https://repo.varnish-cache.org/ubuntu/GPG-key.txt | apt-key add -

$ echo "deb https://repo.varnish-cache.org/ubuntu/ precise varnish-4.0" >> /etc/apt/sources.list.d/varnish-cache.list

$ apt-get update

$ apt-get install varnish

We’ll also need some python mainly to be able to correctly log our nodejs server activity

$ apt-get install python-pip python-dev build-essential
$ pip install --upgrade pip
$ pip install --upgrade virtualenv
$ pip install docopt==0.6.1

$ apt-get install python-pip python-dev build-essential

$ pip install --upgrade pip

$ pip install --upgrade virtualenv

$ pip install docopt==0.6.1

all needed tools are not installed, let’s start the server configuration

Setting up the platform

Before starting, it’s very important to think of a structured organization of your platform, this will make teamwork easier because most of the time, you’ll not be alone on projects requiring such platforms.

event if you are alone, be sure that some months later, you’ll forget how you made things if they are not well structured and documented …

so here is the structure I use, you don’t have to follow the exact same structure, It’s here to give you an idea:

I use a sub domain for each application,it also make migration easy (here we use helloapp.nodeserver.com)
each application resides in /opt/nodeapps/<subdomain name>/, (here we’ll use /opt/nodeapps/helloapp.nodeserver.com
when a service require special configuration file for the application, I use the subdomain name as configuration file name (see bellow the upstart job and monit configurations)
when a variable designate my application in a script I use something like mysubdomain_mydomain_com

the rest of this article will illustrate how I follow this structure.

now we can start the configuration.

« hello world » server

For testing purpose, I’ll create a minimal nodejs http server which represent your real-time nodejs app or you multiplayer game server.
I recommand to put all nodejs apps in a separate directory.
here we’ll use /opt/nodeapps/

our hello world http server

create file /opt/nodeapps/helloapp.nodeserver.com/server.js

var http = require('http');


var server = http.createServer(function (request, response) {
    console.log("received request ", request);
    response.writeHead(200, {"Content-Type": "text/plain"});
    response.end("Hello World\n");
});

// Listen on port 10000
server.listen(10000);


console.log("Server running at http://127.0.0.1:10000/");

var http = require('http');

var server = http.createServer(function (request, response) {

console.log("received request ", request);

response.writeHead(200, {"Content-Type": "text/plain"});

response.end("Hello World\n");

});

// Listen on port 10000

server.listen(10000);

console.log("Server running at http://127.0.0.1:10000/");

Creating upstart job

now we’ll write the upstart job that’ll be responsible of launching our helloapp server.
this will allow you if you want, to schedule the nodejs startup in a system runlevel as any other system service (but it’s not needed since we’ll use monit), the upstart will also send helloapp server output to a logfile so we can inspect what’s hapening, investigate crashes …etc

for logging, I’m using a little python script that you can download here.
this script will pipe the logs to logfiles and ensure that they are not locked.
you can just pipe stdout/strerr directly to logfiles but then you’ll have some troubles (and headache) if you want to make rotating logs.

bellow I suppose that the script is in /opt/nodeapps/scripts/log.py

create/edit file /etc/init/helloapp.nodeserver.com.conf

#!upstart
description "node.js Hello Server"
author "Ezelia"

start on startup
stop on shutdown

env APPNAME="helloapp.nodeserver.com"

script
    export HOME="/opt/nodeapps/$APPNAME"

    echo $$ > /var/run/$APPNAME.pid

/bin/bash <<EOT
exec sudo -u nodeuser /usr/local/bin/node /opt/nodeapps/$APPNAME/server.js | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log
EOT

end script

pre-start script
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log
end script

pre-stop script
    rm /var/run/$APPNAME.pid
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log
end script

#!upstart

description "node.js Hello Server"

author "Ezelia"

start on startup

stop on shutdown

env APPNAME="helloapp.nodeserver.com"

script

export HOME="/opt/nodeapps/$APPNAME"

echo $$ > /var/run/$APPNAME.pid

/bin/bash <<EOT

exec sudo -u nodeuser /usr/local/bin/node /opt/nodeapps/$APPNAME/server.js | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log

EOT

end script

pre-start script

echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log

end script

pre-stop script

rm /var/run/$APPNAME.pid

echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" | /nodeapps/scripts/log.py /var/log/nodeapps/$APPNAME.log

end script

you can use this same upstart script to create multiple applications upstart jobs, if you respect the same structure as mine, all you have to do is to change APPNAME variable

testing the upstart job

$ start helloapp.nodeserver.com

1	$ start helloapp.nodeserver.com

go to your browser and type : http://helloapp.nodeserver.com:10000/

note at this step, the server is accessible throught 10000 port, but this will change with Varnish Cache.

Configuring varnish cache

by default, varnish run on port 6081
if you want your application to be accessible from default HTTP port (or some other port different from 6081) you have to edit /etc/default/varnish
and change

DAEMON_OPTS="-a :6081 \

1	DAEMON_OPTS="-a :6081 \

 DAEMON_OPTS="-a :80 \

1	DAEMON_OPTS="-a :80 \

for port 80

next we need to configure varnish
this is a little tricky
here is my varnish config which is compatible with nodejs, and handle websockets correctly (since websockets are mandatory for the application/game server we are making 😉 )

/* section 1 */
backend helloapp_nodeserver_com {
    .host = "127.0.0.1";
    .port = "10000";
}
/************/

sub vcl_fetch {
    set beresp.ttl = 8h;
    set beresp.grace = 600s;
    if (beresp.status == 404 || beresp.status >= 500 || beresp.status == 503) {
      set beresp.ttl = 0s;
    }
    return (deliver);
}

sub vcl_deliver {
    /* remove those unneeded tags for production */
    remove resp.http.X-Varnish;
    remove resp.http.Via;
    remove resp.http.Age;
    remove resp.http.X-Powered-By;
    return (deliver);
}

sub vcl_recv {
    if (req.url == "/get-server-health") {
        error 200 "Server UP";
    }

    /* section 2 */
    if (req.http.host == "helloapp.nodeserver.com") {
        set req.backend = helloapp_nodeserver_com;
    }
    /*************/

    if (req.http.Upgrade ~ "(?i)websocket") {
        unset req.http.Cookie;
        return (pipe);
    }

    if (req.restarts == 0) {
        if (req.http.x-forwarded-for) {
            set req.http.X-Forwarded-For =
                req.http.X-Forwarded-For + ", " + client.ip;
        } else {
            set req.http.X-Forwarded-For = client.ip;
        }
    }
    if (req.request != "GET" &&
            req.request != "HEAD" &&
            req.request != "PUT" &&
            req.request != "POST" &&
            req.request != "TRACE" &&
            req.request != "OPTIONS" &&
            req.request != "DELETE") {
        /* Non-RFC2616 or CONNECT which is weird. */
        return (pipe);
    }
    if (req.request != "GET" && req.request != "HEAD") {
        /* We only deal with GET and HEAD by default */
        return (pass);
    }



    if (req.http.Authorization || req.http.Cookie) {
        /* Not cacheable by default */
        return (pass);
    }


    return (lookup);
}

sub vcl_pipe {
    if (req.http.upgrade) {
        set bereq.http.upgrade = req.http.upgrade;
    }
}
sub vcl_pass {
    return (pass);
}
sub vcl_error {
     set obj.http.Content-Type = "text/html; charset=utf-8";
     #if (obj.status == 200) return (deliver);
    
     set obj.http.Retry-After = "5";
     synthetic {"
 <?xml version="1.0" encoding="utf-8"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
   <head>
     <title>Server Offline</title>
   </head>
   <body>
     <h2 style="display:block;text-align:center;border:2px solid #c22;color:#911;background:#ff7777">This application is offline :(</h2>
     <pre>
        "} + obj.status + " " + obj.response + {"
     </pre>
    <hr />
   </body>
 </html>
 "};
     return (deliver);
 }

100

101

102

103

104

105

106

107

/* section 1 */

backend helloapp_nodeserver_com {

.host = "127.0.0.1";

.port = "10000";

}

/************/

sub vcl_fetch {

set beresp.ttl = 8h;

set beresp.grace = 600s;

if (beresp.status == 404 || beresp.status >= 500 || beresp.status == 503) {

set beresp.ttl = 0s;

}

return (deliver);

}

sub vcl_deliver {

/* remove those unneeded tags for production */

remove resp.http.X-Varnish;

remove resp.http.Via;

remove resp.http.Age;

remove resp.http.X-Powered-By;

return (deliver);

}

sub vcl_recv {

if (req.url == "/get-server-health") {

error 200 "Server UP";

}

/* section 2 */

if (req.http.host == "helloapp.nodeserver.com") {

set req.backend = helloapp_nodeserver_com;

}

/*************/

if (req.http.Upgrade ~ "(?i)websocket") {

unset req.http.Cookie;

return (pipe);

}

if (req.restarts == 0) {

if (req.http.x-forwarded-for) {

set req.http.X-Forwarded-For =

req.http.X-Forwarded-For + ", " + client.ip;

} else {

set req.http.X-Forwarded-For = client.ip;

}

if (req.request != "GET" &&

req.request != "HEAD" &&

req.request != "PUT" &&

req.request != "POST" &&

req.request != "TRACE" &&

req.request != "OPTIONS" &&

req.request != "DELETE") {

/* Non-RFC2616 or CONNECT which is weird. */

return (pipe);

}

if (req.request != "GET" && req.request != "HEAD") {

/* We only deal with GET and HEAD by default */

return (pass);

}

if (req.http.Authorization || req.http.Cookie) {

/* Not cacheable by default */

return (pass);

}

return (lookup);

}

sub vcl_pipe {

if (req.http.upgrade) {

set bereq.http.upgrade = req.http.upgrade;

}

sub vcl_pass {

return (pass);

}

sub vcl_error {

set obj.http.Content-Type = "text/html; charset=utf-8";

#if (obj.status == 200) return (deliver);

set obj.http.Retry-After = "5";

synthetic {"

<?xml version="1.0" encoding="utf-8"?>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">

<html>

<head>

<title>Server Offline</title>

</head>

<body>

<h2 style="display:block;text-align:center;border:2px solid #c22;color:#911;background:#ff7777">This application is offline :(</h2>

<pre>

"} + obj.status + " " + obj.response + {"

</pre>

<hr />

</body>

</html>

"};

return (deliver);

}

the « section 1 » declare a backend, the « section 2 » identifies requests hostname which will be handled by our backend.

now start varnish

/etc/init.d/varnish start

1	/etc/init.d/varnish start

and go to http://helloapp.nodeserver.com/ (if you used a different port than 80 you have to specify it)
your hello server should respond.

we are almost done !

now what if the server crash or stop ?

let’s simulate a server stop

$ stop helloapp.nodeserver.com

1	$ stop helloapp.nodeserver.com

now when you go to helloapp.nodeserver.com the you get a varnish error page (btw, you can use it to show a message to your users 😉 )

Monit to the rescue

global monit configuration

create/edit file /etc/monit/conf.d/helloapp.nodeserver.com.conf

check process helloapp_nodeserver_com with pidfile "/var/run/helloapp.nodeserver.com.pid"
    start program = "/sbin/start helloapp.nodeserver.com"
    stop program = "/sbin/stop helloapp.nodeserver.com"
    if failed port 10000 protocol HTTP
        request /
        with timeout 30 seconds
        then restart

check process helloapp_nodeserver_com with pidfile "/var/run/helloapp.nodeserver.com.pid"

start program = "/sbin/start helloapp.nodeserver.com"

stop program = "/sbin/stop helloapp.nodeserver.com"

if failed port 10000 protocol HTTP

request /

with timeout 30 seconds

then restart

what this script do is to tell monit to check the presence of helloapp.nodeserver.com process that is listening to local port 10000, if no response within 30 seconds restart the server.

save the file and start monit

service monit start

1	service monit start

now go to to http://helloapp.nodeserver.com/ and you’ll see that the server is responding.
try to stop it : stop helloapp.nodeserver.com

refresh the page …. and he’s here again 🙂

but wait, our nodejs application depend on both nodejs server AND varnish cash
let’s monitor varnish

create/edit file /etc/monit/conf.d/varnish.conf

check process varnish with pidfile "/var/run/varnishd.pid"
    start program = "/etc/init.d/varnish start"
    stop program = "/etc/init.d/varnish stop"
    if failed host 127.0.0.1 port 80 protocol HTTP
      and request "/get-server-health"
      then restart

check process varnish with pidfile "/var/run/varnishd.pid"

start program = "/etc/init.d/varnish start"

stop program = "/etc/init.d/varnish stop"

if failed host 127.0.0.1 port 80 protocol HTTP

and request "/get-server-health"

then restart

here the check condition is different, we actually check varnish presence using a special request we configured in varnish config.

restart monit

service monit restart

1	service monit restart

now if everything is up and running, just make sure that monit is starting on system boot :
update-rc.d monit enable

Final step : configur log rotation

log are very important in such applications, and linux provide an efficient tool to handle log rotations to prevent infinite growing log files.

to configure log rotation for all our nodejs applications

create/edit /etc/logrotate.d/nodeapps

/var/log/nodeapps/*.log {
    daily
    create
    rotate 30
    compress
    size 10M
    missingok
    notifempty
    sharedscripts
}

/var/log/nodeapps/*.log {

daily

create

rotate 30

compress

size 10M

missingok

notifempty

sharedscripts

}

Live example

want to see a live example ? here is doyazan prototype : http://demo.ezelia.com/ which is a proof of concept for a multiplayer HTML5 isometric game 🙂

open the url, enter a random login and click start under the desirez avatar.

Multiple node apps on the same server

Now that you have a working platform, you can easily add other nodejs applications.

for this you have three steps :

Create the upstart job
Configure varnish (declare a new backend and define the request hostname to use)
Create a monit configuration file to monitor the application

You can automate those operations using a python or perl script, but beware, you should make sure that your script are properly tested, a misconfiguration in varnish can make all your node apps unreachable.

Your turn !

this article explained the basics to obtain a self monitored nodejs server for your application.
one important thing that has not been adressed here is security, since it’s a BIG topic… but you can add some security to your current server with a little more work.

First, a good practive would be to drop all incomming connexions to port different from 80, since all your node applications should be accessible throught this port now.

For hack attempts, or bad behaviours, what I usually do, is to let the nodejs application log every suspect activity.
those activities are tagged with some keywords (you can prefix suspect activities log with somthing like [SECURITY-WARN])
then I use fail2ban with a custom rule to detect and ban IPs generating a lot of those security warnings

this can be a good exercice to enhance the platform 🙂

Setting up a NodeJS platform for multiplayer games/realtime application (Ubuntu+Nodejs+Varnish+Monit)

Prerequisites

Used tools

Notes

Setting up the environment

Nodejs

Monit

Varnish cache

Setting up the platform

« hello world » server

Creating upstart job

Configuring varnish cache

Monit to the rescue

Final step : configur log rotation

Live example

Multiple node apps on the same server

Your turn !

Écrit par

Ezelia

Follow us on Twitter

Like us on Facebook