Run node.js app as a daemon process (Linux)

To totally unlock this section you need to Log-in

Let's take a look to the following scenario: we have a node.js server/app and we start it by using npm start & (background mode) command through Putty. But the problem is that if we close Putty (or any other SSH client) the server will be stopped after sometime.

So, our main need, in this scenario, is to instruct the system to keep the nodejs app (app.js), so our application, alive while not connected through SSH on the system itself, in interactive mode. This leads to the need to "transform" the specific app.js (starting it with npm start) command into a daemon, basically into a system service.

To address this scenario there are two possible, reliable, approaches: pm2 and forever.

PM2

PM2 is a daemon process manager that will help you manage and keep your application online 24/7 (https://github.com/Unitech/pm2).

Some advantages of running your application using PM2 are the following:

• PM2 will automatically restart your application if it crashes.
• PM2 will keep a log of your unhandled exceptions - for example, in a file at /home/example_user/.pm2/logs/app-err.log.
• With one command, PM2 can ensure that any applications it manages restart when the server reboots. Basically, your node application will start as a service.

We can install it by simply running the following command in npm:

sudo npm install pm2 -g

To update PM2 the process is simple:

npm install pm2 -g && pm2 update

Then we can update the in-memory PM2 daemon via command:

pm2 update

After the PM2 update is also recommended to update also the PM2 startup script (if used) running:

$ pm2 unstartup
$ pm2 startup

To check the version of PM2 installed on the system, execute the below command:

pm2 --version

Then we can, moving into the working directory of our application on the system (for example in /var/www/html, if we have placed it there), instruct pm2 to keep alive our app by running a similar command (the --name option is not mandatory, but recommended to identify out app later using pm2 command line utility):

pm2 --name HelloWorld start npm -- start

We can list running processes in pm2:

pm2 ps

Each process will have a unique id , using which we can manage a particular process , Instead of restarting the entire app.

Stop it if needed by using the following command, specifying the id of the app that we need to stop:

pm2 delete 0

NOTE: we can use process name instead of ID (if provided).

We can also check the pm2 logs to see if there are issues or problems with apps running with pm2:

pm2 logs

To check the logs for a particular process or for a specific application:

pm2 logs 4
pm2 logs sample-app

To know more details about a particular process, then run this command:

pm2 show 0

To stop a process or an application we can use the below command:

pm2 stop 8
pm2 stop sample-app

To flush all the logs for all the applications, execute the below command:

pm2 flush

To enable the PM2 to automatically start on system bootup (so pm2 will be daemonized (with pm2-root name) and also all the apps that are running in it will be automatically saved for future reboots), the following command will create a automatically-configured startup script for the system:

pm2 startup

To save the lists of processes so that the processes will be automatically started after the system reboot:

pm2 save

To manually restore the previously saved processes:

pm2 resurrect

If you wish to disable PM2 from starting automatically on system boot, Execute the below command:

pm2 unstartup

Another useful aspect is that PM2 offers a declaration file to comprehensively define every app which should be started when using this file (in JSON format).

The following JSON declaration file is an example on how to define two different NodeJS apps, called apps.json:

{
  "apps": [
    {
      "name": "futurestudio-homepage",
      "script": "./homepage/server.js",
      "instances": 2,
      "exec_mode": "cluster",
      "env": {
        "NODE_ENV": "production",
        "PORT": "3000",
      }
    },
    {
      "name": "blog",
      "script": "./blog/index.js",
      "instances": 1,
      "exec_mode": "fork",
      "env": {
        "NODE_ENV": "production",
        "PORT": "4000",
      }
    }
  ]
}

Within the root object, we are defining an apps array containing all apps which get started by PM2 using this declaration file. Each application declaration is wrapped into its own JSON object. Additionally, each object within the apps array defines multiple app properties. These properties represent the options you would use on PM2’s command line utility.

The exemplary declaration file above defines two apps: homepage and blog. We can now start both applications by running this .json file with PM2:

pm2 start apps.json

Of course we can use other PM2 commands referencing this JSON file:

# Start processes
pm2 start apps.json

# Restart processes
pm2 restart apps.json

# (Graceful) Reload processes
pm2 reload apps.json

# Stop processes
pm2 stop apps.json

# Delete processes
pm2 delete apps.json  

Remember: If we are executing the PM2 commands on the app declaration file, our command will affect each defined application. We can restart, stop, or delete a single application using the command line utility, as usual.

The following list depicts available options within our JSON app declaration file:

• name: application name, e.g. "blog"
• cwd: app location from where it will be launched, e.g. "/blog"
• args: additional arguments passed to your app, e.g. ["testing"]
• script: the script which will be executed to start an app, e.g. "/blog/app.js,
• node_args: additional Node.js arguments when launching your app, e.g. ["--harmony", " --max-stack-size=102400000"]
• timestamp: format of dates in your log files, e.g. "YYYY-MM-DD HH:mm Z"
• error_file: location of your apps error log file, e.g. "logs/blog.stderr.log"
• out_file: location of your apps out log file, e.g. "logs/blog.stdout.log"
• pid_file: location of your apps .pid file, e.g. "pids/blog.pid"
• instances: number of worker processes for your app. Allowed values: positive integer including 0 or 'max'
• min_uptime: minimum uptime before PM2 starts to restart an app in case of errors, e.g. "20s" (20 seconds), default value is 1s
• max_restarts: maximum restart count before PM2 gives up to get your app online, e.g. 10, default value is 15
• max_memory_restart: maximum memory amount before PM2 restarts your app, e.g. "100M" (100 megabytes), allowed values: "1G", "50M", "4K"
• cron_restart: cronjob pattern to restart your app, e.g. `"0 1 * * *",
• watch: watch the app folder for file changes in restart the app if a change is detected, default value is false, allowed values are true|false
• ignore_watch: regex list of files or folders to be ignored when watching an app, e.g. ["[\\/\\\\]\\./", "node_modules"]
• merge_logs: defines whether the logs of all app worker instances will be merged into the same file, default value is false, allowed values true|false
• exec_interpreter: interpreter of your app, e.g. "node"
• exec_mode: execution mode of your app (can also be defined by the instances field), e.g. "fork", allowed values fork|cluster
• autorestart: if enabled, PM2 will automatically restart apps on crashes, e.g. false, default is true, allowed values true|false
• vizion: integrates version control metadata with PM2, default value is true, allowed values are true|false
• env: environment varibles for your app, specified as a nested object, like env: { "NODE_ENV": "production", "AWESOME_SERVICE_API_TOKEN": "xxx" }

Every command line option is also available within the declaration file.

It is useful to note that we can create a sample ecosystem.config.js configuration file, using the following command:

$ pm2 init simple

This will generate a sample ecosystem.config.js that we can extend to define our NodeJS apps, but not only, and specific properties:

module.exports = {
  apps : [{
    name   : "app1",
    script : "./app.js"
  }]
}

Once defined multiple apps into this configuration file (that needs to end with the .config.js text to be valid in pm2), we can manage all of them by controlling directly the ecosystem.config.js with the following commands:

# Start all applications
pm2 start ecosystem.config.js

# Stop all
pm2 stop ecosystem.config.js

# Restart all
pm2 restart ecosystem.config.js

# Reload all
pm2 reload ecosystem.config.js

# Delete all
pm2 delete ecosystem.config.js

Finally, we need to consider also that we can define "environments" for each app defined into a JSON (or YAML) configuration file (or into ecosystem.config.js).

Depending on the application setup and execution context we want to make use of different values within your process specific environment variables. Environment variables are a convenient and commonly used way to inject context specific options to the app and pass specific parameters/values to it.

The following example process file defines four different environments, each having their own key-value-pairs: env, env_develop, env_production, env_test. Environments in a PM2 process file follow the schema of env_* where * is the placeholder for your desired environment name.

The following example uses JSON format for the process file. We call the following process file app.json.

{
  "apps": [
    {
      "name"           : "futurestudio-homepage",
      "script"         : "./server.js",
      "exec_mode"      : "cluster",
      "instances"      : 2,
      "env": {
        "DB_USER"      : "root",
        "DB_PASS"      : "root"
      },
      "env_develop": {
        "NODE_ENV"     : "develop"
      },
      "env_production" : {
        "NODE_ENV"     : "production",
        "PORT"         : 2000,
        "DB_USER"      : "username",
        "DB_PASS"      : "secret-password"
      },
      "env_test"       : {
        "NODE_ENV"     : "testing",
        "PORT"         : 8000
      }
    }
  ]
}

Besides the attributes for name, script, exec_mode and instances, we can spot the individual environment configurations.

It is important to know that values within env are set as the default environment. We can override the default values if necessary in other environments.

Different environments within a PM2 process file are defined using the env_* keyword and replacing the star * with the appropriate name, like env_production. We can define as many different environments as you want by using differing names.

With multiple environments set up in your process file we can choose the one to start by adding the --env <environment> to the start command that references the process file.

Using the app.json file from above, we start the development environment like this:

$ pm2 start app.json --env develop

Within your application running in develop mode, the NODE_ENV variable is now accessible with the value develop (as defined), so the application will know that the variable NODE_ENV has develop value. If we will add more environment variables for env_develop within the process file, they will be exposed to our app as well.

Assuming that we want our app to just start in default mode (so using the env values), we can still reference the app.json file with the PM2 command line utility and only the values defined within env are passed and accessible by your application.

$ pm2 start app.json
# will start your app using "env"

Starting your application with default values makes sense for development or testing purposes avoiding the extra typing to specify the environment.

Let's now consider that we want to switch to another environment: that’s straight forward: we need to use the --env <new-environment> argument with the new environment.

If we have already started our app in env_develop mode and we want to switch in env_production we can use the following approach:

$ pm2 restart app.json --env production

Besides pm2 restart, you can leverage every available command like reload to switch environments in zero-downtime manner.

Cluster Mode

An additional very importart feature usable in pm2 is the cluster mode.

The cluster mode allows networked Node.js applications (http(s)/tcp/udp server) to be scaled across all CPUs available, without any code modifications. This greatly increases the performance and reliability of your applications, depending on the number of CPUs available. Under the hood, this uses the Node.js cluster module such that the scaled application’s child processes can automatically share server ports.

To enable the cluster mode, just pass the -i option:

pm2 start app.js -i max

max means that pm2 will auto detect the number of available CPUs and run as many processes as possible:

Or using a js/yaml/json configuration file approach:

module.exports = {
  apps : [{
    script    : "api.js",
    instances : "max",
    exec_mode : "cluster"
  }]
}

NOTE: you need to set the exec_mode to cluster so pm2 know you want to load balance between each instances, by default it will not.

Then to start the process file:

pm2 start processes.json

The -i or instances option can be:

• 0 or max to spread the app across all CPUs
• -1 to spread the app across all CPUs - 1
• number to spread the app across number CPUs

Using PM2 and its cluster mode there is also the possibility to scale the applications in real-time. If we need more or less workers than currently available, we can use PM2’s scale command and adjust the cluster size respectively:

pm2 scale <app-name> <number-of-workers>  

The scale command is only available for processes that already run in cluster mode. PM2 won’t restart your app which currently runs in fork mode (so, the basic process spawning). You need to manually delete and restart it in cluster mode.

The following code block shows how to scale up the number of processes for the examplary sample app:

$ pm2 scale sample 6

Of course, PM2 allows you to scale down any app in real-time (in this case from 6 to 2 workers):

$ pm2 scale homepage 2

The previously running processes are deleted immediately and the number of workers reduced to your defined number.

We can also use a shortcut within PM2 which allows us to scale up or down your number of workers: pm2 scale sample +1 or pm2 scale sample -1: the first command will add another worker to sample and the second one removes a worker process.

Forever

Forever is, as PM2, a process manager for Node.js applications (but not only, as PM2); it keeps node.js applications running on production server and it utomatically restart server if it crashes or close, without any downtime. It is meant to simplify your life in a production environment by managing (starting, stopping, restarting, etc) Node processes and their configurations.

NOTE: one important feature about forever (and other process management tools like it) is that we can use it for non-Node applications as well, like Python, Ruby, etc. The -c, or command flag, let forever how to run your app: we can tell it to use any other interpreter for execution, like Python:

$ forever start -c python py_script.py

So even if we don't like Node, or just need to use a different language for a project, keep in mind that this could be still useful to know.

You can specify these configurations via command line or using a JSON configuration file. Using a JSON file we can configure/define multiple NodeJS processes in a single file, making it easy to launch all of your processes at once.

The general syntax of forever utility is shown below:

 $ forever --help
 usage: forever [action] [options] SCRIPT [script-options]  Monitors the script specified in the current process or as a daemon  actions:
 start               Start SCRIPT as a daemon
 stop                Stop the daemon SCRIPT by Id|Uid|Pid|Index|Script
 stopall             Stop all running forever scripts
 restart             Restart the daemon SCRIPT
 restartall          Restart all running forever scripts
 list                List all running forever scripts
 config              Lists all forever user configuration
 set <key> <val>     Sets the specified forever config  clear <key>         Clears the specified forever config  logs                Lists log files for all forever processes
 logs <script|index> Tails the logs for columns add <col>   Adds the specified column to the output in `forever list`. Supported columns: 'uid', 'command', 'script', 'forever', 'pid', 'id', 'logfile', 'uptime'
    columns rm <col>    Removed the specified column from the output in `forever list`
    columns set <cols>  Set all columns for the output in `forever list`
    columns reset       Resets all columns to defaults for the output in `forever list`
    cleanlogs           [CAREFUL] Deletes all historical forever log files

  options:
    -m  MAX          Only run the specified script MAX times
    -l  LOGFILE      Logs the forever output to LOGFILE
    -o  OUTFILE      Logs stdout from child script to OUTFILE
    -e  ERRFILE      Logs stderr from child script to ERRFILE
    -p  PATH         Base path for all forever related files (pid files, etc.)
    -c  COMMAND      COMMAND to execute (defaults to node)
    -a, --append     Append logs
    -f, --fifo       Stream logs to stdout
    -n, --number     Number of log lines to print
    --pidFile        The pid file
    --uid            DEPRECATED. Process uid, useful as a namespace for processes (must wrap in a string)
                     e.g. forever start --uid "production" app.js
                         forever stop production
    --id             DEPRECATED. Process id, similar to uid, useful as a namespace for processes (must wrap in a string)
                     e.g. forever start --id "test" app.js
                         forever stop test
    --sourceDir      The source directory for which SCRIPT is relative to
    --workingDir     The working directory in which SCRIPT will execute
    --minUptime      Minimum uptime (millis) for a script to not be considered "spinning"
    --spinSleepTime  Time to wait (millis) between launches of a spinning script.
    --colors         --no-colors will disable output coloring
    --plain          Disable command line colors
    -d, --debug      Forces forever to log debug output
    -v, --verbose    Turns on the verbose messages from Forever
    -s, --silent     Run the child script silencing stdout and stderr
    -w, --watch      Watch for file changes
    --watchDirectory Top-level directory to watch from
    --watchIgnore    To ignore pattern when watch is enabled (multiple option is allowed)
    -t, --killTree   Kills the entire child process tree on `stop`
    --killSignal     Support exit signal customization (default is SIGKILL),
                     used for restarting script gracefully e.g. --killSignal=SIGTERM
                     Any console output generated after calling `forever stop/stopall` will not appear in the logs
    -h, --help       You're staring at it

  [Long Running Process]
    The forever process will continue to run outputting log messages to the console.
    ex. forever -o out.log -e err.log my-script.js

  [Daemon]
    The forever process will run as a daemon which will make the target process start
    in the background. This is extremely useful for remote starting simple node.js scripts
    without using nohup. It is recommended to run start with -o -l, & -e.
    ex. forever start -l forever.log -o out.log -e err.log my-daemon.js
        forever stop my-daemon.js

To install forever, using npm:

npm install forever -g

Daemonizing an application using forever:

forever start app.js

And we can just stop it with (we can refer to the app by id (for example, 21390), uid (for example, moMB), pid (for example, 21389), index (for example, 0), or script name (for example, app.js)):

forever stop app.js

We can also inspect the running process with:

forever list

Using forever command line isn't bad, but it can become a bit much when you start adding arguments for log files, different Node executables, working directories, and more. So instead of specifying everything on the command line, we can use a JSON configuration file like the following:

{
    // Comments are allowed!
    "uid": "myapp",
    "append": true,
    "watch": true,
    "script": "index.js",
    "sourceDir": "/home/scott/myapp",
    "command": /Users/scott/.nvm/versions/node/v4.1.2/bin/node
}

Assuming this file is in your current working directory and named development.json, use it like the following:

$ forever start ./development.json

Absolute paths to such configuration files are also supported:

$ forever start /home/myuser/app/forever/development.json

As we can see, you can even tell forever which version of Node to use, which is really convenient when you have multiple apps on a single server that require different versions:

[
  {
    // App using Node v0.11.8
    "uid": "myapp1",
    "append": true,
    "watch": true,
    "script": "index.js",
    "sourceDir": "/Users/example/myapp1",
    "command": "/Users/example/.nvm/versions/node/v0.11.8/bin/node"
  },

  {
    // App using io.js
    "uid": "myapp2",
    "append": true,
    "watch": true,
    "script": "index.js",
    "sourceDir": "/Users/example/myapp2",
    "command": "/Users/example/.nvm/versions/io.js/v2.2.1/bin/node",
    "args": ["--port", "8080"]
  }
]