admin-maintenance(n) 4.99.30 manual "NaviServer Manual"

Using Watchdog

NaviServer has implemented a quite useful watchdog feature that allows the server to restart if it fails. In many situations you are now freed from setting up a special way of handling a server failure that would be solved with a simple restart.

Just start the server with the new -w switch.

Technically speaking the nsd process is forked twice and the first forked instance (the watchdog) controls the second (the worker). The first instance reacts on exit codes and signals caught during the watch and correspondingly restarts the second instance.

Restarting the Server To restart the server from within your Tcl code:

 ns_shutdown -restart

To restart the server from the shell you can send the server the SIGINT.

Pro:

• Built-in feature. No need to setup something else.

• Ability to restart from within your application.

Con:

• First step of implementation. You may need more options for the watchdog as it offers you now.

Systemd

Most Unix-like systems tends to use systemd (see e.g. https://www.freedesktop.org/wiki/Software/systemd). System admins might find the following sample configuration file for NaviServer helpful, which might be saved under /etc/systemd/system/nsd.service or /usr/lib/systemd/system/nsd.service.

 [Unit]
 Description=NaviServer
 After=network.target
 # After=network.target postgresql.service
 # Wants=postgresql.service
 
 [Service]
 Type=forking
 PIDFile=/usr/local/ns/logs/nsd.pid
 Environment="LANG=en_US.UTF-8"
 # In case, a site is using Google Perfortools malloc with the system-malloc patch for Tcl:
 # Environment="LD_PRELOAD=/usr/lib/libtcmalloc.so"
 ExecStartPre=/bin/rm -f /usr/local/ns/logs/nsd.pid
 
 # Standard startup with a non-privileged port (like 8000)
 ExecStart=/usr/local/ns/bin/nsd -u nsadmin -g nsadmin -t /usr/local/ns/conf/nsd-config.tcl
 
 # Startup for privileged port, like 80; the IPv6 address is bound to 2 sockets.
 # ExecStart=/usr/local/ns/bin/nsd -u nsadmin -g nsadmin -t /usr/local/ns/conf/nsd-config.tcl -b YOUR-IPv4-ADDRESS:80,YOUR-IPv6-ADDRESS:80#2
 
 Restart=on-abnormal
 KillMode=process
 
 [Install]
 #WantedBy=multi-user.target

Other approaches

There are several ways to keep your server running. Depending on your particular requirements the watchdog may not be the right thing for you. Common approaches are listed below. Using init with /etc/inittab

Very easy to setup but with its own limitations. init is the parent of all processes. It creates processes from the script /etc/inittab. Add a line like

 ns1:345:respawn:/usr/local/ns/bin/nsd -i -u nsadmin -g www -t /usr/local/ns/config.tcl

and init does the job of restarting the server if it crashes. Pro:

• Very easy to configure.

• Some kind of built-in feature of your OS.

Con:

• You need to be root to edit /etc/inittab and make changes.

• If there's an error during startup init tries to restart the server and then waits for some time. Repeats endlessly.

• There's no simple way to just stop the server if you want to as init tries to keep it up.

Using cron

As you already have (or may have) a rc-script for starting your server during boot time you simply could run a cronjob that checks the status of something like

 rcnaviserver status || rcnaviserver start

as part of a script every 5 minutes by adding a crontab line (crontab -e) like

 */5 * * * *		/root/cronjobs/nsd_crontab

Pro:

• If you wrap the status check in your own script you are more flexible in choosing the right action for the job, e.g. E-Mail notifications.

• Works fine if you don't have a requirement of virtually no downtime.

Con:

• Slightly more work if you have the situation to just stop the server for maintenance etc. (E.g. just restart if a certain status file does not exist)

• You have to touch at least three files: the rc-script, crontab list and restart script.

Using daemontools

Daemontools is a collection of software to control other processes.

Pro:

• Allows more users to control the process, e.g. users of a specific group.

Con:

• Extra package, you need to compile it.

• You need to be root.

Running As root

PLEASE NOTE THAT THIS IS NOT RECOMMENDED FOR WEBSERVER USAGE. The situations where you might want to run NaviServer as user root are rare to none. But NaviServer as a tool is so flexible that it cannot only be used as webserver. It is possible (and has been done) to use it as some kind of application where serving webpages is not the primary target.

So it is possible to tell NaviServer to run as root using the -u switch at startup from the command line.

Of course, you have to be root to do that:

 ./nsd -f -u root -g www -t ../sample-config.tcl

You can see the result in the security info:

 ...
 [-main-] Notice: nsmain: NaviServer/4.99.30 (naviserver-4.99.30) starting
 [-main-] Notice: nsmain: security info: uid=0, euid=0, gid=8, egid=8
 ...

It is simply a convenience for people that need it to not have to rewrite code to achieve this.

Running inside Chroot Environment

Chrooting NaviServer effectively means to run it with a special root directory. When using Tcl commands like file, glob or exec only files "below" the specified path are visible, limiting access to critical files and devices. This is an important means to limit the access attackers of a website might gain.

You don't have to use the command line chroot command (see man page), NaviServer comes with built-in chroot capability.

There are scripts available (http://sourceforge.net/projects/nsdinstallers) with the intention to simplify and speed up the task of creating a chroot environment.

Benefits

• Usually, strong barrier for attackers

• Access to the filesystem is limited

• Hopefully limits the damage of previously unknown, new security holes

Disadvantages

• The setup of a chroot cage is a cumbersome task, you have to find out every library used, absolute paths might be compiled into the code, strace will become your friend...

• Every other application used (e.g. ImageMagick) must be chrooted

• When using a database network support must be enabled or the socket placed into the chroot environment (?)

• Updates and patches for NaviServer or any other chrooted tool becomes more difficult

Alternatives Use Novell AppArmor (http://www.novell.com/products/apparmor/) and create a Profile for NaviServer

Command (interactive) mode

Usually, you start the server and it waits for signals. With testing in mind, a new mode was added for developers: The interactive 'Command mode'.

If you start NaviServer with

 ./nsd -c <other args>

it runs a Tcl shell and waits for commands. You are then able to do more comprehensive tests than going the other way with using a Tcl shell and loading libnsd.so.

Note:

This mode is not intended to be used for a production system. If you need insight on a running server the control port (module nscp) might be the right thing for you.

Just a quick example: The server is started and two commands are issued, ns_info threads and ns_server requestprocs.

 buckel@tulpe:/usr/local/ns/bin> ./nsd -c -u nsadmin -g www -t ../sample-config.tcl
 [-main-] Notice: nsmain: NaviServer/4.99.30 (naviserver-4.99.30) starting
 [-main:conf-] Notice: nsmain: security info: uid=500, euid=500, gid=100, egid=100
 [-main:conf-] Notice: nsmain: max files: soft limit 10240, hard limit 10240
 [-main:default-] Notice: nsd/init.tcl [default]: booting virtual server:  Tcl system encoding: "utf-8"
 [-main:default-] Notice: modload: loading '/usr/local/ns/bin/nssock.so'
 [-main:default-] Notice: modload: loading '/usr/local/ns/bin/nslog.so'
 [-main:default-] Notice: nslog: opened '/usr/local/ns/servers/server1/modules/nslog/access.log'
 [-main:default-] Notice: conf: [ns/server/server1]enabletclpages = 0
 [-main:default-] Notice: nsmain: NaviServer/4.99.30 (naviserver-4.99.30) running
 [-main:default-] Notice: nsmain: security info: uid=500, euid=500, gid=100, egid=100
 [-sched-] Notice: sched: starting
 [-driver:nssock:0-] Notice: starting
 [-driver:nssock:0-] Notice: nssock:0: listening on [0.0.0.0]:8080
 [-driver:nssock:0-] Notice: driver: accepting connections
 
 % ns_info threads
 {-driver- -main- 1084140464 0 1119862183 p:0x40037cfb a:0x0}
 {-sched- -main- 1082039216 0 1119862183 p:0x4004a09a a:0x0}
 {-main- {} 1076887680 1 1119862183 p:0x0 a:0x0}
 
 % ns_server requestprocs
 {GET / * inherit ns:fastget a:0x0} {HEAD / * inherit ns:fastget a:0x0} {POST / * inherit ns:fastget a:0x0}
 %

Examples

 nsd -t nsd.tcl

This is the simplest form of the command line.

 nsd -ft nsd.tcl

NaviServer will be run in the foreground.

 nsd -fkt nsd.tcl -r /newroot

NaviServer will be run in a chroot environment, and any currently-running server will be killed.

Back Up the Pages Directory

The location of the pages directory for each virtual server is determined by the server-specific PageDir entry in the NaviServer configuration file. Normally, it is the /pages subdirectory under the NaviServer home directory.

Use whatever filesystem backup procedure you have in place at your site. To schedule nightly backups, use the Unix cron facility.

Back Up the Access Log

The access log file needs to be backed up regularly. By default, the access log for each virtual server is in the /logs/access.log file under the NaviServer home directory.

The access log can be configured to limit the number of old logs maintained (with the MaxBackup parameter). This parameter sets an upper limit on the amount of disk space the access logs take. However, because old logs beyond the limit configured to be saved by the NaviServer are deleted automatically, you must back up old logs if you require a complete history of access to your site. For example, if the MaxBackup parameter in the configuration file is set to 5, only five old access log files will remain on disk. When a sixth log file needs to be opened, the oldest log is removed.

Back Up the Server Log

Ordinarily, the server log file grows at a slow rate and does not need regular truncation. However, while debugging new applications, you should set the Verbose parameter in the ns/module/nsdb/pool/pool-name section in the configuration file to on instead of off (the default). Every SQL statement sent to the database is logged in the error log and causes the file to grow much more quickly. In this case you may want to back up the error log.

Back Up the Tcl Scripts Directory

The Tcl scripts directory contains the source to the Tcl scripts that provide the server with much of its advanced functionality. Tcl scripts for each virtual server are stored in the /tcl subdirectory by default, and global Tcl scripts are stored in the /modules/tcl subdirectory by default.

If you write new Tcl scripts or edit the existing ones, you must ensure your changes are saved regularly to a safe place. Also, be sure that any external files utilized by your Tcl scripts are backed up too, including files outside the NaviServer home directory.

Back Up the bin Directory

The /bin subdirectory of the NaviServer home is the location of the NaviServer binary and the default location of any dynamically loadable C modules. If your site maintains several interesting loadable modules, you must make sure copies of the modules are backed up to avoid having to recompile them after a filesystem failure. Also, be sure to back up your module source code.

/conf

Directory containing the NaviServer configuration files. This is convention and convenience, the configuration files can be located anywhere as nsd process requires path to configuration file.

Files:

 .tcl files nsd.tcl sample-config.tcl simple-config.tcl

/bin

Directory containing the NaviServer binary and the default directory for any dynamically-loadable C modules.

Files:

 .so files, nsd, tclsh, ...

/include

Directory containing header files for NaviServer. These files are primarily needed for compiling additional C modules.

Files:

 .h files

/lib

Directory containing static libraries used for building customized components to NaviServer. This directory currently only includes the libnspd.a file that can be used to build database proxy daemons (external database drivers).

Files:

 libnsd.so,
 libnsdb.so,
 libnsthread.so,
 libnsproxy.so,
 ...

/logs

Directory containing log files and the server pid file.

Files:

 access.log,
 server.log,
 nsd.pid

/modules

Contains directories for each configured module that operates across servers, such as the "tcl" or "nsperm" folders.

/modules/nsperm

Directory containing user, group, and permissions files which is used to provide access control for NaviServer.

Files:

 passwd, group, hosts.allow, hosts.deny, perms

/modules/tcl

Default directory for private Tcl script library for this server.

Files:

 .tcl

/pages

Default directory where pages and images for the server are stored. Users define typically various subdirectories of this directory.

Files (typically):

 .htm, .html, .shtml, and .adp

/tcl

Default directory for shared Tcl script library. Also contains subdirectories containing Tcl examples and Tcl scripts for various modules. The top-level files of this directory are loaded into NaviServer during startup.

Files:

 .tcl 

General nsadmin Passwords

By default, the nsadmin password for NaviServer is either set to NULL or to a poor password. Set an acceptable password for nsadmin as described below.

Edit the nsadmin entry in the "/modules/nsperm/passwd" file. For example, the default passwd file contains this nsadmin entry:

 nsadmin:CUdnvgBYocLSI:::::

Substitute an alternate encrypted password in place of CUdnvgBYocLSI.

To produce a hash for a new password, use ns_crypt in Command mode (using "nsd -c"):

 % ns_crypt MyNewPassword xx
 xxhR1Y2vt4OOY

Alternatively, the same hash can be produced from the command line using e.g. Perl:

 $ perl -le 'print crypt("MyNewPassword","xx");'
 xxhR1Y2vt4OOY

For more information about the passwd file, see the "Defining Users" section.

Permission Settings

It is more secure to avoid using the nsperm module and use file-level security for ADPs. If you must use the nsperm module, set appropriate permissions records as follows:

1. Maintain the same permission records for GET and POST; they actually provide the same permissions.

2. Remove any permission records related to network publishing (PUT, DELETE, MKDIR, and BROWSE) for all users except nsadmin.

3. Keep in mind the inheritance rules for permission records. In general, a permission record for a directory also applies to the directories underneath it.

To define NaviServer permissions, create permission entries for them in the perms file, which resides in the /modules/nsperm directory. The default perms file does not contain any permission entries, but it contains comments that explain how to add entries to the file.

For more information about setting permissions, see the "Permissions" section.

NaviServer Version

In general, NaviServer versions 4.99 and higher should be used whenever possible, because they are more secure than earlier versions of NaviServer.

• NaviServer can be run in a chroot environment.

• The configuration file, which has a new Tcl format, is executed in a separate, temporary interpreter that is destroyed before startup begins. The configuration file memory buffer is then zeroed after parsing.

• The nsd binary can be stored outside the root directory because NaviServer no longer locates and re-executes itself.

• The configuration file can be stored outside the root directory, because NaviServer opens and reads the configuration file before running chroot().

• The nscp module, which allows connections only from localhost, provides a secure control port interface that allows ad hoc Tcl evaluation and other server administration features. For more information about the control port interface, see the "NaviServer's Control Port Interface" section.

Secure chroot Environment

NaviServer should be run in a secure chroot() environment whenever possible.

In Versions 4.99 or higher, NaviServer supports a -r command line option to run NaviServer in a chroot() environment. It provides the following benefits:

• The chroot() system call updates the process such that all absolute filenames are relative to a new root directory instead of the actual mounted filesystem.

• The chroot() call is irrevocable. Once chroot() returns, the server cannot access any file above the new root directory.

• Although it does not actually protect any of the underlying content, scripts, or protected databases, chroot() is the single most effective tool for protecting the server machine and sensitive information, such as user passwords and configuration files, from view.

To run NaviServer in a chroot() environment, you need only copy a few files and directories to the new root directory. For example, on the SGI platform, you would execute the following commands to create new directories and copy the necessary files to them:

 mkdir $root/dev $root/tmp $root/etc
 chmod 1777 $root/tmp
 $root/dev; /dev/MAKEDEV generic usema
 cp /etc/passwd /etc/resolve.conf $root/etc

Then, you can run NaviServer with the -r option as in this example: nsd -t nsd.tcl -r $root

For more information about the nsd command line, see the "NaviServer Command Line" section.

Restricted Content

Determine whether any of the content available to a NaviServer in a chroot() environment would be restricted. In general, NaviServer should be read-only and everything it can read should be world-readable.

If any of the content available to NaviServer is restricted, the NaviServer administrator needs to define the appropriate permissions with the nsperm module. The administrator should be very clear which areas are blocked off and know both the URL and METHOD for the restricted areas.

It is preferable to allow the GET method for all URLs and have nothing restricted accessible through NaviServer.

Tcl Library

Limit the available Tcl functions to just those functions that are necessary by that particular NaviServer installation. Purge the Tcl library of unnecessary functions. For example, if the site doesn't send e-mail, remove the ns_sendmail procedures.

Some potentially unsafe commands you may want to consider removing are:

• File system related functions, such as open, read, and puts

• The NaviServer ns_sock* Tcl functions

• The Tcl socket routines

• The exec command

• The file command, or at least the delete and rename features

• The exit command

This code example disables the open command:

 static int
 AddCmds(Tcl_Interp, void *arg) {
     Tcl_CreateCommand(interp, "open", BadCmd, NULL, NULL);
     return TCL_OK;
 }
 
 static int
 BadCmd(ClientData dummy, Tcl_Interp *interp, int argc, char **argv) {
     Tcl_AppendResult(interp, "disabled command: ", argv[0], NULL);
     return TCL_ERROR;
 }

Stack Size

Apart from the specific details of integration of your favored language you may have to take a look at the stacksize parameter of your configuration file.

As a rule of thumb the default stack size (in bytes)

 #
 # Thread library (nsthread) parameters
 #
 ns_section ns/threads {
   ns_param   stacksize 64kB   ;# Per-thread stack size.
 }

is almost always not enough. Use e.g. 512kB for the start:

 #
 # Thread library (nsthread) parameters
 #
 ns_section ns/threads {
   ns_param   stacksize 512kB  ;# Per-thread stack size.
 }

If the size of the stack is not enough you may encounter crashes without real explanation.

Database Access

Database access should be restricted with read-only logins to the server and queries through stored procedures.

Control Port Interface

The control port interface (module nscp) should not be used unless absolutely necessary. A user connected to his interface can issue arbitrary commands. When the module is activated, it is recommended to accept only commands via the loopback interface.

For more information about the control port interface, see the documentation of the nscp ("NaviServer's Control Port") module.