admin-tuning - NaviServer Tuning and Scaling Guide
NaviServer is immensely scalable because it has to be. It is based on the web server for the most heavily-used production environment in the world -- AOL. The sections below refer to various configuration parameters. For a complete reference, see the sample configuration files sample-config.tcl.
Modules like nsperm register extra requests that are checked on each connection, for example. Other modules like nscgi and nscp do not need to be loaded if you're not actually using CGI or the control port, respectively.
The nslog module has these turned off by default, but it helps to see if it your configuration has it turned off. Also, the nsperm module uses DNS lookups for access control -- turn this off if you don't use host access control rules with nsperm. The nscgi module also can use DNS but, again, it is off by default (gethostbyaddr setting). If you must use DNS, NaviServer has a DNS cache that can be tweaked -- see dnscache, dnswaittimeout and dnscachetimeout in the config reference.
In the ns/server/$server/adp section, the parameters are cache, cachesize, and threadcache. Use the appropriate $server for you installation (maybe the name default). NaviServer defaults to a 10 MB ADP cache. This cache is used to store parsed ADP pages -- as a consequence such ADP scripts are only parsed once and every subsequent time they are run directly out of memory. This means, though, that you should have a fair amount of core memory on your system to accommodate the cache if you decide to make it very large.
This is a separate cache used to store static HTML pages. The section is ns/server/$server/fastpath, and the options are cache, cachemaxentry, and cachemaxsize. The default is 10 MB. On some systems enabling the mmap parameter can make it work even faster.
Is your site updated rarely or often? On some slower systems, setting checkmodifiedsince to false in the ns/server/$server section can speed up things considerably as NaviServer reads all scripts and data directly from the in-memory cache without checking to see if the file has changed each time a cache entry is hit.
Thread settings are sometimes helpful, sometimes not. If one is running a small configuration (e.g. 100K requests per day), the default settings provided by the sample configuration files are typically sufficient. For larger or very constraint setups the paramamter for the connection theads can and should be tuned. The following parameters can be altered in the section "ns/server/$server" (for a server named "$server"): connsperthread, highwatermark, lowwatermark, maxconnections, maxthreads, minthreads, rejectoverrun, retryafter, poolratelimit, connectionratelimit and threadtimeout. See also connection thread pools.
As mentioned, in most cases the defaults are fine, with the exception of minthreads and maxthreads which should be carefully adjusted based on your load. When minthreads and maxthreads are sent to different values, NaviServer tries to adjust the number of running threads according to the current load. When running applications with a large Tcl blueprint (e.g. OpenACS), starting of new threads can be costly, such that a frequent starting and stopping of threads might not be the best either. Therefore, it is sometimes better to set minthreads equals tomaxthreads.
The parameter maxconnections defines the queue length of a connection pool. This means, requests are received in a situation where no connection thread is available, these requests are added to this queue. Since this queue has a limited size, it might as well overrun. In this situation, one can fine tune the behavior. When rejectoverrun is set, NaviServer will send a 503 (service unavailable) reply to the client. When additionally the parameter retryafter is set, NaviServer provides it value as a "Retry-After" hint to the client. When rejectoverrun is false the NaviServer keeps an additional waiting list, causing eagerly to retry such requests automatically. The latter can lead to a memory bloat on flooding attacks. However, for internal servers (or connection pools), such behavior might be still favorable.
On busy machines, one can define multiple connection thread pools and use the configuration option map to map HTTP method, URL and context filter patterns to certain pools (for details about the mapping specs, the documentation of connection thread pools and the command ns_server. By default, the settings are for the default connection threads pool. Additional connection thread pools can be created via pools.
ns_section ns/server/$server/pools {
#
# To activate connection thread pools, uncomment one of the
# following lines and/or add other pools.
ns_param monitor "Monitoring actions to check healthiness of the system"
ns_param fast "Fast requests, e.g. less than 10ms"
}
ns_section ns/server/$server/pool/monitor {
ns_param minthreads 2
ns_param maxthreads 2
ns_param map "GET /admin/nsstats"
ns_param map "GET /SYSTEM"
ns_param map "GET /ds"
ns_param map "POST /ds"
ns_param map "GET /request-monitor"
}
ns_section ns/server/$server/pool/fast {
ns_param minthreads 2
ns_param maxthreads 2
ns_param map "GET /*.png"
ns_param map "GET /*.PNG"
ns_param map "GET /*.jpg"
ns_param map "GET /*.pdf"
ns_param map "GET /*.gif"
ns_param map "GET /*.mp4"
ns_param map "GET /*.ts"
ns_param map "GET /*.m3u8"
}
ns_section ns/server/$server/pool/bots {
ns_param map "GET /* {user-agent *bot*}"
ns_param map "GET /* {user-agent *rawl*}"
ns_param map "GET /* {user-agent *pider*}"
ns_param map "GET /* {user-agent *baidu*}"
ns_param map "GET /* {user-agent *Knowledge*}"
ns_param minthreads 2
ns_param maxthreads 2
ns_param poolratelimit 1000 ;# 0; limit rate for pool to this amount (KB/s); 0 means unlimited
ns_param rejectoverrun true
}
Different pools can be configured with different parameters. It is for example possible to specify the maximum outgoing traffic rate per for every connection of the pool, or for the total of currently running connections of a pool.
Memory considerations are paramount on heavily-used servers. Use the "ps -leaf" on most systems to look at the "nsd" processes. Nearly all the memory used by nsd should be in RSS (resident set size). If the RSS size of the nsd process is less than 2/3 its SZ, then there is a good chance that your operating system is thrashing, meaning it is spending more time managing memory than allowing the system to work to its capacity. Some operating systems actually limit the total RSS used by any one process. That 1-gigabyte machine might only allow NaviServer to use 500 megabytes of core at any one time -- this is especially the case with SGI servers.
Databases are a bottleneck. Do you use lots of simple queries that return a hoard of data? Investigate using stored procedures or finely-tuned queries so that you get only the data you actually want and make the database do the work it was designed to do. Don't make your system spend its time putting together those ns_getrow structures.
NaviServer has a built-in statistics-gathering system that collects data on the caches, Tcl interps, threads, and other interesting data. We use these at AOL to gather an immense amount of data on how the systems are working and where they need to be improved (more cache, less cache, more memory, more threads, etc). The module nsstats can be dropped into any running server to get a snapshot of how it is doing. The module is available in the modules repository.
A few further parameters might be used for tuning the performance of NaviServer in a certain application specific context.
Administrators might consider allowing concurrent Tcl interpreter create operations. Versions of Tcl up to at least 8.5 are known to crash in case two threads create interpreters at the same time. These crashes were hard to reproduce, but serializing interpreter creation helped. For Tcl versions starting with Tcl 8.6, the default is set to true.
ns_section ns/parameters {
ns_param concurrentinterpcreate true ;# default false
}
Sample documented configuration files: