ns_http - HTTP client functionality
The command ns_http provides HTTP and HTTPS client functionality. It can be used to create, dispatch, wait on and/or cancel requests and process replies from web servers, where NaviServer acts as a web client. This is important, for example, to use web services and REST interfaces.
Cancels queued HTTP/HTTPS request indntified by the id (of the request). The command returns an empty result. The purpose of this command is to terminate runaway requests or requests that have been timed out. Even completed requests can be cancelled if nobody is interested in the request result.
ID of the HTTP request to cancel.
Cancels all pending HTTP/HTTPS requests issued in the current interpreter. At this point, any Tcl channels that have been optionally assigned to a task, will be automatically closed.
Returns the number and slot (socket) usages for active or recent ns_http requests. The following example shows, how the socket opened to google.com is kept still for about 2 seconds. The slots for the sockets are allocated dynamically depending on the system load
% ns_http run -keepalive 2s https://google.com/
% ns_http keepalives
{slot 0 state waiting expire 1.999983 peer google.com:443 sock 82} {slot 1 state free}
The returned information is useful for monitoring and debugging busy HTTP client operations.
Lists running ns_http requests. When id was specified, returns a single list in form of: id url status. If id was not specified, returns a list of lists is return, where the elements have the described format. The value of the status can be one of done, running, or error.
Optional ID of the HTTP request to list.
Opens a connection to the web server denoted in the url and returns (unless a -done_callback is specified) an id, which might be used later in ns_http wait or ns_http cancel to refer to this request. The command supports both HTTP and HTTPS URIs. The request is run in the default task queue in a dedicated per-queue thread.
The description of the available options is in the section OPTIONS below.
Sends an HTTP or HTTPS request and waits for the result. The command ns_http run is similar to ns_http queue followed by ns_http wait. The HTTP request is run in the same thread as the caller.
The description of the available options is in the section OPTIONS below. The result value is described in RETURN VALUE.
Returns statistics from the currently running request in the form of a list of Tcl dictionaries. If the optional id was specified, just one dictionary containing details about the requested task will be returned, or empty if the task cannot be found. Otherwise, a list of dictionaries will be returned. The returned dictionary contains the following keys: task, url, requestlength, replylength, sent, received, sendbodysize, replybodysize, replysize. The task returns the ID of the HTTP task. The url returns the URL for the given task. The requestlength returns the length of the complete HTTP request, including header line, all the headers plus the optional request body. The replylength returns the value of the content-length as returned by the remote. This can be zero if the length of returned data is not known in advance. The member sent returns the number of bytes sent to the remote. This includes the header line, all the headers plus optional request body. The member received contains the number of bytes received from the remote. This includes the status line, all the headers plus the optional reply body. The member sendbodysize returns the number of bytes of the request body sent to the remote so far. The member replybodysize returns the number of bytes of the reply body received from the remote so far. The member replysize returns the number of bytes of the body received from the remote so far. The difference to the replybodysize is that this element tracks the number of body bytes prior to the optional deflate step for compressed contents, whereas the member replybodysize tracks the number of body bytes of the deflated contents. For uncompressed reply content, both replysize and replybodysize will have the same value.
Optional ID of the HTTP request to get statistics for.
Returns a list of Tcl dictionaries containing information about the state of the ns_http task threads. Every dict in this list contains the keys name, running, and requests, where the name is the name of the task thread, running is the number of currently running ns_http tasks, and requests is the total number of requests processed so far by this task thread.
In the example below, there are two task queues defined.
% ns_http taskthreads
{name tclhttp.0 running 42 requests 21925} {name tclhttp.1 running 37 requests 25739}
The total number of task threads can be tailored via the configuration parameter nshttptaskthreads as shown below.
Waits for the queued command specified by the id returned from ns_http queue to complete. The specified -timeout specified the maximum duration of the request to complete. The time can be specified in any supported ns_time format.
On success, ns_http wait returns the same dictionary as the ns_http run. On error, leaves a descriptive error message in the interpreter result. On timeout, sets the Tcl variable to NS_TIMEOUT in addition to leaving the error message. The result value is described in RETURN VALUE.
ID of the HTTP request to wait for.
The commands ns_http run and ns_http wait return a dictionary containing the following elements:
The first three members are always returned, the other elements are conditional. The status contains HTTP/HTTPS status code (200, 201, 400, etc). The time contains elapsed request time. The time value is in the ns_time format. The headers contains the name of the set with response headers.
When none of the output options are used, the result is received either in memory (member body) or in a spool file (member file). This decision depends on the value of -spoolsize. Both members are missing in the result dictionary, when outputchan was used.
For requests with a bodychan or outputchan, these values are added as well to result dictionary. For HTTPS requests, the result contains the member https with some low-level TLS parameters in a Tcl dictionary format.
First, a minimal example to retrieve a page with the HTTP GET method:
% http run http://www.google.com
status 200 time 0:174146 headers d0 body { ... }
Here is the same example, using separate ns_http queue and ns_http wait commands.
% ns_http queue http://www.google.com
http0
% ns_http wait http0
status 200 time 0:177653 headers d0 body { ... }
The second example is a code snippet making a request via HTTPS (note that HTTPS is supported only when NaviServer was compiled with OpenSSL support).
% set result [ns_http run https://www.google.com] % dict get $result status 302
If the returned data is too large to be retained in memory, you can use the -spoolsize to control when the content should be spooled to file. The spooled filename is contained in the resulting dict under the key file.
% set result [ns_http run -spoolsize 1kB https://www.google.com] % dict get $result file /tmp/http.83Rfc5
For connecting to a server with virtual hosting that provides multiple certificates via SNI (Server Name Indication) the option -hostname is required.
The third example is a code snippet making a POST requests via HTTPS and provides url-encoded POST data. The example sets a larger timeout on the request, provides requests headers and returns reply-headers.
##################################################
# Construct POST data using
# query variable "q" with value "NaviServer"
##################################################
set post_data [join [lmap {key value} {
q NaviServer
} {
set _ "[ns_urlencode $key]=[ns_urlencode $value]"
}] &]
##################################################
# Submit POST request with provided "content-type"
# to the "httpbin.org" site using HTTPS
##################################################
set requestHeaders [ns_set create headers "content-type" "application/x-www-form-urlencoded"]
set r [ns_http run -method POST \
-headers $requestHeaders \
-timeout 10.0 \
-body $post_data \
https://httpbin.org/anything]
##################################################
# Output results from the result dict "r"
##################################################
ns_log notice "status [dict get $r status]"
ns_log notice "reply [dict get $r ]"
ns_log notice "headers [dict get $r headers]"
The fourth example is a code snippet that sets a larger timeout on the request, provides an ns_set for the reply headers, and spools results to a file if the result is larger than 1000 bytes.
set requestHeaders [ns_set create headers Host localhost]
set h [ns_http queue -timeout 10.0 http://www.google.com]
ns_http wait -result R -headers $requestHeaders -status S -spoolsize 1kB -file F $h
if {[info exists F]} {
ns_log notice "Spooled [file size $F] bytes to $F"
file delete -- $F
} else {
ns_log notice "Got [string length $R] bytes"
}
The next example is for downloading a file from the web into a named file or passed Tcl channel. Note the -spoolsize of zero, which will redirect all received data into the file/channel. Without the -spoolsize set, all the data would be otherwise stored in memory.
% ns_http run -outputfile /tmp/reply.html -spoolsize 0 http://www.google.com status 302 time 0:132577 headers d2 file /tmp/reply.html
% set chan [open /tmp/file.dat w] % ns_http run -outputchan $chan -spoolsize 0 http://www.google.com status 302 time 0:132577 headers d2 outputchan file22 % close $chan
The behavior of ns_http can be influenced by optional settings in the NaviServer configuration file. The behavior can be tailored partially globally (for all servers) and per server definition. Globally, the number of threads ns_http threads can be configured. These threads are used, when the request is started with ns_http queue or -done_callback are used.
#---------------------------------------------------------------------
# Global NaviServer parameters
#---------------------------------------------------------------------
ns_section ns/parameters {
# ...
# Configure the number of task threads for HTTP client requests
# via ns_http. Per task thread, a separate queue is defined. For
# common (Internet) usage, the default value of 1 is fully
# sufficient. For high-speed file uploads/downloads (10/100G
# networks, fast I/O) the performance might be increased by
# defining multiple task threads.
#
#ns_param nshttptaskthreads 2 ;# default: 1
# ...
}
On the per-server configuration level, one can specify the default keep-alive timeout for outgoing HTTP requests, and the logging behavior. When logging is activated, the log file will contain information similar to the access.log of NaviServer (see nslog module), but for HTTP client requests.
#---------------------------------------------------------------------
# HTTP client (ns_http) configuration
#---------------------------------------------------------------------
ns_section ns/server/$server/httpclient {
#
# Set default keep-alive timeout for outgoing ns_http requests
#
ns_param keepalive 5s ;# default: 0s
#
# Configure log file for outgoing ns_http requests
#
ns_param logging on ;# default: off
ns_param logfile ${logroot}/httpclient.log
ns_param logrollfmt %Y-%m-%d ;# format appended to log filename
#ns_param logmaxbackup 100 ;# 10, max number of backup log files
#ns_param logroll true ;# true, should server log files automatically
#ns_param logrollonsignal true ;# false, perform roll on a sighup
#ns_param logrollhour 0 ;# 0, specify at which hour to roll
}