LRWP Documentation                                         ZeroToaster::HTTPD
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
This is the orginal LRWP documentation from iMatix written by Ewen McNeil.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
XWP00001              LRWP Protocol Update -- White Paper
                        Xitami Team <xitami@imatix.com>

Revision:      $Revision: 0.4 $
Written:       1998/12/16
Last Revised:  $Date: 1999/01/25 01:18:20 $

Updates:
1998/12/16:  First publically available version
1999/01/25:  Added wildcarded application names

Copyright iMatix Corporation, 1998.  This document may be freely
distributed in unmodified form.

Disclaimer

The information contained in this document is distributed on an "as-is"
basis without any warrenty either expressed or implied.  The reader is
responsible for the use of this information, and/or implementation of
any techniques discussed.  


Introduction

The LRWP (Long Running Web Process) Protocol is used to communicate
between a web server (such as xitami), and a peer process which provides
additional functionality to the web server.  LRWP was developed by Robin
P Dunn of Total Control Software (<robind@earthling.net>), including an
initial implementation of the web server component for the Xitami web
server developed by iMatix Corporation (http://www.imatix.com/).

LRWP allows programmable functionality in the manner of cgi-bin programs,
but because the LRWP peer program is persistent (that is, keeps running
across multiple requests), there is the potiental for faster responses.  
In some situations it can have large performance gains over cgi-bin 
programs, especially where the setup time (eg for accessing a database) is a 
large portion of the time required to answer a query.


The Current Protocol

A web server that supports LRWP listens on the LRWP port (currently TCP/IP
port 81) for connections from LRWP peer processes.  A LRWP peer process
registers to handle queries by establishing a TCP/IP connection to the 
LRWP web server.  Once the connection is established, it sends through
an identification string giving the name of the application (queries) it
will handle, and optionally the hostname for which the queries should be
handled (for servers that support virtual hostnames).

The peer process then waits for a response from the LRWP web server.  This
response is either "OK" to indicate that the LRWP peer process is
registered, or an error message indicating what the problem was with the
registration.

Once the LRWP peer process is registered, it then waits for requests to
process to be sent to it by the LRWP web server.  Requests are passed to
the LRWP peer process by passing the environment variables (as would be
set for a cgi-bin program), and the posted data to the peer.  The format
used is a 9 byte length indicator (as an ASCII decimal number), followed 
by the data, for each of the environment and posted data. 

After passing the request to the LRWP peer process, the LRWP web server
waits for a response from the LRWP peer process.  This response consists
of a 9 byte length indicator (as an ASCII decimal number), followed by 
the HTML page to be returned to the web browser.  Optionally this HTML
page may be preceded by HTTP headers to pass to the web browser.

Having sent this response to the LRWP web server, the LRWP web process
returns to waiting for a new request.

The LRWP connection is terminated by either the LRWP web server or the
LRWP peer process closing the TCP/IP network connection.

The LRWP protocol supports multiple simultaneous connections under the
same application name, and the LRWP web server may handle this in any
suitable way.  The Xitami server currently distributes requests to the
LRWP peer processes of the same name on an as-available basis (that is,
if one request is being processed by the first peer process and another
one arrives it will go to the second LRWP peer process).  If there are more
requests in a period of time than peer processes to handle them, then
the remaining requests are queued waiting for a free LRWP web process.

The following diagram may illuminate the protocol:

        LRWP peer process                          LRWP web server

                         --- TCP/IP connection -->

                         --- application name --->
                         --- 0xFF --------------->
                         --- virtual hostname --->
                         --- 0xFF --------------->

                         <-- "OK" ---------------
                               or
                         <-- error message ------

                         ...... time passes .....

                         <-- 9-byte env length --
                         <-- env-length bytes ---
                             of environment
                         <-- 9-byte post length -
                         <-- post-length bytes --
                             of environment

                         ... request processed ..

                         --- 9-byte html len --->
                         --- html-len bytes of ->
                             html response

                         ...... time passes .....

                         <-- 9-byte env length --
                         <-- env-length bytes ---
                             of environment
                         <-- 9-byte post length -
                         <-- post-length bytes --
                             of environment

                         ... request processed ..

                         --- 9-byte html len --->
                         --- html-len bytes of ->
                             html response

                         ........ etc ..........

                         ----- connection ----->
                                 closed

                                 or

                         <---- connection ------
                                 closed


Motivation for Update

There are several perceived limitations to the existing LRWP protocol:

  - The security model is very weak.  Connections can be accepted
    or rejected solely based on source IP address, and possibly
    application name.

    Typically connections are limited to being from the local host
    only, but on a multiuser machine this still potentially allows 
    unauthorised LRWP peer processes to connect and register themselves.

    Generally the LRWP peer process can specify any application name,
    and the LRWP web server will pass all requests for that application
    name to the LRWP peer process.  This potentially allows the
    interception of any request to the LRWP web server which may be
    undesirable.

    Even where the LRWP web server allows only certain application names 
    to be registered, it has no way of ensuring that the LRWP peer process
    claiming to be that application is actually that application.

  - The initial connection exchange between the LRWP peer process
    and the LRWP web server can result in either the string "OK" 
    or an error message (anything other than "OK"), and neither
    string is terminated in any way, making it difficult for the
    LRWP peer process to know when the response is finished.  
   
    Currently most LRWP peer processes either employ a timeout (often
    implicitly by assuming the entire response arrives "at once"), or
    assume any string starting with "OK" indicates a positive
    response, and that any other string will end with the connection
    closing.

  - The protocol has no version information which makes it more 
    difficult to identify different versions of the protocol, and
    hence add enhancements.

  - There is no explicit way to specify that a LRWP peer process will
    handle all URLs that have a given prefix, and no way to specify
    that a LRWP peer process will handle all URLs that have a given
    extension (eg, ".asp" for an ASP emulator).

Proposed Update

We propose to change the initial LRWP peer process to LRWP web server
negotiation to add:

  - a protocol version number

  - a clearly deliminated indication of whether the connection is
    accepted or not

  - an optional challenge/response system to more accurately confirm the 
    identity of the LRWP peer process, and enable the LRWP web server to 
    accept or reject connections based on the identity of the LRWP peer 
    process.

  - expand the possible application names to allow an optional wildcard
    character at the start or end of the application name.

In order for existing LRWP peer processes and LRWP web servers to
continue to inter-operate, it is necessary to maintain backwards
compatibility with the existing LRWP protocol, described above.  The
existing LRWP protocol will be called LRWP 1.0 from here on, and the 
protocol proposed here will be called LRWP 2.0.


Protocol Version Number

As described above, LRWP 1.0 begins with the LRWP peer process sending 
to LRWP web server the string:

applicationname0xFFvirtualhostname0xFF

at which point the LRWP peer process waits for a response.

It is proposed to change this for LRWP 2.0 and higher so that the
initial string is of the form:

0xFFprotocolversion0xFFapplicationname0xFFvirtualhostname0xFF

where the initial 0xFF would be an empty application name in LRWP 1.0
and hence a protocol violation, and thus can be used by LRWP 2.0 and
higher to indicate an non-LRWP 1.0 session.

The protocol version would be a string consisting of a major number and
a minor number, as ASCII digits, initially "2.0".  The major version
number should be changed if the updated protocol is backwards incompatible,
and the minor version number should be incremented for any other changes
in the protocol.


Delimited Indication of Success or Failure

In LRWP 1.0 the response from the LRWP web server to the LRWP peer
process is one of "OK" or an error message, without any terminating
character.

For LRWP 2.0 it we propose to change this so that the response is
terminated by a 0xFF character, and is either the string "OK" (for an
accepted connection) or the string "REJECTED" (in other cases).  Where
the string "REJECTED" is sent, following the 0xFF character a
descriptive error message should be sent followed by another 0xFF
message.

Where the challenge/response system is being used (see below), one other
valid response at this point is "CHALLENGE" followed by an 0xFF and the
challenge information.  If sent, this indicates that the LRWP web server
accepts the connection subject to the challenge, and a final indication
of whether the connection is accepted or not will be sent after the
challenge phase.


Challenge/Response System

Because LRWP is typically used only allowing connections from the local
system, the biggest threat to security is from other users with access
to the system on which the LRWP web server is running.  For this reason
network snooping is not considered a major security threat, whereas
process command line snooping is a potentially significant security
threat.

A major consideration in developing this challenge/response system is ease 
of implementation in as many languages as possible, as LRWP peer processes 
can be (and have been) implemented in a wide range of languages.   In 
addition the desire to avoid export restrictions limits the use of 
cryptography.

The challenge/response system proposed is one that uses a shared secret
that both the LRWP web server and the LRWP peer process know.  It is
anticipated that these shared secrets will either be stored in
configuration files (for the LRWP web server and LRWP peer process), and
be per LRWP peer program values, or will be dynamically generated ("one
time" values) and passed from the LRWP web server to the LRWP peer process 
outside the LRWP connection channel (see below).

Following the registration request from the LRWP peer process, the LRWP
web server may request an authentication phase by sending a challenge to
the LRWP peer process.

This is done by sending the work "CHALLENGE" followed by a 0xFF
character to the LRWP peer process after it has requested registration.
Following this should be a length value encoded as per the LRWP 1.0
protocol (that is, 9 ASCII printable digits giving the length), and a
challenge string.  The challenge string should be at least 8 bytes and
no more than 64 bytes, and should be generated ("randomly") for the
purpose of that challenge.

After sending the challenge to the LRWP peer process, the LRWP web
server waits for a response to the challenge.  The LRWP peer process
assembles this response by XORing the bytes in the challenge string with
the bytes in the shared secret, so that the first byte of each is XORed
together to get the first byte of the response, the second byte of each
is XORed together to get the second byte of the response and so on.  If
the challenge string is longer than the shared secret, then the (N+1)th
byte of the challenge string should be XORed with the first byte of the
shared secret, and so on (where N is the length of the shared secret).

The LRWP peer process then sends this response back to the LRWP web
process by sending the length of it in LRWP 1.0 format (ie, 9 ASCII
printable digits giving the length), and the response string.

The LRWP web server then performs the same transformation internally,
and checks that the response received from the LRWP peer process matches
what it generated internally.  If it does, then the LRWP web server accepts
the connection by sending "OK" followed by 0xFF (as described above in
section "Delimited Indication of Success or Failure").  If it doesn't
the LRWP web server rejects the connection by sending the string
"REJECTED" followed by 0xFF followed by a suitable error message
followed by 0xFF, as described above.

The shared secret can be passed to the LRWP web server and LRWP peer
process in any suitable manner.  For the LRWP web server we
recommend that it either be dynamically generated (see below), or
stored in a configuration file associated with the LRWP application
name.

For the LRWP peer process we recommend that the shared secret
either be read from a configuration file, or read from the standard
input.  We recommend that the shared secret NOT be passed to the
LRWP peer process on the command line, because this is vulnerable to
command line snooping on a multiuser machine.

Dynamically generated shared secrets can be used where:
  - the LRWP peer process supports reading the shared secret from
    standard input (we recommend that all LRWP peer processes
    support this option, optionally indicated by a command line
    flag)

  - the LRWP peer process is spawned by the LRWP web server
    (either when the LRWP web server starts, or on demand), and 
    hence the LRWP web server can control the standard input of 
    the LRWP peer process.

In this instance the LRWP web server should generate a suitable string
to use as a shared secret (for instance a string of 8 to 64 "random"
characters), and then start the LRWP peer process with either a file or
a pipe as the standard input.  It should either place the shared secret
into the file, or write the shared secret out to the pipe, to allow the
LRWP peer process to read that value.  We recommend where possible
that the pipe method be used, to avoid the need for temporary files,
and reduce the potiental for snooping.

We recommend that these dynamical generated shared secrets be one-time 
values, and that the LRWP web server associate them with the LRWP application 
name that the LRWP peer process being started is expected to use.  This
reduces the chances of other LRWP peer processes guessing the values.


Diagram of New Connection Process

        LRWP peer process                          LRWP web server
                         --- TCP/IP connection -->
      
                         --- 0xFF --------------->
                         --- "2.0" -------------->
                         --- 0xFF --------------->
                         --- application name --->
                         --- 0xFF --------------->
                         --- virtual hostname --->
                         --- 0xFF --------------->

                         ... optionally .........
                         <-- "CHALLENGE" --------
                         <-- 0xFF ---------------
                         <-- 9-byte chal len ----
                         <-- chal-len bytes of --
                             challenge string

                         --- 9-byte resp len --->
                         --- resp-len bytes of ->
                             response string
                         ... optionally .........

                         <-- "OK" ---------------
                         <-- 0xFF ---------------
                               or
                         <-- "REJECTED" ---------
                         <-- 0xFF ---------------
                         <-- error message ------                         
                         <-- 0xFF ---------------

                         .. requests processed ..
                         ...... as LRWP 1.0 .....


Wildcarded Application Names

To support handling all URLs with a given prefix, or suffix, application
names may optionally contain a wildcard character at EITHER the start OR 
the end.  The wildcard character is "*", which is interpreted in the 
same manner as a filename wildcard, that is it matches zero or more 
characters at either the start or the end.

This means that an application name of:
    
     *.ssi

would match all requests to obtain a URL that ends in ".ssi", and an
application name of:

     servlet/*

would match all requests to obtain a URL that starts with "servlet/",
and thus allows delegating a whole psuedo directory hierachy to the LRWP
peer process.

It is not necessary that a wildcard at the start of the application name
be followed by a ".", but it is recommended that wildcard prefixes be
used only for matching file extensions of the form ".ext" (where "ext" is
the extension).

Since LRWP 1.0 application names do not include wildcard characters at
present, this change should be backwards compatible, and LRWP 1.0 web
servers could support wildcarded pplication names.  However,
LRWP peer processes should not rely on a LRWP 1.0 server correctly 
understanding their wildcarded application name, and should always use
LRWP 2.0 when requesting a wildcarded application name.


Conclusion

The aim of this proposed update is to both improve the LRWP security 
model, and also provide a framework for adding extra extensions into 
the protocol.  The addition of a version number, and the addition of a 
clear delimitation of the response to the registration request from the 
LRWP peer process make it easier to add extra phases to the connection 
setup at a later point, while remaining backwards compatible with previous
versions.

The proposed challenge system improves the security model of LRWP,
particularly when dynamically generated shared secrets are used.
-=- cut here -=-

-- 
Ewen McNeill, Technical Consultant, iMatix Corporation  http://www.imatix.com/
