Originally Published: Friday, 29 September 2000 Author:
Published to: develop_articles/Development Articles Page: 1/1 - [Printable]

Unix Web Application Architectures - Part 4: Other Issues

To ensure the parameter checking is done for all functions, it should be made as easy as possible. Even better, it should be necessary to write a prototype for a function before the function can be used. I personally like to have each function in a file of its own, and for each function file there exists a prototype file, that defines what parameters the function receives. The parameters are then checked automatically by the application framework.

   Page 1 of 1  

Unix Web Application Architectures
1. Introduction and Basic Approaches
2. The Web Server
3. Sessions, Authentication, and Databases
4. Other Issues
5. An Alternate Approach

11 Request Parameter Validation

I think of Web apps as a group of functions, each of which receives zero or more parameters. Each function implements a different page or operation in the application. Let's say we are using plain CGI, and calling function delete_user with parameter username=johndoe, with session carried in parameters. The HTTP request could then be http://www.foo.com/cgi-bin/delete_user?s=987654321&username=johndoe. I'll talk about a function call when there is a link to some function, a function is the target of a form's ACTION attribute or similar.

There is no standard mechanism for making sure that each function is called with correct set of parameters, whose format is as expected. To use an analogy to traditional programming, there is no standard way of giving a prototype for functions. I consider it important that this checking is done for all functions. Some people may find such a requirement a hindrance, but long term advantages are often acquired by doing more work in the short term.

For example, functions pointed to by an ACTION attribute of a form may receive a large number of parameters, namely the values of the form's INPUT elements. In this situation, it's only a matter of time when there occurs a mismatch between what the function expects and the +parameters with which it is called. Even when calling a function with few parameters, it is easy to make a typo or to fail to encode the parameters properly.

To ensure the parameter checking is done for all functions, it should be made as easy as possible. Even better, it should be necessary to write a prototype for a function before the function can be used. I personally like to have each function in a file of its own, and for each function file there exists a prototype file, that defines what parameters the function receives. The parameters are then checked automatically by the application framework.

Besides checking that those and only those parameters mentioned in the prototype exist in a call, it's also possible to give each parameter a type. Examples of the types would be integer, unsigned integer, float, safe-string (a string consisting only of printable 7 bit ASCII letters and numbers for example) and arbitrary-string.

Truly suspicious input, such as parameters containing the null character, shouldn't be allowed for any input, regardless of prototype. See Phrack Magazine's article "Perl CGI problems" about what can happen if you don't filter away funny characters from the input for perl CGI program.

Sometimes not all of the parameters a function is going to receive can be known before hand. This is the case for example in functions that process a form that has a variable number of input fields in it. For these situations it must be possible to tell in a prototype that the function can receive any additional parameters, or that the name of each additional parameter must match some regular expression, or that their value must be of certain type. These situations tend to be fairly rare however, and may be best handled in the function itself, case by case.

A logical parameter may consist of many form elements. For example, time of the day can be entered as an hour and a minute using select boxes. In this situation it would make sense that the application framework can be told to first validate these two parameters individually, and then combine them into a new parameter. This may be more trouble than it's worth, though. Also when a validity check requires checking relationships between parameters, for example parameter A must be greater than or equal to parameter B, it almost certainly is best not to try automate that.

When the parameter validation detects an error, it can be either a programmer error (such as incorrectly constructed link), or a user error (the user left a mandatory form field empty, say). If it's a programmer error, it's reasonable to simply print an error message detailing what went wrong, and not process the function further. If it's a user error, the function should be told an error has occurred, and perhaps be given an error message string that can be shown to the user in a way the function sees fit. To allow this, each prototype parameter should contain a field telling if a validation error in that parameter indicates a programmer or a user error. If latter, the parameter should also be given a descriptive name that can be used for automatically creating an error message string that is meaningful to the user.

In addition to serving as the application's internal consistency checks and user input checking mechanism, these checks are a very good security measure.

11.1 Minimize the Number of Function Parameters

Even if function parameters are carefully checked for validity, their number should always be minimized. In particular, no state of any sort should be kept in function parameters, but always in session state variables. No redundant information should be passed around. For example, a function that asks for confirmation for removal of a user might want to show the user's full name instead of just the username. The full name can be derived from the username, so the full name should not be passed in the function call, even if the calling function has it readily available.

The reason why the number of function parameters should be kept as low as possible is that in the architecture that this document recommends, the function calls are the primary interfaces between parts of the applications, and any interfaces in programs should always be kept simple. Complicating the interface means the application as a whole is harder to understand, and there are more dependencies between the functions. Needless to say this is bad.

12 Separating Code and Output

In the straightforward CGI coding style, HTML output is generated by having print statements in the code. In the "Code Embedded in HTML" style the code is in middle of HTML. In my opinion, more clarity and structure for the program can be achieved by better separation of code and HTML.

That kind of separation can be done by splitting each HTTP request processing into two major parts: 1) operations mostly independent of HTML, that is, the code, and 2) outputting the HTML, using a HTML template.

The code part does everything not directly related to the HTML that will be output. This typically includes complex request-specific parameter checking, reading and writing the database, and more complicated data manipulations. As the last step, the code sets some variables in the template interface, which will affect what the template outputs.

12.1 HTML Templates

A HTML template is a file containing mostly HTML with some sort of code embedded in it. The code may be simple variable substitutions, loops, if clauses, etc. The language used should be easy to learn, and can be quite high level. The idea is that the template is as simple as possible, concentrates only on the layout of the page, and doesn't need to know how the data it displays is retrieved. The template should never modify data, only output it.

12.2 An Example

An example will probably make this concept easier to understand. The code shall be written in perl, and the template shall be an Embperl file. Embperl is a package that allows embedding perl code in HTML files. It can be used much like PHP, but here we'll use it strictly as a way to implement HTML templates.

In this simple example we'll fetch some data about a user from a database and show it on a HTML page. The HTTP request parameters will be put into a global hash %fdat by the application framework. The code will get the username of the user whose data to be shown from the hash, read full user data from the database, and add that into the hash. The template will similarly read the hash to get the user data to be shown. The code might look as following (comments are in green italics):

Read the user data from the database. $::users is an object of a
class that wraps the DBI interface and offers for instance the method
get_rows() that does SQL quoting automatically.
$user_data = $::users->get_rows (username => $fdat{username},
                                 status => 'active');

Make sure exactly one record was received. Otherwise print an error
message and abort the request processing.
if (! defined $user_data or scalar @$user_data != 1) {
    ::error "can't get user data: $DBI::errstr";
    return 0;

Insert all the user data into %fdat so that the HTML template can
show it. The template can show any or all of the data - that doesn't
concern us.
for my $key (keys %{$user_data->[0]}) {
    $fdat{lc ($key)} = $user_data->[0]->{$key};

Return success code.
return 1;

And the simplified HTML template:

<BODY BGCOLOR="white">

The construct [+ <perl code> +] evaluates the code and
substitutes the construct with the return value of the code.
<H1>Data of user [+ $fdat{username} +]</H1>

$::USER_TABLE_INSIDE_COLOR is a global variable set in application config.
  <TD>Full name:</TD>
  <TD>[+ $fdat{fullname} +]</TD>

[$ if <condition> $] ... [$ endif $] is an Embperl extension.
The block between the statements is shown only if the condition is true.
[$ if $fdat{group} eq 'customer' $]
  <TD>[+ $fdat{password} || '&nbsp;' +]</TD>
[$ endif $]

  <TD>Account created:</TD>
  SHOW_DATE_HOUR_MIN() is a helper function defined elsewhere.
  <TD>[+ SHOW_DATE_HOUR_MIN($fdat{created}) +]</TD>


12.3 Advantages

The advantages of splitting the work into well defined code and template parts include:

  • Request processing has simpler structure. Whenever a non-trivial piece of program can be split into independent parts, that should be done, and this definitely is such a situation. By performing the division, the structure of the HTML page and the code no longer depend on each other, and both are easier to understand and modify. They both also become smaller in size and as such visually easier to comprehend.
  • Editing HTML gets safer. It often happens that the HTML of an application is tweaked seemingly forever. When templates are used, only the template needs to be touched, and there is little danger of breaking the code when of changing the HTML. The worst that can happen is that incorrect HTML will be output, and that is rarely very dangerous.
  • Editing HTML gets simpler. This is especially beneficial if someone else besides the programmer needs to edit the HTML. It's easy to learn to write and especially to modify HTML, and therefore non-programmers often choose to do that instead of delegating the job to the programmer.
  • Templates are reusable. Often many different requests show almost identical output despite having very different code part. In such cases the same template can be shared between the requests.

Sometimes it makes sense to further separate the layout and textual contents of templates. One person could be responsible for the layout of the pages, and other people, possibly with very little computer skills, could edit the texts of the pages. A simple way to do this would be to store the template in one file, and have it include other files that contain the text blocks. The HTML-unaware people would then be allowed to only edit the files containing the text blocks.

13 Locking Data Between Requests

As stated before, HTTP is a very stateless protocol, and the user can and will end his session at any moment, perhaps simply because of a browser crash. Because this can't be detected, care must be taken when locking any data between requests.

Let's take an example of editing order data. A user chooses to edit an order, and is shown the order data in a form. In a traditional application, the order would probably now be locked against modification by other sessions. In web applications this is not a good idea, because the browser of the user may crash while he's editing the form, and therefore the lock will not get released before the session is removed, which might take a week.

Instead, locks should usually allow many users to start editing the same record. When a user is about to submit (confirm) the changes, the user should be notified if a different user has made changes to the data during editing. If this has occurred, the application could give the user a chance to cancel his changes. Sometimes it's possible to merge the changes made by both users, but this kind of heuristics may be not only difficult to implement correctly but also confusing for users.

Another situation that may need to be handled is as following. User A has been shown data of some product on a read-only page. He chooses to start editing that data by clicking an "edit this product" button. At this point the application should make sure, that the data the user had on his read-only screen was not stale, that is, some other user B has not changed the data between user A receiving the read-only page and clicking the edit button. If this has occurred, user A should be warned about this.

When it's not feasible to allow multiple editing sessions to occur at the same time, for example because the record being edited is large and editing consists of several stages, the record must be locked when the editing starts, as with traditional apps. If a record being edited by session A, and another session B attempts to start editing the same record, the user of session B should be told that session A is currently editing the record. User of session B should then have the choices of not starting to edit the record after all, or overriding the lock in case user of session B is certain that session A is no longer active.

Because there is no good locking strategy that would work in all situations, each case must be considered separately, and regrettably often requires a unique solution. Doing this is a lot of work, is difficult to get right, and it's hard to be convinced that all situations have been considered. In reality, it is often possible simply not to lock any records between requests, and let the users know about the implications.

14 Security

In web applications input validation is one of the most important security issues. Nothing that comes from web should be trusted. This means not just the data that is supposed to come from the user of the application, such a form fields. This means also the data that the user is not expected to change. A malicious attacker will attempt changing also that data. Forced validation of request parameters as explained in chapter 11 Request Parameter Validation is a big step to the right direction.

The data sent to the browser should be treated almost as cautiously as the data passed to external commands like the shell. This is because the browser _is_ an external program, that manipulates the data it receives in complex ways. What you think of as text data when you write the application may in fact get substituted by HTML data, or worse, a JavaScript program by an attacker. If that reaches the user without proper escaping, it may, for example, mislead the user into submitting the attacker confidential data when the user in fact thinks he's entering the data to the application.

If an SQL database is used, care must be taken when using input from HTML forms in the SQL queries. Using an interface that automatically quotes strings and makes sure numeric and other arguments have valid format is a good idea. The user the application connects to the database as should have minimum amount of privileges. Read-only requests, which means those that don't change the data in the database, should use a database connection that only allows reading data.

If the application is a daemon separate from the web server, the application should run with a user ID of its own, preferably in a chroot jail. Further partitioning the application into separate processes with well defined interfaces never hurts.

Cryptography is a tricky field. Don't make the mistake of assuming that using encryption or a cryptographic protocol automatically makes things secure, unless you fully understand what you're doing. Getting strongly random numbers is difficult, and many implementations have been broken because of this. If /dev/random exists on the platform, use it. Consider using SSL, preferably with 128 bit keys. Be aware that HTTP Basic Authentication passes the passwords across the network essentially in plaintext.

The points above are of course just the tip of the iceberg. For more information, see the Secure Programming for Linux and Unix HOWTO, one of the many resources available on this topic.

Copyright (c) 2000 by Samuli Kärkkäinen <skarkkai@woods.iki.fi>. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later (the latest version is presently available at http://www.opencontent.org/openpub/).

   Page 1 of 1