Forms, usability, and the W3C DOM

By Peter-Paul Koch

Despite the specification having been around for nearly five years and a workable level of browser support for about three, the average Web developer doesn’t yet have a clear view of what the W3C DOM can do for Web sites.

Sure, in those five years many good and bad, technical and less technical introductions have been written, but these all focus on implementation details. That is understandable—you first have to understand the implementation of any new technology before you can start using it.

Nonetheless, our knowledge of the W3C DOM’s JavaScript implementation in today’s browsers is sufficiently complete to start exploring the practical use of the DOM in Web sites.

The W3C DOM works. What shall we do with it?

What shall we do?

Six months ago, I mused about the practical possibilities of the W3C DOM. For the first time in browser history, you can give users control over their workspace. Users can restructure parts of an XHTML page to personalize it, not only the graphic design but also the presentation of the data and its ordering on the page. Is this useful? If so, for what kinds of sites?

I wrote a simple example script that clones sets of form fields if users so desire. This can be quite useful in a number of circumstances. For instance, when creating a page to rate and review CD’s, how many CD review fieldsets should the page contain? The answer is usually four, sometimes seven, even though most users will want to enter another number of CD reviews.

The script allows users to generate as many fieldsets as they need, and not one more or less. This obviously makes the page more usable, so I concluded that the field of usability will be the first to profit from practical W3C DOM scripts.

When an interesting business case came my way, I decided to see whether the W3C DOM could help with the single most ubiquitous, persistent and unsolvable usability disaster: large and complicated forms. In this article, I’ll present the resulting W3C DOM script. This script:

solves a usability problem of large forms that even usability specialists seem to have given up on.
shows how script data (in this case “form logic” data) can be retrieved from the XHTML itself, which results in extremely simple code that is easy to implement and maintain, even for novice Web developers.
serves as an example of combining an advanced W3C DOM script with 100% accessibility.

First, I’ll take a look at the business case and the main usability question. Then I’ll give my reasons to use the W3C DOM instead of DHTML. After that, I’ll explain a novel way of getting data from the XHTML itself, and give the detailed instructions for using the script, followed by notes on browser compatibility, accessibility, and XHTML validation, and, finally, the conclusion and suggestions for further research.

The case

One of my clients, a large Dutch bank, asked me to create a demo form for mortgage applications, arguably one of the most complicated forms in existence. I implemented my earlier script in the “Other mortgages and loans” section. If the users have several other mortgages and loans, they can generate more sets of form fields to describe all the details.

The form nonetheless remains quite complicated. In a way, this doesn’t matter: everyone expects to apply for a mortgage to be a convoluted process of giving full financial information and endless private details, accompanied by a general gnashing of teeth.

So the users are quite resigned to spending 15 minutes in front of their computers to complete the application. Nonetheless, any method of making these huge forms less disastrous is worth a try. If I could reduce the time spent on the form to, say, 10 minutes because the users instinctively understand what to do, this might give the bank a slight edge over its online competitors.

Usability

The best way to reduce the time the users spend on the form is to enhance its usability. This fact has been recognized in some good articles and best-practice sites. Nonetheless, all these sites conspicuously overlook one possibility of usable online forms.

In large forms, many form fields are required (or, in fact, useful) only in a certain context. For instance, “Date of divorce” fields are only required when the users indicate they are, in fact, divorced. Online forms can easily hide the fields, and show them only when the users check “Marital status – Divorced.”

Curiously, this possibility seems to be structurally absent from mainstream usable forms discussions. This is not surprising for older articles since the technology was not available before 2000. However, newer sites and articles hardly mention it either, not even as a bad idea. I did find a few pages and documents discussing the idea, but they only note it in their wish list or resort to complicated applications to obtain the effect.

Even stranger, I found only one online form that hides unnecessary form fields. I heard of one other residing on a corporate Intranet, and there is my own demo form. Although I don’t think these three forms will actually be the only examples in existence, this meager score certainly indicates that the technique is anything but widespread.

Web developers and usability specialists have too easily copied paper forms to a Web environment, without paying enough attention to the differences between the media. The paper form is printed once and must be usable in any circumstance, which means it must include every possible form field. The online form, though, can be manipulated to a striking degree to reflect the needs and choices of the users, for instance by hiding unnecessary fields.

Note that I mean hiding form fields quite literally: until the users select the correct option, the fields are completely absent, not only from the form’s visual representation but also from the actual XHTML <form> element.

Technically all this is possible. Does it make forms more usable?

The users

Before creating the form, I had a long conversation with a mortgage product manager who patiently explained some complicated features of mortgages and applying for them. In return, I explained my idea about hiding form fields.

He knew exactly what I meant, even though I only vaguely sketched the possibilities over the phone, without any demo being available to show what I meant. In fact, his immediate reaction was that this would be a “blessing” to online application forms, which he thought far too complicated in general.

When I’d completed the prototype script, I proudly showed it to a few friends, who are internet users but not particularly involved with creating Web sites. Their unanimous reaction was “So what?”At first, I was a bit miffed. Afterward, though, I understood that this was the best possible reaction. Not only did my friends immediately understand the concept, they even thought it quite natural. “What’s the fuss? Of course, you can hide form fields!”

Later on I interviewed yet another friend whom I’d asked to test the demo form. He told me about several usability problems, but never even mentioned the hiding of form fields until I brought it up. He thought it a useful effect, and it was clear from his remarks that he understood exactly how it worked. But apparently the effect was so clear and obvious that he didn’t find it necessary to mention it.

Having reached Nielsen’s boundary by successfully testing the idea with exactly five users (one of whom never even saw the form!), I think it’s likely that most end-users won’t even consciously notice the effect. They’ll just use it. That’s the best usability there is.

The theory

So my theory is that large and complicated forms will become much more usable when they hide as many form fields as possible until the users needs them.

To confirm or refute this theory we need to test it. To test it, we need a script.

W3C DOM vs. DHTML

Before writing the script, I had to select technology. To hide form fields from the user’s view, I do not need the W3C DOM. I could use DHTML, too. Nonetheless, for several reasons the W3C DOM is superior to DHTML. Before explaining these reasons, have a look at the differences:

When hiding from fields through DHTML, you change the value of certain style properties. Traditionally, you set the display style of the form elements or their container to none. If necessary, you change it back to its original value. Note that the document structure is not changed in any way. The hidden form fields remain where they are, they’re just hidden.
When hiding from fields through the W3C DOM you remove the form fields and their container element(s) from the form and put them in what I call a “waiting room.” If necessary, you take the form fields out of this waiting room and return them to their original place in the form. So you change the document structure and leave the styles of the form fields and their container untouched, although you’ll have to make sure that the waiting room and its content are hidden from view.

To the users, the net result is the same: now they’re here, now they’re gone. Why is a W3C DOM solution superior to a DHTML solution?

First of all, there’s a technical reason. To keep the script simple, I wrote it to search for certain custom attributes in the XHTML. This part of the script relies on the getAttribute() method which is implemented only in the W3C DOM. Previous technologies do not allow us to search for custom attributes.

Structure vs. presentation

But the most important reason to use the W3C DOM rather than DHTML is that it changes document structure instead of document presentation.

DHTML is the changing of CSS by means of JavaScript. It about presentation and the changing of presentation. “Shall I make the font size larger?” “I’ll highlight this navigation item for you so you can see where you are.” “Look, there’s a Flash banner for cat health food bouncing like mad in your browser window! Ain’t it fun?”
The W3C DOM, on the other hand, is about document structure and the changing of document structure. It allows you to add or remove XHTML tags, or even entirely rewrite the page if you so wish.

If you want to offer the users a logical, usable form, you must change the form structure so that it conforms to the users’ specific situations. Of course, this causes the form presented to change, too, but that is a secondary effect.

If you think this line of reasoning is too theoretical, consider what happens when the users send the names and values of all fields to the server by submitting the form:

If you use DHTML, some form of fields are hidden from view through CSS manipulation. However, they’re still present in the document between the <form> tags and are thus sent to the server.
If you use the W3C DOM, some form fields are entirely removed from the form. They are not sent to the server because they’re not part of the form.

All in all, choosing the W3C DOM over DHTML is more in line with the ultimate purpose of our script: changing the structure of the form according to the users’ needs.

Custom attributes

The most complicating factor in coding large forms is the “form logic.” Of course, this “logic” is anything but logical; it’s a set of essentially arbitrary rules that say “if the users check this radio button, they must fill in that field,” and so on.

Some of these rules make sense to the users, others only to the creators of the form. None of these rules make sense in terms of the document structure though. There are no general rules, like “if the users check the second radio button of a group, show the next TR.” So we must somehow add custom instructions that tell the script when to do what.

Previously there were two options to do this:

Either use endless JavaScript arrays:

var users_InterFace.prototype.data = new Array(
  ’marital-single’,’none’,
  ’marital-married’,’marriageContract’,
  ’marital-divorced’,’divorceDetail’,
  ’and no one’,’on earth’,
  ’will ever’,’find out’,
  ’why changing’,’this value’,
  ’crashes’,’the page’
);

or write custom scripts for every single click action:

<input type="radio" name="marital" value="single"
  onClick="hideMarrContrDivDet()"> Single<br>
<input type="radio" name="marital" value="married"
  onClick="showMarrContr()"> Married<br>
<input type="radio" name="marital" value="divorced"
  onClick="showDivDet()"> Divorced<br>
<input type="radio" name="marital" value="uncertain"
  onClick="anothAbbrdNm()"> Uncertain<br>
<input type="radio" name="marital" value="bliss"
  onClick="whatsThisOneAbout(?)"> Marital Bliss

I don’t like either of these options because they quickly become unmanageable. Instead, my script searches for custom XHTML attributes to get its “form logic” instructions. Although the idea of using custom attributes is not really new, I found only one script that uses attributes in this way. Again, I don’t think this will actually be the only example, but we must once more conclude that the use of this technique is not widespread.

You can add two custom attributes to your XHTML tags:

The show attribute can be added to form fields.
The relation attribute can be added to TR’s containing other form fields.

This will lead to code structures like:

<tr>
  <td class="question">Country</td>
  <td>
    <input type="checkbox" 
       name="country_other" show="othercountry"
      /> Other than The Netherlands
  </td>
</tr>
<tr relation="othercountry">
  <td class="question">Which other country?</td>
  <td><input name="other_country" /></td>
</tr>

Initially all TR’s with a relation attribute are hidden from view. If the users check the checkbox with show="othercountry", the script sees this as a command to show the TR with relation="othercountry". If the users unchecks it, the TR is hidden.

(Why don’t I use the existing rel attribute instead of the custom relation attribute? Because Opera doesn’t accept rel on TR’s.)

This way of delivering instructions to the script is much cleaner and clearer than any other solution. More importantly, the instructions can be changed easily by even the most novice of Web developers. I’m quite taken by the idea and from now on I’ll use it on a large scale. I advise other Web developers to do the same.

Using the script

Using this script is quite simple. You can download the script or study an example form on my site.

Step 1: Include the JavaScript file in your form page, like:

<script src="usableforms.js"></script>

Step 2: Run my function prepareForm() onLoad. For instance:

<body onload="prepareForm()">

Step 3: Outside the form, add a “waiting room” to store unused form fields. This version of the script moves TR’s around the document, so the waiting room should be structurally ready to receive TR’s. Of course, you hide the waiting room from view:

<table><tbody id="waitingRoom" 
      style="display: none"></tbody></table>

The id="waitingRoom" (case sensitive!) is required. The <tbody> tag is required in Safari (v73). If you don’t include this tag, the script won’t work.

Step 4: Now create the form and add the show and relation attributes where you need them.

You can add the show attribute to radio buttons, checkboxes and options. If you add it to radios or options, you must add it to all other radios or options in the same group, too. After all, any click on any of the radio buttons or options should activate the script, even if only to hide the TR’s that are currently visible.
You can add the relation attribute only to TR’s in this test script. This is because the first line in the script reads:
```
var relatedTag = ’TR’;
```
If you want to use another tag, replace the TR by the tag name of your choice. Don’t forget to change the structure of the waiting room.
Browser alert: Do not use the same values for a show/relation attribute and the name attribute of a form field. Internet Explorer on Windows incorrectly equals name and id, causing no end of trouble.

Sample XHTML. We again include a <tbody> for Safari’s sake:

<form>
<table><tbody>
<tr>
  <td class="question">Marital status</td>
  <td>
    <input type="radio" name="marital" value="single"
      show="none" /> Single<br />
    <input type="radio" name="marital" value="married"
      show="marriageContract" /> Married<br />
    <input type="radio" name="marital" value="divorced"
      show="divorceDetails" /> Divorced<br />
  </td>
</tr>
<tr relation="marriageContract">
  <td class="question">Marriage contract?</td>
  <td><input type="checkbox" 
        name="marriage_contract" /> Yes</td>
</tr>
<tr relation="divorceDetails">
  <td class="question">Date of divorce</td>
  <td><input name="datedivorce" /></td>
</tr>
<tr relation="divorceDetails">
  <td class="question">Type of divorce</td>
  <td>
    <input type="radio" name="typedivorce"
      value="messy" show="none" /> Messy<br>
    <input type="radio" name="typedivorce"
      value="very_messy" show="details"
            /> Very messy
  </td>
</tr>
<tr relation="details">
  <td class="question">Gory details</td>
  <td><input name="gory_details"></td>
</tr>
</tbody></table>
</form>

Now if the users click on Marriage, the marriageContract TR is shown. If they click on Divorced, the two divorceDetail TR’s are shown. In addition, any earlier choice is removed, so if the users click on Divorce while the marriageContract TR is visible, this TR disappears.

Note the special show="none" attribute of the Single radio. It’s specifically meant for radios or options that must have a show attribute but should not actually show new form fields. It says “Hide all TR’s associated with the other radios but don’t show any new ones.”

Also note the possibility of nested relations. One of the divorceDetails TR’s contains a form field with a show attribute. When the users hide the divorceDetails TR’s by clicking on Single or Married, the nested TR is also hidden.

Since I define one general onclick event handler for the entire page, you can continue to add your own events to radio buttons and checkboxes.

Select boxes are more difficult. A click event on select boxes or options turns out not to work in all browsers, so I opted for the traditional change event on the select box itself. Any change event you register yourself is overwritten by my script and won’t function.

Browser compatibility

What about browser compatibility? Obviously, a browser not supporting the W3C DOM will not support this script, so the effect does not work in, for instance, Netscape 4, Internet Explorer 4, OmniWeb or iCab. No surprises here.

More unfortunate is the demise of Internet Explorer 5 on Mac, which crashes mercilessly when the users click on one form field too many. The sad conclusion is that IE 5 has reached the end of its W3C DOM capabilities. Its implementation is slow and cumbersome, completely unusable in more complicated applications like this one.

The worst part is that I actually have to use a browser detect. This is not a case of a browser not supporting certain methods and properties, but of a browser crashing on methods and properties it theoretically supports, a fact that no object detection will ever reveal.

Therefore, I don’t run the script in IE 5 on Mac. I intensely dislike browser detects, but I dislike the idea of crashing 2 to 5% of my visitors’ browsers even more.

I tested the script in Internet Explorer 5.0 and 6.0 on Windows, Mozilla 1.3, Opera 7.10 and Safari v73. Opera, especially, has some quirks:

When submitting the form Opera incorrectly sends the form fields residing outside the <form> tags to the server, too.
Nested select boxes don’t work in Opera. Initially nested select boxes are hidden, and Opera curiously gives any hidden select box a selectedIndex of -1, something my script can’t deal with. Since this is a browser bug that will probably be solved pretty soon I haven’t worked around it.

If you use tables for basic form layout, you will notice that Mozilla has some redrawing problems, which are unrelated to this particular script.

The script does not work in Safari in Strict mode.

Accessibility

So some browsers don’t support the script. Is the form still accessible without it?

Yes, it is. In fact, the form is 100% accessible at all times.

The principle for keeping your Web sites accessible even when using advanced scripting is simple: make sure that the initial XHTML is logical and accessible. The startup you’re script only after this logical, accessible XHTML has been delivered to the browser.

So initially you place all form fields in the correct order. The browser receives the XHTML document with this natural structure intact. If the script doesn’t work nothing happens, so the form fields keep their natural, logical positions. Therefore the form is still perfectly accessible to users not having a W3C DOM capable browser.

I added one extra accessibility feature to the script: you can add elements with class="accessibility" to the form, which are hidden automatically if the script works. You can use them for showing the extra text to the users of incompatible browsers. After all, these users see every single form field that could be necessary. They need some extra guidance.

<tr>
  <td class="question">Country</td>
  <td>
    <input type="checkbox" 
      name="country_other" show="othercountry"
      /> Other than The Netherlands
  </td>
</tr>
<tr relation="othercountry">
  <td class="question">
    <span class="accessibility">If other than
      the Netherlands:</span>
    Which other country?
  </td>
  <td><input name="other_country" /></td>
</tr>

Despite this, the form will be less usable without the script, and there’s nothing to be done about it. You’ve done your duty, though. Any device can access the form.

Validation

XHTML validation is the single weakness of this script. Since you use custom attributes not defined in any W3C spec, validators will point out that the XHTML code is invalid. You’ll have to tolerate the error messages concerned with the show/relation attributes.

Nonetheless, the advantages of this script outweigh this weakness. Besides, XHTML is, authorities assure us, eXtensible HTML. So let’s extend it, shall we?

Conclusion and further research

From a usability point of view, hiding as many form fields as possible until the users need them seems to be a good idea. Writing a W3C DOM script that hides form fields turns out to be quite simple. When using this script, the relations between form fields can easily be changed by anyone having basic Web knowledge, because the script instructions are placed in XHTML attributes. In addition, the form remains 100% accessible at all times.

Do the users actually think it’s a good idea? My suggestion for usability testing is the following:

Use the script in a complicated form. Set up a simple usability test.
Do not mention the effect at all. Just let the users play with the form. Expected behavior: they’ll use the effect without commenting on it.
Afterward, try to bring the effect into the conversation naturally. See what they think of it. Expected behavior: they’ll react lukewarm. “Yeah, sure, nice.” Remember that this is actually the best possible reaction. It means: “Why make such a big deal about something so natural?”

Of course some users might react differently, and my theory may even turn out to be incorrect. We’ll see.

We’ve now successfully used the W3C DOM to enhance the usability of large forms. Please keep in mind, though, that this is just one example of the W3C DOM’s power. No doubt other people will write scripts that improve other aspects of Web sites. We only stand at the beginning of using this new technology that will change the way people use Web sites.

Peter-Paul Koch is a freelance Web developer, writes and maintains the JavaScript Section and is an Administrator of the WDF and WDF-DOM mailing lists. Peter-Paul is also the author of Digital Web Magazine’s Keep It Simple column.