User testing is always good, though only if done right. There's a risk if you have a simplistic rule for making two classic mistakes:

a) testing too late in the project lifecycle for the findings to impact fundamental/architectural decisions.

b) low-quality studies either through the use of incompentent staff (for example in a design agency that claims to have usability as a skill without actually knowing proper methdoloogy) or because people who do know what they're supposed to do are not allowed to do it correctly.

So the process has to be thought thorugh a bit more than a simplistic checkbox item "yes, we did usability testing" (but how good?).

@jakob There's an implication that I like from your point A. Would you agree that testing should/could be an ongoing part of the project lifecycle as long as the site you're testing remains flexible enough to support fundamental/architectural decisions? I'm thinking of projects that have an inherently iterative process to them where constant learning/changing is built in.

Until 'management' and upper managers are held accountable, nothing will change or improve agency-wide or at any institutional level. Non-compliance with most parts of the Rehabilitation Act of 1973, as amended, is still part of the Fed's corporate culture. I think that Jakob's point B hits it on the head - and it applies to federal agencies as much as contract vendors. as with most things, employees aren't empowered to get the job done, regardless of how competent and motivated they are.

We don't need more checkboxes and requirements. Inflexible rules create inflexible information and service delivery.

Sadly, Section 508 isn't a new rule or requirement. It was supposed to be implemented by/as of June 21, 2001. EEOC and court decisions well before that even determined that captioning of videos and multimedia was already required under other Sections of the Rehab Act; yet and still, probably a good 25% or more of new multimedia isn't captioned, with another comparable amount poorly captioned. Multimedia players aren't accessible to people who use assistive technology; in ten years of Section 508, I think I've seen maybe one or two audio-described videos.

I like what Ed has to say about "inherently iterative processes." If UX and accessibility testing is required as a box to check, it might lull people into thinking that they're "done" with it.

No one's ever going to get it right the first time. We need a culture (and procurement process) that allows government sites to develop organically by shipping early and often, rather than testing, launching, and walking away.

I hesitate to disagree with you, Jakob, (because I so rarely do, and because so much of what I know about usability I learned from your work), but I think your second point is overcautious.

Yes, people *can* do a bad job of usability testing (most often because they fail to keep their own biases out of it). But I've only seen a handful of instances of this happening, even after years of asking people to send me examples. And almost all of the examples I have gotten have involved testing done by supposed professionals.

I think government sites will inevitably be better if the team has had a chance to watch people try to use them to do mission-critical tasks (while thinking aloud). And, of course, starting very early in the development process, and continuing to do it at regular intervals.

And for this to actually happen economically, I think the teams have to learn to do it themselves.

Please, stop this horrible waste of money. You double the cost of everything when special interest are prioritized above common sense. Stop the govenement waste requiring 508.

Since when are people with disabilities a "special interest?" Much or most of accessibility is just improved usability for everyone. How is ensuring that everyone - employees of the Federal government and members of the public - having access to public information or to the tools to do their job a "horrible waste of money?"

Bringing services to the public should be managed by creating an infrastructure that is data driven, centralized and reusable. In most cases, a data driven environment will enable better usability across the board. However, elevating the 508 special interest, as with any special interest, detracts from the mission of the government to the public by saying information and services can not be publicized until they have passed the "special interest" group whether or not it pertains to the content or the accessibility of the content.

Look at the ridiculous nature of requiring braille at a drive thru bank. This is a secondary proiority, specific to the content of site and it's customers. It should not be required to all sites and content.

First, there is no requirement for braille at a drive-through bank. If you can document, I'd be very interested. Although it may be there as a convenience for passengers, I suspect such machines exist due to the economy of scale - why have accessible ATMs as a walk-up service but an inaccessible version to install in other places. Manufacturers are going to build one version which can be installed anywhere. That's simple economic efficiency!

As for picking and choosing which software applications or websites are made accessible and which aren't, who decides? Which able-bodied person can speak for everyone as to what information is important or necessary for a person with a disability? We wouldn't do that based on race, color, or sex - why do we decide for people with disabilities what's important for them and what's not?

That's why we are broke. You can not prioritize or organize so everthing carries the same priority. Please give extra in you taxes - you owe it.

@jakob - my organization is trying to work through point b. We know we need to be 508 compliant, but because most of the web content developed is done by contractors who are managed by people who don't know anything about the web (biologists, engineers, etc). As the COR's on the contract, they are the ones responsible for cerifying the deliverables, but they have no idea what 508 means, as long as it "looks good" they accept the non-508 compliant deliverable, and then staff is forced to fix the mistakes later. We can't possibly teach all of these people what 508 compliance is. Someone suggested offering an agency wide "508 testing service", but my concern is that whatever contractor is awarded that function will do as poor a job as the contractors that we already have who tell our staff, "yeah, sure, its totally 508 compliant" when it isn't. Not sure what the answer is, but having no way to enforce 508 standards makes them basically non-existant.

What about using Bobby or some other similar tool? I created a website for an NGO that receives federal funds, and used Bobby to ensure compliance and proudly displayed the icon saying it was compliant. As the webmaster, I regularly tested the site with Bobby to maintain that.

Going back to Ed Mullen's comment on my comment:

Yes, a key point is the use of iterative design in a flexible process. Ideally, you build in several milestones for usability studies (and other user research). This doesn't have to be expensive or delay the project, because each step can be quick and cheap, given that there are many of them.

I have often recommended doing user testing with about 5 users, which takes 1 or 2 days, depending on how much you need to test in a session. (OK, may fedreal sites are probaby compex enough that one would run two-hour test sessions, but it still wouldn't take very long to test 5 users.)

The reason you can get away with such a small study is that there'll be another one later. So find (and fix) the most important usability problems now, quickly. Then you can get to the rest a few weeks later in the next round. Fast iterations rule!

You don't even have to call it Agile. Just be fast and flexible! I like these words much better. In the end, many of these projecty ideas amount to much the same, which is to NOT lock yourself down. See my article on Agile and usability at http://www.useit.com/alertbox/agile-user-experience.html

Bobby hasn't been in existence in years; even then, it was an accessibility checker, not a Section 508 checker. Not to say that it wasn't good - it was, then and once it became Watchfire as an enterprise product. It just measured a difference standard.

Thee are automated testing tools, which may be useful as a first check, but much of EIT accessibility requires manual checking. For example, automated tools can tell you if an image has an alt-text or not, but can't tell you if it's useful or meaningful. As long as such tools are understood and used within the bigger context, they do have their place.

I use the chrome and the W3C Markup validation service to check for problums. I ran the us.gov and got 14 Errors and 2 warning. Pages need to be the highest validation, and problums fixed ASAP

Its really sad when you perform usability testing on a website that is already up and working and the testing shows that users have difficulty in using it. Why? Because its very difficult to get agencies to make changes after the website is already up and running. Putting this policy into place would at least give everyone the idea that the website is for the USERS not the managers.

For the guy that said that Section 508 is a special interest and a waste of money, please don't forget that the internet is "blind". It needs to be able to "see" the content on your page. Accessibility best practices such as useful ALT attributes, logically structured Heading tags, descriptive links and the like make your site more visible to technology as a whole. Safari just released the "Reader" function which simplifies the content on a website for example, by analyzing the tags and structure and giving the user only the meat of the page without the navigation and other annoyances.

As for how to do iterative design and afford plenty of early testing without breaking your budget, please also see another idea posted today:

http://web-reform-dialogue.ideascale.com/a/dtd/Fail-on-Paper!/82172-4097

Although I agree with this in principle, it will only become a reality if we have a good process.

That includes

- the Fail-on-Paper idea Jakob points to.

- informal testing (like GSA's First Fridays)

- making usability tests available to content managers easily (like NCI's Open Call)

- periodic larger tests to catch problems on released sites

and generally changing the culture to focus on the audience and usability.

This is a little off-topic, but if our tools and our infrastructure were better at making sites accessible and usable, this would all be easier.

On the accessiblity side, you might want to look at Raising the Floor and the GPII (Global Public Inclusive Infrastructure) http://raisingthefloor.org/gpii

I think the following suggestion is also relevant, because it recommends incorporating usability best practices throughout development; http://web-reform-dialogue.ideascale.com/a/dtd/Commit-to-best-practices/81916-4097.

Although I don't agree with everything in that suggestion, I do like that it supports all usability methods, not just usability testing.

The problem with usability and (to a lesser extent) accessibility is that they are squishy areas. What is perfect for one person may be a horrible user experience for another.

But if you don't address them in your design and through to your deployment, you expose yourself to risks: confused people increasing your call center volume, bad public perception, legal liability (at least for accessibility), etc.

The amount that a test is going to help you mitigate these risks depends on the quality of the tests you conduct. Just requiring "testing" is not necessarily going to ensure that you are producing usable and accessible content and applications. It's just part of a quality lifecycle.

I did give this an "I agree", but only because I think it's vitally important for usability and accessibility to be done well. I disagree that requiring testing is sufficient to achieve this.

Sarah is absolutely right about integrating accessibility throughout the process. It is important however that all of these processes and practices be built on strong policy. Our hope of course is that the new 508 is closely aligned to WCAG2 and that purchasing and procurement policy as well as development practices are built upon that strong, cohesive policy platform.

I agree that the perceived limitation of merely "testing" is false, as accessibility needs to be baked into any web development process.

I mistakenly? presumed this was the intent of the suggestion all along. If not, lets make it so :)

Of course testing is critical, but as pointed out here, we need experienced leadership and staff: http://web-reform-dialogue.ideascale.com/a/dtd/Adopt-and-Support-a-Product-Innovation-Mindset/82223-4097

I have never been called a "special interest" before. As a person with a repetitive stress injury, I use voice-recognition software in order to perform my job. Without an accessible websites and software, I cannot earn a living.

I do want to make a point about user testing, my needs as a person using voice-recognition software are very different than a person who is blind and uses JAWS. Most often, when I experience a problem on a website and call or email the webmaster, they frequently talk about accessability in relation to JAWS, as if that's the only game in town. The assessability needs of users are different based on the type of disability. Therefore, it is important to test using persons with a variety of types of disabilities.

Rita, you've hit the nail on the head. When software developers and webmasters are confronted with the "there's more to screen readers than just JAWS and more to assistive technology than just screen-readers," they pay lip-service but either really don't get it or aren't interested. Yes, this is a generalization, but one based on experience. I'm also a speech recognition user. This experience happens as much within the Fed government as it does with vendors and contractors. Ironically, despite the emphasis on JAWS-compatibility, there is almost no inclusion of Audio-Description in videos and multimedia, despite that being part of the law and standards since 2001. As for captioning, as long as it's there, there's little quality control.

Gary-thanks for validating my experience. I work for the federal government and last Friday when I called the computer help desk, the representative told me I was using voice recognition "at my own risk". This was quite a surprise to me. The person didn't know how to help me and said that she didn't know whether or not Dragon Naturally Speaking, my version of voice-recognition software, was supported by the helpdesk. This was the first time I have ever been told this since 2004 when I first started using voice recognition software. Fortunately, this is a unique experience when I call the helpdesk. But still is rather disconcerting and I did report it to the helpdesk management as a training issue.

This is a great idea and certainly best practice, but I wonder how you would make it "required"? Who would enforce this testing? There isn't a "Section 508 Police" now - it's primarily driven by procurement and policy. Are you suggesting that there should be a more defined enforcement mechanism for 508?

This is a great idea. I see this being a couple jobs created for real. However, Usability and 508 compliance are two different items to be addressed...

In theory, it's a good idea. And section 508 SHOULD be done... voluntarily.

But, you have to ask:

Does it make sense to mandate an agency's web division to spend 35-50% of their web budget on <2% of their web audience? That answer, I believe, is a resounding "NO".

Now, imagine if that budget would be spent on content catered directly to the other 98% of their web visitors. Flash, graphics, mobile apps, interactivity. That's what suffers in all of this.

Glen, you are flat out, crazy wrong.

Even if accessibility did take 35-50% of a site's budget, to not offer equal access to everyone is illegal and immoral discrimination. Period.

Of course, any developer here will tell you that addressing the majority of accessibility needs simply involves following best practices for website development.

With some additional captioning and testing, you're there, and the tyranny of the majority is overcome by fair and equal access to all.

Glen, where do the numbers of 35-50% come from? I think most of us would be very interested in seeing the source. Rarely is the cost of building accessibility into a product or service terribly high - it's only when it's an afterthought and has to be 'added on' that costs become steep.

Depending on the application, it may be an erroneous assumption to assume that less than 2% of the audience require an accessability option. Let me give you an example. My Federal agency contracted with a private company to develop a proprietary software system that was to include a new web site to be used by outside consultants, as well as government employees. Once developed, the proprietary software, including in the web site used a very small font size, 10 point if I recall correctly. Unfortunately, there was no ability to change the font size. And the majority of the users of this software and web site were people over the age of 50 with poor eyesight. These are not the people that you would normally classify as "disabled". The outcry was loud and strong, but no change was made in the software for several years because we were told that there were other more pressing changes it needed to be made in the software. Accessability took a back seat to the content issues. In this example, the company who developed the software should have considered the audience and users, to say nothing of integrating accessibility options into the system automatically. I will add to this that two of the users of this proprietary system were government workers who use voice-recognition software. However, the system was extremely difficult, and in some cases, not possible to use because of accessibility issues. These are the challenges faced when the developers of software and web sites do not test the system prior to implementation.

Glen: I almost forgot... your comment also sets up a false conflict with visual panache and accessibility. A site can be both dynamic, accessible and gracefully degradable when developed using the latest HTML/CSS underpinnings and a bit of javascript/ajax.

and as a side note: When Adobe (the owners of Flash) go out of their way to develop a too; that converts Flash to HTML5 (aka Wallaby), that should be a clue that the future of the web is Flash free.

As a long-time Flash developer, this is painful for me, but I'm working through my grief by learning to use jQuery plugins.

Jeff, I do, in fact, agree that we're going to being flash-free (quickly). But, while a web manager (within DHS, ironically), we worked with a VERY small staff and budget. Out of our <400k budget, over 150k of it was geared to 100% accessibility testing. So, basically, we had to lose a staffing slot because of 508 regulations.

Let me stress this: Should we do 508 testing? Yes, but NOT at the expense of helping our customers... which, I believe, is happening thoughout many agencies right now. Let the "want" justify the "need", because they are indeed two different things.

Glen,

The "testing" in the headline has caused a bit of confusion here, but accessibility isn't achieved by testing, is achieved by developing a website from the ground up with accessibility in mind.

I know first hand that DHS has some very strict accessibility requirements, but I'll just reiterate what many have said here that when accessibility is a consideration from the very beginning of a project, it's not nearly as resource-intensive as trying to retrofit it later in the process.

I'm sure you can see I'm passionate about the cause, and I can only hope that, moving forward, accessibility will be an easier achievement for you.