Writing Better Tests for Job Training: The Issues of Reliability and Validity

It's often, if not always, a good idea to provide some form of test or assessment after providing job training to employees.

In some cases, this may be a written test scored in a pass/fail manner, and in others, it may be a performance test that requires the workers to demonstrate a skill or the ability to perform a procedure in a satisfactory manner before they go back and perform that skill for real on the job.

In either case, it's important for that test to be a good one. And by "good," we mean that it provides you with useful, actionable information about whether or not the employee has truly benefited from the training, can satisfy your learning objectives, and is ready and able to successfully apply the new information or perform the new skill on the job.

Side note: Tests are also beneficial for compliance reasons, to prove the worker understood the training, and also because tests have been proven again and again to improve comprehension and retention (see this article on the "testing effect" for more on this).

There are a number of characteristics that "good tests" like this share. Learning & development experts know the two that we'll talk about in this article as validity and reliability. Read on to learn more about what a valid and reliable test is and why it's important to create valid and reliable assessments.

Valid Tests and Assessments for Workforce Training

A valid test is an accurate test of the employee's knowledge, skill, or ability, and of course the ability of the employee to correctly perform the learning objective for the training.

If a test is valid, and an employee passes that test, the worker will really have the desired knowledge or skill you're testing for. Likewise, if the test is valid, and the employee fails that test, the worker really won't have the desired knowledge or ability. In short, if your test is valid, it measures what you think it's measuring.

On the flip-side, if a test is not valid, a worker who passes the test may not have the desired knowledge or skill, and/or a worker who fails the test may have the desired knowledge or skill.

Let's look at a more specific example: say you've got a team of machine operators, and you give them each a test to see if they can operate the machine. Half pass the test, and half fail.

Then let's say you take those workers out onto the production floor and let them run the machine.

If your test was valid, the half who passed the test will be able to run the machine, and the half who failed the test won't be able to run the machine.

On the other hand, if your test is not valid, some of the half who passed the test won't be able to run the machine, and/or some of those who failed the test will be able to run the machine.

The less valid your test, the fewer who passed will be able to run the machine and the more who failed will be able to run the machine (validity is a spectrum/range, not an absolute).

Tips for Creating Valid Tests

Now that you know what a valid test is and see why it's an important concept, here are a few tips for creating valid tests:

• Make sure your test matches your learning objective. For example, if you want someone to be able to operate a machine, don't create a test that asks them to explain how to operate a machine. Why? Because operating a machine is much different than explaining how to operate the machine. Instead, assess their ability to actually operate the machine. Or, don't test their ability to operate the machine while it's running much more slowly than it will when they need to operate it on the job. Or, don't test them in an environment with fewer distractions than they'll have to deal with on the job. Instead, create a test that evaluates their ability to operate the machine at normal work speed and with normal work distractions.
• Match the difficulty of the test with the difficulty of the real-world task. This is similar to the point above, but bears repeating. Don't make the test easier than the real-life task, or else you won't know if your employees can perform the task on the job. And likewise, don't make it harder, either--don't test someone's ability to run a marathon if they simply have to run 50 yards (that's an analogy, but you get the point).
• Ask real-world experts (so-called subject matter experts) for their input in creating the test to match real-world expectations and experiences.

Reliable Tests for Job Training

Now that we've got validity covered, let's look at our second criteria: reliability.

If validity was about accuracy, reliability is about consistency.

If your test is reliable, it will tell you every time if an employee can satisfy the learning objective, and it will likewise tell you every time if an employee can't meet the learning objective.

In short, when everything else is the same, a reliable test will give the same results every time. A reliable test is a consistent measurement. An unreliable test is not consistent.

For a visual analogy, think of a dart board with four darts all thrown in the same area of the board. For this example, we'll say that all the darts are thrown very near the bulls-eye at the center of the dartboard. Those four darts represent the passing score on a test taken by the same employee four different times. This is a reliable test. The employee gets the same or very similar results each time they take the test.

For a second analogy, think of that same dart board with four darts again. The darts are all closely grouped again. But this time, all the darts are off to the side of the board (near the outside edge at what we'd call 3:00 on the face of a clock). Those four darts represent the failing score of the same employee who's taken the test four times in a row. This is a reliable test again, because the same employee got the same results each time (even though he/she has failed).

And for a third and final visual analogy, think of a dart board with four darts spread all over the dart board. Those darts represent the different scores and results when one worker takes a test four times in a row. This is not a reliable test, because the same employee gets wildly different results each time.

So, your goal is to create tests that led to reliable results--meaning results that consistently show a passing employee has passed and a failing employee has failed. And, on the flip-side, your goal is to NOT create unreliable tests that deliver inconsistent results.

Tips for Creating Reliable Tests

Now that you know what a reliable test is, let's give you some tips for creating them:

• Create tests that are at the correct difficulty level--match the test to the necessary job performance.
• Create tests with enough test items so you can get a true understanding of whether or not your employees can satisfy the objective. In a perfect world, you'll have several test items for each objective (this reduces the chances of something random skewing the results).
• Create well-crafted, well-thought-out, well-designed, well-written tests that aren't vague, misleading, ambiguous, poorly written, or easy to guess. Review the items and have others review them for you before trying them for real with your employees.
• Don't create tests that are difficult to answer or difficult to answer just because of the user-interface. In other words, don't be tricky, don't get too cute, or don't be so sloppy that your employees can't figure out how to answer.
• Create tests with an objective scoring basis.
• Don't leave scoring up to the whims of one evaluator, or the different whims of multiple evaluators. Create a checklist or guide for evaluators so they all know what to look for and how to assess worker performance.

Good Workplace Tests Are Both Reliable and Valid

You may have gathered this already, but we wanted to call it out directly: A good test needs to be both reliable and valid.

This information is important if you're having a training provider create tests for you.

Likewise, it's important if you're using your own learning management system (LMS) to create your own assessments, using tools like the online quizmaker shown below.

LMSs allow you to make your own online quizzes and other online learning activities.

Conclusion: Good Tests for Good Workforce Training

Hopefully understanding the concepts of reliability and validity will help when you evaluate tests or create and evaluate your own tests.

Here are some additional articles on testing for workforce training that you may also want to check out:

• Workforce Training & Testing Best Practices
• Writing multiple-choice questions
• Writing true/false, matching, dragging, and other types of questions
• Using scenario-based learning and assessments
• Testing and fidelity
• The "testing effect" and the forgetting curve
• Interview with learning researcher Dr. Patti Shank on writing better learning assessments and multiple-choice questions

Let us know if you've got some questions about testing (there's a chat feature down in the bottom-right corner) or drop a comment in the Comments section below.