I my previous post, I recalled a recent professional interaction where a research proposal claimed that a particular test was valid. The evidence cited to support this claim was defined in the original study of the test’s development.  I pointed out that the phrase; “the test is valid” although commonly used is psychometrically incorrect.

Firstly, validity is something that is built up over time by experimental evidence; it is not a dichotomous quality that is shown to be present or absent.  In contrast to citing one supposedly seminal study, validity should be supported by many types of evidence collected over an extended period of time.

One type of validity evidence involves the relationship between the test results and measures of other constructs where one would expect a significant statistical relationship to occur. For example, a new test of worker burnout in the workplace might be expected to correlate with some existing tests of anxiety concomitantly administered with the burnout questionnaire.

Validity evidence might also be developed in studies that correlate the present results of this new burnout scale with future events.   Therefore, if one were to discover that the new burnout questionnaire predicted with some accuracy who actually went out on medical leave on burnout, then one would have another source of demonstrated validity.

But more importantly, no questionnaire, scale or test is valid; only the test results obtained in a specific administration context are valid.  The context includes multiple factors such as the sample of people tested, as well as, the psychological environment of that testing.   For example. validating the questionnaire of work satisfaction among professionals working in a stable job employment context does not provide evidence that the test is a valid measure of satisfaction of salespeople assessed during company downsizing. The samples and the context of measurement of those samples are different and new validation studies would be required.

When thinking about administering a test or questionnaire, don’t just ask if it’s valid.  Look into the validity evidence. Check to see if the original test was developed on a sample and conditions similar to your own. Look at what the test might have predicted or successfully classified.   Then, make the decision if the test is right or wrong for your people and your situation.