One would imagine that a part-time academic appointment such as mine with McGill University would bring certain perks such as a free gym membership or reduced rates of campus parking… alas no.  What perks exist are almost all of the intellectually stimulating variety such as when I am called upon to review research grants as part of the university’s scientific or ethical review process.  Although my research days are long behind me, I admire researchers, especially those in the mental health area, since the rewards are meager and the competition is ferocious.

Recently, however, I ran into trouble with a grant proposal that sought to examine the fluctuating course of symptoms in patients with severe and persistent mental disorders.  Since the study was observational and not interventional, the focus of my review was on the measurements themselves – specifically, the numerous questionnaires that the authors wanted the patients to complete not just once, but repeatedly.

The questionnaires were long and involved with many relying on the accurate recall of past events such as previous medical appointments and hospital visits.  The material for some of the items I would have trouble remembering and hopefully I don’t yet display the memory, attentional and other cognitive deficits that have been consistently shown to characterize this chronic population of patients.

I could see why the researchers were loading-up the questionnaires, they simply didn’t want to miss anything.  All to often reviewers can easily criticize a study for failing to measure any number of potentially “interesting” variables.  Of course, the “interest” might rest more in the reviewer’s mind than the researcher’s.

In any event, my own comments questioned whether the battery of tests and questionnaires were too demanding both in terms of cognitive as well as motivational requirements. The investigators retorted that my concerns could be ignored because the “tests were valid” – they had been so proclaimed in the studies that heralded their original publication.

A test being called “valid” is something that I’m used to hearing in my practice as a workplace consultant where senior managers or HR staffers are extremely interested employee attitudes and behaviours purported to be measured by “valid questionnaires.”

This commonly uttered “test valid” statement, as used above, is technically incorrect in two basic aspects.  In my next post, I’ll explain why.