Usefulness Trumps Validity

Validity is perhaps one of the most misunderstood concepts in HR analytics and psychometrics in particular. This is a topic that I have previously written about on this blog but the message has yet to fully resonate with the HR community. The most common question that OPRA gets asked in relation to any solution we sell, be it an assessment, survey or intervention, continues to be “What is the validity?”

On the face of it this is a perfectly reasonable question. However when probed further, it becomes clear that there remains a gap in understanding what validity translates to in terms of business outcomes. The answer to this question is invariably a validity coefficient rolled off the tongue that then suffices some checkbox prescribed for decision making.

Instead of asking about validity, the real question should be “How useful is this assessment or intervention?” This leads to the more focused question of how useful this assessment or intervention is for the problem (or problems) wished to be solved.  Thus the question is not one of simply producing a number to represent artificial validity criteria but is reframed to the business imperative of usefulness.

Ironically this in no way limits the rigour within which an assessment, intervention or survey will be evaluated. On the contrary, the bar now becomes far higher. The reality is that obtaining suitably respectable correlations such as r=0.3 between some measure and an outcome is not particularly difficult. What is far more difficult is to put this figure in context of a system to in turn determine usefulness.  Fortunately there are frameworks that help in this thinking.

One in particular that I’m drawn to is the Key Evaluation Checklist developed by Professor Michael Scriven. This concept has been furthered by more recent work in the field.  The point being that usefulness combines statistics, logic and systems thinking to establish the merit, worth and significance of what is being evaluated. This provides a far more systematic way of thinking about validity and provides an applied framework for determining usefulness.

Likewise, when we are interested in truly understanding construct validity, the standard correlations between like measures simply does not suffice. What is required is to understand the nomological network in which the construct exists and demonstrate not only what it correlates with (convergent validity) but also what it can be discriminated from (discriminant validity). In this way we build a deeper understanding of the construct of interest, and in turn, increase our understanding of its usefulness.

In the world of HR analytics and big data we sometimes forget that usefulness, not validity, is our ultimate goal. Number crunching is but a tool in this process. What is more important is an understanding of the framework in which usefulness will be demonstrated. When good analytics meet good applied thinking and methodologies, progress will ensure. But to confuse validity as an end-point is to render any HR analytics project irrelevant to adding value to an organisation.

Advertisements

One thought on “Usefulness Trumps Validity

  1. Rod J.

    Paul, insightful article. Yes, “usefulness, not validity is our ultimate goal.”

    However, many seek safety in the construct of validity, not usefulness.

    Thanks.

    Reply

Please Comment!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s