How Much Has Really Changed In Two Years?

I was recently cleaning out the forum board from the old OPRA site and found an archived post which sums up the industry very well for me at the moment. This post was written a couple of years ago and covers the changing nature of the testing industry. I think the sentiment is as relevant today as it was back then, and in this regard it reminds me how little things change.

“Hi everyone,

I read with interest the article on the changing nature of testing by Bartram. On reading the article, I would suggest that nothing fundamental at all has changed in our industry; reality is merely kicking in. The paradigm shift represents a removal of the mysticism of testing and the emperor’s clothes are finally off. The reality of testing is still the same.

Key points I noted:

There is a lot to comment on in relation to the article, both in terms of what the themes presented represent for the testing industry, as well as the gap between the rhetoric of testing companies and the reality of commercialisation and the profit driver in the testing industry”.

1) That the past involved scoring keys and interpretation algorithms, and that recovering data from users was a costly procedure.

This is only if you are not using a database driven solution. There are many open source systems that allow users to collect, analyse, and share data at no cost. The model that was referred to in this article was only one model that I believe was maintained in a monopolistic environment that pushed transactional test use. It did not allow users easy access to their data for strategic advantage. This model is clearly changing. The Internet is not the advent of data capture, it merely facilitates data capture.

2) Unsupervised testing does not impact on the properties of the test.

The issue may not be around the properties of the test but more so the ethics of testing. These have been raised and endorsed by Psychological societies, and perhaps only in countries with strong Psychological societies (which have ethics that are enforceable by legislation), such as South Africa, that are continually followed up. In this regard, companies that flaunt testing standards have been severely punished, i.e. those that allowed unsupervised Internet testing with their standard tests.

3) The idea of new test construction.

This is really the only change discussed that I would see as a fundamental development. It fits a broader framework of new models of psychometrics and the old criterion that we used to evaluate tests is definitely being challenged. A further departure is the use of single item psychometrics. What this indicates is that the very measurement (using the term loosely!!) models that have been used to determine robustness may very well be limiting innovation and therefore are in need of change.

4) High volume low cost vs. low volume high cost.

Bartram implies that high volume testing has come with the advent of the Internet. I would suggest that this is not the case but rather it has come with competition. Competition naturally brings down prices. In New Zealand there are many providers, and competition breeds more competition for the good of the marketplace. It destroys artificially higher pricing that exists only in monopoly environments. Low volume is of course a by-product of high cost giving the illusion of exclusivity, which is a mantra that the testing industry has long held on to. However, with the emperor’s clothes off and radically different pricing regimes this mantra is falling on deaf ears and thus the paradigm shifts to high volume.

5) Art or craft and the role of the I/O.

Batram makes the point, and quite rightly IMO, that the industry is becoming less art and more technology. I would rephrase this to say that the industry is becoming more honest and transparent. Personality instruments are simply measurement tools to estimate preferences. The interpretation is, by its very design, colour by numbers to maintain objectivity. The idea that you can develop some mystical skill set to read all in sundry into the profile is nonsense. They are however not ends in themselves but just means to an end. They provide a guide when combined with other data, and the strategic vision of the organisation. They can also be used to make better decisions around such things as selection. It is this later role that the I/O Psychologist fulfils. The tests are just simple mechanisms to aid the I/O Psychologists in this regard, and by their very nature must be measuring similar aspects of personality.

6) What does real change look like?

Real change in this industry is not another personality test or a new delivery mechanism (i.e. the Internet). Change is applications to process data, combining data to increase test effectiveness, computer adaptive testing, new measurement around ‘g’, single item psychometrics, etc. This is the current level of innovation and if we continue to believe the delivery of a test in a different format is innovative then we will never progress.

The largest omission in this paper is that the real driver to the changing standards in the testing industry is not new science but the need for new profit, the catalyst for a new set of ethics. I remember when Internet testing first came out in the mid-late 90’s. Established test producers would not touch it with a barge pole for ‘ethical reasons’. There were public debates against unsupervised Internet testing companies for these very reasons. However once market share is challenged, then ethics have a tendency to be ‘revised’ and a new set of justifications come out as to why it is now all right to test in this manner. This is the reality of this industry and I believe it misleading by omission for Bartram not to emphasise a core driver in this area being the need to meet market demands AND ethics may be seen as secondary to the detriment of the industry. The situation throughout South Africa with companies flaunting rules and then being asked to account for unsupervised testing is a much more real description of how this industry works. This is not to say that unsupervised testing is unethical, just that it contravenes the guidelines of a particular jurisdiction and what appears to be the floating nature of ethics of testing.

In summary, the key points which would be interesting to hear others views on include:

a) The article makes a very valid point that the industry is moving towards a test factory, which involves moving away from Psychometric testing as an art or craft and into a technology that de-skills the practitioner in terms of ‘magically reading reports’. The skill of the practitioner is in APPLYING the data to solve business solutions.

b) It is not the test so much as the data and application of that data that is crucial.

c) The paper omits the role of commercialisation of our industry as a core driver of change. The paper makes it sound like the testing industry by-and-large is driven to break new ground underpinned by non-negotiable ethics and is therefore a paragon of virtue. This is not the case. It is IMO misleading to not discuss in depth the redefinition of ethics that has occurred in testing and the commercial driver for this. In many ways this paper demonstrates for me the reverse engineering disguising as justification that is prevalent in this industry. In the late 90’s when many new test providers came to the market, they often came via the Internet with the idea of non-supervised testing. The incumbent providers then chastised them as acting unethical, thereby destroying competition. Now, of course, it is a different marketplace as the consumer is demanding non-supervised testing, and the practicalities of fast moving industries like recruitment supersede the traditional ethics of the testing industry.

The source of the wonderful quote:

 Sternberg, R. J., & Williams, W. M. (1998). You proved our point better than we did: A reply to our critics. American Psychologist, 53, 576-577.

 “No technology of which we are aware- computers, telecommunications, televisions, and so on- has shown the kind of ideational stagnation that has characterized the testing industry. Why? Because in other industries, those who do not innovate do not survive. In the testing industry, the opposite appears to be the case. Like Rocky I, Rocky II, Rocky III, and so on, the testing industry provides minor cosmetic successive variants of the same product where only the numbers after the names substantially change. These variants survive because psychologists buy the tests and then loyally defend them”.


Please Comment!

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s