Computer Adaptive Testing (CAT) and Ipsative Personality Assessments

There appears to be a resurgence of ipsative testing under the veil of CAT (Computer Adaptive Testing) for personality.  Clearly there is a lot more quality research that must go on to develop a CAT for personality than the simple ipsative tests the market has seen before. In regards to CAT testing, the development of item banks where items are calibrated against a latent variable for that attribute, in terms of “how much” attribute it contains is no small task. In addition, CAT testing will clearly speed up the time to complete the test as fewer items are used to make an estimation of personality.

However the problems of ipsative testing remain in that, again, we have a within-person measure used to make comparisons across people. Moreover, the problem is potentially compounded with CAT testing as the actual construct (e.g. trait being measured) is defined by a much smaller sub set of items. Both these issues should raise major concerns for test users.

There is no evidence of increased validity ( cf. Cherneyshenko, O.S., Stark, S., Drasgow, F., & Roberts, B.W. (2007) Constructing personality scales under the assumption of an Ideal Point response process: toward increasing the flexibility of personality measures. Psychological Assessment, 19, 1, 88-106) and ironically the construct validity is achieved by comparing the results with a standard measure and showing comparability. With these two points in mind, the basis for CAT personality testing is spurious and likely to be underpinned more by a marketing ploy than a commitment to furthering the I/O psychology discipline.

The argument that the value of the new models is seen in terms of administration flexibility and technology rather than in terms of increased predictive validity is worrisome. What this does is send a message to the market that personality can be used with high degree accuracy for screening. We know that this is not the case. While screening out at lower levels has merit the reality remains that personality tests have low criterion validity. The constructs are far more complex from a measurement perspective than ‘g’. Thus, to present a ‘new wave’ of assessments in this regard is very very dangerous as it removes the developmental and coaching component inherent in personality testing.

Arguments around faking are also spurious. As shown in the white paper by Hammond and Barrett (1995): The Psychometric and Practical Implications of the use of Ipsative, forced-choice format, Questionnaires, as within-person measures have in-built bias for faking. All tests scores contain error and with personality testing error is contributed to by motivated distortion. This is not solved by ipsative testing but rather quite the opposite in that error and distortion becomes harder to detect.

The I/O industry must critique these new innovations on the basis of the science that underpins them. The testing industry is highly competitive and as a result everyone is looking for a marketing edge. That is what these things are and frankly, in an industry based on science it demonstrates how market drivers do not necessarily led to advancement. So long as practitioners still use such questionnaires to measure personality then the issues will remain. These are self-report measures of behaviour and the similarities between tests will always outweigh their differences. Test providers know this and use this as the very basis for the constructs they are measuring. In this regard any attempts at innovation must be seen in light of the ‘marketing ploy’ it is attempting to deliver.

I will be proud to congratulate real innovation in our field. CAT testing for narrow constructs like ‘g’ makes real sense and has been around for years. In personality the real interesting work is looking for biological correlates of personality models and other truly revolutionary ways of understanding the personality space of human behaviour. Marketing under the guise of innovation has under-pinned I/O psychology since the advent of the commercialisation of research ideas in this space. Let us never confuse innovation with marketing in this discipline anymore and hopefully the discipline will finally make some real advancement on the understanding of performance.

Advertisements

2 thoughts on “Computer Adaptive Testing (CAT) and Ipsative Personality Assessments

  1. Me

    I think you are confusing things. It’s not because a couple articles used ipsative testing to validate a CAT system that CAT systems are primarily validated in this way.

    Reply
    1. Dr Paul Post author

      Thank you for engaging in the forum. I agree that CAT testing is not primarily validated in this way and I’m a little surprised that this came across. Validation and construction are different and the point of the article is that ipsative methods are being using in the construction of CAT personality testing and the same concerns over this method of measurement apply whether CAT or not.

      Reply

Please Comment!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s