Attacking the Myth That Personality Tests Don’t Add Value

For this month’s myth, the last on the topic of psychometrics, I have chosen a slightly different approach. I’m coming out in defence of personality tools; when they are used correctly and understood in the right context. Rather than reinvent the wheel in this regard, I have chosen to highlight what I believe to be a very reasoned article on the topic in Forbes by Tomas Chamorro-Premuzic.

Before posting a direct link to the article, I want to set the context for the value in psychometrics. For me, it is based on 5 key points:

 

  1. Predicting human behaviour is difficult: The psychometric industry often over steps the mark with the levels of prediction it claims to have. Assessments are not crystal balls and the search for the greatest predictive tool which is easily generalizable across multiple contexts is futile. The corollary, however, is not true that due to human complexity assessments have no application and understanding a little more about a person’s behavioural preference, and understanding a framework of personality, has no value. On the contrary, it is valuable for the very reason that human beings are complex and more information on individual differences and frameworks to help us conceptualise behavioural patterns does add value to the people decisions we need to make. Psychometric tools provide a framework for understanding personality and provides a simple, relative measurement model to assist decision-making..

 

  1. Human beings have free-will: It never ceases to amaze me when I meet people who are sycophantic with respect to their devotion to a particular assessment tool. It is as if they choose to ignore the concept of free-will. Behaviour will inevitably change across situations and with different reinforces, and this is so inherent that it needs no further explanation. What psychometric tools can do however is estimate the likelihood of behavioural change and the preference for behaviour. The assessment does not supersede free-will but rather helps us to understand how free-will be displayed a little bit better.

 

  1. Lying, or distortion, is a problem for any assessment method: Lying is something humans often do! A common argument against personality tools is that people may present themselves in an overly positive light. It should be noted that the same criticism can reside in any assessment methodology, from interviews to CVs.  It affects many dimensions of life, from employment to those hoping to meet Mr or Ms Right via an online dating site. Quality personality tools attempt to mitigate this issue with response style indicators such as measures of social desirability, central tendency and infrequency.

 

  1. Behaviour is an indication between the situation and preference: Much like the comment on free-will, the situation should never be ignored when attempting to understand behaviour. Personality tests provide us with part of the puzzle, and in doing so they help us understand how someone is likely to behave. The keyword in that sentence is ‘likely’, and how ‘likely’ depends on the strength of the behavioural preference and the situation.

 

  1. Personality assessments are a simple, coherent and quick method for shedding light on human complexity: The bulk of personality tools are used for recruitment. When recruiting a person, we need to make an expensive decision on limited information and in a short timeframe. This necessitates the need to look at all the feasible ways of making an informed judgement. At its most basic, the instrument is: a collection of items that have been clustered along psychometric principles, resulting in a degree of reliability over time and internal consistency thus giving meaning to a wider trait. A person’s responses are then compared to others who have taken the tests. Assuming the norm is relevant and up-to-date, and with spread, it gives us an indication of the person’s relative behavioural preference against a comparison group of interest. The information is used to make inferences on likely behaviour together with other information collected. That is the sum total of the process. For argument’s sake, the alternative would be to say that human behaviour is all too complex and we should operate without asking any questions at all. That is equally untenable.

 

The problem is not that personality tests have no value, but that practitioners over estimate their value and predictive power. Psychometric test providers may also confuse the issue by over promoting their assessment, marketing their uniqueness; and extolling the magical powers of their special item set. When understood in the right context, personality assessment can add value. When used as part of a complete system, interlinking recruitment to training to performance management, a deeper understanding of how personality impacts company performance can result. I agree that there are some tools that do not meet minimum psychometric standards and as such their usefulness is limited, but for those assessments that attempt to simply ‘do as they say on the tin’ the problem lies not with the assessment but the practice of the users and unrealistic expectations.

 

I strongly encourage you to read this short piece on the seven common, but irrational, reasons for hating personality tests: http://www.forbes.com/sites/tomaspremuzic/2014/05/12/seven-common-but-irrational-reasons-for-hating-personality-tests/

6 thoughts on “Attacking the Myth That Personality Tests Don’t Add Value

  1. Andrew Munro

    Paul, another of your typically great pieces – and much to agree with.
    But I think, like Ones a while back, Tomas under-estimates the impact of faking – its prevalence and impact on selection decision making.

    Richard Griffith has a decent review in the 2013 Handbook of Personality at Work, where his studies (quite clever) estimate that around 30% of applicants fake, accepting that faking comes in a few different guises. But in a way that affects how selection decisions are made.
    And perhaps slightly worryingly the successful fakers (ie who go undetected) may be the more Machiavellian characters. See the piece by Robert Tett in the same Handbook. “Applicants who are smart, truly lower on desirable and higher on Machiavellian traits pose the biggest challenge to faking researchers.”

    And another piece by Richard Griffith that is worth a look:

    Click to access Tryba%20&%20Griffith_paper_2013.pdf

    Reply
    1. Dr Paul Englert Post author

      Hi Andrew

      Thank you for the kind comments. I agree that the prevalence of faking is potentially higher than is often cited. This is supported by other recent research:

      Zvonimir, G., Zeljko, J., & Maja, P.K. (2012) Do applicants fake their personality questionnaire responses and how successful are their attempts? A case of military pilot cadet selection. International Journal of Selection and Assessment, 20, 2, 229-241.
      Abstract
      This paper presents the results of three interrelated studies investigating the occurrence of response distortion on personality questionnaires within selection and the success of applicants in faking situations. In Study 1, comparison of the Big Five personality scores obtained from applicants in a military pilot cadet selection procedure with participants responding honestly, faking good, and faking an ideal candidate revealed that applicants responded more desirable than participants responding honestly but less desirable than respondents under fake instructions. The occurrence of faking within the military pilot selection process was replicated in Study 2 using the Eysenck Personality Questionnaire and another comparison group. Finally, in Study 3, comparison of personality profiles obtained in selection and ‘fake job’ situations with experts’ estimates indicated that participants were partially successful in faking the desirable profile.

      I don’t think that test publishers have necessarily approached this head on or have deliberately distorted the truth on this issue. A classic example of this is the attempts by test publishers to encourage ipsative tools for selection over the years despite the inherent limitations of such tools for that purpose. I have written extensively on this topic in previous posts:

      Ipsative Tests: Psychometric Properties

      Great Blog; But What are Ipsative Tests?

      Computer Adaptive Testing (CAT) and Ipsative Personality Assessments

      I don’t think the testing community has completely got it’s head around how faking impacts selection decisions.An interesting application of ‘faking’ and the application to selection is the overt integrity measures such as the Stanton http://www.opragroup.com/images/opra/pdf/downloads/TechnicalManuals/ssi.pdf. These measures accept that faking is an issue and it is the relative nature of the faking that is at issue.

      This is similar to the thinking of Hogan on the issue as to how problematic faking may actually be: https://oprablog.wordpress.com/2010/09/21/faking-and-its-implication-for-the-use-of-personality-tests/

      Drawing this together we need to look at what is going through the respondents mind when taking an assessment and there is some recent interesting work in this regard:

      Kleinmann, M., Ingold, P.V., Lievens, F., Jansen, A., Melchers, K.G., & König, C.J. (2011) A different look at why selection procedures work : The role of candidates’ ability to identify criteria. Organizational Psychology Review, 1, 2, 128-146.

      Klehe, U., Kleinmann, M., Hartstein, T., Melchers, K.G., König, C.J., Heslin, P.A., & Lievens, F. (2012) Responding to personality tests in a selection context: The role of the ability to identify criteria and the ideal-employee factor. Human Performance, 25, 4, 273-302.
      Abstract
      Personality assessments are often distorted during personnel selection, resulting in a common “ideal-employee factor” (IEF) underlying ratings of theoretically unrelated constructs. However, this seems not to affect the personality measures’ criterion-related validity. The current study attempts to explain this set of findings by combining the literature on response distortion with the ones on cognitive schemata and on candidates’ ability to identify criteria (ATIC). During a simulated selection process, 149 participants filled out Big Five personality measures and participated in several high- and low-fidelity work simulations to estimate their managerial performance. Structural equation modeling showed that the IEF presents an indicator of response distortion and that ATIC accounted for variance between the IEF and performance during the work simulations, even after controlling for self-monitoring and general mental ability.

      König, C.J., Merz, A., & Trauffer, N. (2012) What is in applicants’ minds when they fill out a personality test? Insights from a qualitative study. International Journal of Selection and Assessment, 20, 4, 442-452.
      Abstract
      Many practitioners fear that applicants will fake if they are asked to fill out a personality test. Although this fear has inspired much research, it remains unknown what applicants think when they fill out a questionnaire. Thus, we conducted a qualitative interview study that was guided by grounded theory principles. We interviewed (a) real applicants directly after filling out a personality test; (b) real applicants who had filled out a personality test in their past job hunt; (c) hypothetical job applicants whom we asked to imagine being an applicant and to fill out a personality test; and (d) hypothetical applicants who had much experience with personality tests. Theoretical saturation was achieved after interviewing 23 people. A content analysis showed that much is going on in applicants’ minds – that which is typically subsumed under the expression ‘faking’ actually consists of many facets. In particular, participants assumed that the interpretation of their responses could be based on (a) the consistency of their responses; (b) the endorsement of middle versus extreme answers; and (c) a certain profile, and these assumptions resulted in corresponding self-presentation strategies. However, these strategies were not used by all participants. Some answered honestly, for different reasons ranging from honesty as a personality trait to the (false) belief that test administrators can catch fakers. All in all, this study questions whether measuring mean changes in classical faking studies captures all important facets.

      With all this in mind I think that there are possible two take home points for the practioner:
      1. That personality testing is not immune to faking. This should be accepted rather than denied.
      2. When using the tools for screening then you are screening out rather than screening in. Yes you will have some false negatives but if you have a large enough selection ration this is less problematic as you can only hire a certain number at the end of the day. Ultimately though any screening will need to be done with caution, and using multiple hurdles prior. If personality data is used as part of a multiple hurdle this should only be employed when the size of the candidate pool and time make this a necessity. The cut score must be sufficiently low to exclude standard Error of Measurement of the scale used. Finally this is always a last resort when other criteria (bio-data, cogs) do not result in a small enough candidate pool.

      Reply
  2. Pingback: The Myth that Training is an Art not a Science | OPRA's Learning Blog

  3. Pingback: Tips to spot a myth | OPRA's Learning Blog

  4. Pingback: Tips to Spot a myth | Paul Englert

  5. Pingback: Tips to Spot a Myth – Dr. Paul Englert

Please Comment!