Why do People Recommend Ipsative Tools for Selection if They Are Not Designed for that Purpose?

Last week we discussed the basics of ipsative testing. This week my blog will look at why ipsative testing has been used for selection and the real driver of ipsative testing: marketing.

Why do people recommend ipsative tools for selection if they are not recommended for that purpose?

There are two reasons that people recommend ipsative measures for selection. The first is a mis-belief that they are less resistant to faking and therefore produce more valid results. The second is that marketing is fundamentally about having a point of difference; This ultimately means that companies selling assessments will present any number of different assessment solutions so long as they can create a story around it.

Faking

 As is often the case, I quote from work by Paul Barrett in response to issues of faking: “The widespread use of ipsative measurement came about as a response to rater bias exhibited in questionnaire personnel ratings. Travers (1951) credits Paul Horst with first proposing the idea of forced-choice format to counter ‘leniency’ and other errors in rating. The forced-choice method involves the presentation of items that have been matched for preference value (e.g. social desirability) yet discriminate differentially on a set criterion, such as leadership quality for a specific task (Gordon, 1951). However, two important assumptions underlie this approach. Firstly, all the choices must be as high in apparent validity as each other, and secondly, the ratees will, on average, ascribe equal status to the irrelevant qualities (Guildford, 1954)”.

Hammond and Barrett then go on to show that the ipsative tools rather than getting around social desirability have in build and systematic problems of social desirability in responding. People will still attempt to respond in ways in which they are motivated to. However with ipsative testing because the scales are interdependent it is much harder to detect social desirability and differentiate it from situational difference. Thus, ipsative testing is not so much more difficult to fake but rather is more difficult to detect making its use in selection more not less problematic.

Marketing

The key reason that ipsative testing has been promoted by a few assessment companies mainly around marketing purposes. Personality testing has far less validity than other forms of testing, such as cognitive ability, despite its common use. With the plethora of assessment companies, the need for market differentiation increases. Couple this with issues around faking in personality testing and you have a clear market driver for ipsative testing.

The key is understanding that this market driver runs counter to the supporting evidence for ipsative testing. Thus, companies who flaunt ipsative testing clearly show a commitment not to psychometric rigor but gaining market share. One cannot have things another way as to promote the use of ipsative testing requires a deliberate ignorance of the independent supporting theory and science related to ipsative assessment.

A more pressing question is how the market place ever got conned into the belief that ipsative testing is indeed valid for selection. The answer lies with whom the control of the message resided and who is the paradigm setting agency. As competition has increased, counter views and ‘independent best-practice’ have been brought to bear in countries like New Zealand where ipsative testing for selection is now rare. The question still remains as to why even the New Zealand market was not told about the issue related to ipsative measurement before and why (in my experience) were so many convinced that the ‘best-practice’ was the other way around (i.e. ipsative for selection and normative for development)?

The resolution is of course professional ethics in that if we as psychologists know there is no issue with normative tools for between person comparisons than our use of ipsative testing will be minimised. As professionals we must therefore not fall folly to marketing diatribe but as always base our decisions on independent science and theory.

Advertisements

10 thoughts on “Why do People Recommend Ipsative Tools for Selection if They Are Not Designed for that Purpose?

  1. wahyupsy

    Dear Dr. Paul.
    I read some articles that recently, company seek employee who possess certain type of personality which suit to job requirement. As I know, personality testing that produce typology of personality is ipsative testing. For example MBTI typology: extrovert-introvert; sensing-intuition; thinking-feeling. Some job require employee who extrovert, intuition and feeling, other job require another types. Is the test like this called ipsative testing? Thank you.

    Reply
    1. Dr Paul Post author

      Thank you for your question. Ipsative tests are not defined by the output (i.e. bi polar scales such as extrovert-introvert). Rather the tests are defined by the way the questions are asked. Ipsative tests have forced choice questions. A respondent is asked to choose between (usually 4 choices) two alternatives (usually an item that is most like them and an item that is least like them). This can be seen in contrast to questionnaires which ask a respondent to rate themselves (say from strongly agree to strongly disagree).

      Both tests may measure the same constructs (say introversion and extroversion) but they will do this in a different way. With an ipsative test your score on one trait is not independent from your score on another and in there lies one of the main problems. For other issues please read the previous posts on ipsative tests in this blog as they provide more detail and references.

      Reply
  2. Joey Wang

    Dear Dr.Paul.
    I have just read this articles and I think it is really helpful for me understanding why some company selling ipsative test.
    Actually I am a student in Asian studying psychology.
    And because I am not quite know about how ipsaitve socore is caculated,
    would you please give me some brief introduction or some links to the articles introducing it?

    Thank you very much and best regards,

    Reply
    1. Dr Paul Post author

      Hi Joey

      Great to see you reading the blogs. All the key references for ipsative tests are in the blogs so that should steer you in the right direction.

      Reply
  3. Tom

    Dr Paul,
    Just finding your blog and enjoyed your posts on ipsative assessments.

    I first was subject of a hiring assessment in the late 1990’s and carried that tool with me to other companies and now utilize it as part of my consulting practice.

    I do not see that you addressed the available options for people interested in conducting hiring or other comparative assessments…ie. Normative Assessments such as The Prevue.

    Reply
    1. Dr Paul Englert Post author

      Hi Tom

      Thanks for the comment. There are a range of normative tools that can be used for selection and I have not gone into these in the posts. The key is that a normative tool is designed to make between person judgments so is suitable for selection (so long as it meets other psychometric standards and is related to the job).

      So in short you are quite right. There are a range of normative tools that can be used in replace of ipsative measures.

      All the best
      Paul

      Reply
  4. Bob Parker

    Ipsative tests ARE approved for selection. It’s a complete myth that they are not. Well constructed (NOT MBTI) ipsative tests are fantastic for making comparisons between people. I do it every day in my practice with a Job Benchmark. Ipsative tests are very difficult to game. Normative tests are quite easy to game. Every claim you make against ipsative testing has been proven false by TTI Success Insights, the over 100,000 companies that use their ipsative tests, and the EEOC compliant process they have in place.

    Amazing that there’s so much ignorance.

    Reply
    1. Dr Paul Englert Post author

      Dear Mr Parker

      Thank you for your response to the blog. We appreciate all commentary, even those that disagree with us.

      In response to your comment it is important to note that the critique is not test specific. The critique is one of fundamental design issues based on the properties of ipsative tests in contrast to normative tools. The argument is as much based on the logic of the design as the measurement properties.

      Secondly I agree that there have been advancements in this space since these blogs were written. The work on the OPQr is a sound and a committed attempt to tackle this problem. Likewise CAT personality tools like TAPAS are clearly extremely well developed and have potential to further develop the industry (Conference Presentation). The central issue is one of between versus within person measurement, which these tests, I agree, are tackling.

      This noted these are but a subset of the ipsative tools on the market. Most do not have this development background and therefore the critiques noted remain applicable and relevant to most.

      Thirdly, the critique is one based on research; not the marketing hype that is so prevalent in this industry. The research continues to point to the problems of ipsative tools. While much of this research is noted in this series of blogs (Ipsative Tests: Psychometric Properties) you may also wish to review Closs S.J. (2011). On the factoring and interpretation of ipsative data, Journal of Occupational and Organizational Psychology, 69 (1), 41–47.Article

      I contrast to this research based approach to discussion I note your own reply that appears to provide as evidence that you “do it everyday” and that “100,000 companies use their ipsative tools”. In fairness I went to your site in response to your critique. I clicked on the topic ‘validity’ http://www.ttisuccessinsights.com/how/validity which then appeared to focus on an interesting, albeit misplaced, run down on the reliability of your tests. While I admit to only having a cursory overview I could not see as to how you, or the company you mention, have tackled the ipsative issue. The marketing was nice though.

      I would be more than happy to review any peer reviewed literature you have on the topic of ipsative measurement for between person judgement and any peer-reviewed research that outlines the process you have taken to solve this conundrum. Case studies, marketing hype, research on other topics, numbers of people tested, pretty graphs, years in the industry, trust me and any other spurious evidence I unfortunately don’t have time for.

      Thanks again for participating and if you wish to personally email the peer reviewed evidence base for your ipsative measure I will happily look at it.

      All the best
      Paul

      P.S. On the issue of faking and ipsative tools I suggest a good read of Hammond & Barrett, (1995). The Psychometric and Practical Implications of the use of Ipsative, forced-choice format, Questionnaires.

      Reply
  5. Ryan

    You seem rather dogmatic in the assertion that ipsative formats are fundamentally broken, and provide no empirical evidence for this. Meanwhile, there is good research indicating that ipsative formats are more resilient to faking in high-stakes contexts and more useful for predicting job performance, e.g. Hirsh, Peterson 2008.

    Ipsative formats may have their problems, but normative formats may well be far worse.

    Reply
    1. Dr Paul Englert Post author

      Hi Ryan

      Thank you for participating in the blog, this is appreciated. You note that there are no references provided. This is not the case and you may have missed the list in this blog: https://oprablog.wordpress.com/2010/10/12/great-blog-but-what-are-ipsative-tests/. For your benefit I note this below:

      Cattell, R.B. (1944) Psychological Measurement: ipsative, normative, and interactive. Psychological Review, 51, 292-303.

      Meade, A.W. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology (2004), 77, 531–552

      Hammond, S., and Barrett, P., (1995). The Psychometric and Practical Implications of the use of Ipsative, forced-choice format, Questionnaires.

      Hugh, L.M., and Ones. D. S., (2001). The structure, measurement, validity and use of Personality variables in Industrial, Work and Organisational Psychology, pg. 233.

      Since this time there have been further papers that could be cited. Two that you might find interesting are Goffin, R.D., Jang, I., & Skinner, E. (2011) Forced-choice and conventional personality assessment: Each may have unique value in pre-employment testing. Personality and Individual Differences, 51, 7, 840-844; Salgado, J.F., Anderson, N., & Tauriz, G. (2014). The validity of ipsative and quasi-ipsative forced-choice personality inventories for different occupational groups: A comprehensive meta-analysis. Journal of Occupational and Organizational Psychology, (2015) 797-834

      Having addressed the assertion that there are no references, let me move on to your second claim that I’m dogmatic on ipsative formats. On the contrary, I note, that my issue is the difference between within and between person measurement and the application of the two. I state clearly that ipsative measurement has its place in the former and is problematic, by design, for the latter. “In Summary, ipsative testing is very applicable when working with an individual. Ipsative testing is inappropriate when used for selection”. I note at this time that these are the same guidelines that you will find in any elementary textbook on testing and have been recently endorsed by the BPS (assessment matters (2015) Vol 7, (4).

      Next, we come to the paper by Hirsh and Peterson (2008). The fact that they call their test ‘ A Fake Proof’ measure is perhaps all that anyone needs to know about how far they may be bending the truth in the promotion of the tool. These are the exact claims that I’m fighting. They are obviously excessive and have no place in science. The paper in no way convinces me that ipsative tests are useful for between person comparisons. The paper does not address the fundamental issues with respect to this. What the paper assess is how predicative validity is affected under fake good conditions. The naturally corollary, shown by later papers, is this predictive validity in a faking condition is simply accounted for by ‘g’ especially in cases with students samples.

      This brings me to your final point, “Ipsative formats may have their problems, but normative formats may well be far worse”. I disagree for all the reasons that I have outlined in the blogs. Ipsative tests, by design, are not meant for between person comparison. The problems with their interpretation, factor structure, and reliability, when used in this setting, make their problems far greater. The International guidelines mean that that those that use them in this manner have a hurdle to defend if challenged. The issue of faking is a moot point, as both are susceptible for faking (any employee who wants to try and fake can as shown in the paper you quote). However, under ipsative conditions, it is harder to track. The push for ipsative formats for selection invariably comes from those who have a test to sell. Claims that anything is ‘fake proof’ is a nonsense and the industry must be wise to this.

      Finally to reiterate my point early and points made in other responses. I see great value in ipsative measure for counselling. Moreover, much of the work that is going on with the likes of the OPQi I see as valuable and would concur that the team behind that instrument are addressing many of the issues for selection. But this same science is not behind most ipsative tools and the claims made are done so for commercial reasons. Thanks again.

      Reply

Please Comment!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s