Hierarchical Bayes Models for Response Time Data
the data constitute the times taken by a subject to react to a succession of stimuli under varying experimental conditions.
Because of the sequential nature of the experiments there are trends (due to learning, fatigue, fluctuations in attentional
state, etc.) and serial dependencies in the data. The data also exhibit extreme observations that can be attributed to lapses,
intrusions from outside the experiment, and errors occurring during the experiment. Any adequate analysis should account for
these features and quantify them accurately. Recognizing that Bayesian hierarchical models are an excellent modeling tool,
we focus on the elaboration of a realistic likelihood for the data and on a careful assessment of the quality of fit that
it provides. We judge quality of fit in terms of the predictive performance of the model. We demonstrate how simple Bayesian
hierarchical models can be built for several RT sequences, differentiating between subject-specific and condition-specific
effects.
- Content Type Journal Article
- DOI 10.1007/s11336-010-9172-6
- Authors
- Peter F. Craigmile, The Ohio State University Department of Statistics 404 Cockins Hall, 1958 Neil Avenue Columbus OH 43210 USA
- Mario Peruggia, The Ohio State University Department of Statistics 404 Cockins Hall, 1958 Neil Avenue Columbus OH 43210 USA
- Trisha Van Zandt, The Ohio State University Department of Psychology Columbus OH 43210 USA
- Journal Psychometrika
- Online ISSN 1860-0980
- Print ISSN 0033-3123
Online Psychometric Test Mini-Course: Lesson 3
Online Psychometric Testing Mini-Course: Session 3
In this session we will explore the following:
1. Why psychometric tests are used and how they are useful. We will do this by referring mainly to alternative methods of assessment.
The short answer to the first part of the above question is that psychometric tests are used because (assuming they are well designed tests) they are a reliable and valid means of assessing people. We will discuss in a future session exactly what is mean by reliability and validity when applied to psychometrics.
Let’s consider a few alternatives to psychometric tests and highlight this issue further.
Unstructured Interviews
Most candidates who apply for a job will expect to have an interview at some stage of the process and indeed, most organisations will work an interview into the process. However, how useful is this interview for predicting performance on the job? This depends a lot on the training of those who will be interviewing. Many people who conduct interviews have never been trained. Perhaps one day a boss asked them to go and interview a candidate for a job and it continued from there. They may have years of experience but experience and competence are not the same. Most people who interview use what is known as the traditional interview. It is also sometimes called an unstructured interview. The idea is that this is a time to meet with and get to know the job applicant. Often the interviewer is thinking things such as:
“Let’s see if he has a firm handshake.“
“Let’s see if he looks me in the eye.”
“I’ll ask him what he does in his spare time.”
The problem is that none of the answers to these questions will predict performance at work. So what if I have a limp handshake? Donald Trump (very successful property tycoon) does not even like to shake hands – he’s worried about germs! Imagine him at a job interview. The shake would be very limp if at all. In some cultures it’s rude to look people in the eye – so we cannot go assuming that those who avoid eye contact will not be good performers or that they are dishonest or hiding something. As for spare time, what about somebody who puts together model cars or aeroplanes on the weekend, does it mean that will be a good designer or engineer. No, this may simply be a low level weekend interest and not something that would keep them entertained as a career. Not to mention the fact that in some parts of the world it’s actually illegal to ask about people’s hobbies in a job selection process!
The point to grasp then is that often the people conducting interviews have little or no training and are running unstructured interviews that have little relevance to job performance and therefore lack both reliability and validity. However, the suggestion is not that we remove interviews totally!
Structured Interviews
Research has shown that interviews have good reliability and validity when run in a particular way by those who have undergone thorough training. These are called structured interviews. The idea here is to align the interview questions to the competencies required of the candidate to be successful in the job. Then the interviewer asks the same or very similar questions to each candidate based on job requirements. Behavioural interviews are one type of structured interview. The questions are designed to elicit a high level of evidence that the candidate has displayed the behaviour associated with competent performance over repeated occasions in the past. Another type of structured interview is Situational interviewing – here the candidate is asked what they would do in certain situations. Situational interviews are generally less valid than Behavioural interviews. The biggest problem with getting HR and Consultants to run structured interviews is the need for training. PsyAsia used to run a 2-day course in behavioural interviewing, but our clients in Asia told us that would require too much time out of the workplace. We thus reduced this to a one-day course (see our behaviour-based interviewing course here if interested) but whilst this satisfies the big decision makers it really only serves as an introduction to interviewing. There needs to be more communication and understanding between HR and those who hold the purse-strings in Asia if we are to increase competence in this area!
Psychometric Tests and Structured Interviews
So thus far, we pointed out that interviews can be reliable and valid but that can only happen if the interviewers have been appropriately trained and where using structured interviews; preferably a behavioural interview. Those using psychometric tools also need to be appropriately trained in order to ensure they remain reliable and valid tools. Assuming training and competence requirements are met for both tests and interviews, why use tests?
Psychometric tests are able to cover a lot more ground in far less time. Aptitude tests give us an indication of numerical, verbal and spatial skills in 18 minutes if using modern tests like the Saville Consulting Aptitude range. There’s no way we could discover this information in even a one-hour interview! Personality assessments can sample and assess personality traits relevant to performance on the job. The average completion time for good personality assessments is 30-40 minutes. There also a few good faster tools available which take around 20 minutes. The amount of information gleaned in this short period of time is a credit to the developers of psychometric tests. However, with particular regard to personality testing, it is necessary to confirm the profile with behavioural evidence from the candidate. So, whilst the profile may suggest somebody who really enjoys multi-tasking, this becomes a basis for an interview question (assuming this is required by the job).
In essence then, psychometric tests are useful because they provide so much more information than an interview can provide in a much shorter period of time. They have been designed by experts using modern statistical techniques aligned with modern personality research and theory. However, psychometric tests are only part of the story and a well designed interview using competent interviewers will add incremental validity to the assessment process. The interview will serve to confirm (or refute) the psychometric profile and provide rich behavioural evidence (that cannot be recorded by psychometric tests) that the person can perform at the level required by the person specification.
Other Methods of Assessment
So far we’ve only looked at different types of interview as an alternative or as complimentary to the assessment process. How about other methods of assessment?
Application forms
We all need to complete one of these to show our intention to apply for a job. Realistically though they are there for this reason alone. They serve as a record of information which the organisation deems important to hold on the individual. Current application forms hold no value as selection tools with the exception perhaps of educational and experiential background. This can be changed by designing application forms that elicit only job relevant responses and preparing a scoring system for the from even before sending it out.
CV/Resume
Candidates like to send their CV/Resume because many people have these on file and it’s easy to quickly update it and print it off on a per-job basis. However, again these are not particularly useful in selection. Research shows that decision-makers are often seduced by smart graphics as well as vocab which sells the applicant by over-inflating their achievements. It’s also possible to lie in a CV, although research has shown that most people don’t lie about their educational qualifications or experience as they know the prospective employer can check up on this. What they do tend to lie about or at least mislead about is their level of competence. We suggest that CVs are not used at any stage of the selection process.
Assessment Centres (ACs)
This is where the candidate is invited to a physical location to partake in a number of exercises with other candidates. Most ACs last a day and during that time the candidates will undergo both group and individual exercises such as presentation exercises, negotiation exercises or in-tray exercises. Assessment Centres have been shown to be highly valid and reliable methods of selection when using well trained assessors.
PsyAsia runs training in Assessment Centres and we also offer consultancy in Assessment Centre Design
References
References lack validity in the assessment process and yet organisations continue to request them! Typically a candidate will not give a potential employer the name of somebody who will give them a poor or perhaps even an honest reference. The tendency is to only offer names of those who they trust will give a great reference. On the other hand, if the current employer really wants the candidate to move on they may fake the reference, making the candidate appear almost angelic! Does this mean we should not use references in the selection process? No. It is possible to improve upon the use of references by designing work–related reference forms that elicit behavioural evidence from the previous employer that is in line with the competency requirements of the new job. However, this may lower the response rate as the referee really needs to think about actual behaviours and write them down rather than sending the standard “he’s a great guy” reference.
Graphology
Most organisations aren’t into this, but an alarmingly high percentage of French organisations are! The idea here is that various personality traits can be seen via somebody’s handwriting. Those traits can then be linked to performance at work. So for somebody that writes with very bold strokes, the graphologist may say they are ambitious. This would be good for a salesperson. However, research has shown a lack of reliability in this method. Not only do people write differently depending on their mood, their culture, their upbringing and so on, but graphologists given the same handwriting to analyse often do not agree with each other about the personality traits of the writer! Graphology thus should not be used as a selection tool.
Phrenology
Phrenologists assume that different aspects of personality are stored in different parts of the brain and that where somebody has more of a particular characteristic, the corresponding part of the brain will be larger and hence cause protrusions on the head! The idea would be that you measure different bumps and indentations on your candidates and then project their personality from that. Of course, this method holds no validity and brain imaging tools such as fMRI and PET scans have refuted it.
Astrology
In Asia, people use astrology to help them decide auspicious dates for business openings, functions, weddings and so on. Does it work for job applicants? No! The idea that people born at the same time, in the same place, where the alignment of stars and planets are similar will work in the same way does not hold any weight. Don’t hire employees based on their star signs!
Psychometric Tests and other Selection Methods
As you can see, there are many ways we can assess people. However each method varies in terms of reliability and validity. Assessment Centres hold very high reliability and validity if done properly, but they are expensive, require lots of resources and skills to run and only assess 6-12 people at a time. We’ve already said that structured interviews are good but again, they take time and resources. Psychometric tools do cost money. However the cost is offset by the number of candidates that can be assessed and the information that can be gathered in the assessment compared to other selection methods. Don’t forget, an interviewer’s time is costly. A panel interview with 3 interviewers is likely to cost around 2-3 times the fee of a psychometric test and yet will not gather as much information. Not to mention the fact that if you are using the right psychometric tool, it’s reliability and validity will already have been assessed and will be good. Whereas we tend to assume that interviews will be reliable and valid if run by trained people – this is rarely tested!
Psychometric Tests for development, coaching, careers advice and team-building
This lesson has focussed on the use of psychometric tests in candidate selection. However, much of what has been raised applies to the use of tests in other scenarios. For example, in careers advice, psychometric tools allow the counsellor to offer advice which is based on a systematic assessment of the individual’s aptitude and personality alongside the information already on file such as achievements thus far, previous experience, educational qualifications and so forth. In coaching, development and team-building, psychometric tools often serve as a reliable and valid basis for the discussion. Not using these tools means the initiator starts off with far less information and is likely to be less systematic. Psychometrics enables the initiator to work from a validated model and a holistic assessment of the people being developed and not to base interventions and advice on subjective insights.
Interested in learning more about psychometric testing for HRM? Keep reading – your next free session is not far away! To ensure you don’t miss a single instalment, we suggest you follow-us on twitter as each new post will be announced there. You may also like to join our face-to-face psychometric training courses in Singapore or Hong Kong – these range from simple introductory courses through to Certification Courses such as the BPS Level A and BPS Level B Certificates of Competence in Occupational Testing. Not in Singapore or Hong Kong? No problem – we also offer both recorded and live online training in psychometrics! For full details please see here or email us.
DO NOT COPY OR SAVE THIS ARTICLE TO YOUR COMPUTER.
THIS ARTICLE IS CLEARED FOR PUBLISHING ON PSYCHOLOGY1 GROUP SITES ONLY. IT REMAINS COPYRIGHT AND INTELLECTUAL PROPERTY OF PSYASIA INTERNATIONAL PTE. LTD. YOU ARE NOT AUTHORIZED TO PUBLISH IT ON ANY OTHER SITE. YOU ARE NOT PERMITTED TO COPY/PASTE THIS ARTICLE OR TO SAVE IT TO YOUR LOCAL DRIVE. YOU ARE ONLY PERMITTED TO READ IT ONLINE AT OUR WEBSITE. VIOLATION OF THESE TERMS WILL RESULT IN BANNING OF OFFENDING IPS AND LEGAL ACTION FOR THOSE WHO REPUBLISH THIS ARTICLE WHETHER IT BE WITH OR WITHOUT A REFERENCE TO THE ORIGINAL AUTHOR.
Free Access to Psychometric Society Journal until 31 July
| Psychometrika – Journal of Psychometrics Society Free Trial Access until July 31, 2010! |
| From now until July 31, 2010, please enjoy free access to the journal. Click here to begin your reading. |
S. Guo & M.W. Fraser (2010). Propensity Score Analysis: Statistical Methods and Applications.
S. Guo & M.W. Fraser (2010). Propensity Score Analysis: Statistical Methods and Applications.
- Content Type Journal Article
- Category Book Review
- DOI 10.1007/s11336-010-9170-8
- Authors
- Peter M. Steiner, Northwestern University Evanston USA
- Journal Psychometrika
- Online ISSN 1860-0980
- Print ISSN 0033-3123
B. S. Everitt (2009) Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences.
B. S. Everitt (2009) Multivariable Modeling and Multivariate Analysis for the Behavioral Sciences.
- Content Type Journal Article
- Category Book Review
- DOI 10.1007/s11336-010-9173-5
- Authors
- Paul M. W. Hackett, Emerson College Boston MA 02116-4624 USA
- Journal Psychometrika
- Online ISSN 1860-0980
- Print ISSN 0033-3123
New Equating Methods and Their Relationships with Levine Observed Score Linear Equating Under the Kernel Equating Framework
assumption of the classical test theory model, that we name curvilinear Levine observed score equating. In fact, by applying
both the kernel equating framework and the mean preserving linear transformation of post-stratification equating, we obtain
a family of observed score equipercentile equating functions, which also includes the classical Levine observed score linear equating and the Tucker linear equating
as special cases.
- Content Type Journal Article
- DOI 10.1007/s11336-010-9171-7
- Authors
- Haiwen Chen, ETS 666 Rosedale Rd. MS 02-T Princeton NJ 08541 USA
- Paul Holland, ETS 666 Rosedale Rd. MS 02-T Princeton NJ 08541 USA
- Journal Psychometrika
- Online ISSN 1860-0980
- Print ISSN 0033-3123
Online Psychometric Test Training – Free 10 Week Online Mini-Course Starts Soon
PsyAsia International is pleased to announce a free, 10-week mini-course in psychometric testing. The course is mainly text based and introduce the basics of psychometrics in 10 weekly articles. You’ll learn about what psychometrics is, where it came from, how it is useful, discrimination, reliability, validity, error and more. PsyAsia is Asia’s leading independent provider of psychometric tests and psychometric training. The course however will be invaluable to readers anywhere. It will be interesting for those in HRM or consulting roles using or considering psychometric tests as well as for those about to undergo testing.
To ensure you don’t miss a single instalment, we suggest you follow-us on twitter as each new post will be announced there.
Interested in learning more about psychometric testing for HRM? Keep reading – your first free session is only a week away! You may also like to join our face-to-face psychometric training courses in Singapore or Hong Kong - these range from simple introductory courses through to Certification Courses such as the BPS Level A and BPS Level B Certificates of Competence in Occupational Testing. Not in Singapore or Hong Kong? No problem – we also offer both recorded and live online training in psychometrics! For full details please see here or email us.
Nested Logit Models for Multiple-Choice Item Response Data
suggested to provide a better approximation to multiple-choice items where the application of a solution strategy precedes
consideration of response options. In practice, the models also accommodate collapsibility across all distractor categories,
making it easier to allow decisions about including distractor information to occur on an item-by-item or application-by-application
basis without altering the statistical form of the correct response curves. Marginal maximum likelihood estimation algorithms
for the models are presented along with simulation and real data analyses.
- Content Type Journal Article
- DOI 10.1007/s11336-010-9163-7
- Authors
- Youngsuk Suh, University of Texas at Austin Department of Educational Psychology 1 University Station D5800 Austin TX 78712 USA
- Daniel M. Bolt, University of Wisconsin-Madison Madison WI USA
- Journal Psychometrika
- Online ISSN 1860-0980
- Print ISSN 0033-3123
Bayesian Analysis of Multivariate Probit Models with Surrogate Outcome Data
to model and analyze ordinal data. A general model structure is assumed to accommodate the information that is obtained via
surrogate variables. A hybrid Gibbs sampler is developed to estimate the model parameters. To obtain a rapidly converged algorithm,
the parameter expansion technique is applied to the correlation structure of the multivariate probit models. The proposed
model and method of analysis are demonstrated with real data examples and simulation studies.
- Content Type Journal Article
- DOI 10.1007/s11336-010-9164-6
- Authors
- Wai-Yin Poon, The Chinese University of Hong Kong Department of Statistics Shatin Hong Kong China
- Hai-Bin Wang, Xiamen University School of Mathematical Sciences Xiamen 361005 China
- Journal Psychometrika
- Online ISSN 1860-0980
- Print ISSN 0033-3123
P. Sprent & N.C. Smeeton (2007). Applied Nonparametric Statistical Methods (4th ed.).
P. Sprent & N.C. Smeeton (2007). Applied Nonparametric Statistical Methods (4th ed.).
- Content Type Journal Article
- Category BOOK REVIEW
- DOI 10.1007/s11336-010-9166-4
- Authors
- Laura M. Schultz, Rowan University Department of Mathematics 201 Mullica Hill Road Glassboro NJ 08028 USA
- Journal Psychometrika
- Online ISSN 1860-0980
- Print ISSN 0033-3123
