IMPROVING THE ACCURACY OF
BIODATA QUESTIONNAIRES. Frank
Kuschnereit and Barry Spinner. University
of New Brunswick.
Biographical information (Biodata) obtained from job applicant forms has been in use since the turn of this century to predict future job success. It has been only relatively recently that there has been a concerted effort to improve Biodata's reliability and validity. Two of the major concerns are how to minimize the incidence of faking and how to reduce inaccurate responses due to social desirability. Theoretical and empirical work have identified several factors which might affect the accuracy of Biodata responses. Using a 2x2 (between) x2x2 (within) design, this study examined the effects of (1) the consequences of inaccurate responses (warning); the amount of detail requested in responses (Detail); (3) the relevance of the item to the job (Relevancy); and (4) whether or not the information could be independently verified (Verifiability). Participants were volunteers registered in Introductory Psychology. It was hypothesized that providing a warning of consequences and requiring greater detail would reduce social desirability of responses, but only for items that were job relevant and potentially verifiable. Results are discussed with reference to the characteristics of desirable Biodata items and the format in which they are presented. Implications for future research are also discussed.
Introduction
There are many costs associated with hiring personnel who subsequently are not successful. These costs can be measured not only in dollar terms but also by their effect on the lives and careers of the persons involved. Many organizations expend a great deal of time and effort to match the right person to the job. There are a variety of personnel selection techniques in common use. The interview is one of the most popular but has been criticized as being too subjective. The assessment centre approach has the applicant complete a battery of selection tests but can be very time consuming and expensive. One technique that is nonsubjective, inexpensive and effective involves obtaining biographical information (Biodata) from job applicants to predict future job success. While resembling the standard job application form, a Biodata questionnaire seeks to obtain more extensive and detailed information on significant events in applicant's lives and especially on their past behaviours. The basis of Biodata's predictive abilities is the theory that past behaviour is the best predictor of future behaviour. Biodata cannot predict all future behaviours but it can give an indication of the types of probable behaviours based on an individual's prior learning history (Mumford & Owens, 1987). Nichols (1994) stated that "Biodata measures may predict performance across so many aspects of behaviour as well as they do because responses to Biodata items may serve to capture previous manifestations of the constructs and mechanisms that ultimately determine predictive relationships with criteria."
While all job application forms collect some form of biographical information and have been in use since the turn of this century, it has been only relatively recently that there has been a concerted effort to improve Biodata's reliability and validity. Two of the major concerns are how to minimize the incidence of faking and how to reduce inaccurate responses due to social desirability.
One recent approach to addressing both of these concerns employs differing the instructions given to the applicants. Kluger and Colella (1993) studied the effects of faking Biodata test items by randomly warning 214 of 429 applicants for a nurse's assistant position against faking. While the warning mitigated the propensity to fake, the specific warning effects depended on item transparency. For transparent items, warning reduced the extremeness of item means and increased item variances. For nontransparent items, warning did not have an effect on item means and reduced item variances. These faking effects were best predicted when transparency was operationalized in terms of item-specific job desirability in addition to the item-general social desirability.
This study examined the instructions given to the role-play applicants to see if a forceful caution could reduce the incidence of faking good. It also examined the hypothesis that asking for additional details in response to multiple-choice type items would increase the accuracy of responses. In addition, the effects of warning conditions and amount of supporting detail requested on faking behaviours were examined in a simulated job application situation using a two by two between and two by two within factorial design as shown below. There were two levels of warning (minimum and maximum warnings). In the maximum warning groups, the subjects were explicitly cautioned that some of their responses will be verified and that any false statements would be grounds for dismissal.
Each item was rated as to the degree it could be perceived as being job related (JR) and verifiable (V). It was expected that the means and correlations would be different for items that are highly job related (HJR) and highly verifiable (HV) as opposed to items that are low (LJR & LV). Within the instrument there was also a "fake" (verifiable) item to further assess faking based on a study by Pannone (1984).
DESIGN
The subjects were 258 Psychology 1000 students. These students
volunteered to participate in return for receiving credit towards their
course participation marks. The participants were asked to role play
as a job applicant responding to a fictional job advertisement.
MATERIALS:
Once the nature of the study was described to the participants and they agreed to take part, each was given a package of materials that included:
A) A fictional job advertisement, indicating the nature of the job and the desired job applicant qualifications.
A note that indicated they had made an initial application for the job, and had been invited by the company to complete the rest of the application process by answering a Biodata form.
The Biodata form, which included 97 items tapping into several aspects of the job applicant's past. Some of the items (High Relevancy) were selected as they related back to desirable characteristics identified in the job advertisement. Others were selected to be less relevant to the job. Similarly, some of the items were selected to be easily verifiable (High Verifiability) while others were not easily verified (Low Verifiability). Item relevancy and verifiability were confirmed by having 48 participants (not in the current study) rate them on each of these dimensions. In addition, as a check on faking, we included a single item which asked participants if they had experience with a (nonexistent) computer software package whose name was fabricated for this study.
Results
Initial Analysis
Univariate outliers (Z>3.0) were examined and adjusted. No multi variate outliers were found. Although, there was no indication from the literature to expect a difference due to sex, an initial analysis was conducted using a series of ANOVAs using sex, warning condition and version as independent variables. All tests were non significant and it was decided to combine the sexes for the main analysis. The individual test items were converted to Z scores and composite means were computed for the four question types - High Verifiability/ High Job Relevance, High Verifiability / Low Job Relevance, Low Verifiability / High Job Relevance and Low Verifiability/Low Job Relevance.
Main Analysis
The SPSS MANOVA procedure was used to examine the 2 X 2 between-subject factors of version type (extended or short version) and warning condition (maximum or minimum warning). It also examined the 2 X 2 within design of item verifiability by job relevance. None of the effects in the design were significant at an alpha of .05.
A separate analysis was conducted for the single item designed to
check faking. While over 20% of the participants indicated they had
experience with this software, there were no differences as a function
of either between-subject factor (warning and extended version).
None of the factors in this design (item verifiability, item-job relevance, the consequences of misreporting, and requesting extended details behind the answers to some items) had a systematic effect on Biodata responses. While it is possible that these conditions have no effect in real-life situations, previous literature suggests that these features should have an effect on responses.
The most likely reason for the absence of significant effects is the nature of the sample and the setting used for the study. It is difficult to determine whether the results would have been different if the participants had been older and more experienced as job seekers. Although there is some indication in the literature that university students can effectively role play as job seekers it makes intuitive sense that first year students are not ideal.
The setting for the role playing should be as realistic as possible. Participants in this study completed the materials in a large group. In retrospect, a smaller classroom could have been set up to better represent a business location rather than have large groups of participants complete the instrument in an amphitheater.
The level of warning failed to have an effect on the results. This could have been an artifact of the warning not being meaningful to the participants. The same could be said for the failure of Relevancy and Verifiability to be significant. The effectiveness of these manipulations relied upon participants "getting into their role", and taking the warnings seriously.
One possible explanation for the lack of differences between conditions is that participants generally responded honestly to all items (therefore conditions designed to reduce faking would have no effect). While this may have happened to some extent (i.e., participants were not motivated enough), it is unlikely that all answers were honest and accurate since over 20% of the participants reported having some experience with a computer program called Intrude 4.0. This was a check item and there is no such program in existence. Clearly the responses must be some form of error but unfortunately the manipulations in the study had no effect on this item as there were no significant differences between the groups.
Unfortunately the absence of significant results prevents any conclusions about the factors investigated in this study. Future research should be orientated towards the factors investigating these factors in a more realistic setting and ideally with participants who are actually applying for a position which they want to obtain.