1. Introduction
Criterion or ‘concrete’ validity expresses the extent to which a measurable entity may be related to an outcome (or result). For example, in a study of potential benefits arising from application of a simple, valid tool derived for assessing broadly the corporative positive wellness climate, particularly due to climate contribution to employee wellbeing, it was shown that a confirmatory factor analysis indicated that a 9-item measure has good model fit (RMSEA = 0.06, CFI = 0.91), with an inter-item consistency of 0.74, and a mean Rwg(j) of 0.87, with the new instrument demonstrating a significantly positively correlation with physical health and wellbeing, and a negative correlation with substance use behavior [1], implying that the 9-item instrument possessed good reliability, construct, and criterion validity. Although concurrent and predictive criterion validity share similarities, the former refers to a comparison between the measurement that is under observation and an outcome that is estimated concurrently. The latter is assessed typically through comparison with a gold standard test.
In a replicative study based upon executives’ estimations of work performance at recruitment, using the JobMatchTalent (JMT) database [2,3] administrators’ estimations of employees’ performance and ‘work-ability’, as assessed by the JMT recruitment instrument [4]. The study involved 258 recruited employees, who had responded to the JMT instrument, consisting of three main scales wherein each scale was comprised of three sub-scales, at recruitment and 94 executiveadministrations. As the basis for these estimations, three dimensions (or criteria) have laid the foundation: (i) “productivity and motivation”, (ii) “quality and work-structure”, and (iii) “cooperativeness”. The formulations of the validation questions resemble those instruments utilised in other studies examining predictive criterion validity [5,6]. Thus, the purpose of the present study was to estimate the predictive validity of the JMT recruitment instrument through applications of pairwise correlational analyses based upon the above three dimensions.
2. Methods and Materials
Predictive validity, based on uni-variate pairwise correlations (r), was compared to multi-variate relationships. The following three predictive criteria were included: Productivity, Quality of work, and Ability to cooperate.
2.1 Expected convergent and divergent correlations
An acceptable coefficient for convergent validity was set to 0.30 (Pearson r) as a ‘rule of thumb’ criterion. According to Cohen's guidelines, this value corresponds to a medium correlation [7]. Criteria for the analogue divergent coefficient was either an uncorrelated relationship (r <.10), or a weak correlation (.10 r <.30) [7]. The expected relationships among predictors and criteria are shown in Table 1 below.
3. Results
Predictive validity, based on uni-variate pairwise correlations (r), was compared to multi-variate relationships. The following three predictive criteria were included: Productivity, Quality-of-work, and Ability-to-cooperate.
In both studies, the results were mainly in accordance with expected relationships. For the 18 divergent relationships (‘non-correlated', or 'weakly correlated'), there were no correlations were higher than the expected values (.30).
For the first study (N=258), the mean value for convergent correlations was 0.37, and -0.071 for divergent. See further details on Table 2 below.
For the follow-up study carried out about four years after the first, the mean value for con-vergent correlations was 0.40, and -0.062 for divergent. See Table 3 below, for details.
The difference between convergent and divergent correlations was also tested for statistical significance by use of confidence intervals (CI). For both studies, this difference was significant (i.e., the 95% CI was >0). See Table 4 below.
4. Discussion
Despite the restrictiveness intrinsic to the test model, implying that only the main scale could influence criterion measures, the applications of independent pairwise associations in the present study was shown to possess utility for the assessment of criterion evaluation among the main scales of JobMatchTalent (JMT), implying that predictive validity was observed markedly. The purpose of deriving optimal predictability of the criterion measures provided a multivariate approach through which a result was obtained that was associated with the theoretical implications obtained from the JMT manual as well as the relevant scientific notions of psychometry in recruitment. The previously arranged associations between the criterion measures and predictors (JMT main scales) were confirmed on the grounds of the generally significant correlations as well as the situation that the relatively high levels for the median value derived for executives (r = 0.48) exceeded the estimated value pertaining to an explained variance of 15% (r = 0.39), which has been reported as an expected highest value from a meta-analysis of criterion measures and personality psychometry [8,9]. Furthermore, these analyses provided convergent and divergent conditions, respectively, thereby implying that the expected association gave broad support to the analyses and median values. The exploratory factor analysis exploratory study confirmed the present findings comprehensively despite possessing a lesser extent of predictive power thereby enhancing markedly the ‘combined’ predictive validity of the JMT-test instrument.
The predictor, “Stress Index” (C) showed throughout low levels of association over the personality categories and was excluded but nevertheless discussed here fleetingly. The finding that this main scale failed to be predictive of the criterion measure “Ability to Co-operate” may be open to several interpretations. For example, stress levels experienced by an employee may be exceedingly difficult to assess since the myriad of stressors affecting individuals induce, not only psychological and behavioural reactions but biological, physiological and epigenetic reactions as well [10-12]. These types of ‘internal’ and ‘extreme’, respectively, responses/reflexes present aspects of individuals’ behavioural repertoire that one attempts, with greater or lesser levels of success, to mask, strategically-tactically, at the place of work. Thus, as to be expected from these notions, the ‘true’ estimation of an individual’s stress level or expression becomes more apparent than real. Even applying the focus upon analyses differentiating executives from non-executives, there emerged interesting aspects of the results pertaining to similarity and stability from several smaller personal categories, such as specialists, administrators and HR-orientated. It appears likely that future estimations of predictive validity from JMT, involving larger, more accumulated samples for these three as well as other arising categories, are expected to provide greater levels of association than those reported here.
5. Conclusion
High levels of pairwise correlations indicate sufficient, indeed impressive, extents of predictive validity inherent to the JMT instrument.
6. Limitations
It is possible that another analysis employing regression analyses may offer further evidence of JMT predictive validity. Also, if a few items to provoke responses concerning questions of logic had been included perhaps more validity may have been forthcoming. Finally, the study may have benefitted from a larger sample size to enhance power.
Competing Interests
The authors declare that they have no competing interests.