Profile
International Journal of Psychology & Behavior Analysis Volume 4 (2018), Article ID 4:IJPBA-150, 8 pages
https://doi.org/10.15344/2455-3867/2018/150
Research Article
JobMatch Talent Construct Validity: Exploratory Factor Analysis

Bengt Jansson1, Klaus Olsen2 and Trevor Archer1*

1Department of Psychology, University of Gothenburg, S-40530 Gothenburg, Sweden
2JobMatch Talent, JobMatch Sweden AB, Skårs Led 3, 400 20 Göteborg, Sweden
Dr. Trevor Archer, University of Gothenburg, Department of Psychology, S-40530 Gothenburg, Sweden; E-mail: trevor.archer@psy.gu.se
12 July 2018; 20 September 2018; 22 September 2018
Jansson B, Olsen K, Archer T (2018) Jobmatch Talent Construct Validity: Exploratory Factor Analysis. Int J Psychol Behav Anal 4: 150. doi: http://dx.doi.org/10.15344/2455-3867/2018/150

Abstract

The JobMatch Talent (JMT) is a person oriented test with a focus upon working life. The test consists of ten main scales, each main scale having three underscores. The main scales (A-J) have the following designations: Work structure (A), Personal drive (B), Stress-index (C), Decision characteristics (D), Activity (E), Drive (F), Acting (G), Tolerance (H), Social interest (I), Communication (J). Previous studies have shown that the JMT has good reliability (Cronbach's alpha) and good criterion validity (both predictive and concurrent). In an initial phase, an exploratory factor model was developed and compared across groupings (N=12,650). In a second phase, correlations between the exploratory and theoretical factors were calculated. Moreover, the exploratory factors were correlated with job performance ratings (N=258). In both analyses, expected (convergent) and non-expected (divergent) relationships were systematically compared. The exploratory factors were stable with good model fit across groupings. Eight exploratory factors were obtained with an explained variance of about 72%. Large similarities, but also a few differences, emerged between the eight exploratory versus the ten theoretical factors. Correlations between corresponding factorings were high (mean = 0.70). Furthermore, correlations between the exploratory factors and job performance ratings were high (mean =0.36). It should be emphasized that these relationships were very similar to those obtained in previous studies involving the theoretical factors of the JMT. The eight exploratory factors were found to be stable and highly correlated with the theoretical factors. They were externally valid and predictive of job performance.


1. Introduction

In the present study, the construct validity of the JMT was investigated. Empirical analysis of construct validity was performed at different steps. In a first step, the scales in a test were examined using exploratory factor analysis (EFA). The derived factor model was compared to a theoretical model of the test scale ('internal' construct validity). Since the JMT instrument is possibly the only recruitment instrument currently available for the sole purpose of occupational recruitment endeavors, it seems necessary to investigate further the construct validity through applications of different statistical methods. It is expected that the theoretical and practical purposes of the JMT instrument will offer greater applicability towards considerations pertaining to the potential of each ‘to-be-recruited’ or recruited individual to develop professionally within the particular position involved. All the ten main scales, taken together with the three subscales for each, were developed from pre-existing clinical and occupational instruments/questionnaires employing items to test various aspects of personality attributes.

In a second step, the exploratory factor model was related to external characteristics with particular relevance to the examined test ('external' construct validity). In general, external features are measured using selected instruments, but the feature scan also be real conditions such as job position or work function [1]. A useful description of 'external' construct validity is provided, if the exploratory factors are well related to external features or ratings.

A distinction between 'internal' and 'external' construct validity implies an important basis for the design of this study. Similarly, the term of construct validity as a superior concept of validity was an important aspect for a few studies to be analyzed in this report. For example, it has been emphasized that all validity is basically some form of validity and constitutes the fundamental meaning of validity [2,3]. Not seldom, construct validity is studied by means of convergent/ divergent correlation matrices, but less often, such methodology for evaluation is used in, for example, predictive criterion analysis, as is the case in this report.

In previous studies [4,5], concurrent and predictive validity of the JMT have been outlined for the JMT instrument. From a methodological point of view, it was important that the factor analysis was performed on a well-defined sample of participants with relatively similar work functions and job positions, and who had responded to the JMT due to similar purposes. A further criterion was a large sample size of the selected group.

The purpose of the study was to analyze the concept validity of the scales in the JMT using EFA. The derived exploratory factors were to be compared to the theoretical model of the JMT, and also, their ability to predict job performance was investigated.

2. Methods and Materials

2.1 Participants

From a norm group (N=12,650) for the JMT, a target group for the factor analyses was selected. This selection had a focus on private business executives (N=3411), and who answered the JMT when being recruited. For comparative reasons, the remaining more heterogeneous grouping (N=9,239) was also included into the analyses as a control group.

The gender distribution of the norm group was 40.0% women and 60.0% men. The corresponding figures for the target group were 38.3% and 61.7%, respectively, and for the control group 40.6% and 59.4%, respectively.

In addition, a previous study was used for analysis of 'external' construct validity for the exploratory JMT factors. This group consisted of recruited persons (N=258) [5].

2.2 Instruments

The JMT is a person-oriented test with a focus on working life and consists of ten main scales, and each main scale having three underscores. The main scales (A-J) have the following designations: Work structure (A), Personal drive (B), Stress-index (C), Decision characteristics (D), Activity (E), Drive (F), Acting (G), Tolerance (H), Social interest (I), Communication (J). Table 1 below contains a complete list of the JMT scales.

table 1
Table 1: The JMT scales with short denotations (a1, a2, a3 etc.).

Reliability of the JMT scales was based on the norm group. For the main scales, the mean of Cronbach's alpha .80 (SD=0.042), and also the median value was 0.80. For the 30 sub-scales, the mean of alpha was .71 (SD=0.046), and median value was 0.72 [5].

Furthermore, a questionnaire (containing 11 items) for leaders' ratings of job performance of recruited persons was used for analysis of the predictive validity of the exploratory model of the JMT. The ratings of leaders' were categorized into three indices: (1) Productivity and motivation, (2) Quality and structure of performed work, (3) Interpersonal skills. Reliability of the indices ranged between 0.75 - 0.90 [5].

3. Design

A procedure for analyzing construct validity in JMT was based on conditions (as below) for the exploratory and theoretical factors. If the conditions were fulfilled, conclusions could be drawn about similarity between an exploratory and theoretical model of the JMT.

The exploratory factors should meet the following conditions:

  1. Consistent loadings and a clear factor structure over groups
  2. Clear expected (convergent) or non-expected (divergent) correlations with'external' criteria (prediction of job performance)
  3. Clear convergent / divergent correlations with the theoretical factors (if the exploratory factors differed from the theoretical)

3.1 Statistical analyses

The 30 sub-scales in the JMT were included in exploratory factor analyses over selections in the norm group. Extraction of factors was done with Principal Component Analysis (PCA). Orthogonal rotation was chosen in order to facilitate interpretation of correlations between exploratory and theoretical factors.

Correlations (Pearson) were calculated between exploratory factors and other scales and concepts. Differences between mean values of convergent and divergent correlations were tested with independent t-test. In order to counteract negative and positive divergent correlations to balance each other in calculating averages, all of the t-tests were performed with absolute values of divergent correlations. As a result, an average of divergent correlations could have an 'unfavorably' high value when being compared to a convergent average value (always based on the positive correlations in this report).

Moreover, since the distribution of Pearson correlations is not normally distributed, Fisher's z-transformation of the correlation values was used for complementary t-test comparisons.

4. Results

Descriptive statistics such as mean values, spread, skewness and flatness for the JMT Scales are found in the Jansson and Winge [5] study. The presentation of analyses of concepts in the JMT has been divided into two main sections:'internal' and'external' construct validity.

4.1 Analysis of 'internal' construct validity

Three types of analyses have been performed to analyze JMT's internal conceptual structure: (i) exploratory factor analyses, (ii) reliability of the exploratory factors, and (iii) correlations between the exploratory factors and the corresponding theoretical factors.

In order to compare exploratory with theoretical factors, a decision rule was used assuming that if two or three subscales were found in an exploratory factor, this factor was considered to be compatible with the corresponding theoretical factor.

4.2 Exploratory factor analyses

The initial factor analysis for the target group of privately employed managers showed eight factors. Mainly, the differences between the theoretical factor structure could be summarized in three points. Firstly, a combination of the factors B, F and G appeared. Secondly, the D factor was represented by a single subscale along with a subscale from the A factor. Thirdly, factors A, C, I and J were supplemented with one or two subscales from other main scales than the target scale. The exploratory model could be described as follows: A': Structure of Work, BFG': Productivity, C': Stress Index, ad': Strategic Behavior, E': Activity, H': Tolerance, I': Social Interest, J': Communication.

Four exploratory factors - A', C', I', J' - added one or two individual subscales. To the factor A' (Work Structure) Concentration Ability (c) and Endurance (d) was added. The factor C' (Stress-index) was expanded with Mood stability (b) and Non-development motivation (f). Furthermore, the factor I'(Social interest) was expanded with Concurring image (h). Finally, the factor J'(Communication) was supplemented with Willpower (d) and Contact creation (i).

For the target group of private managers, 0 27 (out of 30) diagonal factor loadings at (intersections of theoretical and exploratory factors) were greater than 0.50. The 3 other diagonal loadings were between 0.43-0.48. There were 6 (out of 210) cross loadings which were greater than 0.40 (5 in the range 0.40-0.49 and one in the range 0.50-0.59). Table 2 below provides a detailed summary of the factor loadings for the group of private managers.

table 2
Table 2: Exploratory model of the JMT with eight factors for the target group of private managers (N=3,411).

An addition, the EFA performed on the control group showed very similar results to those from the target group. It should be emphasized that loadings >0.40 differed on average 0.0047 between groups (means for these loadings were 0.6627 and 0.6673, respectively). Measurements associated with model fit were similar for the two groups: KMO was between 0.83-0.85; communalities (averages of 30 subscales) were between 0.69-0.71; explained variance was between 71.5% -73.7%.

4.3 Reliability of the exploratory factors

The reliability of the exploratory factors, based on Cronbach's alpha, was similar to the corresponding theoretical values. The mean of alpha for the exploratory factors was 0.80, which was the same value as for the theoretical model [5]. See Table 3 for detailed description.

table 3
Table 3: Standardized values for Cronbach’s alpha of the exploratory factors in the JMT across groups (N=12,650).

5. Analysis of'External' Construct Validity

5.1 Relationship between theoretical and exploratory factors

Weighted factor scores were calculated for the orthogonal exploratory factors. Corresponding composite scores were calculated for the theoretical factors by means of an unweighted mean sum of sub-scales.

The correlations between exploratory and theoretical factors were divided into convergent (expected) and divergent (non-expected) relationships based on matching concepts. Correlations were calculated for the target group and the control group. The results unequivocally showed that correlations between convergent factors were clearly higher than were they between divergent. For the target group of private managers, the mean of convergent correlations was 0.70 (SD=0.15), and for divergent correlations the mean value was 0.16 (SD=0.12), and for the control group, the corresponding values were 0.70 (SD=0.16) and 0.17 (SD=0.13), respectively. See Table 4a below.

table 4a
Table 4a: Correlations between the ten theoretical main scales (A–J) and the eight exploratory factors (A’–J’) for the target group of private managers (N=3,411).

The variation of convergent and divergent correlations was plotted over the exploratory factors and was found to be fairly uniform. The distance between the two categories of relationships was relatively constant. See Figure 1 below, where the correlations show absolute values.

figure 1
Figure 1: Plot of assumed convergent and divergent correlations (n=80) over the exploratory factors, across groups.

Absolute values were used for statistical testing of difference between convergent and divergent correlations. Testing of differences(t-test) between convergent and divergent correlations was based on Fisher's z transformation.For both groups, outcomes of the tests wasclearly significant (p<.001) (Table 4b).

table 4b
Table 4b: Descriptive (upper part) and analytical statistics (lower part) across groups of assumed convergent and divergent correlations (n=80) between theoretical and exploratory factors; correlations were transformed to Fishers z based on absolute values.

5.2 Relationships with ratings of job performance

The exploratory factors were correlated with three predictive criteria, all concerned with leaders' ratings of job performance: (i) Productive action, (ii) Quality and structure at work and (iii) Collaborative ability (N=258).

The results showed that the correlations between the exploratory factors and the predictive criteria were almost equal to the corresponding values for the theoretical factors. The mean of the correlations between the ratings and the exploratory predictors was .36 (where factor C', stress index, was not included). The corresponding value for the theoretical factors was 0.37 [5]. Expected (convergent) and non-expected (divergent) correlations were reported in Table 5a below, where all convergent correlations were greater than 0.30.

table 5a
Table 5a: Correlations between predictive criteria of job performance and summed values from exploratory factors of the JMT (N=258).

The variation of convergent and divergent correlations was plotted over the predictive criteria and the exploratory factors, respectively. Although divergent correlations show a slightly more uneven spread over the exploratory factors, the distances between the two categories of correlations were relatively similar. The same pattern of convergent and divergent correlations was found over the predictive criteria. See further in Figure 2 below.

figure 2
Figure 2: Plot of convergent and divergent correlations (n=24) over JMT’s exploratory factors and predictive criteria of job performance, respectively.

The mean of convergent correlations was 0.36 (SD=0.041) and 0.08 (SD=0.079) for divergent (using absolute values). Test of the difference was based on Fisher’s z-transformation, although the effect of such transformation was marginal since the highest correlations were around 0.40.The difference between convergent and divergent values was highly significant (p<0.001). See Table 5b for anaccount of these results.

table 5b
Table 5b: Descriptive (upper part) and analytical statistics (lower part) of assumed convergent and divergent correlations (n=24) betweenpredictive criteria of job performance and JMT’s exploratory factors; correlations were transformed to Fishers z based on absolute values.

6. Discussion

The study was conducted in two phases. The first included exploratory factor analysis that gave the same result for the target and control group. In the second 'external' phase, the resulting exploratory factors were examined with respect to predictive validity, and in relation to the theoretical factors.

6.1 Relationships with external criteria

The construct validity in the JMT was confirmed by the analytical procedures based on convergent and divergent correlations. The results consistently showed strong significant differences between these two types of relationships. The exploratory factors had the same predictive ability that previously had been found for the theoretical factors [5]. Furthermore, a similarity was found between the theoretical and exploratory factors, and the convergent correlations between the eight exploratory and the theoretical factors were high.

The mean of convergent correlations between the criteria for job performance and JMT's exploratory factors was 0.36 (N = 258). This value can be seen as predictive criteria validity. This, in its context, high value [6] matched well with the corresponding value of the theoretical factors (calculated from [5]). Furthermore, the exploratory factors in general had the same reliability (Cronbach's alpha) as the theoretical factors.

In comparison, the results were consistent with a recent study of construct validity, where the JMT scales were related to customized scales from the IPIP (see 'JMT-IPIP Correlations', 2018). In the latter study, the average correlation between the JMT and IPIP scales was 0.65 (95% Confidence interval: 0.61, 0.69). The median value was 0.67. This indicated that this type of construct validity in the JMT was sufficient-to-superior.

6.2 Number of factors

The stability of the proposed exploratory model with eight factors seemed good. In all, the results showed that a factor analysis model with eight factors could be described as well adapted to data, and that it exhibited similar patterns of loadings across two groups. Nevertheless, there remains reason to question whether or not the exploratory factor model could not have maintained the initial ten factors, but with some alteration of the partition of the 30 sub-scales. In the literature, similar problems have been investigated.

In Loehlin& Goldberg [7], two studies analyzed the impact additional factors had upon the existing factors in a model. Two kinds of effects were discussed. One effect was whether an increased number of factors gave rise to a hierarchical effect upon any existing factor. For example, would a factor like BFG' (Productive behavior) be influenced? The other effect was whether an additional factor instead gave rise to a new concept. Extensive analyzes indicated that the second option occurred. Such an effect was found in this study, where an increase in the number of factors, with one or two, had no effect on e.g. BFG' (this analysis was not reported in the Result section). The factor BFG'remained intact.

6.3 Broad and narrow concepts

A restructuring of the 30 subscales of the JMT was done by the exploratory factor analysis. From being influenced by ten theoretical concepts/factors, the subscales were subsumed into an eight factor model. The smaller number of factors caused that some factors became broader, i.e. more subscales were included, while a few other factors became narrow, and some remained unchanged.

In general, a main scale maybe characterized as broad, and the facets/subscales as narrow [8]. Studies have shown some contradictory results regarding the ability of concepts to predict the dependent variable. Both broad and narrow concepts have proved to be effective in various ways [8,9]. In the present study, two combined factors emerged, one with broad features, BFG' (Productive behavior), and another with narrow characteristics, ad' (Strategic behavior). The latter factor consisted of two separate sub-scales, while the first contained seven subscales. In terms of predictive ability, it was found that the exploratory factors BFG' and ad' were essentially as effective as the corresponding theoretical main scales, when predicting job performance [5].

The consistency of the correlations between exploratory and theoretical factors must be emphasized also in relation to broad and narrow concepts. The two exploratory combination factors BFG' (Productive behavior) and ad' (Strategic Behavior) correlated only with the theoretical main scales they were composed of. This distinguishing feature therefore did not depend on the broadness and narrowness, respectively, of each factor. In addition, some of the exploratory factors had been supplemented by single subscales from other main scales than 'their own'. This was for A' (Work structure), C' (Stress index), I' (Social interest) and J' (Communicative behavior). For these four exploratory factors, the additional sub-scale(s) became a customized complement instead of an independent object or substitute. In formal terms, these exploratory factors became broader, but this did not affect the assumed convergent correlation with the corresponding theoretical main scale. The additional single sub-scale of these four factors did not correlate with their original theoretical main scale, but instead contributed to providinga semantic nuance to the meaning of the exploratory factor that became their new inheritance.

Moreover, this study showed that the confluence of related concepts into a broad concept retained a shared predictive ability, if the individual scales turned out to be predictive of job performance. This observation can be contrasted with the analyzes by Hastings and O'Neill [10] that were conducted with broad and narrow concepts for the prediction of job performance on personality. They showed that while broad concepts, like Big Five, were poor predictors of some aspects of job performance, the facets of the larger concepts performed better.

A conclusion that could be drawn from their (above) analyzes, and the present study, where in a broad or joined concept is predictive only if the (most or all) individual factors of the concept are predictive. This conclusion presents an interesting object for further research.

In sum, a crucial result of this study was that the two models of JMT's test structure were formally different, but functionally similar. This similarity was based on (i) highly correlated concepts between the theoretical and exploratory factors, using a MTMM-based method, and on (ii) same high validity to predict job performance. Behind the predictive ability of the exploratory model was the finding, that predictivity of JMT main scales was preserved when the scales became broad, or narrow. Thus, both broad and narrow exploratory factors had high predictive accuracy.

7. Conclusion

The focus of this study was construct validity for the ten theoretical scales presented in the JMT. Based on analyzes of 'internal' and 'external' construct validity, the following three main conclusions may be drawn:

Firstly, the exploratory model with eight-factor presentation showed a good adaptation to data and stability across the target and control groups.

Secondly, the eight exploratory factors showed good predictive criterion validity for job performance. Mean values of the correlations were similar to the corresponding high values from studies with the ten theoretical factors.

Thirdly, there was a strong relationship between the eight exploratory and the ten theoretical factors. The exploratory factor structure seemed to be relevant for the theoretical model of the JMT.

8. Limitations

The measurement invariance of the factor structure across the target and control group was not tested at levels configuration, loadings, intercepts and residual variances of the latent factors [11].

Competing Interests

The authors declare that they have no competing interests.


References

  1. Raykov T, Marcoulides A (2011) Introduction to psychometric theory. New York: Routledge
  2. Guion RM (1977) Content validity: Three years of talk-what’s the action? Public Personnel Management 6: 407-414
  3. Streiner DL, Norman GR (2003) Health measurement scales: A practical guide to their development and use (3rded.). Oxford: OUP. [Google Scholar]
  4. Garcia D, Nima AA, Rappe C, Ricciardi MR, Archer T, et al. (2014) The relationship between the JobMatch Talent test and the neo pi-r: construct validation of an instrument designed for recruitment of personnel. PLoS ONE 9: e90309 [CrossRef] [Google Scholar] [PubMed]
  5. Jansson B, Winge H (2013) Prediktiv kriterievaliditet hos JMT genom chefsskattningar: en korrelationsbaserad ansats
  6. Viswesvaran C, Ones DS, Schmidt FL (1996) Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology 81: 557-574 [CrossRef] [Google Scholar]
  7. Loehlin JC, Goldberg LR (2014) Do personality traits conform to lists or hierarchies? Pers Individ Dif 70: 51-56 [CrossRef] [Google Scholar] [PubMed]
  8. Salgado JF, Moscoso S, Berges A (2013) Conscientiousness, Its Facets, and the Prediction of Job Performance Ratings: Evidence against the narrow measures. International Journal of Selection and Assessment 21: 74-84 [CrossRef] [Google Scholar]
  9. Ashton MC, Paunonen SV, Lee K (2014) On the validity of narrow and broad personality traits. Personality and Individual Differences 56: 24-28 [CrossRef] [Google Scholar]
  10. Hastings SE, O’Neill TA (2009) Predicting workplace deviance using broad versus narrow personality variables. Personality and Individual Differences, 47: 289-293 [CrossRef] [Google Scholar]
  11. Kline RB (2011) Principles and practice of SEM (3rded.). New York: The Guilford Press