Because the assessment industry is largely unregulated, the market is flooded with cheap, ineffective assessments that claim to measure personality but have no foundation in personality psychology or supporting evidence. Therefore, careful selection of assessment tools is essential to success—otherwise it’s just money down the drain.
At Hogan, we pride ourselves on the science behind the assessments that drive our powerful talent solutions. We strive to be as transparent as possible, so you can be confident in selecting the right assessment product for your business needs.
The following guide provides a standard set of considerations against which you can compare potential assessment providers.
What is the goal of measurement using personality assessment?
The goal of measurement is prediction. While this may seem obvious, many personality assessments on the market do not measure what they claim. Assessment providers should offer clear scientific evidence that supports the predictive validity and reliability of their assessments.
Hogan’s products predict performance. More specifically, we design our assessments to help organizations evaluate people for basic employability and job requirements, and to provide a solid basis for coaching around career development. This data feeds into our reports, which help organizations differentiate job candidates based on characteristics related to job performance and organizational alignment.
Based on decades of research on personality psychology and measurement, Hogan’s assessments have been validated against external criteria, including job performance and peer ratings (e.g., 360-degree performance review), allowing us to describe a job candidate’s likely reputation in the workplace. We also compare the way a job candidate scores compared to thousands of other people to determine how that person will likely behave at work. Continuous evaluation of our assessments against performance indicators clearly shows that our tools can improve overall business outcomes.
What backgrounds and professional affiliations do the assessment developers have?
Proper assessment development is a rigorous process that requires the supervision of qualified professionals with backgrounds in psychology and research methods. One way to ensure that a tool adheres to the highest standards possible is to look at memberships the tool’s developers have with scientific or professional organizations.
At Hogan, we employ PhD- and master’s-level professionals with backgrounds in industrial-organizational and quantitative psychology. Our professional affiliations span several divisions of the American Psychological Association, including Division 5 (Measurement and Assessment), Division 8 (Personality and Social Psychology), and Division 14 (Industrial and Organizational Psychology).
Our team of professionals participate in conferences and communities of practice to share their research and stay informed about legal developments and industry standards. Our employees regularly receive invitations to participate in boards and panels alongside other industry leaders in workplace assessment, and they have been recognized for their research on topics such as machine learning and natural language processing.
What kind of algorithm do the assessments use?
More specifically, do they use a traditional statistical algorithm, or do they use an algorithm based on artificial intelligence? Three considerations are important when it comes to the algorithms that are used in tools for making decisions about people: accuracy, potential for bias, and transparency.
To understand the differences between traditional and AI algorithms, it is first helpful to recognize that every algorithm follows a simple input-process-output model. For example, given the numerical values 2 and 2 as inputs and instructions to process with addition, both humans and computers output the value 4. Note that in this example, we specified both the inputs and the process instructions—this is precisely how traditional algorithms work.
The advantages of using traditional algorithms for making talent decisions are that humans can determine the relevance of the input, the accuracy of the process, and the validity of the output. Questions (i.e., inputs) on such assessments are screened for job relevance, statistical algorithms built from responses to these questions are empirically linked to job performance, and the output from these algorithms can be scrutinized to tell you why one candidate was evaluated favorably compared to another.
By comparison, modern artificial intelligence systems work in the opposite direction. The human tells the AI what kind of output is desired; then the AI tries to find inputs from a potentially vast and potentially irrelevant array and processes that best match to the output requested by the human. In these cases, the output can be evaluated for validity, but the inputs and process instructions are completely up to the AI. In other words, AI algorithms tend to try to champion accuracy, but they often lack transparency and have the potential for bias.
Have the assessments been peer-reviewed or reviewed by unbiased third parties?
An unbiased party, such as a psychological council or association, should evaluate any scientific tool designed to measure personality. Appearance in peer-reviewed academic journals is also a good indicator of a tool’s quality. As part of peer review, panels of impartial experts conduct reviews of manuscripts to ensure scientific accuracy. Given recent technological changes, such as AI, newer entrants to the assessment market might not have submitted their assessments for unbiased review yet.
Agencies across four continents have examined Hogan’s assessments. Our assessments have received positive reviews from well-known assessment evaluation bodies, including the Buros Institute of Mental Measurements, British Psychological Society, Health Professions Council of South Africa, Federal Psychology Council of Brazil, and DNV GL (Norway). In addition, our tools regularly appear in top peer-reviewed journals, including the Journal of Applied Psychology, Personnel Psychology, Human Performance, and the International Journal of Selection and Assessment.
Do the assessments adhere to any employment guidelines or standards?
Assessment publishers should provide information regarding the compliance of their tools with international guidelines. In the United States, employment assessments need to conform to guidelines established by the Equal Employment Opportunity Commission (EEOC), the government agency that enforces laws making it illegal to discriminate against job applicants based on race, color, religion, sex, national origin, age, or disability. These legal requirements apply to traditional assessments, as well as those that use artificial intelligence or other software. Assessment providers should provide manuals that conform to the Uniform Guidelines on Employee Selection Procedures, or the regional equivalent.
Each assessment manual we publish contains detailed information on the development, reliability, validation, and norms for each assessment; we use the Uniform Guidelines, Society for Industrial and Organizational Psychology (SIOP) Principles for the Validation and Use of Personnel Selection Procedures, and American Psychological Association (APA) Standards for Educational and Psychological Testing to guide the development of our manuals and technical reports. In addition to ensuring our products predict job performance, we regularly monitor the assessments for the presence of adverse impact, or discriminatory effect, on the candidate. This due diligence allows us to state with confidence that the assessments are predictive of performance, and they do not discriminate based on EEOC criteria.
Are the assessments accompanied by technical reports or validation studies?
Another good way to evaluate the quality of an assessment is by reviewing the support materials that the publisher provides. This can include technical manuals or reports and research studies published from the data. These publications provide transparent technical information on the scientific processes used to develop the assessments, as well as the results of the research. Publishers unwilling to publish this data may lack evidence to support their claims or have something to hide.
Hogan’s research library contains thousands of technical reports describing competent validity studies that our researchers prepare in accordance with the Uniform Guidelines, SIOP Principles, and APA Standards. These reports are based on research conducted with employees in real organizations around the world. We provide technical manuals for all of our assessments, and we produce technical reports for every local validation study or competency solution that we develop.
Note that support materials, such as technical reports and validation studies, are not the same as marketing materials. When evaluating assessments, be cautious of marketing claims, especially regarding AI and other new technologies. Such claims can be misleading. Many companies claim to use AI because it’s attractive to investors, but these claims do not mean they are using AI in any meaningful way.
Are the assessments appropriate for the job under consideration?
Various jobs require different profiles; therefore, assessment providers should be able to provide evidence that their assessments are appropriate to determine if candidates match the job requirements.
While many new assessment providers focus only on one job area (e.g., sales or aviation), Hogan’s assessments have validity data supporting their use for virtually any job. Our research archive contains studies from more than 20 different industries and represents 95% of the industry coverage from the Occupational Information Network (O*NET).
Using studies from our archive, we can match worker requirements of a target job with similar jobs in the archive, allowing us to adapt assessments to predict job performance. We use industry-standard research methodologies to aggregate results from hundreds of studies and thousands of job candidates to provide predictive profiles that have been pre-validated for multiple broad job families and selection criteria (e.g., safety behaviors). In addition to job family profiles, our research team regularly conducts local validation studies that organizations use to screen job candidates based on job- specific criteria.
How is the performance of the assessments measured?
As cultures and norms shift over time, personality assessments should be adapted to ensure that measurements align with changing job requirements and expectations for candidates.
Hogan works closely with clients to update and revalidate selection profiles on a regular basis. We continuously monitor our assessments to ensure that items are functioning properly and that item clusters measure what they should be measuring. Our data analytics team tests new items and replaces items that become outdated, stop working, or are otherwise problematic. The revalidation process involves collecting evidence that the assessments and recommendations are appropriate for the selected jobs.
In addition to monitoring our assessments' ability to predict job performance, we also leverage tools to detect errant responses. Our assessments automatically flag careless response patterns, and we have a proprietary tool that allows us to detect suspected cheating.
Are the assessments and products adapted to different cultures and supported locally?
International use of personality assessments requires adaptation to local languages and cultures. For an assessment to be relevant in multiple cultures, it must match local norms, be sensitive to nuances of language, etc. Additionally, local support must be available to respond to culturally specific needs in a timely manner.
Hogan works with an extensive global network of industry experts and practitioners who are fluent in both English and the target language, understand the psychological concepts under consideration, and know the target culture well. This ensures quality adaptations of our assessments and other accompanying materials. Hogan develops local norms to support and respect cultural differences and has translated assessments and reports for use in nearly 50 languages. We ensure consistency and cultural sensitivity in our translation process by using qualified translators who are fluent speakers of the target language and a combination of forward and backward translation to ensure the original and translated versions are congruent.
Still have questions?
We’d be delighted to answer them. As the global leader in personality assessment, Hogan offers a robust product line that transforms psychometric assessments into predictive results. We take pride in the depth of measurement of our assessments, and our multitude of report options means that we have the solution for your business needs. All of our products come with support from our team of consultants. Visit us at hoganassessments.com, or give us a call at 918.749.0632. We look forward to hearing from you.