Boalt School of Law / Hastings College of the Law
The Law School Admission Project: Looking Beyond the LSAT
Marjorie M. Shultz and Sheldon Zedeck, Principal Investigators
Eunice Chang, Graduate Research Assistant
Two Berkeley faculty members have undertaken a multi-year study that seeks to create a law school admission test that will try to predict who would be an effective lawyer. This project was funded by the Law School Admissions Council beginning in July 2001 and the investigation is led by Co-Principal Investigators Sheldon Zedeck (Department of Psychology) and Marjorie Shultz (Boalt Hall School of Law). The Institute of Industrial Relations has generously sponsored Eunice Chang, the graduate student researcher on this project, for the last two years.
The overall goal of the project is to develop predictors of attorney competence that could be used in choosing which applicants to admit to law school. In order to develop such predictors it was necessary to undertake a 3-part research process. In Phase I of the project, we 1) defined what factors were important to lawyering effectiveness and 2) specified methods for measuring those factors. Based on the Phase I factors and measurement scales, we developed and selected eight assessments ranging from a Situational Judgment Test to an Emotion Recognition Instrument in Phase II of our research. We are currently in Phase III of our project, in which we are administering our online test battery to Boalt Hall and Hastings Law School alumni to assess whether performance on those predictor tests correlates with actual lawyering effectiveness.
- Rationale for the Research
Law schools not only choose students, they are the main gate-keepers deciding who becomes a lawyer. But law schools admissions rely more heavily on academic criteria than do graduate departments that train academics. Law schools also weight test scores more heavily than do other professional schools. The LSAT only evaluates three cognitive skills (reading, analysis and logic). Although it is the most effective method to predict first year law school grades, the test is narrow in method and in goal. Commentators on legal education have urged that the criteria of merit be broadened, but no one has yet developed reliable ways of identifying, assessing or predicting other relevant skills. We seek to do so. If we succeed, law schools could add a "lawyering effectiveness index score" to existing admissions criteria. We believe that selecting prospective lawyers on the basis of a broader range of competencies would improve the profession's performance of its many tasks in society and in the justice system.
- Research Design
Phase I (2001-2003)
Phase I identified the range of competencies needed to be an effective lawyer and developed methods to assess people in regard to those professional work factors. The products of Phase I are: (1) A comprehensive list of 26 Effectiveness Factors that are important to effective lawyering; (2) A set of 715 behavioral examples of performance that describe or illustrate poor to excellent performance on the 26 factors. These descriptive examples enabled us to create the next product: (3) A number of flexible Behaviorally Anchored Rating Scales (BARS) that an evaluator could use to assess the effectiveness of any given practicing lawyer. The products from Phase I provided the informational foundation that was necessary for Phase II.
People affiliated with Boalt provided the material to create these products through individual interviews, focus group interviews, and a large sample questionnaire. The research participants included judges, clients, students, faculty, and lawyers in many practice fields, at various career stages, and in various career settings.
Phase II (2003 - 2006)
In Phase II, we developed tests that could, to some useful degree, predict who has the potential to excel in the factors identified in Phase I. In choosing tests, we will both identified and adapted existing tests that are productive for these purposes and we created new tests designed specifically with this goal in mind. Our test battery includes a Situational Judgment Test, a Biographical Inventory, a Law School Perception Index, three Hogan Personality Inventories, an Accomplishment and Experience Record, and an Emotion Recognition Test. These assessments are not based on legal knowledge or lawyering experience per se. Rather, they seek to predict who will have and/or could develop the competencies identified in Phase I as essential for effective lawyering.
Phase III (2006)
We are in the process of assessing whether performance on our predictor tests correlates with actual lawyering effectiveness. This validation process involves several steps. 1) We are administering those predictor tests to a sample of practicing lawyers; 2) we are having supervisors of those same individuals evaluate their effectiveness in professional practice, using the factors and scales identified in Phase I as tools for measurement; 3) if people who score high (or low) on given predictors of given factors also receive high (or low) evaluations in ratings of their effectiveness on those factors in practice, we will have demonstrated the necessary correlation between the predictor test(s) and actual lawyering competence. We expect that some of the factors will be more readily predicted than others, and that some tests developed in this phase will be discarded as not sufficiently valid. We are also conducting a similar validation study on upper division law students, administering the predictor tests to second and third year law students, and then having faculty assess (to the extent that the academic or clinical setting allows them sufficient exposure to the student's work) the effectiveness those students demonstrate in their final year of professional training.
In sum, Phases I and II have been completed. Once we have completed collecting the validation data in Phase III, we hope that law schools will be able to use the results to select components of a "lawyering effectiveness index score" to aid in their admissions decisions. Adding such predictors to the selection process would broaden the operative definition of merit to include not simply academic factors (LSAT and UGPA) but also professional performance factors in choosing the lawyers of the future. Individual law schools could use varying strategies in combining academic-based and professional-based predictors, and could also select particular professional effectiveness factors in light of the particular mission of their own school. We believe the result of this research would be to select better lawyers in a more principled way than is available to law schools today.