No CrossRef data available.
Article contents
Audit Experiments of Racial Discrimination and the Importance of Symmetry in Exposure to Cues
Published online by Cambridge University Press: 06 May 2024
Abstract
Researchers are often interested in whether discrimination on the basis of racial cues persists above and beyond discrimination on the basis of nonracial attributes that decision makers—e.g., employers and legislators—infer from such cues. We show that existing audit experiments may be unable to parse these mechanisms because of an asymmetry in when decision makers are exposed to cues of race and additional signals intended to rule out discrimination due to other attributes. For example, email audit experiments typically cue race via the name in the email address, at which point legislators can choose to open the email, but cue other attributes in the body of the email, which decision makers can be exposed to only after opening the email. We derive the bias resulting from this asymmetry and then propose two distinct solutions for email audit experiments. The first exposes decision makers to all cues before the decision to open. The second crafts the email to ensure no discrimination in opening and then exposes decision makers to all cues in the body of the email after opening. This second solution works without measures of opening, but can be improved when researchers do measure opening, even if with error.
- Type
- Article
- Information
- Copyright
- © The Author(s), 2024. Published by Cambridge University Press on behalf of The Society for Political Methodology
Footnotes
Edited by: Jeff Gill
References
Andersen, S. C., and Guul, T. S.. 2019. “Reducing Minority Discrimination at the Front Line—Combined Survey and Field Experimental Evidence.” Journal of Public Administration Research and Theory 29 (3): 429–444.CrossRefGoogle Scholar
Bartels, L. M. 2008. Unequal Democracy: The Political Economy of the New Gilded Age. Princeton: Princeton University Press.Google Scholar
Bertrand, M., Chugh, D., and Mullainathan, S.. 2005. “Implicit Discrimination.” American Economic Review 95 (2): 94–98.CrossRefGoogle Scholar
Bertrand, M., and Mullainathan, S.. 2004. “Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” American Economic Review 94 (4): 991–1013.CrossRefGoogle Scholar
Bischof, D., et al. 2022. “Advantages, Challenges and Limitations of Audit Experiments with Constituents.” Political Studies Review 20 (2): 192–200.CrossRefGoogle Scholar
Bueno de Mesquita, E., and Tyson, S. A.. 2020. “The Commensurability Problem: Conceptual Difficulties in Estimating the Effect of Behavior on Behavior.” The American Political Science Review 114 (2): 375–391.CrossRefGoogle Scholar
Butler, D. M., and Broockman, D. E.. 2011. “Do Politicians Racially Discriminate against Constituents? A Field Experiment on State Legislators.” American Journal of Political Science 55 (3): 463–477.CrossRefGoogle Scholar
Butler, D. M., and Crabtree, C.. 2017. “Moving beyond Measurement: Adapting Audit Studies to Test Bias-Reducing Interventions.” Journal of Experimental Political Science 4 (1): 57–67.CrossRefGoogle Scholar
Butler, D. M., and Homola, J.. 2017. “An Empirical Justification for the Use of Racially Distinctive Names to Signal Race in Experiments.” Political Analysis 25 (1): 122–130.CrossRefGoogle Scholar
Crabtree, C. 2018. “An Introduction to Conducting Email Audit Studies.” In Audit Studies: Behind the Scenes with Theory, Method, and Nuance, edited by Gaddis, S. M., Volume 14 of Methodos Series, Chapter 5, 103–117. Cham: Springer.CrossRefGoogle Scholar
Crabtree, C., and Dhima, K.. 2022. “Auditing Ethics: A Cost–Benefit Framework for Audit Studies.” Political Studies Review 20 (2): 209–216.CrossRefGoogle Scholar
Daniel, W. W. 1968. Racial Discrimination in England: Based on the P.E.P. Report, Volume 1084. Harmondsworth: Penguin.Google Scholar
Desposato, S. 2022. “Public Impacts from Elite Audit Experiments: Aggregate and Line-Cutting Harms.” Political Studies Review 20 (2): 217–227.CrossRefGoogle Scholar
Devine, P. G. 1989. “Stereotypes and Prejudice: Their Automatic and Controlled Components.” Journal of Personality and Social Psychology 56 (1): 5–18.CrossRefGoogle Scholar
Elder, E. M., and Hayes, M.. 2023. “Signaling Race, Ethnicity, and Gender with Names: Challenges and Recommendations.” The Journal of Politics 85 (2): 764–770.CrossRefGoogle Scholar
Fenno, R. F. 1978. Home Style: House Members in Their Districts. Boston: Little, Brown & Co.
Google Scholar
Frangakis, C. E., and Rubin, D. B.. 2002. “Principal Stratification in Causal Inference.” Biometrics 58 (1): 21–29.CrossRefGoogle ScholarPubMed
Fryer, R. G., and Levitt, S. D.. 2004. “The Causes and Consequences of Distinctively Black Names.” The Quarterly Journal of Economics 119 (3): 767–805.CrossRefGoogle Scholar
Gaddis, S. M. 2018. “An Introduction to Audit Studies in the Social Sciences.” In Audit Studies: Behind the Scenes with Theory, Method, and Nuance, edited by Gaddis, S. M., Volume 14 of Methodos Series, Chapter 1, 3–44. Cham: Springer.CrossRefGoogle Scholar
Gaddis, S. M. 2019. “Understanding the ‘How’ and ‘Why’ Aspects of Racial–Ethnic Discrimination: A Multimethod Approach to Audit Studies.” Sociology of Race and Ethnicity 5 (4): 443–455.CrossRefGoogle Scholar
Heckman, J. J. 1998. “Detecting Discrimination.” Journal of Economic Perspectives 12 (2): 101–116.CrossRefGoogle Scholar
Holland, P. W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81 (396): 945–960.CrossRefGoogle Scholar
Hughes, D. A., Gell-Redman, M., Crabtree, C., Krishnaswami, N., Rodenberger, D., and Monge, G.. 2020. “Persistent Bias among Local Election Officials.” Journal of Experimental Political Science 7 (3): 179–187.CrossRefGoogle Scholar
Imbens, G. W., and Rubin, D. B.. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. New York: Cambridge University Press.CrossRefGoogle Scholar
Kirgios, E. L., Rai, A., Chang, E. H., and Milkman, K. L.. 2022. “When Seeking Help, Women and Racial/Ethnic Minorities Benefit from Explicitly Stating Their Identity.” Nature Human Behaviour 6 (3): 383–391.CrossRefGoogle ScholarPubMed
Labov, W. 1972. Language in the Inner City: Studies in the Black English Vernacular. Philadelphia: University of Pennsylvania Press.Google Scholar
Landgrave, M., and Weller, N.. 2020. “Do more Professionalized Legislatures Discriminate Less? The Role of Staffers in Constituency Service.” American Politics Research 48 (5): 571–578.CrossRefGoogle Scholar
Landgrave, M., and Weller, N.. 2022. “Do Name-Based Treatments Violate Information Equivalence? Evidence from a Correspondence Audit Experiment.” Political Analysis 30 (1): 142–148.CrossRefGoogle Scholar
Leavitt, T., and Rivera-Burgos, V.. 2024. “Replication Data for: Audit Experiments of Racial Discrimination and the Importance of Symmetry in Exposure to Cues.” Harvard Dataverse V1. https://doi.org/10.7910/DVN/R3JGWS
CrossRefGoogle Scholar
Li, X., and Ding, P.. 2017. “General Forms of Finite Population Central Limit Theorems with Applications to Causal Inference.” Journal of the American Statistical Association 112 (520): 1759–1769.CrossRefGoogle Scholar
McClendon, G. H. 2012. “Ethics of Using Public Officials as Field Experimental Subjects.” The Experimental Political Scientist 3 (1): 13–20.Google Scholar
Miratrix, L. W., Sekhon, J. S., and Yu, B.. 2013. “Adjusting Treatment Effect Estimates by Post-Stratification in Randomized Experiments.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 75 (2): 369–396.CrossRefGoogle Scholar
Neyman, J. 1923. “Sur les Applications de la théorie Des probabilités Aux Experiences Agricoles: Essai Des Principes.” Roczniki Nauk Rolniczych 10: 1–51.Google Scholar
Pedulla, D. S. (2018). Emerging Frontiers in Audit Study Research: Mechanisms, Variation, and Representativeness. In Gaddis, S. M. (Ed.), Audit Studies: Behind the Scenes with Theory, Method, and Nuance, Volume 14 of Methodos Series, Chapter 9, 179–195. Cham: Springer.CrossRefGoogle Scholar
Perry, T., and Delpit, L.. 1998. The Real Ebonics Debate: Power, Language, and the Education of African-American Children. Boston: Beacon Press.Google Scholar
Riach, P. A., and Rich, J.. 2004. “Deceptive Field Experiments of Discrimination: Are They Ethical?” Kyklos 57 (3): 457–470.CrossRefGoogle Scholar
Rivera-Burgos, V., and Rubio, J. M.. 2024. “Responsiveness to Coethnics and Cominorities: Evidence from an Audit Experiment of State Legislators.” Journal of Race, Ethnicity, and Politics 9 (1): 55–79.CrossRefGoogle Scholar
Rubin, D. B. 1974. “Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 66 (5): 688–701.CrossRefGoogle Scholar
Slough, T. 2023. “Phantom Counterfactuals.” American Journal of Political Science 67 (1): 137–153.CrossRefGoogle Scholar
Wienk, R. E., Reid, C. E., Simonson, J. C., and Eggers, F. J.. 1979. Measuring Racial Discrimination in American Housing Markets: The Housing Market Practices Survey, Volume 444. Washington, DC: Division of Evaluation, US Department of Housing and Urban Development, Office of Policy Development and Research.Google Scholar
Leavitt and Rivera-Burgos supplementary material
Leavitt and Rivera-Burgos supplementary material
File
70.7 KB