Suspect Development Systems: Databasing Marginality and Enforcing Discipline
Algorithmic accountability law—focused on the regulation of data-driven systems like artificial intelligence (AI) or automated decision-making (ADM) tools—is the subject of lively policy debates, heated advocacy, and mainstream media attention. Concerns have moved beyond data protection and individual due process to encompass a broader range of group-level harms such as discrimination and modes of democratic participation. While a welcome and long overdue shift, the current discourse ignores systems like databases, which are viewed as technically “rudimentary” and often siloed from regulatory scrutiny and public attention. Additionally, burgeoning regulatory proposals like algorithmic impact assessments are not structured to surface important –yet often overlooked –social, organizational, and political economy contexts that are critical to evaluating the practical functions and outcomes of technological systems.
This Article presents a new categorical lens and analytical framework that aims to address and overcome these limitations. “Suspect Development Systems” (SDS) refers to: (1) information technologies used by government and private actors, (2) to manage vague or often immeasurable social risk based on presumed or real social conditions (e.g. violence, corruption, substance abuse), (3) that subject targeted individuals or groups to greater suspicion, differential treatment, and more punitive and exclusionary outcomes. This framework includes some of the most recent and egregious examples of data-driven tools (such as predictive policing or risk assessments), but critically, it is also inclusive of a broader range of database systems that are currently at the margins of technology policy discourse. By examining the use of various criminal intelligence databases in India, the United Kingdom, and the United States, we developed a framework of five categories of features (technical, legal, political economy, organizational, and social) that together and separately influence how these technologies function in practice, the ways they are used, and the outcomes they produce. We then apply this analytical framework to welfare system databases, universal or ID number databases, and citizenship databases to demonstrate the value of this framework in both identifying and evaluating emergent or under-examined technologies in other sensitive social domains.
Suspect Development Systems is an intervention in legal scholarship and practice, as it provides a much-needed definitional and analytical framework for understanding an ever-evolving ecosystem of technologies embedded and employed in modern governance. Our analysis also helps redirect attention toward important yet often under-examined contexts, conditions, and consequences that are pertinent to the development of meaningful legislative or regulatory interventions in the field of algorithmic accountability. The cross-jurisdictional evidence put forth across this Article illuminates the value of examining commonalities between the Global North and South to inform our understanding of how seemingly disparate technologies and contexts are in fact coaxial, which is the basis for building more global solidarity.
In October 2020, a small group of protesters dressed in black and carrying black umbrellas to protect their privacy marched through downtown Phoenix, Arizona.1Kaila White, 18 Arrested During Protest Against Police in Downtown Phoenix on Saturday, Ariz. Republic (Oct. 18, 2020, 4:21PM), https://www.azcentral.com/story/news/local/phoenix-breaking/2020/10/18/downtown-phoenix-protest-police-arrest-nearly-every-attendee/3705969001/, [https://perma.cc/JK73-5R5B]; see Henry E. Hockeimer, Jr., Terence M. Grugan, Bradley M. Gershel & Jillian L. Andrews, Ballard Spahr, LLP, Report to the City, City Manager’s Office of an Investigation into the Criminal Street Gang Charges Filed Against “ACAB” 9–10 (2021) https://www.phoenix.gov/citymanagersite/Documents/Gang-Related-Summary-Report-08-12-2021.pdf [https://perma.cc/TB9A-5R2M]. This was one of many protests across the globe against police brutality and racism that continued into the fall of 2020 following the murder of George Floyd in May of that year.2White, supra note 1; see Hockeimer et al., supra note 1, at 20, 33. The demonstrators were followed by a group of police officers and subsequently arrested in the largest group arrest in Phoenix since June 2020 when the racial justice protests began.3White, supra note 1; see Hockeimer et al., supra note 1, at 20,33. The demonstrators were charged with several offenses, including unlawful assembly,4White, supra note 1; Hockeimer et al., supra note 1, at 4. and were surreptitiously added to the state gang database as members of a gang called “ACAB” which law enforcement officials designated as an “extremist” group with “violent tendencies.”5Dave Biscobing, ‘Prime for Abuse’: Lack of Oversight Lets Phoenix Police Add Protesters to
Gang Database, ABC 15 Ariz., (June 5, 2021, 3:08PM), https://www.abc15.com/news/local-news/investigations/protest-arrests/prime-for-abuse-lack-of-oversight-lets-phoenix-police-add-protesters-to-gang-database, [https://perma.cc/YZD9-2W8B]. But local news reports revealed that the ACAB gang was fictional.6. Id. Phoenix police and county prosecutors colluded to target and arrest the protestors, and invented a moniker based on the common protest chant “All Cops Are Bastards.”7. Id.; Hockeimer et al., supra note 1, at 3–4. This incident is one of a growing number of instances where government officials take advantage of nebulous databases to profile, target, and punish unfavorably viewed or politically marginalized individuals or groups.
For some databases, it is the mere fact of inclusion that places individuals and groups in a “suspect” category that can produce harmful results. For others, it is the omission from the database, which can often be deliberate, that places excluded individuals into a “suspect” category where they face punitive and harmful consequences. For example, digital ID databases, like India’s Aadhaar and Uganda’s Ndaga Muntu, are used by multiple facets of those respective governments and are meant to cover all residents or citizens for all relevant government programs. Yet these claims of universal coverage are not reflected in the databases’ implementation. In practice, communities most reliant on state support and some historically marginalized groups bear a vastly disproportionate burden of database errors and tangible harms, such as exclusion from essential services and government benefits.8. See, e.g., Sadiq Naqvi, In Its Current Form, Aadhaar Is Very Coercive and Invasive: Jean Dreze, Catchnews (Aug. 5, 2017, 7:51 PM), http://www.catchnews.com/india-news/in-its-current-form-aadhaar-is-very-coercive-and-invasive-jean-dreze-76224.html, [https://perma.cc/3BWR-L3Q4]
(“One major problem is the exclusion of a significant minority of people for whom the system does not work. Even in Ranchi district, where the system has been in place for a whole year, more than 10% of cardholders are still unable to buy their monthly rations . . . . [I]t translates into something like 2.5 million people in Jharkhand being deprived of their food rations.”);
Ctr. for Hum. Rts. and Glob. Just., Initiative for Soc. and Econ. Rts., & Unwanted Witness, Chased Away and Left to Die, 9 (2021)
(“This research confirms that Uganda’s national ID has become an important source of exclusion for the poorest and most marginalized . . . . Based on the government’s own data and other official sources, the reliability of which we cannot guarantee, we have calculated that anywhere between 23% and 33% of Uganda’s adult population has not yet received a National Identity Card (NIC).”). Exclusion from these databases also facilitates stereotyping. This results because government narratives about the purpose of these systems suggest that those excluded are non-citizens, security risks, fraudsters, or members of other suspect categories, apparently justifying punitive outcomes of exclusion.9. See, e.g., Ctr. for Hum. Rts. and Glob. Just., Initiative for Soc. and Econ. Rts., & Unwanted Witness, supra note 8 (“Because Ndaga Muntu was primarily designed to be a national security system and not a social development program, it is perhaps not surprising that one major effect of the national ID system has been to exclude those who are considered non-Ugandans, security risks, or criminals.”); Davis Langat, Huduma Namba to Boost Security, Kenya News Agency (Dec. 18, 2020), https://www.kenyanews.go.ke/huduma-namba-to-boost-security/ [https://perma.cc/ENZ2-MPUG] (describing how government officials claimed Kenya’s Huduma Namba ID database would be a crime fighting tool used to boost public security); Jean Dreze, Reetika Khera, & Anmol Somanchi, Balancing Corruption and Exclusion: A Rejoinder, Ideas for India, (Sept. 28, 2020) https://www.ideasforindia.in/topics/poverty-inequality/balancing-corruption-and-exclusion-a-rejoinder.html [https://perma.cc/TT2D-JCHM] (highlighting how identity fraud by individuals in the welfare delivery systems was a primary justification for India’s Aadhaar database despite scant evidence that such fraud was a “serious problem”).
These databases have not received the heightened public concern, institutional priority and funding,10There are multiple legislative and other institutional efforts focused on AI. See, e.g., Advancing Artificial Intelligence Research Act of 2020, S.3891, 116th Cong. (2020) (proposing a national program to study AI and promote AI research); FUTURE of Artificial Intelligence Act of 2020, H.R.7559, 116th Cong. (2020) (proposing an advisory committee to advise president on AI issues); William M. (Mac) Thornberry National Defense Authorization Act for Fiscal Year 2021, H.R. 6395, 116th Cong. (2020) (passing some parts of the previously cited bills through omnibus legislation); Intn’l Telecomm. Union (ITU), United Nations Activities on Artificial Intelligence (2019) (identifying frameworks for trustworthy and inclusive AI systems as a key priority across UN agencies). There are also a range of philanthropic and educational organizations dedicating resources to assessing the social impacts of AI’ technologies. See, e.g., Press Release, Knight Found., The Ethics and Governance of Artificial Intelligence Fund Commits .6 Million to Organizations That Bolster Civil Society Efforts Around the World (July 10, 2017), https://knightexaminedfoundation.org/press/releases/the-ethics-and-governance-of-artificial-intelligence-fund-commits-7-6-million-to-organizations-that-bolster-civil-society-efforts-around-the-world/ [https://perma.cc/X5AE-QYYF]; Stanford Univ. Ctr. for Human-Centered A.I., https://hai.stanford.edu/ [https://perma.cc/MVF5-GJHL] (last visited May 8, 2022); Press Release, Univ. of Oxford, University Announces Unprecedented Investment in the Humanities (June 19, 2019), https://www.ox.ac.uk/news/2019-06-19-university-announces-unprecedented-investment-humanities [https://perma.cc/3AKJ-NVN7]; Grants Database, Alfred P. Sloan Found., https://sloan.org/grants-database [https://perma.cc/VBP8-TJ5X] (last accessed Apr. 6, 2022) (showing fourteen grants with the keyword “artificial intelligence”). or policy reform-based research11. See, e.g., Karen Yeung & Martin Lodge, Algorithmic Regulation: An Introduction, in Algorithmic Regulation 2–3 (Karen Yeung & Martin Lodge eds., 2019); Ada Lovelace Inst. & DataKind U.K., Examining the Blackbox: Tools for Assessing Algorithmic Systems (2020) https://www.adalovelaceinstitute.org/wp-content/uploads/2020/04/Ada-Lovelace-Institute-DataKind-uk-Examining-the-Black-Box-Report-2020.pdf [https://perma.cc/GCY8-D9UK]; David Freeman Engstrom & Daniel E. Ho, Algorithmic Accountability in the Administrative State, 37 Yale J. Regul. 800 (2020); Deirdre K. Mulligan & Kenneth A. Bamberger, Procurement as Policy: Administrative Process for Machine Learning, 34 Berkeley Tech. L. J. 773 (2019). directed at artificial intelligence (AI) or algorithmic/automated decision-making systems (ADS). For example, when the Chicago Police Department announced that it would no longer use its controversial predictive policing program in response to public criticism, it proudly announced plans to revamp its heavily criticized gang database to minimal outcry.12Annie Sweeney and John Bryne, Chicago Police Announce New Gang Database as Leaders Hope to Answer Questions of Accuracy and Fairness, Chi. Trib. (Feb. 26, 2020, 4:29 PM), https://www.chicagotribune.com/news/breaking/ct-chicago-police-gang-database-overhaul-react-20200226-gisz55rytzbsdkyy4kmbb4jrou-story.html [https://perma.cc/8XC5-L3WD]; Jeremy Gorner & Annie Sweeney, For Years Chicago Police Rated the Risk of Tens of Thousands Being Caught Up in Violence. That Controversial Effort Has Quietly Been Ended, Chi. Trib. (Jan. 24, 2020, 8:55 PM) https://www.chicagotribune.com/news/criminal-justice/ct-chicago-police-strategic-subject-list-ended-20200125-spn4kjmrxrh4tmktdjckhtox4i-story.html [https://perma.cc/4RDH-ARMP]; Chicagoans for an End to the Gang Database v. City of Chicago, MacArthur Just. Ctr. (last accessed May 7, 2022), https://www.macarthurjustice.org/case/chicagoans-for-an-end-to-the-gang-database/ (criticizing the gang database for lacking guidelines when adding people to the database); Rashida Richardson & Amba Kak, It’s Time for a Reckoning About This Foundational Piece of Police Technology, Slate (Sept. 11, 2020) https://slate.com/technology/2020/09/its-time-for-a-reckoning-about-criminal-intelligence-databases.html [https://perma.cc/NXA7-ZEUV]; see also Joseph M. Ferguson & Deborah Witzburg, City of Chi., Off. of Inspector Gen., Follow-Up Inquiry On The Chicago Police Department’s “Gang Database” 29 (2021) (“CPD has consistently maintained that its collection of gang information is critical to its crime fighting strategy but has not yet clearly articulated the specific strategic value of the data to be collected in the planned CEIS.”). Meanwhile, arguably all of the most controversial technology projects across the Global South in the last decade have been large-scale database projects, which receive relatively little media attention and research funding.13. See Payal Arora, The Bottom of the Data Pyramid: Big Data and the Global South, 10 Int’l J. of Commc’n 1681 (2016) (arguing that database projects in the Global South have been neglected in critical big data discourse despite their profound impacts on surveillance, privacy, and equity). Instead, in popular and policy discourse, databases are characterized as foundational and passive raw material that enables the creation of more advanced algorithmic systems that can sort, prioritize, predict, and so on.14For accounts that describe databases as enabling more sophisticated technologies, see generally Woodrow Hartzog & Evan Selinger, I See You: The Databases That Facial-Recognition Apps Need to Survive, Atlantic (Jan. 23, 2014), https://www.theatlantic.com/technology/archive/2014/01/i-see-you-the-databases-that-facial-recognition-apps-need-to-survive/283294/ [https://perma.cc/WE8F-82WQ]; Kevin Driscoll, From Punched Cards to “Big Data”: A Social History of Database Populism, 1 commc’n +1, 1, 2 (2012) (“Implicit in this metaphor is a database—or, more likely, a network of databases—from which the engine (code) draws its fuel (data.)”); Martin Lodge & Andrea Mennicken, Reflecting on Public Service Regulation by Algorithm, in Algorithmic Regulation 178, 185 (Karen Yeung & Martin Lodge eds., 2019) (“[T]he use of different databases built from tax returns, complaints data, social media commentary, and such like—offers the opportunity to move away from a reliance on predefined performance metrics towards bringing together different types of data.”). Because AI and ADS are viewed as more complex technologies, databases are invariably positioned by many scholars as necessary for the creation and maintenance of, but “subordinate” to, these newer systems.15Driscoll, supra note 14, at 2
(“In the emerging scholarship concerning the role of algorithms in online communication, databases are often implicated but rarely of principle concern. This subordinate position may be due to the ambiguous relationship between algorithm and database. Whereas an algorithm, implemented in running code, is self-evidently active, a database appears to serve a largely passive role as the storehouse of information.”). This is compounded by a lack of definitional clarity in public discourse around what does or does not “count” as AI, given vague and evolving technical thresholds.16P.M. Krafft, Meg Young, Michael Katell, Karen Huang & Ghislain Bugingo, Defining AI in Policy Versus Practice, arXiv cs.CY, Dec. 2019, https://arxiv.org/pdf/1912.11095.pdf [https://perma.cc/RH4E-MND2]); Ian Bogost, ‘Artificial Intelligence’ Has Become Meaningless, Atlantic (Mar. 4, 2017), https://www.theatlantic.com/technology/archive/2017/03/what-is-artificial-intelligence/518547/ [https://perma.cc/T65B-6WEG]. This definitional ambiguity has policy implications as well, where the meanings ascribed to the terms AI or ADS can determine the scope of any regulatory effort.17Rashida Richardson, Defining and Demystifying Automated Decision Systems, 81 Md. L. Rev. 1, 3–9 (s2022).
Recent policy discourse, now commonly referred to under the rubric of “algorithmic accountability,”18. See Yeung & Lodge, supra note 11, at 9–10; see also Ada Lovelace Inst. & DataKind U.K., supra note 11; Engstrom & Ho, supra note 11; Mulligan & Bamberger, supra note 11. has drawn considerable attention to how algorithmic systems serve to entrench or exacerbate systemic and historical discrimination against marginalized groups. While this emphasis on a deeper and contextual understanding of social harms is welcome, emergent policy frameworks for evaluating these systems, like Algorithmic Impact Assessments (AIAs) and algorithmic audits, are limited in their scope of review and analysis; these proposals still struggle to unearth the complete spectrum of structural technological inequities and tacit modes of discipline, control, and punishment.19. See generally Jacob Metcalf, Emanuel Moss, Elizabeth Watkins, Ranjit Singh & Madeleine Clare Elish, Algorithmic Impact Assessments and Accountability: The Co-Construction of Impacts, in FAccT ‘21: Proceedings of the 2021 ACM Conf. on Fairness, Accountability, and Transparency 735 (2021) (exploring limitations of impact assessments in algorithmic and non-algorithmic domains); Rashida Richardson, Racial Segregation and the Data-Driven Society: How Our Failure to Reckon with Root Causes Perpetuates Separate and Unequal Realities, 36 Berkeley Tech. L. J., 101, 126–34 (forthcoming 2022) (demonstrating how evaluations of algorithmic systems in the wild overlook relevant historical and social contexts that affect algorithm performance); Alfred Ng, Can Auditing Eliminate Bias from Algorithms?, Markup (Feb. 23 2021) https://themarkup.org/ask-the-markup/2021/02/23/can-auditing-eliminate-bias-from-algorithms [https://perma.cc/ANS2-BSMB] (highlighting how bias audits were mischaracterized by companies, which brings their use into question); Ada Lovelace Inst. , Algorithmic Accountability for the Public Sector: Learning From the First Wave of Policy Implementation 21–28 (2021), https://www.opengovpartnership.org/wp-content/uploads/2021/08/algorithmic-accountability-public-sector.pdf [https://perma.cc/46SV-Y2KS] (highlighting real-world use cases of algorithmic impact assessments and audits, and discussing the limitations and assumptions of these frameworks).
Responding to this combination of definitional ambiguity for policy intervention and the lack of a systematic way to evaluate harms, this Article proposes “Suspect Development Systems” (SDS) as both a definitional category and framework for analysis. SDS can be defined as (1) information technologies used by government and private actors, (2) to manage vague or often immeasurable social risks20Social risk is a normative concept. Because “institutions are not organized around a
single, cohesive notion of order,” there is no universal definition of social risks. Instead, institutions have “unique definitions of risk” and logics for managing or dealing with risks. See, e.g., Richard V. Ericson & Kevin D. Haggerty, Policing the Risk Society 43 (1997); see also, Glossary, in Criminalization, Representation, Regulation: Thinking Differently About Crime 443 (Deborah Brock, Amanda Glasbeek & Carmela Murdocca eds., 2014) (defining risk as “[a] particular discourse that has emerged alongside neoliberalsm through which events like crime are imagined.”). based on presumed or real social conditions (e.g., violence, corruption, substance abuse), (3) that subject targeted individuals or groups to greater suspicion, differential treatment, and punitive and exclusionary outcomes.
We conceptualize SDS as a normative category that acknowledges how technologies like AI, ADS, and databases amplify structural inequities and the modes through which they discipline and control individuals and groups.21For a Foucauldian analysis of “disciplinary power” and “population management power,” see Dean Spade, Normal Life: Administrative Violence, Critical Trans Politics, and the Limits of Law 52–72 (2015)
(“These programs operate through purportedly neutral criteria aimed at distributing health and security and ensuring order. They operate in the name of promoting, protecting, and enhancing the life of the national population and, by doing so, produce clear ideas about the characteristics of who the national population is and which ‘societal others’ should be characterized as ‘drains’ or ‘threats’ to that population.”). Systems described and understood as “databases” are by no means the only kinds of SDS. Indeed, more recent ADS tools like predictive policing are clear examples of SDS and would benefit from the multi-pronged analysis set forth in this Article. Overall, SDS offers a broader framework within which to build an advocacy agenda that is inclusive of this diverse range of systems, rather than focusing only on the most recent or egregious examples of information technologies. However, in this Article, we focus predominantly on databases as they have received relatively less attention than AI or ADS and would especially benefit from strategic reframing. This Article is in conversation with criminology and surveillance studies scholarship that illuminates the reasons why databases might have proliferated as a key technique of penal governance within the criminal justice system and beyond.22. See Katja Franko Aas, From Narrative to Database: Technological Change and Penal Culture, 6 Punishment & Soc’y 379 (2004); Michaelis Lianos & Mary Douglas, Dangerization and the End of Deviance: The Institutional Environment, 40 Brit. J. of Criminology, 261 (2000); Toshimaru Ogura, Electronic Government and Surveillance-Oriented Society, in Theorizing Surveillance: The Panopticon and Beyond (David Lyon ed., 2006); Gary T. Marx, Undercover: Police Surveillance in America, 208-29 (1988). For an analysis of the impact of databases in social work, see Nigel Parton, Changes in the Form of Knowledge in Social Work: From the ‘Social’ to the ‘Informational’?, 38 Brit. J. of Soc. Work 253–69 (2008). This work explores how the decontextualized and “byte-like”23 . Scott Lash, Critique of Information 2 (2002). mode of managing information within the seemingly objective structures of the computerized database has enabled the ground-level state apparatus to be more detached and unaccountable for the consequences for these decisions. This veneer of objectivity and routineness associated with databases contributes to the lack of public and regulatory scrutiny when they are introduced.
This visual illustrates that SDS can be applied to a broad range of technological systems, including systems that are identified in popular and policy discourse as Automated Decision Systems (ADS), databases, machine-learning based systems, or paper-based information systems. SDS provides a normative categorization that deliberately de-centers the technological form of the system, instead calling attention to its pernicious social impacts of creating suspect categories of people. While the wealth of examples in this Article demonstrates the ubiquity of SDS across social domains, the visual clarifies that not all algorithmic systems or databases will be a type of SDS. For example, driver’s license databases, school assignment algorithms, or price-discriminating machine-learning-based algorithms on e-commerce platforms do not fit our definition of SDS, even as they might attract other forms of critical inquiry.
Criminal intelligence databases, which have been used by police agencies for over a century to profile and target based on the risk of criminality, are paradigmatic examples of SDS.24. See David L. Carter, U.S. Dept. of Just., Law Enforcement Intelligence: A Guide for State, Local, and Tribal Law Enforcement Agencies 31–49 (2d ed. 2009) (detailing the history of U.S. law enforcement intelligence practices including the creation and use of criminal intelligence databases). “Criminal intelligence databases are populated with information about people who should be watched or monitored because they might have committed a crime or might commit a future crime.”25James B. Jacobs, The Eternal Criminal Record 15 (2015) Notable examples of criminal intelligence databases include sex offender registries, prescription drug monitoring programs, London’s Crimint, and the FBI’s National Crime Information Center, which hosts the agency’s controversial Terrorist Watchlist and the Violent Person file. In this Article, we examine different variants of such databases in the U.S., the U.K., and India. Despite different socio-political contexts and institutional histories, these databases share a range of common features, conditions, and consequences.
Modern gang databases in the U.S. have existed since at least the late 1980s26Stacey Leyton, The New Blacklists: The Threat to Civil Liberties Posed by Gang Databases, in Crime Control and Social Justice: The Delicate Balance 109, 111 (Darnell F. Hawkins, Samuel L. Myers, Jr. & Randolph N. Stone, eds., 2003). and continue to evolve and proliferate as crime-fighting tools alongside newer modalities, such as predictive policing. Gang databases exist as both centralized and decentralized information systems primarily used by criminal justice actors and institutions to accumulate, analyze, and disseminate information about gangs and gang members for a variety of interests and priorities.27James B. Jacobs, Gang Databases: Context and Questions, 8 Criminology & Pub. Pol’y 705, 705–08 (2009). These databases are compiled and used by various criminal justice actors and institutions based on the belief that they function as a “force” or institutional multiplier increasing the overall efficiency, speed, and performance of all agencies without having to increase staffing or expend additional funds.28. See James Lingerfelt, Technology as a Force Multiplier, in Technology for Community Policing Conference Report 29 (1997). Gang databases exist to make intelligence and investigative information accessible to various government actors and institutions, though the needs and rationales for use vary.29. See Leyton, supra note 6, at 109–12; Jacobs, supra note 27, at 705–07, Joseph M. Ferguson, City of Chi. Off. of Inspector Gen., Review of the Chicago Police Department’s “Gang Database” 25–31 (2019). Gang databases have been used to secure mass arrests and indictments, but since such actions have disproportionately included Black and Latinx youth,30. See, e.g., Citizens for Juv. Just., We are the Prey: Racial Profiling of Youth in New Bedford, 20–22 (2021); K. Babe Howell, Gang Policing: The Post Stop-and-Frisk Justification for Profile-Based Policing, 5 U. Denv. Crim. L. Rev. 1, 16–17 (2015). these databasing efforts have also been the source of litigation31. See, e.g., Complaint, Chicagoans for an End to the Gang Database v. City of Chicago, No. 18-cv-4242 (N.D. Ill. June 19, 2018) (voluntarily dismissed Sept. 2, 2020) https://www.macarthurjustice.org/wp-content/uploads/2018/06/cpd_gang_database_class_action_complaint.pdf [https://perma.cc/PQ7E-HK8N]; Steph Machado, Community Group Files Suit Over Providence ‘Gang Database’, WPRI (July 23, 2019, 9:59 PM), https://www.wpri.com/news/local-news/providence/community-group-files-suit-over-providence-gang-database/ [https://perma.cc/LLQ2-TAVY]. and highly critical public reports.32. See, e,g,, Ferguson & Witzburg, supra note 12.
In the U.K., while intelligence files on gangs and gang members have been maintained for decades, the early 2000s saw the computerization of gang databases in large cities. In this Article, we focus on the Gangs Violence Matrix (“Gangs Matrix”), a database of purported gang members launched in 2012 as part of the U.K. Government’s “war on gangs.” The Gangs Matrix is used as a risk management tool to assess and rank suspected gang members deemed most likely to commit a violent crime and to inform local police strategies to suppress violent crime.33James A. Densley & David C. Pyrooz, The Matrix in Context: Taking Stock of Police Gang Databases in London and Beyond, 20 Youth Just. 11, 17–18 (2020); Amnesty Int’l, Trapped in the Matrix: Secrecy, Stigma, and Bias in the Met’s Gangs Database 2–3 (2018) https://www.amnesty.org.uk/files/reports/Trapped%20in%20the%20Matrix%20Amnesty%20report.pdf [https://perma.cc/QJE8-JUGH]. According to recent accounts, at any given point, there are an estimated 3,000–4,000 individuals listed in the Gangs Matrix.34Densley & Pyrooz, supra note 33, at 11; Amnesty Int’l, supra note 33, at 15. The purpose of this databasing effort is ostensibly to “audit” the gang landscape of cities, target individuals with heightened surveillance, as well as deterrence messaging or “nudges” to leave gang life.35. See Amnesty Int’l, supra note 33, at 2–4, 20–22; Gang Violence Matrix, Metro. Police, https://www.met.police.uk/police-forces/metropolitan-police/areas/about-us/about-the-met/gangs-violence-matrix/ [https://perma.cc/GMS8-KBCQ] (last visited May 7, 2022).
In India, there are a range of police databases with records of individuals who are believed to be deserving of heightened surveillance on account of their perceived dangerousness and likelihood of committing future crimes.36. See Mrinal Satish, “Bad Characters, History Sheeters, Budding Goondas and Rowdies”: Police Surveillance Files and Intelligence Databases in India, 23 Nat’l L. Sch. India Rev. 133 (2011). These databases have their roots in colonial legislation that targeted specific communities that were designated as “criminal tribes.”37The Criminal Tribes Act of 1871 led to the branding of entire Indian communities as “criminal tribes,” followed by record-keeping of their details and physical movements. These requirements imposed severe restrictions on movement, routine physical surveillance, and “limited…access to legal redress.” See Mark Brown, Postcolonial Penality: Liberty and Repression in the Shadow of Independence, India c. 1947, 21 Theoretical Criminology 186, 186 (2017). Other accounts argue that notions of group criminality associated with these tribes in fact pre-dates colonial India as a fall-out of the Hindu caste system, and that the introduction of the CTA simply systematized this discrimination and made these groups vulnerable to the constant threat of surveillance and violence at the hands of state actors. See, e.g., Shivangi Narayanan, Guilty Until Proven Guilty: Policing Caste Through Preventive Policing Registers in India, 5 J. Extreme Anthropology 111 (2021); Mukul Kumar, Relationship of Caste and Crime in Colonial India: A Discourse Analysis, 39 Econ. & Pol. Wkly. 1078 (2004); Anastasia Piliavsky, The “Criminal Tribe” in India Before the British, 57 Compar. Stud. in Soc’y & Hist. 373 (2015). This policing approach towards marginalized communities continues, in part through the operation of surveillance databases.38Satish, supra note 36, at 135. The current material form of these databases39For an explanation of the terms “surveillance databases” and “surveillance registers” to describe these record keeping systems, see Mrinal Satish, “Bad Characters, History Sheeters, Budding Goondas and Rowdies”: Police Surveillance Files and Intelligence Databases in India, 23 Nat’l L. Sch. India Rev. 133 (2011). continues to be paper records stored in official files called “registers,”40For a listing of different categories of surveillance registers, see Santana Khanikar, State, Violence, and Legitimacy in India, 41 (2018). but there are ongoing efforts to digitize these records.41Ameya Bokil, Avaneendra Khare, Nikita Sonavane, Srujana Bej, & Vaishali Janarthanan, Settled Habits, New Tricks: Casteist Policing Meets Big Tech in India, TNI (May 2021), https://longreads.tni.org/stateofpower/settled-habits-new-tricks-casteist-policing-meets-big-tech-in-india [https://perma.cc/3CKG-7B2B]. This is part of a broader institutional push for digitization of government functions in low and middle-income countries as a metric of economic development. There are multiple kinds of surveillance databases maintained by each police station that cover suspect individuals residing in that particular precinct. While the official and colloquial names for such databases and the criteria for inclusion vary from state to state, common categories include the “history sheet,” “bad character” registers, and “rowdy” registers.42Satish, supra note 36, at 135, 140–41, 146. Common amongst these databases is their inclusion of individuals who lack existing criminal records on the grounds that police believe they are likely to commit crimes or “disturb the peace.” Being included in these databases triggers a range of consequences like being subjected to heightened physical surveillance (e.g., regular home visits), increased chances of arrest, and unfavorable bail and sentencing decisions.43. Id. at 133–34.
These databases can all be helpfully understood as SDS in the way that they manage vague and often immeasurable social risks, subjecting targeted individuals to greater suspicion, differential treatment, and punitive and exclusionary outcomes. While they present as bureaucratic systems of record-keeping and classification, we explore the myriad forms of disciplinary power they exert over marginalized and historically stigmatized groups. The bureaucratic patina of these systems can conceal the violent nature of this form of control, given the looming threat of scrutiny and brutality that often accompanies their use.44On the violent character of bureaucratic systems see Spade, supra note 21, at 22–29. Analyses of databases, like SDS, should be differentiated from the literature on “automated suspicion algorithms”45. See Michael L. Rich, Machine Learning, Automated Suspicion Algorithms, and the Fourth Amendment, 164 U. Pa. L. Rev. 871 (2016). or “big data blacklists.”46Margaret Hu, Big Data Blacklisting, 67 Fla. L. Rev. 1735 (2016). The latter provides legal frameworks to analyze the use of automated statistical tools that predict and identify individuals deserving of additional scrutiny and other restrictions on account of the social risk they present. Instead of focusing on technical tools, our framework of SDS, shaped by case studies on criminal intelligence databases, centers on the grounded social, political, and economic contexts, organizational practices and technical features that structure databases. In many of these systems, the decision of who is “suspicious” is not solely determined by statistical or other automated tools, but instead by government officials who are in turn influenced by organizational, legal, and social practices. In taking this approach, we hope to address not just emergent and advanced forms of government databases, but also provide a category that is broad enough to encompass the impact of so-called ‘legacy’ or manual systems that continue to operate alongside more recent “database” modes.
The SDS category and framework can help create a common discourse around systems that cut across spheres of governance and bridges geographical and temporal divides. Similar to the creation and maintenance of “suspect” categories of people like the habitual offender or the gang member, racialized tropes coming out of political discourse, like “the welfare-dependent mother” or “the illegal immigrant” are replicated in database use and in other sectors of governance and the private sector as well.47Philip Alston, Report of the Special Rapporteur on Extreme Poverty & Human Rights 10, U.N. Doc. A/74/493, (Oct. 11, 2019) (noting that conservative politicians have historically employed
tropes to discredit inclusive welfare policy); U.S. Fed. Trade Comm’n, Data Brokers: A Call for Transparency and Accountability 29, C-3 (2014) (describing how negative stereotypes and tropes can be replicated in consumer profiles created and shared by data brokers). The SDS framework illuminates the connections between recent moves towards digitizing welfare databases in the U.K.;48 . Amos Toh, Human Rts. Watch, Automated Hardship: How the Tech-Driven Overhaul of the U.K.’s Social Security System Worsens Poverty (2020). the new Argentinian database that facilitates targeting of minors that are “alleged offenders”;49Karen Hao, Live Facial Recognition is Tracking Kids Suspected of Being Criminals, MIT Tech. Rev. (Oct. 9. 2020), https://www.technologyreview.com/2020/10/09/1009992/live-facial-recognition-is-tracking-kids-suspected-of-crime/ [https://perma.cc/3UZZ-SWN5]. the extensively detailed police database maintained by Chinese police agencies that is crucial to the surveillance, persecution and large-scale internment of Muslims in the Xinjiang region;50Yael Grauer, Revealed: Massive Chinese Police Database, Intercept (Jan. 29, 2020, 3:00 AM), https://theintercept.com/2021/01/29/china-uyghur-muslim-surveillance-police/ [https://perma.cc/384Y-N8MM] the proliferation of nationwide mandatory biometric ID databases in developing countries;51. See Access Now, National Digital Identity Programmes: What’s Next? 7–17 (Nov. 2019), https://www.accessnow.org/cms/assets/uploads/2019/11/Digital-Identity-Paper-Nov-2019.pdf [https://perma.cc/6YJJ-X29J] (highlighting case studies from India, Tunisia, and Estonia). and the massive citizenship databases being introduced for immigration control in the European Union, India, and the U.S.52. See Databases for Deportation, Statewatch, https://www.statewatch.org/deportation-union-rights-accountability-and-the-eu-s-push-to-increase-forced-removals/deportations-at-the-heart-of-eu-migration-policy/databases-for-deportations/ [https://perma.cc/64CY-HKCR] (explaining how “[t]he political decision to try to step up expulsions from the EU has led to the transformation of existing databases and the introduction of new ones”). See also infra Section II.C, for descriptions of databases in U.S. and India. This framework also allows us to draw connections between historical practices and modern database usage/developing “suspect” categories of people, such as linking current gang database practices in the U.K. to the British colonial strategy of criminalizing entire communities (designated “criminal tribes”)53Jasbinder S. Nijjar, Echoes of Empire: Excavating the Colonial Roots of Britain’s “War on Gangs”, 45 Soc. Just. 147, 151 (2018). and extrapolating lessons from the use of watchlists to suppress political dissent in the U.S. during the Civil Rights and anti-Vietnam war movements.54. See, e.g., Electronic Surveillance Within the United States for Foreign Intelligence Purposes: Hearing on S. 3197 Before the Subcomm. On Intel. And the Rts. Of Ams. Of the S. Select Comm. On Intel., 94th Cong. (1976); Frank Kusch, Battleground Chicago: The Police and the 1968 Democratic National Convention (2008); Jeffrey Haas, The Assassination of Fred Hampton: How the FBI and the Chicago Police Murdered a Black Panther (2010). Each of these government data projects has defined who is included and who is excluded from the vision of the nation state and has facilitated the conditions for profiling, stigmatizing, or even eliminating those who are excluded. For criminal intelligence databases, inclusion into these databases trigger these consequences, whereas, in welfare or citizenship databases, exclusion or removal from databases is used to punish or otherwise disempower individuals and groups.
The Article is organized as follows. In Section I, through a detailed examination of the use of criminal intelligence databases in the U.S., U.K., and India, we present a framework of features that are crucial to understanding and evaluating SDS. In this Section, we explore five categories of features: technical, legal, political economy, organizational, and social. Each category distills the common features and insights observable in these SDSs, which remain underexplored in existing algorithmic accountability discourse. Section II demonstrates the value of SDS as a framework applied to contemporary government databases outside of the criminal justice system. We apply the framework analysis to other systems that could helpfully be understood as SDS: welfare system databases, universal or ID number databases, and citizenship databases. Finally, we conclude with a range of insights regarding how our analytical framework of SDS offers conceptual, legal, and strategic insights towards addressing the harms caused by these systems or doing away with them altogether where such harms cannot be remedied. SDS, as a defined term and analytical framework, provides an expanded lens with which to define and identify ADS systems by offering a systematic understanding of the socio-technical context, without which the harms of these systems cannot be addressed.
I. Suspect Development System Framework
SDSs are typically unique to the jurisdictions in which they are used because local conditions influence their development and use. However, as their use and consequences become more prevalent globally, such jurisdictional distinctions may be less salient and practical. Policymakers, advocates, and scholars are increasingly interested in identifying, evaluating, and addressing the implications of emergent technologies in systematic and concentrated ways, an approach that some say can only be achieved if there is a categorical framework for analysis.55. See, e.g., Joshua P. Meltzer, Cameron Kerry, & Alex Engler, Submission to the EC White Paper on Artificial Intelligence (AI): The Importance and Opportunities of Transatlantic Cooperation on AI (2020), https://www.brookings.edu/wp-content/uploads/2020/06/AI_White_Paper_Submission_Final.pdf [https://perma.cc/QW2R-WLK3] (emphasizing the importance of global cooperation on regulation to allow governments to maintain legal rules and values); Inst. of Elec. & Elec. Eng’rs, IEEE-USA Position Statement: Artificial Intelligence Research, Development, & Regulation 2 (2017), https://ieeeusa.org/wp-content/uploads/2017/07/FINALformattedIEEEUSAAIPS.pdf [https://perma.cc/RB8P-XXNC] (recommending that a federal interagency panel should determine how to coordinate and enforce federal AI regulation); Jennifer Kuzma, Jordan Paradise, Gurumurthy Ramachandran, Jee-Ae Kim, Adam Kokotovich & Susan M. Wolf, An Integrated Approach to Oversight Assessment for Emerging Technologies, 28 Risk Analysis 1197 (2008) (proposing an integrated oversight assessments approach for evaluating emerging technologies).
In this Section we present a framework for SDS that includes five categories of features: technical, legal, political economy, organizational, and social.56. See generally Stephen D. Mastrofski & James J. Willis, Police Organization Continuity and Change: Into the Twenty-First Century, 39 Crime & Just. 55, 79 (2010) (noting that technologies have a number of components including material, logical, and social). We use the term “features” expansively to include typical characteristics, structural conditions, consequences of SDS development, and use. These categories were developed by examining criminal intelligence databases used by law enforcement and other government actors to profile and target people considered to be a social risk for criminality in the U.S., U.K., and India.57We examined the Gangs Violence Matrix in the U.K., at least six state and local gang databases in the U.S., and a range of surveillance databases in at least five states in India. These databases have different colloquial terminologies (such as gang or habitual offender) and rhetorical accounts that convey who will be targeted and the social risks they pose to a wide variety of audiences, yet they share common or universal features and collectively provide insights that remain unexamined in legal scholarship58Most legal scholarship regarding data-driven technologies generally and the criminal justice system specifically tend to focus on specific technologies or sectoral concerns. Though there are a few notable, recent publications that seek to explore legal and social issues through a categorical framework. See, e.g., Aziz Z. Huq, Racial Equity in Algorithmic Criminal Justice, 68 Duke L.J. 1043 (2019); Vincent M. Southerland, The Intersection of Race and Algorithmic Tools in the Criminal Legal System, 80 Md. L. Rev. 487 (2021). and policy discourse. Thus, we chose to examine countries in the Global North and South that represent different stages of SDS development to explore where such commonalities emerge and how they can expand our understanding of how these seemingly disparate technologies are coaxial.
Our framework is not exhaustive. Instead, we have identified three key features in each category to demonstrate the analytical utility of each category and the framework, while leaving other features open to future development and expansion. Additionally, no category can or should be understood in isolation. The interactions within and between these framework categories influence how SDSs function, the ways they are used, and the outcomes they produce. For example, as political theorist Langdon Winner has argued, “the adoption of a given technical system actually requires the creation and maintenance of a particular set of social conditions as the operating environment of that system.”59 . Langdon Winner, The Whale and the Reactor: A Search for Limits in an Age of High Technology 32 (1986). For a discussion of SDS social features, see infra Section I.E. Similarly, structures and practices explored in these organizational categories can help explain the practical implications of SDS use, in addition to gaps or oversights in relevant legal rules and regulations reviewed in the legal category. Thus, our framework can enable robust and tactical assessments that promote a more nuanced and holistic understanding of the uses, risks, and consequences of SDS, particularly providing insight into how these technologies amplify cumulative disadvantage60Cumulative disadvantage is defined as “the ways in which a decision based on the evaluation of the groups in which an individual has been assigned by chance, or by ill-informed choice, shapes the opportunities that are available to her.” Oscar H. Gandy, Jr., Coming to Terms with Chance: Engaging Rational Discrimination and Cumulative Disadvantage 74 (2009). and structural inequities. This framework can both inform civil society advocacy and research agendas, as well as be operationalized by state and private actors in deciding whether (and how) to develop and use such systems or governing relevant spheres of influence.
A. Technical Features
The technological features of SDSs shape and structure how these systems generate information and knowledge for the entities that use them. This includes tangible characteristics of these systems such as: types of physical or digital interfaces; modes of collecting, storing, and managing information; and statistical techniques that might be incorporated into these systems. The objective here is not to create technical thresholds for what does and does not count as an SDS. Indeed, this framework seeks to de-center technical descriptors that are often used to create harmful hierarchies of analysis between more and less “advanced” systems. Through a detailed examination of criminal intelligence databases across contexts, this category highlights the range and variability in how such systems are designed, the intensity of technical tasks they afford, and the complex relationship between paper-based records and digital databases. These features are typically communicated in technical jargon or mathematical terms that cause misinterpretations outside of the disciplinary contexts in which they were coined.61. See Stephen C. Rea, A Survey of Fair and Responsible Machine Learning and Artificial Intelligence: Implications of Consumer Financial Services, 20–25 (2020) (describing semantic gaps with respect to how AI concepts are used or misinterpreted in different disciplines and contexts); Richardson, supra note 17, at 788. (explaining how the misappropriation of technical jargon from different disciplines and contexts is ill-advised and causes confusion). This jargon can also serve to mystify or obfuscate these systems in ways that prevent critique or broader public engagement.62Tressie McMillan Cottom, Where Platform Capitalism and Racial Capitalism Meet: The Sociology of Race and Racism in the Digital Society, 6 Socio. Race & Ethnicity 441, 443 (2020) (arguing that the use of “needlessly complex technical jargon” is one aspect of an obfuscation strategy used by individual and institutional actors to inhibit access to information that could reveal inherent biases in technology). For these reasons, we focus this category on relating particular technical features in terms of the functionality they afford their users and the consequences these design choices present.
First, distilled to their most basic function, SDSs include simple classification systems that can be manual or digital. Criminal intelligence databases are manual or digital classification systems that categorize individuals or groups into labels, such as habitual offenders or gang members. Along with names, these records can include a range of details about a person including photographs or other biometric information, contact information (e.g., residence, phone number), demographic information (e.g., age, gender, religion, race), identifying marks (e.g. tattoos), socioeconomic information (e.g. marital status), criminal history and membership of a particular group, and names of affiliates (either members of the same gang or informal associations with others on the list). Surveillance databases in India, for example, have a great deal of continuity with those used decades earlier during colonial rule that included the early use of visual cues like different colored ink to categorize an individual’s level of dangerousness.63Radhika Singha, Punished by Surveillance: Policing ‘Dangerousness in Colonial India, 1872–1918, 49 Mod. Asian Stud. 241, 245 (2015) (“The process . . . was one which turned the badmaash of popular discourse into a ‘red-ink ‘or’ black-ink’ badmaash, according to the colour of the ink used to categorize him in a police surveillance register.”). “Badmaash” here refers to the Hindi language colloquial term for person with bad character, or miscreant. See Alasatir Richard McClure, Violence, Sovereignty, and the Making of Colonial Criminal Law in India, 1857-1914, 6 (July 2017) (PhD dissertation, University of Cambridge) https://www.repository.cam.ac.uk./bitstream/handle/1810/268185/McClure-2017-PhD.pdf?sequence=3&isAllowed=n [https://perma.cc/Y36H-957X]. Color-coding continues to be used to highlight different degrees of risk, like in California’s gang database, CalGang, and the United Kingdom’s Gangs Matrix.64. See Leyton, supra note 26, at 111 (“The system also includes photographs . . . color coded to show associates’ gang affiliations.”); Amnesty Int’l, supra note 33, at 6–7 (“The ‘harm score’ assiigned to each individual on the matrix is labelled red, amber or green.”). Paper-based systems in India are intended to be digitized as part of a broader digital policing reform project,65. See, e.g., Delhi Police, Mission Mode Project: RFP for System Integrator (I), https://silo.tips/download/delhi-police-government-of-delhi [https://perma.cc/EA5G-8KAD] (The RFP states that the CCTNS digital policing project will include police records, including of habitual offenders, “since inception”, and that such system should be designed to “capture the details of History Sheets/dangerous/ habitual offenders.”). but evidence from the U.S. suggests that the transition from manual to digital systems is not a binary shift, but instead a messy and negotiated process given that police officers often have greater familiarity and comfort with and fidelity to paper files. For example, accounts of CalGang’s upgrades describe how many officers were reluctant to replace paper files or input these records into the database,66Cal. Dept. of Justice, Technology Acquisition Project Case Study: California Department of Justice CAL/GANG System 6 (Draft Report), (forthcoming) (on file with authors). and the Chicago Police Department’s (CPD) CLEAR system is still described as a patchwork of digitized paper files along with other more advanced computer application modules.67. See Ferguson, supra note 29, at 18–21.
Second, SDSs can incorporate additional technical affordances like relational or link analysis, search functionality, and interoperability with other databases. Beyond the basic functions of recording and categorizing individuals, gang databases across the U.S. include additional features or third-party applications that may be integrated into the web browser-like interfaces.68. See, e.g., N.Y.C. Police Dep’t, Criminal Group Database: Impact & Use Policy 8 (2021) https://www1.nyc.gov/assets/nypd/downloads/pdf/public_information/post-final/criminal-group-database-nypd-Impact-and-use-policy_4.9.21_final.pdf [https://perma.cc/TW2B-UJCU] (“The NYPD purchases Criminal Group Database associated equipment or Software as a Service (SaaS)/software from approved vendors.”). One such additional functionality of such systems is the ability to use a search tool to query the database for records of a particular individual, members of a particular gang, or entries from a particular geographic area.69. See Ferguson, supra note 29, at 18–23. These databases also typically enable inter-and intra-government information sharing capabilities which allow police and other officials to query records across databases.70. See Leyton, supra note 26, at 113; Ferguson, supra note 29, at 24–26. Notably the plans to digitize policing databases in India and Nigeria highlight search and interoperability as a primary objective. See Ministry of Home Affairs Nat’l Crime Records Bureau, Gov. of India, CCTNS Good Practices and Success Stories 107 (2018), https://ncrb.gov.in/sites/default/files/Compiled-Compendium.pdf [https://perma.cc/Q45K-W33W] (positing that the CCTNS database “is a Google type search for Police Department.”). Another functionality observable across multiple gang databases is the mapping of relationality between individuals within the database. CalGang, whose design became the template for the commercially available GangNet used widely in the U.S. and Canada, was created to be this kind of relational database accessible through web portals.71. See Leyton, supra note 26, at 111. See also, Raymond Dussault, GangNet: A New Tool in the
War on Gangs, Gov’t Tech. (Dec. 31, 1997), https://www.govtech.com/magazines/gt/GangNet-A-New-Tool-in-the.html [https://perma.cc/ZY5K-SAQF] (detailing uses of the commercially licensed GangNet system). It allows for over two hundred data fields for individuals listed in the database, and “can generate ‘link diagrams’ of gang associates out to three levels, which include photographs and are color coded to show associates’ gang affiliations.’”72. See Leyton, supra note 26, at 111. The Chicago Police Department’s system also includes modules for “link analysis,” which claim to perform social network analysis by connecting “individuals with records of crimes for which they were arrested as well as individuals with whom they were arrested.”73. Citizen and Law Enforcement Analysis and Reporting (CLEAR), Ash Ctr. for Democratic Governance and Innovation (Jan. 1, 2007) https://ash.harvard.edu/news/citizen-and-law-enforcement-analysis-and-reporting-clear [https://perma.cc/W2QN-XELY]. Ferguson, supra note 29, at 18. The information available in link analysis can then be linked with “[ballistic] evidence or other data to link individuals to crimes.”74 . Ferguson, supra note 29, at 95.
Finally, SDSs can generate statistical insights about individuals and groups. Despite being characterized as passive systems, databases include algorithmic tools that generate statistical insights about individuals and groups that are included in the database. While some gang databases in the U.S. generate summary reports with descriptive statistics on geographic maps of areas with individuals identified as “gang-involved,”75. Id. at 9, 15–19. the U.K. Gangs Matrix goes further to offer risk assessments that guide and even prioritize policing resource allocations. In the Gangs Matrix, a “harm score” is assigned to each individual, which is essentially a color-coded score (red, amber, or green) based on the “level of violence [that they] have shown.”76 . Amnesty Int’l, supra note 33, at 6 (alterations in original). The algorithm that produces the harm score relies on “police information about past arrests, convictions, and ‘intelligence related to violence/weapons access,’” social media, and other informal sources.77. Id. The Metropolitan Police, the territorial police force responsible for London’s thirty-two boroughs, have not disclosed the criteria and weights for the automated scoring algorithm they developed.78. Id. at 15. Some sources suggest that the score depends on how many crimes the person was involved during a three-year period, weighted according to the seriousness of the crime and its recency.79. Id. at 13. The harm scores are then ranked within each borough and the individuals with the top ten scores of each list are prioritized for enforcement.80. Id. at 13–14. While criminal intelligence databases are often viewed as being the foundation for more algorithmically advanced systems, the Gangs Matrix example, in particular, demonstrates the need to avoid any rigid semantic distinction between databases and algorithmic decision systems.
B. Legal Features
This category highlights common legal features and contexts that structure and potentially constrain SDSs design, use, and outcomes. These conditions are necessary for evaluating how databases are operationalized in practice, how they are viewed and understood by the public, and how they impact society.
The first SDS legal feature observable across legal frameworks applicable to criminal intelligence databases is the vague and subjective labels and criteria for determining who should be included in these databases, which both enables and conceals the biased notions of group criminality embedded in these systems. As we explore in this Section, this definitional flexibility is neither benign nor accidental. The operationalization of these legal regimes demonstrates how amorphous legal definitions have created a system ripe for the disproportionate targeting of historically marginalized social groups. Despite the veneer of individualized assessment, we explore how such labels and criteria both enable and maintain biased notions of group criminality.
Defining what constitutes a gang or gang members has been a significant challenge for law enforcement in the U.S., particularly because law enforcement definitions and practices tend to foreground the criminal activities of gangs, whereas researchers and social welfare practitioners “tend to emphasize the social and cultural aspects of gang formation and activity.”81Leyton, supra note 26, at 114; see also Mercer L. Sullivan, Maybe We Shouldn’t Study “Gangs”: Does Reification Obscure Youth Violence?, 21 J. of Contemp. Crim. Just. 170, 171 (2005) (“Youth violence takes many organizational forms. Lumping these together as ‘gang’ phenomena carries distracting baggage . . . . It can, and sometimes does, cloud our view of what we should be placing front and center, the problem of youth violence.”); Charles M. Katz, Issues in the Production and Dissemination of Gang Statistics: An Ethnographic Study of a Large Midwestern Police Gang Unit, 49 Crime & Delinq. 485, 487 (2003)
(“[P]olice do not necessarily document individuals because of their behavior but rather document individuals according to their own ideas and beliefs about gang members. . . . [T]his leads to officers documenting individuals based solely on where individuals live, with whom they associate, what they look like, or what clothes they wear.”);
Forrest Stuart, Code of the Tweet: Urban Gang Violence in the Social Media Age, 67 Soc. Probs. 191, 194 (2019) (describing how police assign residents to gang and other criminal databases based on social media activity that is misinterpreted as evidence of criminal activity). This schism can be partially attributed to law enforcement’s traditional approach to criminal profiling, which relies “on the correlation between behavioral factors and the past experience of law enforcement in discovering criminal behavior associated with those factors. Thus, profiling rests on the perceived accuracy of the profile as a predictor of criminality.”82William M. Carter, Jr., A Thirteenth Amendment Framework for Combating Racial Profiling, 39 Harv. C.R.-C.L. L. Rev. 17, 22 (2004).
Definitions are inconsistent across the country and are likely to reflect the political, social and financial pressures of any given jurisdiction.83Katz, supra note 81, at 486–89 (2003). A national study on gang-related laws found that only fifteen states have statutory definitions for gang members and five of these definitions are relatively generic.84. Id. See also Audit Div., L.A. Police Dep’t Chief of Police, AD No. 18-016, CalGang Criminal Intelligence System Audit 1, 7 (2019), http://www.lapdpolicecom.lacity.org/091019/BPC_19-0252.pdf [https://perma.cc/77SM-FUUR] (finding the most common criteria for including individuals in the CalGang System was whether they were “seen frequenting gang areas” and the second most frequent was whether they had “been seen associating with documented gang members.”). For instance, the statutory definitions have different requirements for how many individuals must participate in criminal activity to qualify as a gang—most states require three or more individuals, some require at least five individuals, and some do not specify a required number of members.85Julie Barrows & C. Ronald Huff, Gang and Public Policy: Constructing and Deconstructing Gang Databases, 8 Criminology & Pub. Pol’y 675, 683–85 (2009). Self-identification as part of a gang is the only common criteria, where no other corroboratory evidence is necessary. Research suggests this is a fraught category that is particularly vulnerable to abuse.86. See, e.g., Citizens for Juv. Just., supra note 30, at 22 (highlighting incidents where police threaten and coerce young people to self-identify as gang-involved); Barbara Bradley Hagerty, The Other Police Violence, Atlantic (Sept. 17, 2020) https://www.theatlantic.com/ideas/archive/2020/09/other-police-violence/616363/ [https://perma.cc/J2CR-3QXN] (describing how official misconduct in interrogations lead to false self-admissions that disproportionately send innocent Black men to prison); #GuiltyPleaProblem, Innocence Project, https://guiltypleaproblem.org/ [https://perma.cc/S5AZ-7DG4#stats] (“nearly 11 percent of the nation’s 362 DNA-based exonerations since 1989 involved people who pleaded guilty to serious crimes they didn’t commit. Furthermore, according to the National Registry of Exonerations, 18 percent of known exonerees pleaded guilty”) (last visited June 24, 2022). With police officers increasingly interpreting gang membership based on content posted by individuals on social media, there is a significant risk of misunderstanding cultural practices and signals, where “[w]hat may be meant as a joke or recognized as a lyric to a favorite rap song is instead interpreted by outsiders as inculpatory.”87 . Priscilla Bustamante & Babe Howell, Report on the Bronx 120 Mass “Gang” Prosecution 27 (2019), https://static1.squarespace.com/static/5caf6f4fb7c92ca13c9903e3/t/5cf914a3db738b00010598b8/1559827620344/Bronx%2B120%2BReport.pdf [https://perma.cc/35YY-CLDX]; see also Stuart, supra note 81.
Definitional ambiguity results in heightened discretion for those implementing DBS and is often justified by socially constructed notions of the groups being targeted. Unlike other forms of organized crime, such as mafias or mobs, which are hierarchical groups with strict codes of conduct that exist for the criminal enterprise, gangs are more amorphous and characterized by their fluidity in membership, geographic mobility, and differential organizational structures. Criminal activity is not necessarily the centralizing focus.88Barrows & Huff, supra note 85, at 678–79 (2009); see also Malcolm W. Klein, Cheryl L. Maxson & Jody Miller, Introduction, in The Modern Gang Reader, viii–ix (Malcom W. Klein, Cheryl L. Maxson & Jody Miller eds., 1995) (noting distinctions between street gangs, prison gangs, and criminal syndicates); Richard C. McCorkle & Terance D. Miethe, Panic: The Social Construction of the Street Gang Problem, 202–09 (2002) (problematizing the concept of gangs as coherent groups of individuals with shared goals and synthesizing research that has identified focuses other than criminality that gangs “organize” themselves around including territoriality, dress and grafitti); Judith Greene & Kevin Pranis, Just. Pol’y Inst., Gang Wars: The Failure of Enforcement Tactics and the Need for Effective Public Safety Strategies 10 (2007) (“Most experts agree that drug trafficking is a secondary interest for street gang members.”); Leyton, supra note 26, at 115 (“Most gangs are loosely structured, and many young people may join solely for safety or acceptance reasons rather than to participate in the gang’s criminal activities.”). For example, then-New York City Police Department (NYPD) Commissioner, Dermont Shea, stated in his testimony about the Department’s gang database that gangs had evolved from traditional structure and noted: “their lack of a defined structure makes it difficult to predict their activities or document their association.”89Dermot Shea, Chief of Detectives, N.Y.C. Police Dep’t., Address at New York City Council Committee on Public Safety (June 13, 2018), https://councilnyc.viebit.com/player.php?hash=cYVufTvzRw2v [https://perma.cc/73UP-TGBM]. This fluidity has, in turn, become a justification for ensuring legal definitions allow maximum discretion for police officers. In 2015, the U.K. Home Office also sought to simplify and expand the definition of a gang and gang member “to make it less prescriptive and more flexible” and the new definition stated that a gang should be defined merely “as having one or more characteristics that enable its members to be identified as a group by others.”90Home Office & Karen Bradley, Changes in Legislation Reflected in New Gang Definition, Gov.UK (June 8, 2015), https://www.gov.uk/government/news/changes-in-legislation-reflected-in-new-gang-definition. Thus, this definitional ambiguity has provided police with unfettered discretion to use scant evidence to not only label individuals as gang members but also to promote beliefs in non-existent gangs, such as the “ACAB gang” referenced in the introduction.91. See Biscobing, supra note 5.
Finally, the existence of individualized assessment criteria for who is included within a database can distract from or even disguise biased notions of group criminality embedded in these systems. The use of such criteria in India is demonstrative of this. The widely criticized colonial-era Criminal Tribes Act (that criminalized marginalized groups) was replaced with a Habitual Offenders Act in several states post-independence, a move that has been widely described as “window dressing changes of nomenclature, from criminal tribes to habitual offenders.”92Brown, supra note 37, at 198. In states that have it, the Habitual Offenders Act determines the criteria for when a history sheet record is created against an individual, which subsequently leads to their inclusion into the surveillance database.93. See Satish, supra note 36.
While these statutes do generally require at least three convictions to trigger a history sheet record, this obscures the fact that police still typically secure such convictions under colonial-era provisions often referred to as the “bad livelihood” sections, which apply to “suspicious persons” and “habitual offenders” without any requirement of a pre-existing criminal record.94. See Madhya Pradesh Police Regulations: Chapter VII (Security for Good Behaviors): Rule 839 Bad Livelihood Cases, (demonstrating that these provisions are referred to as the “bad livelihood” provisions in police regulations). Id. at 250.
(“Flexible as it was, the procedure of a bad-livelihood enquiry could erect quite a rigid frame of ‘habituality’. Someone ordered to furnish security under Section 109 or 110 was no longer simply a ‘suspected person’. He acquired a record and, for purposes of police surveillance or jail discipline, was often categorized with convicted habituals.”). Courts, too, have noted the use of these preventive provisions, specifically Sections 109 and 110 of the current Criminal Procedure Code, to arrest and harass individuals who often lack the financial resources to deposit the “good behavior” bond required under these provisions.95. See Madhya Pradesh Police Regulations: Chapter VII (Security for Good Behaviors): Rule 839 Bad Livelihood Cases. Interview by Amba Kak with Nikita Sonawane and Ameya Bokil, Attorneys, Criminal Justice & Police Accountability Project (Oct. 7, 2020) (transcript and recording on file with authors); see also, Sabari alias Sabarigiri v. Asst. Comm. of Police (2018), https://indiankanoon.org/doc/159399368/ [https://perma.cc/2SSS-XWQP] (“The Police seems to be adopting the practice of registering First Information Reports against the persons under Sections 109 and 110 of CrPC, just to open the history sheet and to justify the continuance of the name of the persons in the history sheet”). In states that do not have their own Habitual Offenders Act, the criteria for the creation of a history sheet record is stipulated via “standing orders”96. See generally, Police standing orders are rules or mandates that offer guidance to officers of police departments in India. They are issued at the state level and are administrative in nature. See Deputy Inspector General of… v. P. Amalanathan, (1966) 203 AIR (1964) (per Ramakrishnan), https://indiankanoon.org/doc/41451/ [https://perma.cc/GH66-DDTS].; B. Tilak Chandar, HC Asks Police to Review History Sheets Regularly, Hindu (Sept. 26, 2018, 8:45 PM), https://www.thehindu.com/news/cities/Madurai/hc-asks-police-to-review-history-sheets-regularly-madurai/article25050457.ece (Under Police Standing Order 746, history sheet can be opened against those involved in crime habitually). or prison manuals that are even more vague and include direct references to the repealed colonial-era regime, such as classifying individuals as “habitual criminals” based solely on whether someone has a bad livelihood (undefined) or even whether they are a member of an erstwhile criminal tribe.97Madhya Pradesh Jail Manual, 1987, Rule 411–12, http://jail.mp.gov.in/sites/default/files/Part%202_2.pdf [https://perma.cc/Q82J-V7Z8]. A recent ethnographic account of the implementation of surveillance databases in New Delhi includes a revealing quote from officers who admitted “some of the communities are still criminal, but they are no longer targeted as a group but as individuals.”98Khanikar, supra note 40, at 42.
The second legal feature of SDS is that despite these vague, flexible, and biased criteria, inclusion into a database still inevitably translates into serious penal or other punitive consequences outside the carceral system. The consequences manifest in similar ways across the jurisdictions we have examined. Police rely on these databases to allocate surveillance resources, determine which geographical areas or homes to target, and which persons of interest to investigate and arrest, leading to the chronic over-policing of those in the databases.99For the U.S. see, for example, Ben Popper, How the NYPD is Using Social Media to Put Harlem Teens Behind Bars, Verge, (Dec. 10, 2014), https://www.theverge.com/2014/12/10/7341077/nypd-harlem-crews-social-media-rikers-prison [https://perma.cc/HR8P-2FD] (describing the case of Jelani Henry, a young black man from Harlem that was incarcerated at Rikers Island for 19 months based on the NYPD’s use of a gang database and social media monitoring to label him as a criminal affiliate). For India, see for example, Satish, supra note 36, at 133 (describing how surveillance databases play a critical role in justifying the rounding up and detaining of “anti-social elements” by
the police before festivals, elections, and other political events). For the U.K. see, for example, StopWatch, Being Matrixed: The (Over)policing of Gang Suspects in London (August 2018) https://www.stop-watch.org/uploads/documents/Being_Matrixed.pdf [https://perma.cc/C2EY-JG9E] (describing how police officers continually patrol the same postcodes and routinely stop and search the same individuals on the Gang Matrix, and as a result, they are more likely to get picked up and charged for minor offences). These databases or the information within are also shared with other actors in the criminal justice system, which can exacerbate collateral consequences of the criminal justice system.100For example, a 2021 U.S. Immigration and Customs Enforcement Memorandum on federal immigration enforcement priorities, lists individuals deemed a “threat to public safety”, which includes individuals with a gang conviction or “who intentionally participated in an organized criminal gang”, as a priority for deportation. U.S. Immigr. & Customs Enf’t, U.S. Dept. of Homeland Sec., Interim Guidance: Civil Immigration Enforcement and Removal Priorities (Feb. 18, 2021), https://www.ice.gov/doclib/news/releases/2021/021821_civil-immigration-enforcement_interim-guidance.pdf. [https://perma.cc/UT2L-P6XH]. The directive is written in a way that can implicate individuals included on a gang database, and advocates warned that reliance on this criterion can exacerbate the discriminatory impact of police databasing efforts on Black, Latinx, and immigrant communities. Letter from GANGS Coalition, to Alejandro Mayorkas, Sec’y, Dept. of Homeland
Sec. (Apr. 1, 2021) https://gangscoalition.medium.com/coalition-letter-on-interim-ice-guidance-7275abadfb67 [https://perma.cc/YA5B-D3C9]. Prison and jail officials rely on databases to make appropriate classifications and other decisions for security and institutional order.101For the U.S. see, for example, Jacobs, supra note 27, 705–07 (2009); Leyton, supra note 26 at 122. For the U.K. see, for example, Amnesty Int’l, supra note 33, at 6. For India see, for example, Telephone Interview by Karishma Maria, Research Assistant to Amba Kak, AI Now Institute at New York University with Jeeyika Shiv, Associate Director, Fair Trial Initiative Project39A (Sept. 12, 2020) (transcript on file with author) (describing how there are often separate barracks for “history sheet cases” and it is more difficult to obtain permission for visitation). Prosecutors rely on these databases to inform and craft criminal charges and plea bargains.102For the U.S., see, for example, Jacobs, supra note 27, at 705–07; Jeffrey Lane, The Digital Streets 128–149 (2019); Leyton, supra note 26, at 122; For the U.K. see, for example, Amnesty Int’l, supra note 33, at 8 (“the Gangs Matrix features information provided by the police to the Crown Prosecution Service (CPS) at the point when the CPS makes charging decisions.”). Judges rely on them to inform bail and sentencing decisions.103For the U.S. see, for example, Jacobs, supra note 27, at 705–07. For India see, for example, Satish, supra note 36 at 147.
Arrests and indictments of individuals in these databases are also often implemented at a mass or group level through the use of conspiracy or other group criminality statutes. In the U.S., gang databases have been used to build conspiracy cases against groups of alleged gang members, where individuals are implicated in alleged criminal activity for associating with or living near individuals or groups involved in a crime.104. See Bustamante & Howell, supra note 86; Josmar Trujillo & Alex S. Vitale, Gang Takedowns in The De Blasio Era: The Dangers of ‘Precision Policing’, 10–17 (2019); see also Shea, supra note 89, (“Since 2016, the NYPD has engaged in approximately 100 long term gang investigations resulting in 1,259 arrests”). In the U.K., conspiracy-based laws, called the joint enterprise doctrine, allow for the prosecution of groups solely on the basis of gang membership or association, even though gang membership or association alone are not crimes.105. See Patrick Williams & Becky Clarke, Ctr. For Crime and Just. Stud., Dangerous Associations: Joint Enterprise, Gangs and Racism (2016); Amnesty Int’l, supra note 33 at 8. Joint enterprise allows more than one person to be prosecuted for the same offense, even if the person was not involved or in the proximity of the crime, and it only requires the prosecution to prove the person had the foresight to believe a crime would occur.106. See Williams & Clarke, supra note 105, at 17–21. Similarly, in India there are recent accounts of large groups of anti-caste protestors belonging to marginalized communities being classified as history sheeters or habitual offenders and consequently being subject to restrictions on their movements.107. See Sukanya Shantha, Arrests, Summons, Externment: Maharashtra Police Moves Against 200 Activists, Wire, (Jan. 21, 2021), https://thewire.in/rights/bhima-koregaon-arrests-summons-extrernment-notices-maharashtra-police-activists [https://perma.cc/XP4U-RJMG]; Interview by Amba Kak, Director of Global Policy & Programs at the AI Now Institute at New York University, with Disha Wadekar, Independent Legal Aid Attorney (Sept. 12, 2020) (transcript and recording on file with author) (describing how in the Bhima Koregaon protests in 2019, two hundred activists that were protesting caste-based violence, were classified as habitual offenders and added to surveillance databases subjecting them to restrictions and heightened physical surveillance, even though most did not have any prior criminal record). These loosely based charges mean people can be criminalized for who they know, where they live, and indeed their membership in a particular gang.
The harmful consequences of inclusion into an SDS database go well beyond the threat of incarceration. The collateral consequences of social censure and stigma can result in multiple, interacting forms of harm that impact various facets of a person’s life. For instance, school officials, public housing authorities, immigration officials, and other non-law enforcement government actors that have access to or receive information from gang databases can use the information for decisions about community safety, disciplinary action in schools, tenant applications, and assignment of counseling resources.108For the U.S., see Leyton, supra note 26, at 122–23; Becki R. Goggins & Dennis A DeBacco, SEARCH, Survey of State Criminal History Information Systems, 2016: A Criminal Justice Information Policy Report (2018); Irene Romulo, ‘Gang Contracts’ in Cicero and Berwyn Schools Raise Concerns About Criminalization of Youth, Injustice Watch, (May 26, 2021) https://www.injusticewatch.org/news/juvenile-justice/2021/cicero-gang-contracts/ [https://perma.cc/MCD4-DAJ4] (describing the strict consequences of gang membership for students in U.S. schools). For India, see Thirumagan v. Superintendent of Police Madurai Dist. (2020), Madras HC, 20207-44 (Apr. 3, 2020) (per Seshasayee) https://indiankanoon.org/doc/156221879/ [https://perma.cc/FC3G-NQE2] (“What is not adequately appreciated is that history-sheeting leaves civil consequences in that it becomes a barrier for obtaining anything from a passport to securing employment.”).
The harsh punitive consequences of being classified under criminal intelligence databases, both within and outside of the carceral system, holds important insights into the role and function of SDS. The legal and administrative rules, as well as the bureaucratic modes of these classification systems, can often obscure the range of punitive, and often violent, consequences they trigger. Indeed, it is the looming threat of harm that accompanies inclusion in the database and is meant to discipline the individuals and groups that it targets through these labels. While these systems are often justified for their ability to predict crime and identify budding criminality, this predictive quality creates a feedback loop. Categorization into an SDS triggers hyper-surveillance, more police contacts, and a range of penal and social consequences for individuals and groups that make it difficult, if not impossible, to evade arrests, indictments, and other forms of punishment. In doing so, they also strengthen the correlation between criminality or deviance and the factors often associated with such classifications and identity markers (e.g., poverty and membership of a marginalized community) that are used to justify other forms of disproportionate targeting and surveillance.109. See, e.g., Gary T. Marx, The Engineering of Social Control: Policing and Technology, 1 Policing 46 (2001); Marie-Eve Sylvestre, Crime and Social Class: Regulating and Representing Public Disorder, in CRIMINALIZATION, REPRESENTATION, REGULATION: THINKING DIFFERENTLY ABOUT CRIME, supra note 20, at 222–29; Ericson & Haggerty, supra note 20, at 256–261 (1997).
The third legal feature of SDS is the lack of robust legal oversight and constitutional scrutiny of these databases, leaving few avenues to challenge their operation or introduce structural safeguards. Most criminal justice constitutional doctrines focus on the abuse of due process at the level of individual action or actors. Such legal frameworks are ill-suited for the broad-based targeting and harm that stem from database uses, which typically involve several institutional actors and multiple intersecting laws. The stage at which databases are used in the criminal investigation process is typically considered within the discretionary power of police. 110. See Elizabeth E. Joh, The New Surveillance Discretion: Automated Suspicion, Big Data, and Policing,10 Harv. L. & Pol’y Rev. 15, 38 (2016); Erin Murphy, Databases, Doctrine & Constitutional Criminal Procedure, 37 Fordham Urb. L.J. 803, 826–29 (2010). In the U.S., such uses occur before constitutional scrutiny, primarily via the 4th amendment, is triggered.111Joh, supra note 110, at 38; Murphy, supra note 110, at 826–29; see also, Karen S. Glover, Racial Profiling: Research, Racism, and Resistance, 15–19 (2009) (arguing that many concerning policing practices like racial profiling occur at informal stages of the criminal justice system rendering them to less scrutiny in criminology studies).
Legal scholar K. Babe Howell has argued that these databases are actively employed to avoid constitutional requirements of due process.112Howell, supra note 30. In Gang Policing: The Post Stop-and-Frisk Justification for Profile-Based Policing, she details how in 2013 the NYPD increased use of its gang database and covert surveillance to advance a new policy priority of policing “crews,” a law enforcement term for gangs that are loosely organized, neighborhood-based and primarily composed of young people.113. Id. at 2–7. She argues that this increased reliance on gang databases was intentional because their use was less likely to be subject to review or legally challenged because they are not governed by constitutional or statutory requirements like the NYPD’s recently invalidated stop and frisk regime. 114. Id. at 11–14. The lack of serious constitutional challenges to these legal regimes can also be explained by the lack of transparency and visibility into databasing practices that would be necessary to build or foreground such a challenge. These systems are typically decentralized (explored in Section d), vary greatly across different states and municipalities, and there is typically no public access to these records and systems due to sector-specific data confidentiality policies.
C. Political Economy Features
This category highlights common political economy features that emerge and are produced by SDS use. These features explore and illuminate the relationships between modes of economic production, governance, customs and practices, and the vested interests of private enterprise and political institutions. Political economy features are necessary for understanding the drivers of SDS development and use, as well as their disproportionate impact on particular communities.
The first political economy feature of SDSs is that they are a modern form of the commodification of security and risk management, where private-public partnerships and other arrangements are created to extract profit from the development and use of these public administrative databases and actuarial practices. The 1970s mark an important period where economic and political conditions engendered the development of this feature. During this period, the U.S., U.K., and India experienced economic volatility and declines due to deindustrialization, foreign economic policies, and global stagflation.115. See generally William Julius Wilson, When Work Disappears: The World Of The New Urban Poor (1997); Ira Katznelson, City Trenches: Urban Politics And The Patterning Of Class In The United States (1982); David Byrne, Deindustrialization and Dispossession: An Examination of Social Division in the Industrial City, 29 Socio. 95 (1995); Atul Kohli, From Breakdown to Order: West Bengal, in State and Politics in India 352–53 (Partha Chatterjee ed., 1999). This period is also when each country dramatically shifted penological policymaking and practices towards more punitive and “law and order” approaches that emphasized categorical and group-based risk management methods as more efficient.116. See generally Stanley Cohen, Visions of Social Control: Crime, Punishment and Classification (1985) (detailing post-1960s theories and practices in social control and crime-control in North America and Europe); Katherine Beckett, Making Crime Pay: Law and Order in Contemporary American Politics (1997) (documenting how political rhetoric and policies regarding crime wars were used to expand imprisonment as a primary criminal justice policy and to disguise racial motivations); Nijjar, supra note 53, at 147–62 (2018) (describing how post-colonial India adopted and evolved colonial practices of social control and disciplinary measures); Jonathan Simon, Governing Through Crime: How the War on Crime Transformed American Democracy and Created a Culture of Fear (2009) (arguing that elected officials substituted substantive social policy governance with punitive crime control rhetoric and policies.). As a result, each government enacted harsh criminal justice policies or practices that legitimized and thinly disguised the targeting of certain social groups. As inequality in access to security grew (especially in marginalized communities), a burgeoning private market in security, risk management, and information technology developed.117. See generally Simon, supra note 116; see also, Satish, supra note 36; Khalil Gibran Muhammad, The Condemnation of Blackness: Race, Crime, and the Making of Modern Urban America 226–46 (2010) (discussing the use of racialized crime statistics); Robert Reiner, Crime and Control in Britain, 34 Socio. 71, 81–86 (2000); Erin Murphy, Paradigms of Restraint, 57 Duke L. J. 1321, 1326–44 (2008). Over time this resulted in the creation and growth of government administrative databases and actuarial practices that often required collaboration with private enterprises that financially benefited from the arrangement.
The U.K. enacted national policy reforms that required intelligence systems, crime audits, and statistically based performance monitoring and systematic analysis of police crime-fighting efficiency.118. See Reiner, supra note 117; Home Off. Rsch. & Stat. Directorate, Reducing Offending: An Assessment of Research Evidence on Ways of Dealing With Offending Behaviour, 23–36 (1998), http://www.nomsintranet.org.uk/roh/official-documents/HomeOfficeResearchStudy187.pdf [https://perma.cc/H4VJ-PG79]. While most of the information technology infrastructure was created and used by the government, these policy shifts produced a market for private security and insurance (whose policies deterred victims from reporting crime).119 . Home Off. Rsch. & Stat. Directorate, supra note 118, at 41–43 (1998); Reiner, supra note 117, at 81–86 (2000); Robert Reiner, Policing, Popular Culture and Political Economy 416 (2011) In the U.S., large urban police departments like the NYPD and CPD engaged in public-private partnerships with large technology companies to develop or procure computerized information systems that could manage and statistically manipulate a growing body of police administrative data.120Brian Jefferson, Digitize and Punish: Racial Criminalization in the Digital Age 93–120 (2020). It is notable that during this period, the securities and technology sectors experienced significant growth compared to rest of the labor market.121. Id. at 94–97; Charles S. Gascon & Evan Karson, Growth in Tech Sector Returns to Glory Days of the 1990s, Fed, Reserve Bank of St. Louis (July 25, 2017), https://www.stlouisfed.org/publications/regional-economist/second-quarter-2017/growth-in-tech-sector-returns-to-glory-days-of-the-1990s [https://perma.cc/4DD2-6SDU]. In India, the Crime and Criminal Tracking Network System (CCTNS) was created to digitize criminal administrative data, and is now reportedly being opened up to commercial background check services to generate revenue.122Neeraj Chauhan, Pvt Firms, PSUs Set to Get Access to Crime Database, Hindustan Times (Jan. 2, 2021, 12:18 AM), https://www.hindustantimes.com/india-news/pvt-firms-psus-set-to-get-access-to-crime-database/story-4JqLbe8qgtOuVSujVl3n9O.html#:~:text=The%20National%20Informatics%20Centre%20(NIC,to%20pay%20a%20prescribed%20fee [https://perma.cc/H3T5-382D] (“The idea… is to monetise this data; companies will be charged for the background check”); Centre to Give Private Companies, PSUs Access to Crime Database for ‘Background Check’: Report, Deccan Herald (Jan. 2, 2021, 10:07 PM)”) https://www.deccanherald.com/national/centre-to-give-private-companies-psus-access-to-crime-database-for-background-check-report-934465.html
The second SDS political economy feature is that these systems facilitate administrative opacity, which shields private and public actors from dissent, democratic inquiries, and various other appeals for transparency, access, due process, and equity. “Administrative opacity is a deliberate strategy to manage regulatory environments” by controlling access to information regarding technical systems.123Cottom, supra note 62. Sociologist Tressie McMillian Cottom argues that a prominent technique of administrative opacity is obfuscation through privatization, where “data that would have previously been public, publicly available or legally discoverable” is either privatized or made practically inaccessible.124. Id.; see also Frank Pasquale, The Black Box Society 3–14 (2015). SDSs, or the information within, can be privatized and made inaccessible when private companies that develop the systems assert trade secrets claims or withhold documentation,125Robert Brauneis & Ellen P. Goodman, Algorithmic Transparency for the Smart City, 20 Yale J.L. & Tech. 103, 152–160 (2018). See generally Rebecca Wexler, Life, Liberty and Trade Secrets: Intellectual Property in the Criminal Justice System, 70 Stan. L. Rev. 1334 (2018). and when public agencies claim that the information within or the system itself is classified or meets exceptions to public transparency mandates.126. See Brauneis & Goodman, supra note 125, at 160–63; see also Pasquale, supra note 124, at 153–60 (discussing federal intelligence practices that evade public transparency). SDSs are made practically inaccessible through various means, but one notable strategy is the use of reasonable, yet ethically questionable, rationales to quell inquiries regarding the underlying policies or logics that undergird SDS design or use. Regardless of which approach to administrative opacity is used, the consequences are the same—they hamper public inquiries and contestations.
Evidence of the second SDS political economy feature varies in each jurisdiction. In the U.K., critics of the Gangs Matrix argue that it is overinclusive and question whether the “right people” are targeted by or included in the system.127 . Off. for Policing and Crime, Mayor of London, Review of the Metropolitan Police Service Gangs Matrix 7 (Dec. 2018) https://www.london.gov.uk./sites/default/files/gangs_matrix_review_-_final.pdf [https://perma.cc/4MA8-PVPR]; see also Amnesty Int’l, supra note 33; Densley & Pyrooz, supra note 33, at 17–18 (2020). Critics rightly assert that gang membership in and of itself is not a crime and can be mutually exclusive of criminal activity.128. See Amnesty Int’l, supra note 33; Densley & Pyrooz, supra note 33. Yet the Gangs Matrix remains administratively opaque in two ways. First, an Enforcement Notice regarding violations of data protection laws revealed that inconsistencies in the Gangs Matrix’s operating policies, along with informal practices by police, distorted public perception about how the systems was used and possibly concealed legal violations.129Sebastian Klovig Skelton, ICO Finds Metropolitan Police’s Gangs Matrix Seriously Breaches Data Protection Laws, ComputerWeekly (Nov. 22, 2018, 11:30 AM), https://www.computerweekly.com/news/252452971/ICO-finds-Metropolitan-Polices-Gangs-Matrix-seriously-breaches-data-protection-laws [https://perma.cc/LM3Q-KNTK]. Second, the government employed a reasonable, but controversial, rationale to quell scrutiny of the Gang Matrix’s net-widening effects. The government reasoned that it adopted a preventative public health model of violence interruption to justify why the Gangs Matrix included individuals with no record of violence, though this rationale and practice were challenged in the aforementioned Enforcement Notice.130Densley & Pyrooz, supra note 33. In the United States, police engage in several tactics to keep details regarding gang databases opaque. Indeed, a coalition letter to the NYPD’s Inspector General and testimony at legislative hearings on the lack of transparency regarding the NYPD’s gang database revealed that individuals in the database never receive notice of this fact, and that this information is not included in their criminal history or rap sheet, documents that are publicly accessible.131. See Coalition Letter Calls on the NYPD Inspector General to Audit the NYPD “Gang Databases,” Brennan Ctr. for Just. (Sept. 22, 2020), https://www.brennancenter.org/our-work/research-reports/coalition-letter-calls-nypd-inspector-general-audit-nypd-gang-database [https://perma.cc/434M-398Z]; Hearing Testimonies, Before the New York City Council Comm. on Pub. Safety, Comm. Room, City Hall (June 13, 2018), Written Comments of the Bronx Defenders, 27–31. In India, various local police manuals maintain that the existence of history sheets and other surveillance databases is confidential and public Right to Information requests have been denied.132. See Satish, supra note 36, at 140; S. Vijay Kumar, When RTI Came to the Aid of History-Sheeter, Hindu (Dec. 2, 2019) https://www.thehindu.com/news/national/tamil-nadu/when-rti-came-to-the-aid-of-history-sheeter/article30131824.ece [https://perma.cc/RCN6-PZ37]; In Malak Singh v. State of P & H (1981) 1 SCC 420 (India) (holding that surveillance register entries may be kept confidential so long as the Superintendent of Police “entertain[s] a reasonable belief that persons whose names are to be entered … are habitual offenders or receivers of stolen property” before individuals are included in the surveillance register).
The third political economy feature is that SDSs facilitate and serve a carceral economy133We use the term carceral economy to refer to the role and relationship between criminalization, incarceration, and economic interests. of cheap labor or other exploitative labor dynamics by accelerating criminalization and incarceration, in addition to serving as a mechanism that maintains a high correlation between poverty, local social hierarchies, and criminalization. This feature may seem more apparent because several criminal justice policies, practices, and technologies facilitate similar outcomes, and research demonstrates that changes in incarceration and the carceral economy are linked to social and economic structural changes, rather than actual crime rates.134. See, e.g., Jackie Wang, Carceral Capitalism (2018) (collection of essays highlighting how different social and economic policies and phenomena contribute to the carceral economy); Ruth Wilson Gilmore, Golden Gulag: Prisons, Surplus, Crisis, and Opposition in Globalizing California 138-172 (2007); Dario Melossi, Gazette of Morality and Social Whip: Punishment, Hegemony and The Case of the USA, 1970–92, 2 Soc. & Legal Stud. 259 (1993); Georg Rusche, Labor Market and Penal Sanction, 10 Crime & Soc. Just. 252, 2–8 (1933). SDS use accelerates criminalization and incarceration, which creates a surplus labor force that can be subjected to exploitive labor arrangements both within and outside of carceral institutions. Individuals with database designations like “gang member” or “habitual offender” are subjected to heightened law enforcement scrutiny and increased punitive outcomes with potentially fewer constitutional protections due to issues discussed in Section B and poor data quality practices.135. See, e.g., Leyton, supra note 26, at 122–40; Joshua D. Wright, The Constitutional Failure of Gang Databases, 2 Stan. J. C.R. & C.L. 115 (2005); Murphy, supra note 110, at 813; Kenneth C. Laudon, Data Quality and Due Process in Large Interorganizational Record Systems, 29 Comm’cns of the ACM 4 (1986). For example, in the U.S., the NYPD invested in the development of criminal intelligence databases and other data-driven administrative systems, like Compstat, to enable greater data sharing and “decentralize decision making,” which altered policing from “a logic of functional specialization (detectives, forensics, youth officers) to one of territorial specialization (census blocks, hot spots, street segments).”136Jefferson, supra note 120, at 115. This provided data-based pretext to conceal otherwise biased targeting and micromanaging of certain groups and communities that politicians blamed for social disorder. This also helped justify government divestment that significantly reduced economic opportunities for those communities.137. Id. at 111–20; see also Richardson, supra note 19, at 120–22 (describing how racial segregation concentrated social problems in certain neighborhoods, which resulted “in a greater degree of law enforcement presence, targeting and surveillance practices”); see generally Richard Rothstein, The Color of Law: A Forgotten History of How Our Government Segregated America 153–76 (2017) (describing how social and economic policies that entrenched racial segregation depressed the incomes of Black Americans).
In U.S. prisons, incarcerated workers are subjected to low pay and exploitative work conditions and practices; such labor arrangements are constitutionally permissible via the Thirteenth Amendment and beyond the reach of labor laws.138. See Whitney Benns, American Slavery Reinvented, Atlantic (Sept. 21, 2015), https://www.theatlantic.com/business/archive/2015/09/prison-labor-in-america/406177/ [https://perma.cc/6SMK-ZREL]. In society, markers of involvement in the criminal justice system, whether criminal records or criminal intelligence designations, can also diminish job opportunities relegating those individuals to low-wage and exploitative jobs.139. See Leyton, supra note 26, at 120–23; Jacobs, supra note 25, at 227–74; Katz, supra note 81, at 513.
SDSs also ensure that specific forms of marginalization, particularly poverty, race, and caste, are targeted by criminalization process, which in turn results in these marginalized groups being disproportionately represented in the criminal justice system. This is because criminal intelligence databases reinforce and rationalize differential law enforcement practices and treatment, and groups disproportionately targeted by database-enabled law enforcement often have fewer resources and less power to contest their treatment and outcomes.140. See Jefferson, supra note 120, at 111–116 (2020); see also Ericson & Haggerty, supra note 20, at 256–59. For instance, India’s Criminal Procedure Code allows magistrates to issue bonds to individuals with a history sheet record as an assurance for good behavior.141. See Code of Criminal Procedure, 1973, § 110 (India). Yet, because history sheets are indiscriminately used against poor and marginalized communities, they are often unable to pay the bonds, this failure to pay is then used to legitimize their inclusion in the registers and subsequent differential treatment.142. See K.D. Gaur, Poor Victim of Uses And Abuses Of Criminal Law And Process In India, 35 J. Indian L. Inst. 183, 208 (1993).
The development, use, and impact of criminal intelligence databases lie in the intersection of political economy interests and concerns. The features of this SDS category provide several facets of analysis that can uncover often opaque interests and drivers of SDSs development and use, which can aid identification of unique points for policy intervention and public contestation. This analysis can also deepen understanding of the structural implications of SDS use and the outcomes they produce.
D. Organizational Features
This category examines the organizational structures, practices, and decisions (e.g., deferential actions) that are relevant to understanding how SDSs are operationalized by and within institutions. Though SDSs are demonstrably technological in character, they are still created by parts of the criminal justice process and this is best understood through an examination of the organizational and operational features of the institutions designing and using SDSs.143. See, e.g., Juho Pääkkönen, Matti Nelimarkka, Jesse Haapoja & Airi Lapinen, Bureaucracy as a Lens for Analyzing and Designing Algorithmic Systems, Proceedings of the 2020 CHI Conf. on Human Factors in Computing Systems, April 2020, at 1, 5 (indicating that socio-technical systems are influenced by the institutional context and can be viewed as extensions of bureaucratic administration).
Police departments, like other government agencies whose missions are oriented around addressing social issues, are hierarchical bureaucracies. Yet because police departments are increasingly becoming the primary or at least initial government response to a myriad of social problems,144. See, e.g., Jonathan Ben-Menachem, Pulling Back The Curtain On Boston’s ‘Operation Clean Sweep’, The Appeal (Aug. 15, 2019), https://theappeal.org/boston-police-clean-sweep-arrests/ [https://perma.cc/8WNV-DXPH]; see also Lauren Chambers, Unpacking The Boston Police Budget, ACLU Massachusetts, https://data.aclum.org/2020/06/05/unpacking-the-boston-police-budget/ [https://perma.cc/8RJW-EMSS] (indicating that the police receive a significant majority of municipal funding). their outwardly facing organizational structure distorts perceptions of how policing actually operates. This distorted view is amplified with the use of criminal intelligence databases. The public views these agencies as operating under formal rules with oversight because of their hierarchical organizational structure.145. See Michel Crozier, The Bureaucratic Phenomenon (1964) (arguing that bureaucracies tend to centralize control to reduce uncertainty in the organization’s operation). But in reality, the majority of police work consist of hyper-local, unsupervised, and discretionary practices of front-line or patrol level police officers, which shapes and sustains criminal intelligence database use and outcomes.146. See, e.g., Michael Lipsky, Street-Level Bureaucracy: The Dilemmas of the Individual in Public Service (1983) (arguing that decision-making and policy implementation in street-level bureaucracies, like policing, necessarily involves front-line workers having reflexive discretion); see also Albert J. Reiss, Jr., Police Organization in the Twentieth Century, 15 Crime & Just. 51 (1992) (observing that American police departments became more bureaucratized while leaving considerable discretion to front-line officers with ineffective accountability systems). Indeed, some surveillance scholars have argued that criminal intelligence databases and other information technologies have facilitated the growth of police work as information work,147. See Ericson & Haggerty, supra note 20, at 31–38; see also Reiss, supra note 146, at 82 (“The core technology of police organization is the production and processing of information.”). and that the integration of these technologies further distorts policing practices and decision-making to seem more impartial.148. See Pääkkönen, Nelimarkka, Haapoja & Lapinen, supra note 143, at 1, 6 (describing how government actors’ discretionary judgments continue with the use of algorithmic systems but give the appearance of impartiality). Thus, the first organizational feature of SDS is a hierarchical and “top-down” organizational structure with formal rules and policies that practically operate as minimally supervised, highly discretionary, local units of front-line workers.
This first SDS organizational feature means that despite the existence of formal policies and leadership structures that should presumably shape and oversee SDS use, it is actually the superficially supervised, discretionary practices of front-line police officers that give meaning to and concretize SDS contents, logics, uses, and outcomes, which are often noncompliant with local legal rules or requirements. In several states in India, there are police manuals and “standing orders” that provide executive instructions regarding the use of “history sheets” for habitual offenders. However, these policies include a range of arbitrary and subjective standards that are often at odds with constitutional and statutory standards and give police officers complete discretion regarding who can be added and removed from the database. For example, one standing order permits police to open a suspect history sheet for any person who is convicted of any crime in the Indian Penal Code, even if it is their sole offense.149. See Ganesan v. The District Superintendent of Police, (2010) 6 CTC 507, https://indiankanoon.org/doc/1411102/ [https://perma.cc/4CYV-U7LU]. Although individuals with history sheets are considered habitual offenders when they’ve been charged more than once, High Courts have found that the commission of an isolated crime cannot be characterized as a habitual act.150. Id. at ¶ 35. The practical implications of such discretionary policing and database practices mean that database inclusion and overall composition reflect police biases.151. See Nikita Sonawana & Ameya Bokil, How Poverty-Struck Tribals Become ‘Habitual Offenders’, Article 14 (May 28, 2020) https://www.article-14.com/post/born-a-criminal-how-poverty-struck-tribals-become-habitual-offenders. [https://perma.cc/3U3U-VNNN]. Similar effects were found in gang databases in the U.S.. A report on the New Bedford, Massachusetts Police Department’s gang database found that the over-representation of Black and Latinx men in the police department’s gang database was the result of implicit bias amongst the police regarding which young people will be part of a gang. Citizens for Juv. Just., supra note 30, at 22. In one case, an Indian court highlighted a police database record suggesting that a woman might commit theft and should be subjected to surveillance because she was unmarried and “is always found fashionably dressed without any income and therefore, her activities should be watched.”152S. Vani v. The Superintendent Of Police, (2008) Madras HC 1525, at ¶ 10.4, 20, https://indiankanoon.org/doc/115135/ [https://perma.cc/XF3N-KBY3] (holding that the supervising officer “acted in a mechanical fashion” and failed to review these entirely arbitrary reasons for subjecting an individual to enhanced surveillance). Similar observations were made in other jurisdictions examined. One example is CalGang, which provides database guidelines to individual police officers to ideally reduce discretion.153Leyton, supra note 26, at 115. Despite these guidelines, a 2016 audit of CalGang noted that an “inadequate leadership structure” resulted in the database not complying with several legal requirements governing its use.154 . Cal. State Auditor, The CalGang Criminal Intelligence System, Report 2015-130, at 23 (2016).
A second organizational feature is that while SDS are developed and appear to centralize information and oversight, in practice, they are diffused with some localities having duplicative, informal systems because of the decentralized organizational structure of law enforcement. In the U.K., the Gangs Matrix is overseen by the Trident Gang Command, a unit of a Metropolitan Police Service, but it is locally managed by the police and local governments in each of London’s 32 boroughs.155 . Amnesty Int’l, supra note 33, at 6. Although the Gangs Matrix is considered a central database, police officers in a given borough only have access to information in their jurisdiction, so if an individual in the database moves to another borough, the local agencies are encouraged to pass information along rather than access it directly through the database.156. Id. at 8–10. The centralization of information is also hampered by the fact that practices for adding and removing individuals to the database vary by borough so “[e]ach borough effectively has its local matrix.”157. Id. at 10. Similar challenges were found in California’s CalGang, which is considered a “pointer system,” where records within the database “point or refer” to external source documents that provide the basis for official police actions.158 . L.A. Police Dep’t Chief of Police. supra note 84, at 1 (2019). The CalGang data entry process is sorted through local and regional agency nodes so that all data uploaded to the central database is viewable by all user agencies but the information can only be added through specific regional nodes.159. See CalGang Architecture Infographic, Cal. Off. of Att’y Gen. (May 23, 2018), https://oag.ca.gov/sites/all/files/agweb/pdfs/calgang/architecture-infographic.pdf [https://perma.cc/SST6-ZTN2]. While this filtered process exists to provide more supervision and oversight, police departments still use duplicative, informal systems and practices locally, which can threaten the integrity of the central database.160 . Cal. State Auditor, supra note 154, at 11 (noting inconsistent local records and flaws in controls for accurate recordkeeping). Indeed, a 2020 CalGang audit revealed significant misuses by the Los Angeles Police Department (LAPD), which resulted in the department’s permanent withdrawal from the program and the state’s Attorney General’s revocation of statewide access to the department’s records, which makes up nearly 25 percent of the databases.161Press Release, State of Cal. Dep’t of Just., Attorney General Becerra Restricts Access to LAPD-Generated CalGang Records, Issues Cautionary Bulletin to All Law Enforcement, and Encourages Legislature to Reexamine CalGang Program (July 14, 2020), https://oag.ca.gov/news/press-releases/attorney-general-becerra-restricts-access-lapd-generated-calgang-records-issues [https://perma.cc/M8TD-FZRW]. Notably, less than a year before this finding of significant misuse, the LAPD performed its first department audit of CalGang, which found the LAPD had a 95 percent compliance rate with most CalGang policy objectives.162 . L.A. Police Dep’t Chief of Police, supra note 84, at 4. This revelation is significant because it highlights how the decentralized organizational structure of policing can impede centralized oversight of SDSs.
The third SDS organizational feature is that SDS enable compliance-based administrative practices that serve to justify law enforcement’s suspect profiling practices.163. See Carter, supra note 82, at 22 (“Traditional criminal profiles rely on the correlation between behavioral factors and the past experience of law enforcement in discovering criminal behavior associated with those factors. Thus, profiling rests on the perceived accuracy of the profile as a predicator of criminality” (internal citation omitted)). Compliance-based policing is a risk management practice that is executed and articulated through the administrative function of policing.164. See generally Ericson & Haggerty, supra note 20, at 48–52 (1997) (summarizing scholarship and theories on compliance-based law enforcement). Scholars have distinguished this model of policing from deterrence-based models because it is more interested in regulating undesirable activity rather than individual acts.165. See Albert J. Reiss Jr., Consequences of Compliance and Deterrence Models of Law Enforcement for the Exercise of Police Discretion, 47 Law & Contemporary Problems 83, 121–22 (Fall 1984). Criminal intelligence databases systematize this model by encoding current law enforcement profiling practices and policies, as well as constraining law enforcement’s understanding and response to a predetermined social problem. This process is made simple because police administrative practices are primarily retrospective, in that they document and, in many ways, justify what has happened.166. See generally Ericson & Haggerty, supra note 20, at 33 (“Paperwork is a way of retrospectively justifying what has been done for administrative purposes…”). When this data is entered and organized in criminal intelligence databases, the data entry requirements, labels, and other classification schemes shape how information is entered and presented, which in turn structures and legitimizes how police view their prior actions and subsequent investigations.167. See id. (“Police work is . . . prospectively structured by the categories and classifications of risk communication and by the technologies for communicating knowledge internally and externally. The communication formats provide the means through which the police think, act, and justify their actions”). Thus, criminal intelligence databases can prospectively shape the nature of a law enforcement inquiry and therefore what is subsequently entered.168. See Mastrofski & Willis, supra note 56, at 88–89 (2010) (“Computerized systems can police the inquiry by requiring that certain data entry fields be completed before the officer may proceed to complete other fields, and the range of acceptable responses can be controlled as well.”). If police are familiar with the data entry requirements of a database, they may standardize the sorts of questions they ask or evidence they look for in anticipation of expected administrative duties. Data in criminal intelligence databases can also unnecessarily limit law enforcement investigations, both “supplant[ing]” traditional investigative methods and serving as pretext for subjective and politically influenced policing practices (e.g., meeting “quotas”).169. See id. at 88–93; see also Oscar H. Gandy, Jr., Quixotics Unite! Engaging the Pragmatists on Rational Discrimination, in Theorizing Surveillance: The Panopticon and Beyond 319 (David Lyon ed., 2006) (describing how information about the past and past decisions can be used as pretext for further discrimination).
This third SDS organizational feature manifested in various ways among the cases reviewed. In the U.K., this feature influenced practices and decisions made by several criminal justice actors. In a speech regarding his concerns with the British criminal justice system, British Parliament member David Lammy noted how information in the Gangs Matrix is used in decision-making without verification:
[T]he Gangs Matrix features information provided by the police to the Crown Prosecution Service (CPS) at the point when the CPS makes charging decisions. The inclusion of this information suggests that prosecutors regard it as pertinent to whether defendants are charged, or what they are charged with. If cases make it as far as court, the Gangs Matrix could than [sic] [be] used by the prosecution in cases involving Joint Enterprise. It is deployed to substantiate claims that individuals are part of a gang and therefore played their part in a crime.170David Lammy, Member of Parliament, Speech to London Councils (July 15, 2016), https://www.gov.uk/government/speeches/review-of-bame-representation-in-the-criminal-justice-system.
Similar practices have been observed in the U.S. and India. For example, an investigative report on the NYPD’s gang database practices, Gang Takedowns in the De Blasio Era: The Dangers of Precision Policing, revealed that gang database designations are used to justify heightened police harassment.171 . Trujillo & Vitale, supra note 104, at 13. See also Citizens for Juv. Just., supra note 30, at 22 (highlighting incidents where the New Bedford Police Department harassed and violated constitutional rights of family members of a person listed in a gang database); see also, Kathleen McGrory & Neil Bedi, Targeted, Tampa Bay Times, (Sept. 3, 2020), https://projects.tampabay.com/projects/2020/investigations/police-pasco-sheriff-targeted/intelligence-led-policing/ [https://perma.cc/HYN7-63AQ] (“They [Pasco County Sheriff’s Office] swarm homes in the middle of the night, waking families and embarrassing people in front of their neighbors. They write tickets for missing mailbox numbers and overgrown grass, saddling residents with court dates and fines. They come again and again, making arrests for any reason they can.”). The report highlights several incidents where gang designation instigated hyper-policing of minor offenses like jaywalking, and how data entered from these hyper-policing practices were used to substantiate fabricated gangs and justify serious criminal charges.172 . Trujillo & Vitale, supra note 104 at 11–15 (2019). This phenomenon also resembles the fabricated ACAB gang discussed in the introduction. See Biscobing, supra note 5. Moreover, case law and testimony from criminal lawyers across Indian states suggest that the surveillance registers are used to meet and thus justify politically motivated law enforcement goals or targets.173Interview with Sonawane & Bokil, supra note 95 (“The habitual offender status is used to justify why a particular individual or group was arrested for a particular crime. The fact that there is a narrative of criminality already associated with these communities makes it easy for the police to build a case against them.”). Ethnographic accounts have observed that local politicians regularly put pressure on police departments to use the registers to ‘solve’ cases.174 . Khanikar, supra note 40 at 41. Similarly, a 2020 Madras High Court decision noted that the creation of a history sheet against the plaintiff without cause represented a “shortcut method” to meet police targets.175Thirumagan v. Superintendent of Police Madurai Dist. (2020), Madras HC, 20207-44 (Apr. 3, 2020) (per Seshasayee) https://indiankanoon.org/doc/156221879/ [https://perma.cc/FC3G-NQE2].
SDS organizational features help clarify a number of misrepresentations regarding policing practices, including the use of information technologies like SDS. The analysis enabled through SDS organizational features excavates pertinent, yet under-examined, aspects of SDS development, use, and outcomes, which can also function as sites for institutional reform. Such reforms can both mitigate abuses of SDS and their negative outcomes as well as address broader systemic concerns.
E. Social Features
This category explores conditions and features within society that structure the development, use, and outcomes of SDSs. Some social features are historical, so they precede and influence the development and use of databases, while others are contemporaneous with database use and thus constrain or influence how databases are operationalized and their broader social impact.
The first social feature is that SDSs are developed or enacted after major social change or crisis, which often include demographic, political, and economic shifts. In India, the practices and policies that shaped current surveillance and database practices stem from colonial police governance. Following independence, the Parliament replaced the Criminal Tribes Act with the Habitual Offenders Act, which established the rules that still govern surveillance databases in several states.176Habitual Offenders (Control And Reform) Act, Act No. XI of 1956 (India), https://www.indiacode.nic.in/bitstream/123456789/4911/1/habitual_offenders_act.pdf [https://perma.cc/MND3-Z4AJ]; see Brown, supra note 37. As scholar Mark Brown discusses, there was a paradigm shift from the group criminality model of classification (“grounded in concepts of tribes, castes, or gangs”) to the individual model (“based upon the idea of habituality”) that functions to normalize these discriminatory practices as part of “ordinary criminal law”.177Brown, supra note 37 at 197–98. Individualizing the category of the habitual offender could not overcome the group classification that remains embedded in a society ordered by a rigid and discriminatory caste system. In fact, in 2007 the UN Committee on Elimination of Racial Discrimination (CERD) asked the Indian government to repeal the Habitual Offenders Act, concluding that “the so-called denotified and nomadic, which are listed for their alleged ‘criminal tendencies’ under the former Criminal Tribes Act (1871), continue to be stigmatized under the Habitual Offenders Act.”178Rep. of the Comm. On the Elimination of Racial Discrimination on its Seventieth and Seventy-First Session, U.N. Doc. A/62/18, at 36. (2007), https://www.refworld.org/pdfid/473424062.pdf [https://perma.cc/4KR9-BLVC]. In the U.S., increased investments in gang suppression tactics and information technologies have occurred after mass migrations and immigration of racial and ethnic minorities that fundamentally altered the demographic makeup of the country and its major cities.179Referring to Black Americans escaping racial terrorism of Jim Crow in southern states and Puerto Ricans seeking economic opportunities migrated to northern and western cities of the U.S. The Immigration and Nationality Act of 1965 removed racially discriminatory barriers in American immigration policy and significantly altered immigration demographic in the United States. See generally Tukufu Zuberi, The Population Dynamics of the Changing Color Line, in Problem of the Century: Racial Stratification in the United States (Elijah Anderson & Douglas S. Massey eds., 2001). For a description of how these patterns of migration led to an increase in gang activity amongst minority groups in the U.S. see McCorkle & Miethe, supra note 88 at 45–49. These demographic changes also coincided with major economic declines, high unemployment rates induced by deindustrialization, and severe austerity measures encouraged by the financial sector as a condition of post-deindustrialization bailouts.180. See generally Wilson, supra note 115; see also Elizabeth Hinton, From the War on Poverty to the War on Crime: The Making of Mass Incarceration in America, 263–70 (2016); Jefferson, supra note 120, at 93–108 (2020). National and local crime rates rose as a consequence of government divestment and economic insecurity, but politicians employed racialized and inflammatory rhetoric to scapegoat the victims of their social policies as drivers of crime and social disorder.181. See Michael K. Brown, Martin Carnoy, Elliott Currie, Troy Duster, David B. Oppenheimer, Marjorie M. Schultz, & David Wellman, White-Washing Race: The Myth of a Color-Blind Society, 154 (2003) (describing research that demonstrated that rising crime rates were associated with rising economic inequality which confirmed strain theories that suggested crime was most likely to grow from relational socioeconomic inequality (citing Richard Fowels & Mary Merva, Wage Inequality and Criminal Activity: An Extreme Bounds Analysis for the United States 1976-1990, 34 Criminology 163–182 (1996))); see also Jefferson, supra note 120, at 111–28 (detailing how New York City and Chicago politicians scapegoated marginalized groups as symbols of disorder); Dan Baum, Legalize It All, Harper’s Mag. (Apr. 2016), https://harpers.org/archive/2016/04/legalize-it-all/ [https://perma.cc/YQY3-WPLP] (“The Nixon campaign in 1968, and the Nixon White House after that, had two enemies: the antiwar left and black people…We knew we couldn’t make it illegal to be either against the war or black, but by getting the public to associate the hippies with marijuana and blacks with heroin, and then criminalizing both heavily, we could disrupt those communities. . . . Did we know we were lying about the drugs? Of course we did.”). These same politicians then lobbied for technological upgrades to target and suppress gang violence and other criminal justice priorities.182. See generally Jefferson, supra note 120, at 111-118. In the U.K., the Gangs Matrix was created following nationwide civil riots that the government labeled as gang crime, though 81 percent of those arrested were not identified as gang members.183 . Amnesty Int’l, supra note 33, at 5. The government used this crisis to announce new anti-gang strategies that would involve more intelligence creation and sharing to identify and target known gang members and individuals at risk of violence.184. See id.at 5–6.
The second social feature is that SDS development and use is normalized through media amplification and dramatization of crime-related moral panic, which in turn shapes public opinion and encourages public acquiescence to related government practices and policies. Law enforcement officials and politicians often use vague but sensationalized rhetoric and terms like “gangs” and “habitual” or “prolific” offenders to suggest that their criminal intelligence database classifications are justified because these terms imply that their targets have extensive or violent criminal records.185. See e.g., McGrory & Bedi, supra note 171 (describing how the Pasco County Sheriff’s use of a “prolific offender” system targeted innocent, young people with aggressive policing tactics); Chip Brownlee & Ann Givens, After Years of Gang List Controversy, The NYPD has a New Secret Database. It’s Focused On Guns., Gothamist (May 12, 2022), https://gothamist.com/news/nypd-grip-list-guns [https://perma.cc/JT36-7DYH] (describing the NYPD’s “Gun Recidivist Investigation Program list”, where the title suggests individuals included are recidivist although the actual criteria can result in bystanders of shootings being added to the list). Yet these suggestive labels are misnomers since many individuals in the systems reviewed have no criminal record or fail to meet the criteria for inclusion.186. See Amnesty Int’l, supra note 33, at 5–7 (2018) (finding that the Gang Matix includes many individuals who have never been involved in violent crime); Bustamante & Howell, supra note 87, at 2–3 finding the majority of defendants in the Bronx 120 prosecution were not alleged gang members or convicted of violent crimes). These mischaracterizations also allow the government to avoid public scrutiny because they focus attention on symbols of disorder and evade interrogation of how the government contributes to the outcomes they profess to correct via policing.187. See generally Gandy Jr., supra note 60, at 153 (describing how policy promoters overemphasize aspects of a policy that are not easily measured or subject to falsification to avoid scrutiny of the net costs and benefits of a particular policy); see also Jefferson, supra note 120, at 99–128 (detailing examples of how criminal justice datasets and data-driven technologies are used to sidestep analysis of the criminal justice apparatus and how this contributes to the social problems police target); Gandy Jr., supra note 60, at 153 (arguing that policy promoter overemphasize aspects of a policy that are not easily measured or subject to falsification to avoid analysis of the net costs and benefits of a particular policy). Despite these mischaracterizations and fallacies, the media (e.g. journalism, scripted fiction, true crime) is drawn to the sensational nature of the subject and tends to overemphasize and amplify the problem, which then distorts public perceptions about actual crime rates and activity.188. See Susan R. Takata & Richard G. Zevitz, Divergent Perceptions of Group Delinquency in a Midwestern Community: Racine’s Gang Problem, 21 Youth & Soc’y 282 at 287–91 (1990) (finding some adults described the threats of gangs as very serious because they relied on local news reporting for their information about gang activity in their city); see also Ericson & Haggerty, supra note 20, at 54 (1997) (“The media are part of the legal institution in this regard, helping it to dramatize myths . . .”) McCorkle & Miethe, supra note 88, at 91–94 (describing how media influence public perceptions of the gang problem and related public policy); Gandy Jr., supra note 60, at 162 (“The ways in which risks are communicated to the public will influence the nature of their support for public policies designed to bring risks under control.”). For example, research shows that there are fewer gangs today despite narratives of growing gang activity.189Greene & Pranis, supra note 88, at 5. In the U.S. and the U.K., Black, Indigenous, and People of Color (BIPOC) and Black, Asian, and minority ethnic (BAME) communities190While these acronyms may not appear inclusive of every minority group adversely affected by these databases, they are terms precise to the local meaning and are used for that reason. are the public faces of the gang problem because they are overrepresented in these databases, although white individuals make up a considerable portion of gang and organized crime activity.191 . Greene & Pranis, supra note 88, at 4-5; Amnesty Int’l, supra note 33, at 15–19 (2018); Donna Ladd, Only Black People Prosecuted Under Mississippi Gang Law Since 2010, JACKSON FREE PRESS (Mar. 29, 2018, 1:32 PM) https://www.jacksonfreepress.com/news/2018/mar/29/only-black-people-prosecuted-under- mississippi-gan/ [https://perma.cc/S8XU-BGDE] (finding that between 2010 to 2017 only Black people were arrested under the Mississippi Gang Law, even though the Mississippi Association of Gang Investigators declared that 53 percent of verified gang members are white). Similarly, fictional crime dramas frequently mention or depict law enforcement use of SDSs in ways that suggest law enforcement use is always valid and lawful, and that these technologies are both effective and harmless, which normalizes and distorts public perceptions regarding these tools. Episode seven of the canceled crime drama Tommy portrayed a fictional account of the murder of Los Angeles rapper Nipsey Hussle, including the posthumous revelation that the rapper was listed in CalGang.192. See Tommy: Vic (CBS television broadcast, Apr. 2, 2020); Sidney Madden & Rodney Carmichael, Caught in the System: Nipsey Hussle, The LAPD and the Inescapable Trap of Gang Affiliation, NPR (Dec. 12, 2020, 7:01 AM), https://www.npr.org/2020/12/12/945454343/caught-in-the-system-nipsey-hussle-lapd-affiliation [https://perma.cc/8YRB-9A99]. Towards the end of the episode the main character, Tommy, who is also the first female police chief of the LAPD, starts to clear the LAPD’s gang database of individuals with no actual gang affiliation. This episode is one example of how crime dramas distort public perceptions of SDSs because it suggests that law enforcement officials actively audit and maintain these systems, undermines real-world incidents of LAPD abuse of gang databases, and reinforces myths that “good cops” can fix systemic issues.193The show’s distortion of reality is especially apparent when compared to actual stories of police abusing gang databases. See Stella Chan, Prosecutors Say Three LAPD Officers Falsified Gang Information, CNN (July 10, 2020, 8:47 PM), https://www.cnn.com/2020/07/10/us/los-angeles-police-officers-charged/index.html [https://perma.cc/U88F-PE28].
The third social feature is that SDSs encourage and rely on public participation in databasing efforts to legitimize its goals. This third feature is related to the second SDS social feature because public participation and reactions to perceptions of social problems are influenced by government and media representations. Who people know can be limited by neighborhoods and social networks, which are shaped and constrained by segregation.194. See generally, S.J. Smith, Political Interpretations of ‘Racial Segregation’ in Britain, 6 Env’t & Planning D: Soc’y & Space 423 (1988); Elise C. Boddie, Racial Territoriality, 58 UCLA L. Rev. 401, 434-42 (2010); Jessica Trounstine, Segregation by Design: Local Politics and Inequality in American Cities (2018); Elijah Anderson, The Cosmopolitan Canopy: Race and Civility in Everyday Life (2011); Rothstein, supra note 137; Robert J. Sampson, Neighbourhood Effects and Beyond: Explaining the Paradoxes of Inequality in the Changing American Metropolis, 56 Urb. Stud. 3, 7–9 (2019). Thus, public participation in databasing efforts often reproduces systemic and societal disparities. The proliferation of neighborhood communication systems like Nextdoor, Amazon’s Ring Neighbors app, and municipal 311 apps represent an emerging and direct form of public participation in profiling and surveillance practices that can lead to biased SDS inclusion.195. See David McCabe, Amazon’s Neighborhood Watch App Raises Discrimination, Privacy Fears, Axios (May 3, 2019), https://www.axios.com/amazons-neighborhood-watch-app-raises-discrimination-privacy-fears-ce765490-748a-4759-8627-4067ffd040c8.html [https://perma.cc/ZWE2-MUPV]; Michael Harrriot, The Racist Nextdoor, The Root (June 28, 2019, 11:17 AM), https://www.theroot.com/the-racist-nextdoor-1835939264 [https://perma.cc/9SYK-LWPC]; Dae Shik Kim Hawkins, An App for Ejecting the Homeless, Atlantic (June 28, 2018, 7:23 AM), https://www.theatlantic.com/technology/archive/2018/06/an-app-for-ejecting-the-homeless/563849/ [https://perma.cc/MT5Y-SC5U]. Moreover, informant arrangements remain a longstanding and prominent form of public participation in criminal justice investigations and practices. These arrangements are often coercive or incentivized, “exacerbat[ing] some of the most problematic features of the criminal justice process,” particularly its racial and caste disproportionality.196. See Alexandra Natapoff, Snitching: The Institutional and Communal Consequences, 73 U. Cin. L. Rev. 645–46. (2004). For instance, in some informant arrangements police will ignore or tolerate criminal activity if certain objectives are met.197While use of informants typically includes allowance of some criminality, use in the context of criminal intelligence databases involves value-laden judgment of which crimes or forms of criminality are “worse” for society and which harms are permissible in order to address “worse” crimes. Natapoff, supra note 196, at 647–50. This under-enforcement and tolerance of criminality, especially in historically overpoliced areas, skews criminal justice data and distorts public perceptions of who is a criminal or other symbolic terms that communicate criminality or some form of social risk (e.g. “gang” or “rowdie”).198. Id. One example from the U.S. is the NYPD’s Five Point Plan to prevent gangs from re-establishing a foothold in communities. Under this plan, the department notifies elected officials and community members about arrests and activity to effectively solicit community support and involvement in suppression efforts.199. See Shea, supra note 89, at 3. Similarly, in India, interviews with legal practitioners in multiple states revealed that the informant system is critical to the police’s process for determining who is included in surveillance databases.200Interview with Sonawane & Bokil, supra note 95; Interview with Wadekar, supra note 107. For example, in the state of Madhya Pradesh, the informant system colloquially known as “mukhbiiri” (word-of-mouth) typically involves individuals from targeted communities being coerced into becoming informants, but this cooperation does not preclude these informants from being targeted by police in the future.201Interview with Sonawane & Bokil, supra note 95, at 8. This menacing dynamic also contributes to the same communities being disproportionately represented in these systems and the range of punitive consequences that follow.
SDS social features highlight important contexts and dynamics that influence both how SDSs are developed and used, and how their use or impact is perceived publicly. These features also clarify why database use remains opaque to the public and why their outcomes are not subjected to the same level of scrutiny as other controversial technologies, such as predictive policing. The analysis enabled through this category and the broader framework should inform policy advocacy agendas and academic scholarship by broadening issue analysis beyond current practices and consequences.
II. Suspect Development Systems Beyond
the Criminal Justice System
While criminal intelligence databases are a paradigmatic example of an SDS, our categorical definition and analytical framework have value beyond the criminal justice system. The framework and analysis presented in Section I offers a systematic way to interrogate, evaluate, and eventually inform policy proposals to address database and related information technologies being used across government and private spheres. The key components of the definition of an SDS that we have put forth in this Article are (a) the use of information technologies to (b) manage social risk by (c) targeting individuals or groups to greater suspicion, differential treatment, and more punitive and exclusionary outcomes. There is a range of non-criminal databases that present these characteristics, especially as governments across the world turn to punitive measures to respond to a range of socio-economic problems in domains like social welfare, education, and immigration.202. See Simon, supra note 116; Usha Ramanathan, Ostensible Poverty, Beggary and the Law, 43 Econ. & Pol. Wkly. 33 (2008); see also David Lyon, Surveillance Studies: An Overview 184–85 (2007) (“What require direct attention are the classification and profiling processes . . . that, favouring and confirming the formation of social stereotypes, determine both the attribution of privileges and rights and social exclusion.”). As “technologies, discourses, and metaphors of crime and criminal justice”203 . Simon, supra note 116, at 4. have crept into multiple spheres of governance, we show how the category and framework of SDS can be employed helpfully across various domains.
In this Section, we identify three key categories of databases across jurisdictional contexts (social welfare, national ID, and citizenship) and argue that revisiting them through the SDS framework offers tactical benefits and nuances to our understanding of how these technologies can amplify structural inequities and cumulative disadvantage. For example, despite the origins of welfare distribution as a socially inclusive and non-punitive area of governance, the use of databases in this sphere has coincided with a global shift to a welfare state characterized by assumed scarcity, and an increased focus on eliminating “risky” people and behaviors.204. See generally Katherine Beckett & Bruce Western, Governing Social Marginality: Welfare, Incarceration, and the Transformation of State Policy, 3 Punishment & Soc’y 43 (2001). In this context, welfare databases present characteristics of an SDS and we analyze them in Part A of this Section through the features and categories distilled in Section I of this Article. Similarly, national biometric ID systems that are promoted across the Global South under an inclusive development mandate have had disproportionately exclusionary outcomes and legitimize differential treatment of those who are already subject to intersecting forms of marginality.205. See generally Reetika Khera, Dissent on Aadhaar : Big Data Meets Big Brother (2018); infra Part II(a). Here, the act of naming these systems as SDS can offer tactical benefits to civil society’s resistance to these projects.206For an example of resistance efforts to these technologies in the global south, see Rethink Aadhaar, https://rethinkaadhaar.in/ [https://perma.cc/MQ96-QWVP] (last visited May 7, 2022); Nation Team, Hudumba Namba Drive Off to a Slow Start Amid Protests, Nation (June 28, 2020), https://nation.africa/kenya/news/Hitches-rock-registration-for-Huduma-Namba-/1056-5057440-g1iynuz/index.html?fbclid=IwAR2peaCW5m8Rptyb3WRu_3MeCpbnGn9ITBJaMriooCgW-GdoTfIdWboOjXE [https://perma.cc/DD9R-AK65]. Finally, the fact that immigration policies and practices are increasingly focused on securitization and criminalization is well-studied as a global phenomenon,207. See, e.g., Robert Hartmann McNamara, The Criminalization of Immigration: Truth, Lies, Tragedy, and Consequences (2020); The Securitisation of Migration in the EU: Debates Since 9/11 (Gabriella Lazaridis & Khursheed Wadia eds., 2015). and in Part C of this Section, we use the SDS framework to explore how recent citizenship database projects facilitate broad-based suspicion, exclusionary outcomes, and new forms of precarity for those who are deemed suspect.
A. Welfare System Databases
Welfare state databases are uniquely ripe for SDS analysis because “welfare state development involves defining the boundaries of national membership,” particularly “who is entitled to the benefits and social protection that the state will offer[.]”208Robert C. Lieberman, Shaping Race Policy: The United States in Comparative Perspective, 27 (2005). The welfare state was initially conceived as an inclusive policy regime209Beckett & Western, supra note 204, at 44 (“Inclusive regimes emphasize the need to improve and integrate the socially marginal and tend to place more emphasis on the social causes of marginality.”). that could engender social solidarity, but the persistent political commitment to austerity economics has left governments with fewer resources and has recast the welfare state as an exclusive policy regime210. See id. (“[E]xclusionary regimes emphasize the undeserving and unreformable nature of deviants, tend to stigmatize and separate the socially marginal, and are hence more likely to feature less generous welfare benefits and more punitive anti-crime policies.”). that is more suspicious and punitive in practice and focused on eliminating people and behaviors that are considered risky or deviant.211. See Lieberman, supra note 208, at 27–39; Beckett & Western, supra note 204, at 45–47; Reetika Khera, Impact of Aadhaar on Welfare Programs, Economic & Political Weekly (Dec. 16, 2017). Databases are a historical and contemporary feature of welfare administration because welfare policies have increasingly necessitated large data collection and use. Because these databases have become digitized, in whole or part, during the aforementioned policy regime shift, these “systems of social protection and assistance are increasingly driven by digital data and technologies that are used to automate, predict, identify, surveil, detect, target and punish” in countries that have adopted them.212Alston, supra note 47, at 4.
Digitized welfare databases and systems are becoming more prevalent globally, but their uses can vary. Some systems are used to implement new welfare policies and administrative practices, while others are used to detect noncompliance and undesirable behavior. India (“Aadhaar”) and Kenya (“Huduma Namba”) have developed national biometric identification databases (discussed in detail in Section II(b)) that are used for verifying welfare recipients and to distribute benefits, services, and subsidies.213 . Vrinda Bhandari, The Ctr. for Internet & Soc’y, Digital ID for the Delivery of Welfare (2020), https://digitalid.design/docs/CIS_DigitalID_WelfareCaseStudy_2020.07.pdf [https://perma.cc/6UC2-ZMQA] (evaluating the use of digital identity systems in Kenya, India, and Estonia for the purpose of delivery of welfare across jurisdictions); Presidency, National Integrated Identity Management System, Exec. Order No. 1 (issued in June 2018) (Kenya). The U.K. introduced its Universal Credit system as a “digital-by-default” social welfare reform intended to integrate and automate the administration of six “legacy” welfare benefits through a singular platform where benefits were consolidated and allocated monthly.214. Nothing Left in the Cupboard: Austerity, Welfare Cuts, and the Right to Food in the UK, Hum. Rts. Watch (May 20, 2019) https://www.hrw.org/report/2019/05/20/nothing-left-cupboards/austerity-welfare-cuts-and-right-food-uk [https://perma.cc/MA2F-W2LD]; see also, Lizzie Coles-Kemp, Debi Ashenden, Amelia Morris & Jeremy Yuille, Digital Welfare: Designing for More Nuanced Forms of Access, 3 Pol’y Design and Prac. 177, 177–88 (2020). Unlike previous policies, Universal Credit emphasizes the individual responsibility of benefits claimants through the imposition of additional requirements to receive benefits and harsh sanctions.215. See Statement on Visit to the United Kingdom, by Professor Philip Alston, U.N. Special Rapporteur on Extreme Poverty and Human Rights 4–7 (Nov. 16, 2018), https://www.ohchr.org/sites/default/files/Documents/Issues/Poverty/EOM_GB_16Nov2018.pdf [https://perma.cc/JS69-97UZ]; Lina Dencik, Arne Hintz, Joanna Redden & Harry Warne, Data Scores as Governance: Investigating Uses of Citizen Scoring in Public Services 113 (2018); Amos Toh, Human Rights Watch, Automated Hardship: How the Tech-Driven Overhaul of the UK’s Social Security System Worsens Poverty (2020), https://www.hrw.org/report/2020/09/29/automated-hardship/how-tech-driven-overhaul-uks-social-security-system-worsens [https://perma.cc/4AG2-J3CD]. In contrast, the Netherlands implemented the System Risk Indication (SyRI) system to predict individuals most likely to commit benefits fraud,216. See Brief of the U.N. Special Rapporteur on Extreme Poverty and Human Rights as Amicus Curiae in the case of NJCM c.s./De Staat der Nederlanded (SyRI) before the District Court of The Hague (case number: C/09/550982/ HA ZA 18/388) at 3–5, Ktg. Hague 5 februari 2020, NJ 2020, 386 m.nt. E.J. Dommering (NJCM c.s./De Staat der Nederlanden (SyRI)) (Neth.) [hereinafter Brief of the U.N. Special Rapporteur on Extreme Poverty and Human Rights as Amicus Curiae] (detailing the social policy history that led to the development and use of the SyRI system), https://www.ohchr.org/Documents/Issues/Poverty/Amicusfinalversionsigned.pdf [https://perma.cc/Z44T-4YG8]. though a national district court recently found the use of the system unlawful because it violated the right to privacy.217. See Ktg. Hague 5 februari 2020, NJ 2020, 386 m.nt. E.J. Dommering (NJCM c.s./De Staat der Nederlanden (SyRI)) (Neth.). Digitized welfare systems have also been a source of legal challenges in the U.S. Systems used to determine and allocate benefits and automated fraud detection and benefits disqualification systems have been met with litigation.218Rashida Richardson, Jason M. Schultz & Vincent M. Southerland, Litigating Algorithms 2019 US Report: New Challenges to Government Use of Algorithmic Decision Systems (2019) (highlighting several legal challenges to digitized welfare systems in the United States).
Critical analysis by the U.N. Special Rapporteur on Extreme Poverty and Human Rights, Philip Alston, and successful legal challenges have illuminated significant concerns regarding the development and use of digital welfare state systems. Understanding these systems as SDSs, however, can expand public understanding overall and better inform holistic policy interventions. Here, we demonstrate how the five SDS framework categories can broaden our collective understanding of welfare databases as SDSs.
The SDS technical features demonstrate that database design exists on a spectrum and digitized welfare systems can exist in various forms. In the U.S., the government of Idaho used a formula in Microsoft Excel to determine and terminate Medicaid benefits for individuals with developmental disabilities,219Jay Stanley, Pitfalls of Artificial Intelligence Decisionmaking Highlighted in Idaho ACLU Case, ACLU (June 2, 2017, 1:30 PM), https://www.aclu.org/blog/privacy-technology/pitfalls-artificial-intelligence-decisionmaking-highlighted-idaho-aclu-case. [https://perma.cc/7TNF-CTBE]. whereas the Netherlands’ SyRI system used deep learning and data mining methods for fraud detection.220. NJCM, NJ 2020, 386. The SDS legal features broaden our understanding of the legal and social consequences of welfare databases. In particular, SDS legal features reveal that these systems are often put into place without robust legal oversight or constitutional scrutiny, which means that their overall legality is questionable. Furthermore, a lack of meaningful legal safeguards may enable new modes of governance and social exclusion. Kenya’s Huduma Namba system amplifies decades of social policy aimed to exclude certain ethnic groups, which was previously found unconstitutional, and the failure to produce suitable government identification can lead to an automatic denial of government services, which is out of line with constitutional principles.221. See Rasna Warah, Huduma Namba: Another Tool to Oppress Kenyans?, The Elephant (Apr. 20, 2019), https://www.theelephant.info/op-eds/2019/04/20/huduma-namba-another-tool-to-oppress-kenyans/ [https://perma.cc/SF66-YQER] (detailing the concerns of social exclusion and marginalization associated with the Huduma Namba); Alston, supra note 47, at 8 (highlighting that although Huduma Namba registration is voluntary, the Kenyan government is “reportedly threatening to withdraw unregistered individuals’ access to benefits and the right to vote.”). Similarly, the U.K.’s Universal Credit system impedes appeal procedures, particularly those regarding disability assessments.222. See Sophie Howes & Kelly-Marie Jones, Child Poverty Action Group, Computer Says ‘No!’ Stage Two: Challenging Decisions (2019) (detailing the harms and legal concerns associated with the implementation of the Universal Credit system). Prior to the Universal Credit system, citizens could appeal decisions directly to an independent tribunal, but now under the system, appeals are a two-stage process that introduces a variety of impediments, including the system losing appeal requests and dissuasion by government officials.223. Id. at 3–4. Thus, the SDS legal features help reveal that in practice, welfare databases can operate as a new mode of governance or social exclusion. The lack of scrutiny such databases received upon their introduction has resulted in them lacking meaningful legal safeguards, which means that these databases effectively determine welfare rights and the allocation of resources without providing avenues for legal recourse.224. See Khera, supra note 211.
The SDS political economy features demonstrate how welfare databases are another form of the commodification of risk management for private-sector financial benefit. In the U.S., many state welfare systems are developed by private companies. For example, Indiana entered a $1.3 billion contract with IBM to privatize and automate the state’s welfare eligibility process.225. See Virginia Eubanks, Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor, 39–83 (2018) (describing Indiana’s failed attempt to automate its welfare system); Judge: IBM owes Indiana M for failed welfare automation, Bus. Standard (Aug. 7, 2017, 8:28 PM), https://www.business-standard.com/article/pti-stories/judge-ibm-owes-indiana-78m-for-failed-welfare-automation-117080701115_1.html [https://perma.cc/4NFE-XRKW] (describing a ruling that IBM owes Indiana million in damages stemming from its cancelled .3 billion contract to privatize and automate the state’s welfare system). IBM also sells a customizable, off-the-shelf software package to automate eligibility decisions.226Alston, supra note 47, at 9. In Kenya, there are allegations that banks associated with the Kenyan President’s family could benefit from a credit lending platform linked to the Huduma Namba.227Wycliffe Nyamasege, Huduma Namba Is a Fraud Scheme Aimed at Benefiting Kenyatta Family—David Ndii, Kahawa Tungu (Apr. 5, 2019), https://www.kahawatungu.com/huduma-namba-fraud-benefiting-kenyatta-family-david-ndii/ [https://perma.cc/W9WH-MMTT]. The SDS organizational features reveal that while welfare systems are often digitized to centralize information and overall governance, in practice, these systems facilitate greater diffusion and delegation. Indeed, the U.K.’s Universal Credit system included delegation of unanticipated government responsibilities as well as automation of some system functions, which ultimately led to further diffusion. The government failed to anticipate the additional administrative burdens that would accompany a “digital by default” policy change, so digital assistance for digitally excluded and digitally illiterate citizens was “outsourced to public libraries and civil society organizations.”228. See Statement on Visit to the United Kingdom, supra note 215, at 7–8. The Universal Credit system also requires automated calculations of benefits via the Real-Time Information system, but this automation was error-ridden and subsequently required a team of fifty full-time civil servants to rectify 2% of millions of incorrect transactions every month.229. Id at 9.
Finally, SDS social features aid our understanding of how politicians and media employ and amplify sensational rhetoric to engender public acquiescence or support of these controversial systems and their disproportionate effects.230. See Alston, supra note 47, at 10 (noting how conservative politicians have historically employed tropes to discredit inclusive welfare policy). Government officials and media invoke tropes like, “ghost beneficiaries” and “fraudsters”231. See, e.g., Jean Drèze, Are Ghosts Really Getting Midday Meals?, The Wire (Mar. 28, 2017), https://thewire.in/government/midday-meals-aadhaar [https://perma.cc/DQA2-HJPH]; Dutch Councils Use Algorithms to Identify Potential Social Security Fraudsters, Dutch News.nl (Apr. 9, 2018), https://www.dutchnews.nl/news/2018/04/dutch-councils-use-algorithms-to-identify-potential-social-security-fraudsters/ [https://perma.cc/AWN8-SKQ3]. to suggest that certain individuals or communities are undeserving of government benefits and to justify exclusionary practices and effects. The Netherlands’ SyRI system was promoted as an intervention to identify potential “fraudsters,” but the system was exclusively used in municipalities with “a high proportion of low-income residents, migrants, and ethnic minorities.”232Press Release, Office of the United Nations High Commissioner for Human Rights, The Netherlands Is Building a Surveillance State for the Poor, Says UN Rights Expert (Oct. 16, 2019), https://www.ohchr.org/en/press-releases/2019/10/netherlands-building-surveillance-state-poor-says-un-rights-expert?LangID=E&NewsID=25152 [https://perma.cc/6NQU-B296]; see Brief for United Nations Special Rapporteur on Extreme Poverty and Human Rights as Amicus Curiae, supra note 216. This not only meant that the harsher consequences associated with the system would disproportionately affect more vulnerable communities,233Brief for United Nations Special Rapporteur on Extreme Poverty and Human Rights as Amicus Curiae, supra note 216; Press Release, Office of the United Nations High Commissioner for Human Rights, supra note 232. but also that it increased the likelihood of socially-biased and false conclusions that these communities have a greater propensity for crime or are undeserving of government benefits.
B. National Biometric ID Databases
National biometric identity (ID) systems that generate a unique identifier for each person in the country and create a corresponding biometric database are being rolled out at “a dizzying pace across the developing world.”234Masood Ahmed, Preface to Alan Gelb & Anna Diofasi Metz, Identification Revolution: Can Digital ID Be Harnessed for Development?, at ix (2017). The Indian ID system, known as “Aadhaar,” was launched in 2009. With more than one billion enrollments, it is currently the largest biometric database in the world.235Ranjit Singh & Steven J. Jackson, From Margins to Seams: Imbrication, Inclusion, and Torque in the Aadhaar Identification Project, in CHI ‘17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems at 4776, 4777 (2017). The “Aadhaar model” is also being promoted in West Africa236Bronwen Manby, ‘Legal Identity for All’ and Statelessness: Opportunity and Threat at the Junction of Public and Private International Law, 2 Statelessness & Citizenship Rev. 248, 264 (2020). and the Middle East.237. See Aditi Gyanesh, IIITB to Develop Aadhaar-like Database for Morocco, Times of India (Aug. 30, 2018, 7:36 AM), https://timesofindia.indiatimes.com/city/bengaluru/iiitb-to-develop-aadhaar-like-database-for-morocco/articleshow/65599637.cms [https://perma.cc/Y3MS-V9LK]. Along with demographic information, these systems typically collect biometrics like fingerprints, iris scans, or facial scans that are used for one-time enrollment into an ID database and in many cases, as a continuing means of authentication.238For example, the Indian Aadhaar system uses biometric information for continuing authentication whereas Kenya’s project “Huduma Namba” only uses it for one-time enrolment. These projects have been promoted by traditional development organizations like the World Bank as a “universal identity gateway” for all of a person’s interactions with state and private bodies and touted as a solution to the problems of corruption and fraud in service delivery in the Global South.239. See Ursula Rao & Vijayanka Nair, Aadhaar: Governing with Biometrics, 42 South Asia: J. South Asian Stud. 469, 469 (2019). Yet, across countries these projects are met with stiff civil society resistance and multiple legal challenges on grounds of hardship and exclusion experienced by vulnerable people due to errors in implementation, unwarranted bodily surveillance, privacy, and data security risks.240. See Warah, supra note 221 (describing the problems with Kenya’s system, Huduma Namba, noting “in the absence of a law protecting personal data from abuse or misuse, what guarantee do Kenyans have that their data will not be sold off to a third party for political or commercial reasons?”). Judicial responses to these systems are mixed. After years of public resistance and strategic litigation against the Aadhaar project, in 2018 the Indian Supreme Court found the project to be constitutional and put several limits on the use of the database by private companies.241. See Vrinda Bhandari & Renuka Sane, A Critique of the Aadhaar Legal Framework, 31 Nat’l L. Sch. India Rev 72, 73, 78 (2019). In 2019, the Jamaican Supreme Court struck down Jamaica’s centralized, mandatory biometric ID system.242. See Julian Robinson v. Att’y Gen. of Jamaica,  JMFC Full 04 ¶¶ 249-54 (Jam.), https://supremecourt.gov.jm/sites/default/files/judgments/Robinson%2C%20Julian%20v%20Attorney%20 General%20of%20Jamaica.pdf. In January 2020, the Kenyan High Court also paused enrollment into their biometric ID system that proposed to collect a range of biometric information including face, voice sample, and even DNA.243Case Study: Kenya’s Biometric ID System, Catalysts for Collaboration, https://catalystsforcollaboration.org/case-study-kenyas-biometric-id-system/ [https://perma.cc/E8DS-HNAU] (last visited Mar. 17, 2022).
Despite heated public and legal resistance, debates around ID databases remain largely siloed from broader Western-dominated discourse on data-driven systems like AI or ADS.244For example, the premier conference on fairness and accountability concerns with AI/ADS, ACM’s FaccT conference has typically not included papers on biometric ID systems. See ACM FAccT 2022 Accepted Papers, ACM FAccT Conf., https://facctconference.org/2022/acceptedpapers.html [https://perma.cc/CP4R-ME38] (last visited June 25, 2022). Meanwhile a major civil society coalition against unregulated ID systems, “Why ID” has relatively low participation from Western civil society organizations that focus on AI/ADS. #WhyID, https://www.accessnow.org/whyid/ [https://perma.cc/DG6J-JNP9] (last visited May 7, 2022). In this context, the SDS categorical definition and framework for conceptualizing and evaluating these systems can offer two strategic benefits. First, the analysis helps interrogate and challenge the narratives of universality and inclusion that have legitimized these projects in the Global South by focusing on the ways they can further entrench marginality and lead to the exclusion of certain groups from the vision of the nation state. Second, the analysis creates linkages with other SDSs across the world that can bridge counterproductive siloes between the Global South and Global North and prevents these developments from being relegated to the margins of mainstream discourse on AI and other so-called “advanced” systems.
As demonstrated by the technical features of the SDS framework, databases can be structured across a range of technical affordances. National biometric ID systems in different countries have varying technical designs, often distinguished based on the degree of centralization of the database, the use of biometrics (either one time at enrollment or as a continuing means of enrollment), and the degree to which the identifier allows linkages between different government (and even private) databases.245. See Digital Identities: Design and Uses, The Ctr. for Internet & Soc’y (June 11, 2019), https://cis-india.github.io/digitalid.design/ [https://perma.cc/966K-H4QW]. In many countries, these systems have led to the digital transformation of legacy, paper-based civil registration systems,246. See Manby, supra note 236, at 263–64. and the impact of these shifts should be explored. The SDS framework discourages the creation of technical thresholds and instead analyzes the ways in which these varying features and technological legacies shape the logic and outcomes of these databases.
The SDS legal analysis illuminates the complex relationship of law and regulation to national ID systems. The authorizing legislation for many of these projects has been characterized by vague and open-ended definitions of the purpose of these systems, the types of biometric data to be collected, and implementation protocols on the ground.247. See Robinson,  JMFC Full 04 ¶ 349; Bhandari & Sane, supra note 241, at 84–85. As the first legal feature of our framework explains, this definitional ambiguity works to frustrate the aims of due process and rule-of-law that legal and policy frameworks purport to fulfill. The third legal feature speaks to the lack of any meaningful form of legal redress, given that SDS require structural reforms rather than procedural safeguards at the individual level. When governments like India and Kenya were faced with constitutional challenges that posed existential threats to their respective ID projects, they argued that data protection regulation would be sufficient to address any perceived risk.248. See Amba Kak, The State of Play and Open Questions for the Future, in Regulating Biometrics: Global Approaches and Open Questions 16, 423–25 (Amba Kak ed., 2020) (“The mere existence of procedural safeguards like data security or consent can obscure the root of the problem, only serving to legitimize the continued existence of these systems.”). But this argument ignores broader structural challenges to these biometric ID databases including the exclusion of individuals from basic entitlements, the hyper-surveillance enabled by these systems, and the potential to facilitate undemocratic forms of governance.249. See Rao & Nair, supra note 239. As we explore in Section I(b), legal frameworks that apply to SDS are often cosmetic and obscure or even legitimize more structural concerns (e.g. discrimination and privacy intrusions).
The SDS organizational features reveal that while biometric ID systems are touted as centralized systems with clearly defined national protocols, in practice these rules did not always govern the behavior of those operating the biometric systems on the ground, and there were no oversight safeguards to monitor non-compliance. For example, the Indian government attempted a system of “manual overrides” to address the issue of biometric errors that lead to exclusion from government benefits and systems.250Geetanjali Krishna, PDS Focus Should Be on People, Not Fingerprints, Hindu Bus. Line, (Aug. 25, 2017, 2:04 AM), https://www.business-standard.com/article/economy-policy/pds-focus-should-be-on-people-not-fingerprints-117082500068_1.html [https://perma.cc/4XAW-VRYG]. https://thewire.in/government/rti-campaign-aadhaar-nrega. However, managers of these systems often failed to exercise this option and refused people access to services because of lack of interest in overcoming technical errors.251. See Bidisha Chaudhuri, Paradoxes of Intermediation in Aadhaar: Human Making of a Digital Infrastructure, 42 S. Asia: J. S. Asian Stud. 572, 572–73 (2019). As Anthropologist Ursula Rao has noted, these “mega-projects . . . aim to reduce the space for digression in decision-making. Yet, in practice, they cannot be implemented and do not function without the creative improvisation of intermediaries who adapt the system . . . sometimes having to break the rules in the process.”252Rao & Nair, supra note 239, at 481.
The first political economy feature of SDS guides exploration of how a range of private interests played a covert, yet foundational, role in shaping the design, development, and current use of these government projects. While the Indian government and development agencies promoted Aadhaar primarily as a means of government service delivery, the project was designed from the start to facilitate new forms of market activity and commercial surveillance,253. See Aria Thaker, The New Oil: Aadhaar’s Mixing of Public Risk and Private Profit, The Caravan (Apr. 30, 2018), https://caravanmagazine.in/reportage/aadhaar-mixing-public-risk-private-profit [https://perma.cc/4HNX-4K55]; Usha Ramanathan, Who Owns the UID Database?, MediaNama
(May 6, 2013), https://www.moneylife.in/article/aadhaar-who-owns-the-uid-database-part-ii/32440.html [https://perma.cc/4QFE-D8G6]; Pam Dixon, A Failure to “Do No Harm”—India’s Aadhaar Biometric ID Program and Its Inability to Protect Privacy in Relation to Measures in Europe and the U.S., 7 Health & Tech. 539, 546–47 (2017). and make the crucial shift of turning “citizens” into “customers.”254Bidisha Chaudhuri & Lion König, The Aadhaar Scheme: A Cornerstone of a New Citizenship Regime in India?, 26 Contemp. S. Asia 127, 137 (2018). Private companies (e.g. banks, telecom companies, background check services) were using the Aadhaar databases for authentication and tracking of individuals with little scrutiny or safeguards until the Indian Supreme Court intervened in 2018.255. See Bhandari & Sane, supra note 241, at 77–79. Meanwhile, in Ghana, Rwanda, Tunisia, Uganda, and Zimbabwe, ID databases have facilitated “citizen scoring” exercises like credit reference bureaus to emerge at scale.256. See Nicolas Kayser-Bril, Algorithm Watch, Identity-Management and Citizen Scoring in Ghana, Rwanda, Tunisia, Uganda, Zimbabwe and China (2019), https://algorithmwatch.org/wp-content/uploads/2019/10/Identity-management-and-citizen-scoring-in-Ghana-Rwanda-Tunesia-Uganda-Zimbabwe-and-China-report-by-AlgorithmWatch-2019.pdf [https://perma.cc/938C-TT6R]. Finally, the social features of SDS enable an exploration of the narratives that accompany and promote these projects. Faced with growing criticism and multiple legal challenges, the head of the Aadhaar project would regularly publish opinion pieces in the country’s most respected newspapers stating that privacy concerns were a “Luddite paranoia” and numerous advertisements on television channels portrayed wealthy upper-caste Indians vouching for the fact that Aadhaar is the most trusted ID in India.257Anandita Thakur & Karan Saini, Selling Aadhaar: What the UIDAI’s Advertisements Don’t
Tell You, The Wire (Aug. 23, 2018), https://thewire.in/rights/aadhaar-advertisements-identity-citizenship-rights [https://perma.cc/8S36-ULKA]. Meanwhile, in Kenya, political leadership infamously announced that the Huduma Namba would be the “single source of truth” about every Kenyan.258. Huduma Namba, Republic of Kenya, https://www.hudumanamba.go.ke/ (last visited Mar. 18, 2022) (“The purpose of this initiative is to create and manage a central master population database which will be the ‘single source of truth’ on a person’s identity.”) [https://perma.cc/QH2F-2FQV]; see also Special Correspondent, The Truth About the ‘Single Source of Truth About Kenyans’: The National Digital Registry System, Collateral Mysteries and the Safaricom Monopoly, The Elephant (Apr. 18, 2019), https://www.theelephant.info/features/2019/04/18/the-truth-about-the-single-source-of-truth-about-kenyans-the-national-digital-registry-system-collateral-mysteries-and-the-safaricom-monopoly/ [https://perma.cc/M7TG-4QHK]. Paying close attention to these popular narratives is critical to an understanding of how SDSs are justified and legitimized through the use of strategic imagery and rhetoric.
C. Citizenship Databases
In this Section, we analyze certain kinds of citizenship databases using the SDS framework. These government databases aid immigration policy enforcement and can be used to record and verify citizenship status or maintain information relevant to immigration enforcement priorities (e.g. deportation of individuals with criminal charges or convictions).259The databases we cover in this Section are distinct from government databases that contain citizenship status information but are not primarily used for immigration affairs, such as the U.S. Census databases. Citizenship databases represent a burgeoning global use case, as many of these systems are in different stages of development and use.260. See, e.g., Databases for Deportation, statewatch, https://www.statewatch.org/deportation-union-rights-accountability-and-the-eu-s-push-to-increase-forced-removals/deportations-at-the-heart-of-eu-migration-policy/databases-for-deportations/ [https://perma.cc/64CY-HKCR] (last visited Jan. 28, 2021) (highlighting the European Union’s use of databases and networked information systems for migration policy enforcement). Here, we review India’s National Register of Citizens and the United States’ Homeland Advanced Recognition Technology System (HART). Because these databases are not fully implemented, this Section demonstrates how the SDS framework can be used to forecast analysis of concerns and harms before they occur based on available information and local historical precedent.
1. National Register of Citizens (NRC)
In December 2019, massive nationwide protests erupted across India with protestors holding signs that read “freedom from databases”261Ramya Chandrasekhar, Identity As Data: A Critique of the Navtej Singh Johar Case and the Judicial Impetus Towards Databasing of Identities, 12 NUJS L. Rev. 1, 25 n.129 July–Dec. 2019, at (2019) (noting that protestors carried signs reading “Database se Azadi” which means “freedom from databases” in Hindi). and “freedom from the [National Register of Citizens].”262Nayanika Mathur, “NRC se Azadi”: Process, Chronology, and a Paper Monster, S. Asia Multidisciplinary Acad. J., Winter 2019–20, at 1. Several months after the announcement, there is still no clear timeline for implementation of the NRC. The protestors were responding to the Home Affairs Minister’s announcement of the intention to create a National Register of Citizens (NRC).263. See Special Correspondent, No Room for Complacency, India Still in Danger of Biased, Arbitrary Citizenship Regime: N. Ram, The Hindu (Jan. 4, 2021, 2:06 AM), https://www.thehindu.com/news/national/no-room-for-complacency-india-still-in-danger-of-biased-arbitrary-citizenship-regime-n-ram/article33485293.ece [https://perma.cc/RCN6-PZ37]. The NRC would require every person who self-identifies as a citizen of the country to furnish documentary evidence that confirms them as Indians, culminating in a computerized database that would be the definitive determinant of citizenship status.264. See Niraja G. Jayal, The Misadventure of a New Citizenship Regime, Hindu (Nov. 27, 2020, 12:33 AM), https://www.thehindu.com/opinion/lead/the-misadventure-of-a-new-citizenship-regime/article30090226.ece [https://perma.cc/4VBQ-L32G] (“[T]he objective of this latest initiative is to count citizens – specifically to sift and sort citizens from non-citizens, to include and exclude, and having done so to weed out ‘infiltrators’ destined for detention camps and potential deportation.”). The National Population Register (NPR), a parallel civil registration project, was also announced as means to identify the “doubtful” citizens that might be subject to deregistration in the NRC.265For an explanation of the relationship between the NPR and NRC, see K. Venkataraman, Explained: What connects the NPR, NRIC and Census?, The Hindu (Dec. 26, 2019, 9:41 PM), https://www.thehindu.com/news/national/what-connects-the-npr-nric-and-census/article61602475.ece [https://perma.cc/Q7W6-WX3P]. See also Mudita Girotra, National Population Register Will Serve as Database for NRC, Oppose It: Arundhati Roy, New Indian Express (Dec. 25, 2019, 9:12 AM), https://www.newindianexpress.com/nation/2019/dec/25/national-population-register-will-serve-as-database-for-nrc-oppose-it-arundhati-roy-2080828.html [https://perma.cc/LT8T-HCR3]. The announcement provoked memories of the horrors of the NRC that had been conducted in the Indian state of Assam starting in 2005, a process that was eventually shelved, but marked by “[k]afkaesque bureaucracy,”266. See Mathur, supra note 262, at 1–2. lengthy tribunal processes for those appealing exclusions, and eventually, four million people rendered stateless.267Soutik Biswas, What Happens to India’s Four Million ‘Stateless’ People?, BBC News, (July 30, 2018), https://www.bbc.com/news/world-asia-india-45002670 [https://perma.cc/B87M-S5ZY].
The consequence of not being included in the NRC in Assam was mass disenfranchisement, and many believe a nationwide NRC will inevitably replicate this effect at a frightening scale. Across India, the poor, illiterate, women, and members of marginalized communities (like Dalits) lack access to documentary evidence of citizenship, which will leave them disproportionately excluded from the final database.268. See Debraj Banerjee, The Nation and its Aliens: Contentions of Citizenship under a Hindu Nationalist Regime, at 5 (2020) (M.A. thesis, Central European University), http://www.etd.ceu.edu/2020/banerjee_debraj.pdf; Mathur, supra note 262. The national NRC also had an explicitly discriminatory stance against members of the Muslim community in India within an atmosphere of growing Hindu majoritarianism.269. See Jayal, supra note 264. The announcement of the NRC was accompanied by the swift enactment of the Citizenship Amendment Act (CAA)270The Citizenship (Amendment) Act, 2019, Bill No. 47 of 2019 (Dec. 12, 2019) (India). which specifically offered citizenship to Hindus, Christians, Jains, Sikhs, Buddhists, and Parsis from neighboring countries in South Asia.271. Id. at § 2; see also Dharwi Vaid, One Year of India’s Citizenship Amendment Act, DW, (Dec. 11, 2020), https://www.dw.com/en/one-year-of-indias-citizenship-amendment-act/a-55909013 [https://perma.cc/P5MM-X94U]. Muslims were categorically left out of this declaration.
The NRC functions as an SDS, both in terms of its final consequences for those excluded from the database, as well as for the process which is designed to cultivate suspicion and distrust against a much broader network of people.272For a discussion on NRC-as-process and procedure as violence, see generally Mathur, supra note 262; Suddenly Stateless Conversation Series III: Procedure as Violence, The Polis Project (Mar. 27, 2020) https://www.thepolisproject.com/listen/suddenly-stateless-conversation-series-iii-procedure-as-violence/ [https://perma.cc/A8WL-DE7L] (Podcast Transcript). The SDS framework categories provide a systematic way to unravel and evaluate the multifaceted forms of marginality that such databasing creates and reinforces. The timeline for implementation of the NRC has still not been announced, and with the COVID-19 pandemic it is likely to be delayed considerably. This provides a window of opportunity where the SDS framework can also be used to strengthen advocacy against the project before its harms are realized.273. See Special Correspondent, supra note 263.
The SDS technical category aids inquiries into how the computerized form of the NRC274Namrata Ahuja, Govt Plans Aadhaar-Type Digital Database for NRC, Week, (Oct. 9, 2019, 8:08 PM), https://www.theweek.in/news/india/2019/10/09/govt-plans-aadhaar-type-digital-database-for-nrc.html [https://perma.cc/R6FL-86CB]. interacts with legacy paper-based citizenship records and expands the scope of its consequences. For example, the networked database form of the NRC would allow inter-and intra-government sharing, which will likely make the consequences of exclusion more immediate, broader, and definitive.275. See Srinivas Kodali, Digital India on Steroids: How Aadhaar Infra Enables the NPR and the NRC, Wire (Dec. 24, 2019), https://thewire.in/tech/aadhaar-infra-npr-nrc [https://perma.cc/R6FL-86CB]. The SDS legal category elucidates how the blurred line between colloquial (and often socially charged) phrases and legal categories can create the conditions for the discriminatory and discretionary implementation of SDS. In election manifestos and public speeches, the ruling party in India (the BJP) repeatedly used the term “persecuted minorities” for the groups that it plans to grant citizenship under the NRC process.276Aditya Sharma, Six Clauses of Citizenship Bill Don’t Mention ‘Persecuted Minorities’ But BJP Does. Experts Wonder Why, News18 (Dec. 11 2015), https://www.news18.com/news/india/6-clauses-of-citizenship-bill-do-not-mention-persecuted-minorities-only-bjp-does-repeatedly-2419673.html [https://perma.cc/2JW6-MR47]. Yet the CAA law itself has no mention of definition of the word persecuted, leaving it to the discretion of street-level bureaucrats to determine whether persecution will be a prerequisite,277Banerjee, supra note 268, at 87. and how it will be defined. Similarly, the category of “doubtful citizens” in the NPR is left undefined and is up to the discretion of the “[l]ocal [r]egistrar who can decide that someone is doubtful’ even if they do possess the ‘particulars’ required.”278Mathur, supra note 262, at 4 (emphasis in original).
The SDS organizational category decenters the analysis from national-level policy frameworks and instead examines the organizational practices of the front-line officers who, through performing their respective administrative functions, give meaning to and concretize the logics and outcomes of SDS. With the NRC, it will be crucial to study how a historical culture of suspicion regarding claims of citizens (and a particular mistrust of paper documents) influences how bureaucrats implement these systems and the discriminatory outcomes it produces.279. See id. Indeed, the third SDS organizational feature speaks to how organizational priorities drive biased implementation and the Assam NRC experience suggests that similar problems will manifest in the national NRC. Accounts explain how the extreme pressure on bureaucracy to meet the centrally dictated deadlines of the Assam NRC translated into even more arbitrary and ad-hoc practices.280. See id. Applying the SDS political economy analysis to the NRC shows that this system reflects a common SDS feature regarding the commodification of security and risk management via private-public partnerships because the database system is to be designed and implemented by large private technology vendors.281. See Priya Pathak, NRC Protests: Wipro Celebrates Company’s Role in NRC Project in Assam, First Deletes Page Then Restores It, India Today, (Dec. 23, 2019, 2:41 PM) https://www.indiatoday.in/technology/news/story/wipro-celebrates-its-role-in-nrc-project-in-assam-first-deletes-page-then-restores-it-1630865-2019-12-23 [https://perma.cc/GMW2-4FVG]. More generally, it may be useful to examine how the NRC interacts with broader economic trends towards nationalism and economic self-reliance.282. See Fiona B. Adamson & Gerasimos Tsourapas, The Migration State in the Global South: Nationalizing, Developmental, and Neoliberal Models of Migration Management, 54 Int’l Migration Rev. 853, 864–66 (2020) (offering political economy analysis of migration management in the Global South).
Finally, the SDS social features reveal how sensationalized and often misleading rhetoric is used to engender political support for the NRC, and in particular the framing of the NRC as a system that would not take away citizenship but instead present the “privilege of Indian citizenship” to many.283Banerjee, supra note 268, at 5. Political leadership highlighted the granting of citizenship to Hindus and other religious groups while obscuring the fact that Muslims, the country’s largest minority group, were categorically left out of this inclusive paradigm.284. See id. at 85–87. This, too, needs to be situated in the historical context of the “partition” of British-ruled India in 1947 that resulted in the creation of a separate state of Pakistan that was meant to be a “homeland for Muslims.”285Ayesha Jalal, The Struggle for Pakistan: A Muslim Homeland and Global Politics, 6 (2014). The partition was accompanied by gruesome communal violence, and the Muslims who remained in newly independent India have faced systematic discrimination, prejudice, and violence since, despite constitutional protections.286. See generally Nisid Hajari, Midnight’s Furies: The Deadly Legacy of India’s Partition (2015); Saeed Naqvi, Being the Other: The Muslim in India (2016).
2. Homeland Advance Recognition Technology System (HART)
The Homeland Advanced Recognition Technology System (HART) is a multimodal identity management system for U.S. citizens and foreign nationals that is developed and managed by the Office of Biometric Identity Management (OBIM) in the U.S. Department of Homeland Security (DHS) along with “mission partners” including other internal DHS components, other federal government agencies, and international partners.287 . U.S. Dep’t. Homeland Sec., dhs/obim/pia-004, Privacy Impact Assessment for the Homeland Advanced Recognition Technology System (HART) Increment 1 PIA 1–2 (2020), https://www.dhs.gov/sites/default/files/publications/privacy-pia-obim004-hartincrement1-february2020_0.pdf. [perma.cc/W286-JUPQ]. The system is being developed to replace and modernize DHS’ decades-old legacy biometric database, IDENT, which is at risk of failure and requires significant investments to remain operational.288 . U.S. Gov’t Accountability Off., GAO-21-175, DHS Annual Assessment: Most Acquisition Programs Are Meeting Goals but Data Provided to Congress Lacks Context Needed for Effective Oversight 37 (2021). This modernization project will allow DHS and partner institutions to integrate new biometric technology capabilities, expand the types of information collected, and better identify relationships between individuals and groups.
HART is expected to store and process biometric data (fingerprints, face, iris, voice, etc.), contextualizing data (demographic data or camera quality), biographic data (names, signature, physical identifying details), derogatory information (warrants, suspected terrorist designation, immigration violations), encounter history (metadata associated with the collection of biometric or biographic data), and other associated data from disparate sources, which then links each entry with OBIM unique enumerators.289 . U.S. Dep’t. of Homeland Sec., supra note 287, at 16–17. Mission partners may be authorized users, data providers, or both, and authorized users can either query the system and upload data or function as “search only” users.290. Id. at 4–6. While HART will purportedly offer many services and modalities that aid law enforcement and investigatory activities by a range of national and international institutions, existing documentation suggests that users will most likely use the system to identify individuals and verify the identity of known individuals.
HART is currently being developed in four incremental phases291Aaron Boyd, Homeland Security’s Biometrics Database is on Its Way to the Amazon Cloud, Nextgov (May 6, 2020), https://www.nextgov.com/it-modernization/2020/05/homeland-securitys-biometrics-database-its-way-amazon-cloud/165186/ [https://perma.cc/MSB3-FJCX]. to minimize impact on its institutional users. The first phase (Increment 1), which is currently underway, focuses on technical and infrastructural upgrades (e.g. migration of the database to cloud services and the integration of biometric matching capabilities292Specifically matching fingerprints, facial photos, and irises with government issued unique identifiers, such as Social Security Numbers and Alien Numbers. U.S. Dep’t Homeland Sec., Privacy Threshold Analysis (PTA) Version No. 01-2014 passim, https://epic.org/foia/dhs/hart/EPIC-2018-06-18-DHS-FOIA-20190422-Production.pdf [https://perma.cc/7AMF-TWTM].) that are necessary to operate HART.293 . U.S. Dep’t Homeland Sec., supra note 287; see also U.S. Gov’t Accountability Off., supra note 288. The subsequent phases will focus on increasing interoperability, providing new interface capabilities, and improving analysis and reporting tools.294 . U.S. Dep’t Homeland Sec., supra note 287, at 3; U.S. Gov’t Accountability Off., supra note 288, at 35–36. It is projected that by 2022, HART will include information on at least 259 million unique identities, making the system the world’s second-largest biometric database. However, whether the system will meet these projections and include anticipated capabilities is questionable, especially since a 2021 U.S. Government Accountability Office report revealed that this program is in breach of its cost and schedule goals, and failed to begin its transition from IDENT as originally planned.295 . U.S. Gov’t Accountability Off., supra note 288, at 37. (“The HART program anticipates an affordability gap of approximately 2 million between fiscal years 2022 and 2025, which officials primarily attributed to changes in the program’s schedule.”).
The primary functions and expected uses of HART make it a more apparent SDS because it is used to profile individuals and groups for immigration enforcement actions that are punitive and exclusionary in effect, but analysis of the SDS framework can still help broaden understandings of its potential risks and impact. While HART’s development is incremental, the early phases emphasize the technical affordances illuminated by the SDS technical features. The first two phases of HART development focus on increasing interoperability, matching capabilities, expanding capacity for additional modalities, and other improvements that add functionality and facilitate greater data sharing.296 . U.S. Dep’t. of Homeland Sec., supra note 287, at 3.
Analysis of the SDS legal features indicates that vague criteria or faults in data governance practices make systems susceptible to inaccuracies thus increasing the risk of biased and unlawful outcomes. There are several bias and accuracy concerns regarding HART,297. See Jennifer Lynch, HART: Homeland Security’s Massive New Database Will Include Face Recognition, DNA, and Peoples’ “Non-Obvious Relationships,” Electronic Frontier Found. (June 7, 2018), https://www.eff.org/deeplinks/2018/06/hart-homeland-securitys-massive-new-database-will-include-face-recognition-dna-and [https://perma.cc/Q2B4-AHTL]. but the lack of rigorous standards regarding data entered and stored in HART is notable under the SDS legal features. HART is exempt from accuracy and other data quality requirements mandated by the Privacy Act of 1974 (Privacy Act), and OBIM only recommends data providers follow certain standards rather than systematically verifying the accuracy and quality of data comprising HART.298 . U.S. Dep’t. of Homeland Sec., supra note 287, at 22. Absent and low standards for data entry and storage can increase the likelihood of misidentification in HART query results, which the World Privacy Forum notes “is remarkable for a system . . . that will have a high impact on individuals’ civil liberties.”299World Priv. F., Comments of World Privacy Forum to Department of Homeland Security regarding Proposal to Establish a New DHS System of Records, Department of Homeland Security/ALL-041 External Biometric (EBR) System of Records and Proposal to Exempt New DHS External Biometric (EBR) from Key Provisions of the Privacy Act of 1974, 3 (May 25, 2018), https://www.regulations.gov/document?D=DHS-2017-0040-0005 [https://perma.cc/E7LU-PS28]. SDS legal feature analysis also brings to light that there are few legal remedies available due to a lack of robust legal oversight or other safeguards. This is because HART’s legal governance structure is fragmented. Operationally, OBIM considers itself HART’s “system owner and data steward,” which means that the partner institutions own and are legally responsible for the data they enter into the system, and OBIM only exerts ownership over the unique enumerators300 . U.S. Dep’t. of Homeland Sec., supra note 287, at 2, 5. “OBIM owns unique numbers or sequence of numbers and characters, also known as enumerators, that HART generates to link individuals with encounters, biometrics, records, and other data elements.” Id. at 5. used to link data within HART.301. Id. at 2. As a result, DHS seeks to exempt the database that populates HART from elements of the Privacy Act, individual Privacy Impact Assessments, Systems of Records Notice, and information sharing access agreements that govern the data collection, processing, and storage rights of HART, DHS components, and partner agencies.302Privacy Act of 1974: Implementation of Exemptions; Department of Homeland Security/ALL-043 Enterprise Biometric Administrative Records System of Records, 85 Fed. Reg. 14805 (Mar. 16, 2020) (to be codified at 6 C.F.R. pt. 5), https://www.govinfo.gov/content/pkg/FR-2020-03-16/pdf/2020-04985.pdf [https://perma.cc/4DQ8-XSZX].
HART also reflects a common SDS political economy feature regarding the commodification of security and risk management via private-public partnerships. HART is being developed and maintained by private companies through significant government grants. The aerospace and defense company Northrop Grumman was awarded a $95 million contract to develop and integrate the first two phases of HART, and the system is hosted on Amazon Web Services GovCloud, though the costs of such services are undisclosed and not standardized.303Press Release, Northrop Grumman, Northrop Grumman Wins Million Award from Department of Homeland Security to Develop Next-Generation Biometric Identification Service System (Feb. 26, 2018), https://news.northropgrumman.com/news/releases/northrop-grumman-wins-95-million-award-from-department-of-homeland-security-to-develop-next-generation-biometric-identification-services-system [https://perma.cc/AB5T-2JEP]; see also AWS GovCloud (US) FAQs, Cost & Pricing, Amazon Web Servs., https://aws.amazon.com/govcloud-us/faqs/ [https://perma.cc/WRL3-273C] (last visited June 25, 2022).
The SDS organizational features clarify that, despite formal rules and policies, a system’s use can be minimally supervised and highly discretionary. While HART has several formal rules regarding use and operation of the system, use by its various and fairly autonomous users lacks oversight, which enables discretion and intensifies inaccuracy issues. As previously mentioned, each mission partner is responsible for the documentation of their collection and processing of data in HART via their own SORNs and PIAs and this also means that each entity is responsible for notification requirements to individuals in the system.304 . U.S. Dep’t. of Homeland Sec., supra note 287, at 27. As a result, OBIM has no obligation to oversee or ensure compliance. When inaccurate or flawed data is entered and stored in HART, timely correction or updating of information is dependent on the prerogative of mission partners.305. See id. at 24. This lack of oversight combined with the exemption of HART components from elements of the Privacy Act means that individuals in the system can face significant legal consequences (e.g., false arrest or deportation) with few means of redress.306Nat’l Immigr. L. Ctr. & Nat’l Immigr. Project of the Nat’l Laws. Guild, Comment on Notice of a New System of Records: Department of Homeland Security/ALL-041 External Biometric Records (EBR) System of Records 3 (May 23, 2018), https://www.regulations.gov/document?D=DHS-2017-0040-0003 [https://perma.cc/7FC9-EHHN].
Finally, the SDS social features can aid understanding of the impetus for developing HART. Like other SDSs, motivations to develop HART followed several converging social dynamics and changes. HART’s development was brought about by the intersection of growing social and political hostility regarding immigration enforcement and reform,307. See Julia Preston, In Debate Over ‘Sanctuary Cities’ a Divide on the Role of Local Police, N.Y. Times (Sept. 1, 2016) https://www.nytimes.com/2016/09/02/us/in-debate-over-sanctuary-cities-a-divide-on-the-role-of-the-local-police.html [https://perma.cc/CGV2-6EHN]; see also Nicholas Fanfos, Paul Ryan Says He Won’t Work With Obama on Immigration Reform, N.Y. Times (Nov. 1, 2015), https://www.nytimes.com/2015/11/02/us/politics/paul-ryan-says-he-wont-work-with-obama-on-immigration-reform.html [https://perma.cc/A44T-5FGR]. significant advancements and declining costs in biometric technology development (driven by exponential economic growth in the technology sector), and the government’s growing interest in technology-enabled surveillance practices. In 2015, OBIM began planning the creation of a more robust and efficient biometric database that could take advantage of and integrate new technological advancements to support DHS core missions.308U.S. Dep’t. of Homeland Sec., supra note 287, at 1–2. Since biometric technology development requires enormous start-up costs, the sweeping advancements that OBIM aimed to integrate into this newly conceived system were due to billions of dollars of investments via government subsidies, grants, and private-public partnerships.309Shoshana Amielle Magnet, When Biometrics Fail: Gender, Race, and the Technology of Identity, 58, 62 (2011). Indeed, early-stage investments funded the creation of IDENT and other fledgling government biometric systems.310. Id. Over time, the costs of development decreased, the variety of biometric applications expanded, and their performance in controlled settings improved.311. See, e.g., Tod Newcombe, Facing the Privacy Costs of Biometric Identification, Govtech
(July/Aug. 2019), https://www.govtech.com/products/Facing-the-Privacy-Costs-of-Biometric-Identification.html [https://perma.cc/L44R-JMCT] (detailing the declining costs and improvements in biometric technology development); Jayshree Pandya, Hacking Our Identity: The Emerging Threats From Biometric Technology, Forbes (Mar. 9, 2019), https://www.forbes.com/sites/cognitiveworld/2019/03/09/hacking-our-identity-the-emerging-threats-from-biometric-technology/?sh=1d758b4f5682 [https://perma.cc/3PVN-UNJY] (highlighting the varied uses of biometric technologies); The Current and Future Application of Biometric Technologies: Joint Hearing Before the Subcomm. on Rsch. & the Subcomm. on Tech. of the H. Comm. on Sci., Space, and Tech., 113th Cong. (2013), https://www.govinfo.gov/content/pkg/CHRG-113hhrg81193/html/CHRG-113hhrg81193.htm [https://perma.cc/LW93-FT2N] (describing advancements in biometric technology development and future applications of the technologies). At the same time, the U.S. government’s interest in automation, which can be significantly enhanced by biometric technologies,312Shoshana Amielle Magnet, When Biometrics Fail: Gender, Race, and the Technology of Identity, 58, 65 (2011). spurred interest to reinvest in new interagency databases, like HART, and integrate use of these technologies into immigration and migration enforcement practices and policies, like “biometric exit” programs.313. See Harrison Rudolph, Laura M. Moy & Alvaro M. Bedoya, Not Ready for Takeoff: Face
Scans at Airport Departure Gates, Geo. L. Ctr. on Priv. and Tech. (Dec. 21, 2017), https://www.airportfacescans.com/ [https://perma.cc/5CJJ-MPQW].
Digitization and databases are often thought of as the raw material for algorithmically-determined futures, even as they are already doing this kind of prognostic work in database form. SDS, both as a categorical term and analytical framework, resolves this misconception by elucidating the varied contexts, features, policies, and practices that must be evaluated to understand how these seemingly amorphous technologies function in practice, the ways they are used, and the outcomes they produce. While our framework is not exhaustive, throughout this Article we have demonstrated the analytical utility of the framework as a whole and its distinct constitutive parts in bringing forth necessary and novel analysis. For instance, the perspicacity enabled through this definition and framework allows one to recognize SDS design, use, and impact as a new and less obvious form of predatory inclusion practices in the technology sector, where marginalized groups are included in ostensibly democratic schemes on extractive terms. While there is emergent scholarship examining these practices in the gig economy platforms and credit scoring techniques,314. See, e.g., Cottom, supra note 62, at 441–49 (2020); Mikella Hurley & Julius Adebayo, Credit Scoring in the Era of Big Data, 18 Yale J.L. & Tech. 148 (2017). our SDS framework demonstrates that an important characteristic of the power of SDSs is that through inclusion or exclusion in the databases one is practically excluded from full democratic participation, as a direct result or collateral consequence. In the database context, exploitative processes surrounding inclusion or exclusion are obscured and legitimized because of their primary use in state administrative functions or law enforcement.
In the criminal justice context, inclusion in any of the criminal intelligence databases reviewed individuals to increased police contacts, harassment, detrimental inferences, and other serious punitive outcomes, particularly when information in these databases is used to inform government decision-making like sentencing decisions or public housing eligibility determinations. These consequences are made worse by the fact that the databases or information within is shared with other government and non-governmental actors, the lack of consistent notification practices means individuals are often unaware of the cause of their differential treatment, and existing means for redress are hollow or futile.315. See, e.g., Stacey Leyton, The New Blacklists: The Threat to Civil Liberties Posed by Gang Databases, in Crime Control and Social Justice: The Delicate Balance 119–20 (Darnell F. Hawkins et al. ed., 2003); Satish, supra note 36; Patrick Williams, StopWatch, Being Matrixed: The (Over)Policing of Gang Suspects in London, 26-31 (2018), https://www.stop-watch.org/what-we-do/research/being-matrixed-the-overpolicing-of-gang-suspects-in-london/ (view report by clicking “Downloads” at the bottom of the page) [https://perma.cc/C2EY-JG9E]. In the welfare context, inclusion or exclusion from these databases can have deleterious effects. Inclusion in the United Kingdom’s Universal Credit system subjected individuals to onerous paperwork and documentation requirements, and flaws in the system resulted in extreme cuts to and fluctuations in monthly benefit payments which forced some people into debt.316 . Amos Toh, Human Rts. Watch, Automated Hardship: How the Tech-Driven Overhaul of the UK’s Social security System Worsens Poverty (2020), https://www.hrw.org/report/2020/09/29/automated-hardship/how-tech-driven-overhaul-uks-social-security-system-worsens# [https://perma.cc/Z2NN-VY36]. Exclusion from universal databases that are used to administer welfare programs like India’s Aadhaar and Kenya’s Huduma Namba, can lead to further marginalization by making participation in the database a precondition for carrying out business or the exercise of democratic rights, such as the right to vote.317. E.g., Warah, supra note 221.
In addition to clarifying the full spectrum of uses, risks, and consequences of SDS, our definition and framework can inform evaluations of relevant policy proposals and interventions. A common axiom in technology policy is that technological innovation outpaces the law, but this is only true if one takes a myopic view of which laws and regulations are relevant to a given technology. This axiom ignores the fact that SDS are multifarious assemblages and, as the analysis in Section I demonstrates, how the legal, political economy, organizational, and social features of these systems are equally constitutive as technical features. Therefore, one must examine the policies, practices, and conditions that precede SDS development to understand their design, use, and impact. This analysis can also aid assessments of technological solutions for complex social policy concerns, specific technology policy proposals, and can help identify alternative opportunities or locations for interventions that would produce similar outcomes as new legal or regulatory frameworks.
First, our analysis demonstrates that SDSs are often not an appropriate solution to complex social issues. While certain efficiencies can be gained through digitized databases and data-driven analysis, these purported benefits must be evaluated and balanced with the risks and costs to society, along with whether their use or outcomes undermine governance goals or missions. In the criminal justice context, gang and other criminal intelligence databases sow community distrust, particularly in marginalized communities that bear an outsized burden of databasing practices.318. See U. S. Dep’t. of Just. Civ. Rts. Div. & U.S. Att’y’s Off. N. Dist. of Ill., Investigation of the Chicago Police Department 15–16 (Jan. 13, 2017), https://www.justice.gov/opa/file/925846/download [https://perma.cc/L63S-N2ND] (highlighting that CPD gang database practices “significantly jeopardize CPD’s relations with the community”). This distrust means individuals are less likely to cooperate with police investigations, report crime, and some consider leaving their communities to avoid police encounters.319. See, e.g., Williams, supra note 316, at 37–42; Trujillo & Vitale, supra note 104, at 13–30. All of these outcomes ultimately make communities less safe, which is antithetical to the law enforcement mission of public safety.320. See Amy E. Lerman & Vesla M. Weaver, Staying out of Sight? Concentrated Policing and Local Political Action, 651 Annals Am. Acad. Pol.& Soc. Sci. 202, 202–19 (2014); Blaine Bridenball & Paul Jesilow, What Matters: The Formation of Attitudes Toward the Police, 11 Police Q. 151 passim (2008). Outside of the criminal justice context, universal ID systems are promoted under the developmental goal of inclusion. In practice, however, their operation facilitates the exclusions of historically marginalized groups and creates suspect categories for citizens in different sectors (i.e., social welfare and housing).
While we focus our case studies on database systems, the SDS framework can also bring useful insights to technology policy debates around a broader range of data-driven tools, like the use of pretrial risk assessments as a prevailing solution to cash bail. Pretrial risk assessments attempt to statistically measure a defendant’s dangerousness, flight risk, or risk of recidivism to determine whether they should be detained before their trial. These tools can be categorized as an SDS because they are information technologies that exist in various technical forms but are primarily used to measure vague social risks to justify heightened scrutiny and punitive outcomes like pre-trial detention or electronic monitoring.321. See Sandra G. Mayson, Dangerous Defendants, 127 Yale L. J. 490 (2017); Sean Hill, Bail Reform & The (False) Racial Promise of Algorithmic Risk Assessment, 68 UCLA L. Rev. 910 (2021). These actuarial tools have been used in criminal justice institutions for decades,322Matt Henry, Risk Assessment: Explained, Appeal (Dec. 14, 2019), https://theappeal.org/the-lab/explainers/risk-assessment-explained/ [https://perma.cc/822K-VZ4A]. but as they have become digitized or automated they are increasingly proposed as the alternative to cash bail.323. See John Logan Koepke & David G. Robinson, Danger Ahead: Risk Assessment and the Future of Bail Reform, 93 Wash. L. Rev. 1725 (2018). For example, a 2020 California ballot initiative, Proposition 25, sought to uphold contested legislation (Senate Bill 10) that would replace the state’s cash bail system with a pretrial risk assessment.324 . Sec’y of State of Ca., California General Election November 3, 2020 Official Voter Information Guide, Prop 25 Referendum on Law that Replaced Money Bail with System Based on Public Safety and Flight Risk, https://voterguide.sos.ca.gov/propositions/25/ [https://perma.cc/Q6MK-RKE7]. When attempting to balance the purported benefits of pre-trial risk assessments with the risks and costs to justice-involved communities and governance integrity, analysis of our SDS framework can offer incisive observations. One example is the political economy category, which can illuminate the entanglement of government and private interests in SDS design, use, and outcomes. In California, only one pretrial risk assessment tool (developed by the Arnold Ventures philanthropy, an arm of the Laura & John Arnold Foundation) met the validation requirements mandated in the controversial Senate Bill 10.325Letter from Cory T. Jasperson, Dir., Governmental Affairs, Jud. Council of Cal., to Hon. Reginald B. Jones-Sawyer, Sr., Cal. State Assembly (June 30, 2017), at 5 https://www.courts.ca.gov/documents/ga-position-letter-assembly-sb10-hertzberg.pdf [https://perma.cc/8X7X-C8JH].
Second, the SDS definition and framework can aid evaluations or implementation of prevailing policy proposals. The definition and framework expand public conceptions of what technologies warrant scrutiny and should be included in existing AI and ADS accountability or oversight mechanisms. Our analysis can also broaden the scope and substance of current or emergent government assessment practices of technologies, such as cost-benefit analysis or algorithmic impact assessments (AIA). Analysis of the SDS framework can illuminate assumptions and oversights typically made in these evaluations. For instance, SDS organizational features can bring attention to inefficiencies in government operations or practices that undermine the objectives of using a given technology or will reduce its overall efficacy. Similarly, SDS political economy features can expand conventional audits by revealing how the startup costs of some technologies are balanced by exploitative or extractive labor practices, which distort the overall costs to society.326. See, e.g., Ann Cavoukian, Biometrics and Policing are Not Strangers to Each Other, in Shoshana Amielle Magnet, When Biometrics Fail: Gender, Race, and the Technology of Identity 65–66 (2011) (arguing that cost-efficiency claims of biometric technologies are only possible because labor practices of privatization “paves the way for reductions in pay”); The White House, Modernizing Regulatory Review, Presidential Memo (Jan. 20, 2021), https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/20/modernizing-regulatory-review/ [https://perma.cc/UG43-KJZV] (calling for reforms to the federal regulatory review process, which includes cost benefit analysis, to ensure new regulations affirmatively promote social welfare, racial justice, human dignity, equity, and other values). Moreover, as demonstrated in Section II(C), the SDS framework can be utilized for proactive examinations of concerning technologies before they are fully developed or implemented. This temporal expansion is important since AIA evaluations tend to occur after the technology is developed, procured, or in use.
Our analysis can also inform policy agendas seeking to mitigate and redress the harms of SDS. Given the opacity and secrecy regarding SDS use, some advocates call for transparency as an intervention.327. See, e.g., Camille Ochoa & Dave Maass, Demand California Fix CalGang, A Deeply Flawed Gang Database, Elec. Frontier Found. (Aug. 16, 2016), https://www.eff.org/deeplinks/2016/08/demand-california-fix-calgang-its-deeply-flawed-gang-database [https://perma.cc/BA44-U9BT]; Coalition Letter Calls on the NYPD Inspector General to Audit the NYPD “Gang Databases,” Brennan Ctr. for Just. (Sept. 22, 2020), https://www.brennancenter.org/our-work/research-reports/coalition-letter-calls-nypd-inspector-general-audit-nypd-gang-database [https://perma.cc/FCW7-U9PK]. Transparency can be a helpful diagnostic tool for understanding the ecosystem of technologies used for governance, but knowledge regarding the existence of these systems does not forestall their harmful effects or empower communities seeking redress.328Karen Hao, Live Facial Recognition is Tracking Kids Suspected of Being Criminals, MIT Tech. Rev. (Oct. 9. 2020), https://www.technologyreview.com/2020/10/09/1009992/live-facial-recognition-is-tracking-kids-suspected-of-crime/ [https://perma.cc/6GRP-KEYD] (detailing the use of a publicly accessible SDS used in Argentina that still produced harmful outcomes and lacked oversight); Jacobs, supra note 25; see also N.Y. Police Dep’t, Public Oversight of Surveillance Technology (POST) Act Draft Impact and Use Policies for Public Comment (2021), https://www1.nyc.gov/site/nypd/about/about-nypd/public-comment.page [https://perma.cc/8P4Y-AZT3] (NYPD draft impact and use policies for surveillance technologies for public comment that include boilerplate language regarding technology use and potential community impact and undermine the purpose of the transparency disclosures). Several categories and features of the SDS framework reveal that SDS are brought about by specific or gradual social policy changes; our analysis can redirect advocacy towards structural reforms of core social policies as complementary or alternative solutions.329 . Marshall Buxton, Abigail Fradkin & Antwuan Wallace, Nat’l Innovation Serv. & N.Y.C. Mayor’s Off. for Crim. Just., Measuring Community Safety in NYC (Oct. 2021), https://static1.squarespace.com/static/615225be3ba6a33dade929c2/t/6179b43d1d6e9b33b804e5e9/1635365951103/10-27-2021_MOCJ+Measuring+Safety+and+Thriving+Report.pdf https://perma.cc/MPQ8-KREG] (highlighting a new approach and metrics for measuring and evaluating community safety in Black and Brown communities in New York City).
We encourage technology developers, policymakers, legal practitioners, government officials, scholars, advocates, and community members to use our SDS definition and analytical framework to inform and actively change the trajectory of SDS design, use, and outcomes. As demonstrated in Section II, our analysis can be used to examine important features and contexts that are pertinent to identifying and evaluating the legal and social implications of such technologies in addition to the development of meaningful legislative and regulatory interventions. Thus, the SDS framework can and should evolve as it is used to evaluate emerging or under-examined technologies in sensitive social domains, such as education and child welfare, where the use of databases and information technologies are both evolving and expanding.330. E.g., Rashida Richardson & Marci Lerner Miller, The Higher Education Industry is Embracing Predatory and Discriminatory Student Data Practices, Slate (Jan. 13, 2021), https://slate.com/technology/2021/01/higher-education-algorithms-student-data-discrimination.html [https://perma.cc/9DLM-8RUX]; J. Khadijah Abdurahman, Comment, Calculating the Souls of Black Folk: Predictive Analytics in the New York City Administration of Child Services, 11 Colum. J. of Race & L. 75, 102–08 (2021); Anjana Samant, Aaron Horowitz, Sophie Beiers & Kath Xu, Family Surveillance https://www.aclu.org/news/womens-rights/family-surveillance-by Algorithm: The Rapidly Spreading Tools Few Have Heard -algorithm-the-rapidly-spreading-tools-few-have-heard-of, ACLU (Sept. 29, 2021), https://www.aclu.org/news/womens-rights/family-surveillance-by-algorithm-the-rapidly-spreading-tools-few-have-heard-of [https://perma.cc/K7DU-KHH5].
While SDS use is global and our cross-jurisdictional analysis demonstrates common challenges, there is no universal panacea. Still, the SDS framework can help advance policy discourse and reforms internationally and locally. Since there is a growing global discourse regarding banning or significantly limiting the use of some AI technologies that are demonstrably harmful (e.g., autonomous weapon systems or facial recognition technologies),331. See Kashmir Hill, Another Arrest, and Jail Time, Due to a Bad Facial Recognition Match, N.Y. Times (Dec. 29, 2020), https://www.nytimes.com/2020/12/29/technology/facial-recognition-misidentify-jail.html [https://perma.cc/54HS-KP6T]; Davide Castelvecchi, Is Facial Recognition Too Biased to be Let Loose?, Nature (Nov. 18, 2020), https://www.nature.com/articles/d41586-020-03186-4 [https://perma.cc/K977-G255]; Autonomous Weapons: An Open Letter from AI [Artificial Intelligence] & Robotics Researchers, Future of Life Inst. (July 28, 2015), http://futureoflife.org/open-letter-autonomous-weapons/ [https://perma.cc/VWC2-24Z7]; Hum. Rts. Watch, Heed the Call: A Moral and Legal Imperative to Ban Killer Robot (Aug. 21, 2018), https://www.hrw.org/report/2018/08/21/heed-call/moral-and-legal-imperative-ban-killer-robots# [https://perma.cc/27S2-VXGS]. the SDS framework can aid these conversations in creating international norms that can inform domestic policies. However, for solutions to be effective and address the myriad of issues reviewed in this Article, they must be designed and implemented locally while centering the needs and interests of the individuals and communities harmed. Our SDS framework analysis makes clear that SDS development, use, and outcomes are context-driven; therefore, the development of interventions and solutions will require critical analysis of such contexts to ensure the goals of policy reforms are not merely focused on efficiency or short-term wins.