Suspect Development Systems: Databasing Marginality and Enforcing Discipline

Rashida Richardson & Amba Kak* June, 2022

Abstract

Algorithmic accountability law—focused on the regulation of data-driven systems like artificial intelligence (AI) or automated decision-making (ADM) tools—is the subject of lively policy debates, heated advocacy, and mainstream media attention. Concerns have moved beyond data protection and individual due process to encompass a broader range of group-level harms such as discrimination and modes of democratic participation. While a welcome and long overdue shift, the current discourse ignores systems like databases, which are viewed as technically “rudimentary” and often siloed from regulatory scrutiny and public attention. Additionally, burgeoning regulatory proposals like algorithmic impact assessments are not structured to surface important –yet often overlooked –social, organizational, and political economy contexts that are critical to evaluating the practical functions and outcomes of technological systems.

This Article presents a new categorical lens and analytical framework that aims to address and overcome these limitations. “Suspect Development Systems” (SDS) refers to: (1) information technologies used by government and private actors, (2) to manage vague or often immeasurable social risk based on presumed or real social conditions (e.g. violence, corruption, substance abuse), (3) that subject targeted individuals or groups to greater suspicion, differential treatment, and more punitive and exclusionary outcomes. This framework includes some of the most recent and egregious examples of data-driven tools (such as predictive policing or risk assessments), but critically, it is also inclusive of a broader range of database systems that are currently at the margins of technology policy discourse. By examining the use of various criminal intelligence databases in India, the United Kingdom, and the United States, we developed a framework of five categories of features (technical, legal, political economy, organizational, and social) that together and separately influence how these technologies function in practice, the ways they are used, and the outcomes they produce. We then apply this analytical framework to welfare system databases, universal or ID number databases, and citizenship databases to demonstrate the value of this framework in both identifying and evaluating emergent or under-examined technologies in other sensitive social domains.

Suspect Development Systems is an intervention in legal scholarship and practice, as it provides a much-needed definitional and analytical framework for understanding an ever-evolving ecosystem of technologies embedded and employed in modern governance. Our analysis also helps redirect attention toward important yet often under-examined contexts, conditions, and consequences that are pertinent to the development of meaningful legislative or regulatory interventions in the field of algorithmic accountability. The cross-jurisdictional evidence put forth across this Article illuminates the value of examining commonalities between the Global North and South to inform our understanding of how seemingly disparate technologies and contexts are in fact coaxial, which is the basis for building more global solidarity.

Introduction

In October 2020, a small group of protesters dressed in black and carrying black umbrellas to protect their privacy marched through downtown Phoenix, Arizona.¹ This was one of many protests across the globe against police brutality and racism that continued into the fall of 2020 following the murder of George Floyd in May of that year.² The demonstrators were followed by a group of police officers and subsequently arrested in the largest group arrest in Phoenix since June 2020 when the racial justice protests began.³ The demonstrators were charged with several offenses, including unlawful assembly,⁴ and were surreptitiously added to the state gang database as members of a gang called “ACAB” which law enforcement officials designated as an “extremist” group with “violent tendencies.”⁵ But local news reports revealed that the ACAB gang was fictional.⁶ Phoenix police and county prosecutors colluded to target and arrest the protestors, and invented a moniker based on the common protest chant “All Cops Are Bastards.”⁷ This incident is one of a growing number of instances where government officials take advantage of nebulous databases to profile, target, and punish unfavorably viewed or politically marginalized individuals or groups.

For some databases, it is the mere fact of inclusion that places individuals and groups in a “suspect” category that can produce harmful results. For others, it is the omission from the database, which can often be deliberate, that places excluded individuals into a “suspect” category where they face punitive and harmful consequences. For example, digital ID databases, like India’s Aadhaar and Uganda’s Ndaga Muntu, are used by multiple facets of those respective governments and are meant to cover all residents or citizens for all relevant government programs. Yet these claims of universal coverage are not reflected in the databases’ implementation. In practice, communities most reliant on state support and some historically marginalized groups bear a vastly disproportionate burden of database errors and tangible harms, such as exclusion from essential services and government benefits.⁸ Exclusion from these databases also facilitates stereotyping. This results because government narratives about the purpose of these systems suggest that those excluded are non-citizens, security risks, fraudsters, or members of other suspect categories, apparently justifying punitive outcomes of exclusion.⁹

These databases have not received the heightened public concern, institutional priority and funding,¹⁰ or policy reform-based research¹¹ directed at artificial intelligence (AI) or algorithmic/automated decision-making systems (ADS). For example, when the Chicago Police Department announced that it would no longer use its controversial predictive policing program in response to public criticism, it proudly announced plans to revamp its heavily criticized gang database to minimal outcry.¹² Meanwhile, arguably all of the most controversial technology projects across the Global South in the last decade have been large-scale database projects, which receive relatively little media attention and research funding.¹³ Instead, in popular and policy discourse, databases are characterized as foundational and passive raw material that enables the creation of more advanced algorithmic systems that can sort, prioritize, predict, and so on.¹⁴ Because AI and ADS are viewed as more complex technologies, databases are invariably positioned by many scholars as necessary for the creation and maintenance of, but “subordinate” to, these newer systems.¹⁵ This is compounded by a lack of definitional clarity in public discourse around what does or does not “count” as AI, given vague and evolving technical thresholds.¹⁶ This definitional ambiguity has policy implications as well, where the meanings ascribed to the terms AI or ADS can determine the scope of any regulatory effort.¹⁷

Recent policy discourse, now commonly referred to under the rubric of “algorithmic accountability,”¹⁸ has drawn considerable attention to how algorithmic systems serve to entrench or exacerbate systemic and historical discrimination against marginalized groups. While this emphasis on a deeper and contextual understanding of social harms is welcome, emergent policy frameworks for evaluating these systems, like Algorithmic Impact Assessments (AIAs) and algorithmic audits, are limited in their scope of review and analysis; these proposals still struggle to unearth the complete spectrum of structural technological inequities and tacit modes of discipline, control, and punishment.¹⁹

Responding to this combination of definitional ambiguity for policy intervention and the lack of a systematic way to evaluate harms, this Article proposes “Suspect Development Systems” (SDS) as both a definitional category and framework for analysis. SDS can be defined as (1) information technologies used by government and private actors, (2) to manage vague or often immeasurable social risks²⁰ based on presumed or real social conditions (e.g., violence, corruption, substance abuse), (3) that subject targeted individuals or groups to greater suspicion, differential treatment, and punitive and exclusionary outcomes.

We conceptualize SDS as a normative category that acknowledges how technologies like AI, ADS, and databases amplify structural inequities and the modes through which they discipline and control individuals and groups.²¹ Systems described and understood as “databases” are by no means the only kinds of SDS. Indeed, more recent ADS tools like predictive policing are clear examples of SDS and would benefit from the multi-pronged analysis set forth in this Article. Overall, SDS offers a broader framework within which to build an advocacy agenda that is inclusive of this diverse range of systems, rather than focusing only on the most recent or egregious examples of information technologies. However, in this Article, we focus predominantly on databases as they have received relatively less attention than AI or ADS and would especially benefit from strategic reframing. This Article is in conversation with criminology and surveillance studies scholarship that illuminates the reasons why databases might have proliferated as a key technique of penal governance within the criminal justice system and beyond.²² This work explores how the decontextualized and “byte-like”²³ mode of managing information within the seemingly objective structures of the computerized database has enabled the ground-level state apparatus to be more detached and unaccountable for the consequences for these decisions. This veneer of objectivity and routineness associated with databases contributes to the lack of public and regulatory scrutiny when they are introduced.

This visual illustrates that SDS can be applied to a broad range of technological systems, including systems that are identified in popular and policy discourse as Automated Decision Systems (ADS), databases, machine-learning based systems, or paper-based information systems. SDS provides a normative categorization that deliberately de-centers the technological form of the system, instead calling attention to its pernicious social impacts of creating suspect categories of people. While the wealth of examples in this Article demonstrates the ubiquity of SDS across social domains, the visual clarifies that not all algorithmic systems or databases will be a type of SDS. For example, driver’s license databases, school assignment algorithms, or price-discriminating machine-learning-based algorithms on e-commerce platforms do not fit our definition of SDS, even as they might attract other forms of critical inquiry.

Criminal intelligence databases, which have been used by police agencies for over a century to profile and target based on the risk of criminality, are paradigmatic examples of SDS.²⁴ “Criminal intelligence databases are populated with information about people who should be watched or monitored because they might have committed a crime or might commit a future crime.”²⁵ Notable examples of criminal intelligence databases include sex offender registries, prescription drug monitoring programs, London’s Crimint, and the FBI’s National Crime Information Center, which hosts the agency’s controversial Terrorist Watchlist and the Violent Person file. In this Article, we examine different variants of such databases in the U.S., the U.K., and India. Despite different socio-political contexts and institutional histories, these databases share a range of common features, conditions, and consequences.

Modern gang databases in the U.S. have existed since at least the late 1980s²⁶ and continue to evolve and proliferate as crime-fighting tools alongside newer modalities, such as predictive policing. Gang databases exist as both centralized and decentralized information systems primarily used by criminal justice actors and institutions to accumulate, analyze, and disseminate information about gangs and gang members for a variety of interests and priorities.²⁷ These databases are compiled and used by various criminal justice actors and institutions based on the belief that they function as a “force” or institutional multiplier increasing the overall efficiency, speed, and performance of all agencies without having to increase staffing or expend additional funds.²⁸ Gang databases exist to make intelligence and investigative information accessible to various government actors and institutions, though the needs and rationales for use vary.²⁹ Gang databases have been used to secure mass arrests and indictments, but since such actions have disproportionately included Black and Latinx youth,³⁰ these databasing efforts have also been the source of litigation³¹ and highly critical public reports.³²

In the U.K., while intelligence files on gangs and gang members have been maintained for decades, the early 2000s saw the computerization of gang databases in large cities. In this Article, we focus on the Gangs Violence Matrix (“Gangs Matrix”), a database of purported gang members launched in 2012 as part of the U.K. Government’s “war on gangs.” The Gangs Matrix is used as a risk management tool to assess and rank suspected gang members deemed most likely to commit a violent crime and to inform local police strategies to suppress violent crime.³³ According to recent accounts, at any given point, there are an estimated 3,000–4,000 individuals listed in the Gangs Matrix.³⁴ The purpose of this databasing effort is ostensibly to “audit” the gang landscape of cities, target individuals with heightened surveillance, as well as deterrence messaging or “nudges” to leave gang life.³⁵

In India, there are a range of police databases with records of individuals who are believed to be deserving of heightened surveillance on account of their perceived dangerousness and likelihood of committing future crimes.³⁶ These databases have their roots in colonial legislation that targeted specific communities that were designated as “criminal tribes.”³⁷ This policing approach towards marginalized communities continues, in part through the operation of surveillance databases.³⁸ The current material form of these databases³⁹ continues to be paper records stored in official files called “registers,”⁴⁰ but there are ongoing efforts to digitize these records.⁴¹ There are multiple kinds of surveillance databases maintained by each police station that cover suspect individuals residing in that particular precinct. While the official and colloquial names for such databases and the criteria for inclusion vary from state to state, common categories include the “history sheet,” “bad character” registers, and “rowdy” registers.⁴² Common amongst these databases is their inclusion of individuals who lack existing criminal records on the grounds that police believe they are likely to commit crimes or “disturb the peace.” Being included in these databases triggers a range of consequences like being subjected to heightened physical surveillance (e.g., regular home visits), increased chances of arrest, and unfavorable bail and sentencing decisions.⁴³

These databases can all be helpfully understood as SDS in the way that they manage vague and often immeasurable social risks, subjecting targeted individuals to greater suspicion, differential treatment, and punitive and exclusionary outcomes. While they present as bureaucratic systems of record-keeping and classification, we explore the myriad forms of disciplinary power they exert over marginalized and historically stigmatized groups. The bureaucratic patina of these systems can conceal the violent nature of this form of control, given the looming threat of scrutiny and brutality that often accompanies their use.⁴⁴ Analyses of databases, like SDS, should be differentiated from the literature on “automated suspicion algorithms”⁴⁵ or “big data blacklists.”⁴⁶ The latter provides legal frameworks to analyze the use of automated statistical tools that predict and identify individuals deserving of additional scrutiny and other restrictions on account of the social risk they present. Instead of focusing on technical tools, our framework of SDS, shaped by case studies on criminal intelligence databases, centers on the grounded social, political, and economic contexts, organizational practices and technical features that structure databases. In many of these systems, the decision of who is “suspicious” is not solely determined by statistical or other automated tools, but instead by government officials who are in turn influenced by organizational, legal, and social practices. In taking this approach, we hope to address not just emergent and advanced forms of government databases, but also provide a category that is broad enough to encompass the impact of so-called ‘legacy’ or manual systems that continue to operate alongside more recent “database” modes.

The SDS category and framework can help create a common discourse around systems that cut across spheres of governance and bridges geographical and temporal divides. Similar to the creation and maintenance of “suspect” categories of people like the habitual offender or the gang member, racialized tropes coming out of political discourse, like “the welfare-dependent mother” or “the illegal immigrant” are replicated in database use and in other sectors of governance and the private sector as well.⁴⁷ The SDS framework illuminates the connections between recent moves towards digitizing welfare databases in the U.K.;⁴⁸ the new Argentinian database that facilitates targeting of minors that are “alleged offenders”;⁴⁹ the extensively detailed police database maintained by Chinese police agencies that is crucial to the surveillance, persecution and large-scale internment of Muslims in the Xinjiang region;⁵⁰ the proliferation of nationwide mandatory biometric ID databases in developing countries;⁵¹ and the massive citizenship databases being introduced for immigration control in the European Union, India, and the U.S.⁵² This framework also allows us to draw connections between historical practices and modern database usage/developing “suspect” categories of people, such as linking current gang database practices in the U.K. to the British colonial strategy of criminalizing entire communities (designated “criminal tribes”)⁵³ and extrapolating lessons from the use of watchlists to suppress political dissent in the U.S. during the Civil Rights and anti-Vietnam war movements.⁵⁴ Each of these government data projects has defined who is included and who is excluded from the vision of the nation state and has facilitated the conditions for profiling, stigmatizing, or even eliminating those who are excluded. For criminal intelligence databases, inclusion into these databases trigger these consequences, whereas, in welfare or citizenship databases, exclusion or removal from databases is used to punish or otherwise disempower individuals and groups.

The Article is organized as follows. In Section I, through a detailed examination of the use of criminal intelligence databases in the U.S., U.K., and India, we present a framework of features that are crucial to understanding and evaluating SDS. In this Section, we explore five categories of features: technical, legal, political economy, organizational, and social. Each category distills the common features and insights observable in these SDSs, which remain underexplored in existing algorithmic accountability discourse. Section II demonstrates the value of SDS as a framework applied to contemporary government databases outside of the criminal justice system. We apply the framework analysis to other systems that could helpfully be understood as SDS: welfare system databases, universal or ID number databases, and citizenship databases. Finally, we conclude with a range of insights regarding how our analytical framework of SDS offers conceptual, legal, and strategic insights towards addressing the harms caused by these systems or doing away with them altogether where such harms cannot be remedied. SDS, as a defined term and analytical framework, provides an expanded lens with which to define and identify ADS systems by offering a systematic understanding of the socio-technical context, without which the harms of these systems cannot be addressed.

I. Suspect Development System Framework

SDSs are typically unique to the jurisdictions in which they are used because local conditions influence their development and use. However, as their use and consequences become more prevalent globally, such jurisdictional distinctions may be less salient and practical. Policymakers, advocates, and scholars are increasingly interested in identifying, evaluating, and addressing the implications of emergent technologies in systematic and concentrated ways, an approach that some say can only be achieved if there is a categorical framework for analysis.⁵⁵

In this Section we present a framework for SDS that includes five categories of features: technical, legal, political economy, organizational, and social.⁵⁶ We use the term “features” expansively to include typical characteristics, structural conditions, consequences of SDS development, and use. These categories were developed by examining criminal intelligence databases used by law enforcement and other government actors to profile and target people considered to be a social risk for criminality in the U.S., U.K., and India.⁵⁷ These databases have different colloquial terminologies (such as gang or habitual offender) and rhetorical accounts that convey who will be targeted and the social risks they pose to a wide variety of audiences, yet they share common or universal features and collectively provide insights that remain unexamined in legal scholarship⁵⁸ and policy discourse. Thus, we chose to examine countries in the Global North and South that represent different stages of SDS development to explore where such commonalities emerge and how they can expand our understanding of how these seemingly disparate technologies are coaxial.

Our framework is not exhaustive. Instead, we have identified three key features in each category to demonstrate the analytical utility of each category and the framework, while leaving other features open to future development and expansion. Additionally, no category can or should be understood in isolation. The interactions within and between these framework categories influence how SDSs function, the ways they are used, and the outcomes they produce. For example, as political theorist Langdon Winner has argued, “the adoption of a given technical system actually requires the creation and maintenance of a particular set of social conditions as the operating environment of that system.”⁵⁹ Similarly, structures and practices explored in these organizational categories can help explain the practical implications of SDS use, in addition to gaps or oversights in relevant legal rules and regulations reviewed in the legal category. Thus, our framework can enable robust and tactical assessments that promote a more nuanced and holistic understanding of the uses, risks, and consequences of SDS, particularly providing insight into how these technologies amplify cumulative disadvantage⁶⁰ and structural inequities. This framework can both inform civil society advocacy and research agendas, as well as be operationalized by state and private actors in deciding whether (and how) to develop and use such systems or governing relevant spheres of influence.

A. Technical Features

The technological features of SDSs shape and structure how these systems generate information and knowledge for the entities that use them. This includes tangible characteristics of these systems such as: types of physical or digital interfaces; modes of collecting, storing, and managing information; and statistical techniques that might be incorporated into these systems. The objective here is not to create technical thresholds for what does and does not count as an SDS. Indeed, this framework seeks to de-center technical descriptors that are often used to create harmful hierarchies of analysis between more and less “advanced” systems. Through a detailed examination of criminal intelligence databases across contexts, this category highlights the range and variability in how such systems are designed, the intensity of technical tasks they afford, and the complex relationship between paper-based records and digital databases. These features are typically communicated in technical jargon or mathematical terms that cause misinterpretations outside of the disciplinary contexts in which they were coined.⁶¹ This jargon can also serve to mystify or obfuscate these systems in ways that prevent critique or broader public engagement.⁶² For these reasons, we focus this category on relating particular technical features in terms of the functionality they afford their users and the consequences these design choices present.

First, distilled to their most basic function, SDSs include simple classification systems that can be manual or digital. Criminal intelligence databases are manual or digital classification systems that categorize individuals or groups into labels, such as habitual offenders or gang members. Along with names, these records can include a range of details about a person including photographs or other biometric information, contact information (e.g., residence, phone number), demographic information (e.g., age, gender, religion, race), identifying marks (e.g. tattoos), socioeconomic information (e.g. marital status), criminal history and membership of a particular group, and names of affiliates (either members of the same gang or informal associations with others on the list). Surveillance databases in India, for example, have a great deal of continuity with those used decades earlier during colonial rule that included the early use of visual cues like different colored ink to categorize an individual’s level of dangerousness.⁶³ Color-coding continues to be used to highlight different degrees of risk, like in California’s gang database, CalGang, and the United Kingdom’s Gangs Matrix.⁶⁴ Paper-based systems in India are intended to be digitized as part of a broader digital policing reform project,⁶⁵ but evidence from the U.S. suggests that the transition from manual to digital systems is not a binary shift, but instead a messy and negotiated process given that police officers often have greater familiarity and comfort with and fidelity to paper files. For example, accounts of CalGang’s upgrades describe how many officers were reluctant to replace paper files or input these records into the database,⁶⁶ and the Chicago Police Department’s (CPD) CLEAR system is still described as a patchwork of digitized paper files along with other more advanced computer application modules.⁶⁷

Second, SDSs can incorporate additional technical affordances like relational or link analysis, search functionality, and interoperability with other databases. Beyond the basic functions of recording and categorizing individuals, gang databases across the U.S. include additional features or third-party applications that may be integrated into the web browser-like interfaces.⁶⁸ One such additional functionality of such systems is the ability to use a search tool to query the database for records of a particular individual, members of a particular gang, or entries from a particular geographic area.⁶⁹ These databases also typically enable inter-and intra-government information sharing capabilities which allow police and other officials to query records across databases.⁷⁰ Another functionality observable across multiple gang databases is the mapping of relationality between individuals within the database. CalGang, whose design became the template for the commercially available GangNet used widely in the U.S. and Canada, was created to be this kind of relational database accessible through web portals.⁷¹ It allows for over two hundred data fields for individuals listed in the database, and “can generate ‘link diagrams’ of gang associates out to three levels, which include photographs and are color coded to show associates’ gang affiliations.’”⁷² The Chicago Police Department’s system also includes modules for “link analysis,” which claim to perform social network analysis by connecting “individuals with records of crimes for which they were arrested as well as individuals with whom they were arrested.”⁷³ The information available in link analysis can then be linked with “[ballistic] evidence or other data to link individuals to crimes.”⁷⁴

Finally, SDSs can generate statistical insights about individuals and groups. Despite being characterized as passive systems, databases include algorithmic tools that generate statistical insights about individuals and groups that are included in the database. While some gang databases in the U.S. generate summary reports with descriptive statistics on geographic maps of areas with individuals identified as “gang-involved,”⁷⁵ the U.K. Gangs Matrix goes further to offer risk assessments that guide and even prioritize policing resource allocations. In the Gangs Matrix, a “harm score” is assigned to each individual, which is essentially a color-coded score (red, amber, or green) based on the “level of violence [that they] have shown.”⁷⁶ The algorithm that produces the harm score relies on “police information about past arrests, convictions, and ‘intelligence related to violence/weapons access,’” social media, and other informal sources.⁷⁷ The Metropolitan Police, the territorial police force responsible for London’s thirty-two boroughs, have not disclosed the criteria and weights for the automated scoring algorithm they developed.⁷⁸ Some sources suggest that the score depends on how many crimes the person was involved during a three-year period, weighted according to the seriousness of the crime and its recency.⁷⁹ The harm scores are then ranked within each borough and the individuals with the top ten scores of each list are prioritized for enforcement.⁸⁰ While criminal intelligence databases are often viewed as being the foundation for more algorithmically advanced systems, the Gangs Matrix example, in particular, demonstrates the need to avoid any rigid semantic distinction between databases and algorithmic decision systems.

B. Legal Features

This category highlights common legal features and contexts that structure and potentially constrain SDSs design, use, and outcomes. These conditions are necessary for evaluating how databases are operationalized in practice, how they are viewed and understood by the public, and how they impact society.

The first SDS legal feature observable across legal frameworks applicable to criminal intelligence databases is the vague and subjective labels and criteria for determining who should be included in these databases, which both enables and conceals the biased notions of group criminality embedded in these systems. As we explore in this Section, this definitional flexibility is neither benign nor accidental. The operationalization of these legal regimes demonstrates how amorphous legal definitions have created a system ripe for the disproportionate targeting of historically marginalized social groups. Despite the veneer of individualized assessment, we explore how such labels and criteria both enable and maintain biased notions of group criminality.

Defining what constitutes a gang or gang members has been a significant challenge for law enforcement in the U.S., particularly because law enforcement definitions and practices tend to foreground the criminal activities of gangs, whereas researchers and social welfare practitioners “tend to emphasize the social and cultural aspects of gang formation and activity.”⁸¹ This schism can be partially attributed to law enforcement’s traditional approach to criminal profiling, which relies “on the correlation between behavioral factors and the past experience of law enforcement in discovering criminal behavior associated with those factors. Thus, profiling rests on the perceived accuracy of the profile as a predictor of criminality.”⁸²

Definitions are inconsistent across the country and are likely to reflect the political, social and financial pressures of any given jurisdiction.⁸³ A national study on gang-related laws found that only fifteen states have statutory definitions for gang members and five of these definitions are relatively generic.⁸⁴ For instance, the statutory definitions have different requirements for how many individuals must participate in criminal activity to qualify as a gang—most states require three or more individuals, some require at least five individuals, and some do not specify a required number of members.⁸⁵ Self-identification as part of a gang is the only common criteria, where no other corroboratory evidence is necessary. Research suggests this is a fraught category that is particularly vulnerable to abuse.⁸⁶ With police officers increasingly interpreting gang membership based on content posted by individuals on social media, there is a significant risk of misunderstanding cultural practices and signals, where “[w]hat may be meant as a joke or recognized as a lyric to a favorite rap song is instead interpreted by outsiders as inculpatory.”⁸⁷

Definitional ambiguity results in heightened discretion for those implementing DBS and is often justified by socially constructed notions of the groups being targeted. Unlike other forms of organized crime, such as mafias or mobs, which are hierarchical groups with strict codes of conduct that exist for the criminal enterprise, gangs are more amorphous and characterized by their fluidity in membership, geographic mobility, and differential organizational structures. Criminal activity is not necessarily the centralizing focus.⁸⁸ For example, then-New York City Police Department (NYPD) Commissioner, Dermont Shea, stated in his testimony about the Department’s gang database that gangs had evolved from traditional structure and noted: “their lack of a defined structure makes it difficult to predict their activities or document their association.”⁸⁹ This fluidity has, in turn, become a justification for ensuring legal definitions allow maximum discretion for police officers. In 2015, the U.K. Home Office also sought to simplify and expand the definition of a gang and gang member “to make it less prescriptive and more flexible” and the new definition stated that a gang should be defined merely “as having one or more characteristics that enable its members to be identified as a group by others.”⁹⁰ Thus, this definitional ambiguity has provided police with unfettered discretion to use scant evidence to not only label individuals as gang members but also to promote beliefs in non-existent gangs, such as the “ACAB gang” referenced in the introduction.⁹¹

Finally, the existence of individualized assessment criteria for who is included within a database can distract from or even disguise biased notions of group criminality embedded in these systems. The use of such criteria in India is demonstrative of this. The widely criticized colonial-era Criminal Tribes Act (that criminalized marginalized groups) was replaced with a Habitual Offenders Act in several states post-independence, a move that has been widely described as “window dressing changes of nomenclature, from criminal tribes to habitual offenders.”⁹² In states that have it, the Habitual Offenders Act determines the criteria for when a history sheet record is created against an individual, which subsequently leads to their inclusion into the surveillance database.⁹³

While these statutes do generally require at least three convictions to trigger a history sheet record, this obscures the fact that police still typically secure such convictions under colonial-era provisions often referred to as the “bad livelihood” sections, which apply to “suspicious persons” and “habitual offenders” without any requirement of a pre-existing criminal record.⁹⁴ Courts, too, have noted the use of these preventive provisions, specifically Sections 109 and 110 of the current Criminal Procedure Code, to arrest and harass individuals who often lack the financial resources to deposit the “good behavior” bond required under these provisions.⁹⁵ In states that do not have their own Habitual Offenders Act, the criteria for the creation of a history sheet record is stipulated via “standing orders”⁹⁶ or prison manuals that are even more vague and include direct references to the repealed colonial-era regime, such as classifying individuals as “habitual criminals” based solely on whether someone has a bad livelihood (undefined) or even whether they are a member of an erstwhile criminal tribe.⁹⁷ A recent ethnographic account of the implementation of surveillance databases in New Delhi includes a revealing quote from officers who admitted “some of the communities are still criminal, but they are no longer targeted as a group but as individuals.”⁹⁸

The second legal feature of SDS is that despite these vague, flexible, and biased criteria, inclusion into a database still inevitably translates into serious penal or other punitive consequences outside the carceral system. The consequences manifest in similar ways across the jurisdictions we have examined. Police rely on these databases to allocate surveillance resources, determine which geographical areas or homes to target, and which persons of interest to investigate and arrest, leading to the chronic over-policing of those in the databases.⁹⁹ These databases or the information within are also shared with other actors in the criminal justice system, which can exacerbate collateral consequences of the criminal justice system.¹⁰⁰ Prison and jail officials rely on databases to make appropriate classifications and other decisions for security and institutional order.¹⁰¹ Prosecutors rely on these databases to inform and craft criminal charges and plea bargains.¹⁰² Judges rely on them to inform bail and sentencing decisions.¹⁰³

Arrests and indictments of individuals in these databases are also often implemented at a mass or group level through the use of conspiracy or other group criminality statutes. In the U.S., gang databases have been used to build conspiracy cases against groups of alleged gang members, where individuals are implicated in alleged criminal activity for associating with or living near individuals or groups involved in a crime.¹⁰⁴ In the U.K., conspiracy-based laws, called the joint enterprise doctrine, allow for the prosecution of groups solely on the basis of gang membership or association, even though gang membership or association alone are not crimes.¹⁰⁵ Joint enterprise allows more than one person to be prosecuted for the same offense, even if the person was not involved or in the proximity of the crime, and it only requires the prosecution to prove the person had the foresight to believe a crime would occur.¹⁰⁶ Similarly, in India there are recent accounts of large groups of anti-caste protestors belonging to marginalized communities being classified as history sheeters or habitual offenders and consequently being subject to restrictions on their movements.¹⁰⁷ These loosely based charges mean people can be criminalized for who they know, where they live, and indeed their membership in a particular gang.

The harmful consequences of inclusion into an SDS database go well beyond the threat of incarceration. The collateral consequences of social censure and stigma can result in multiple, interacting forms of harm that impact various facets of a person’s life. For instance, school officials, public housing authorities, immigration officials, and other non-law enforcement government actors that have access to or receive information from gang databases can use the information for decisions about community safety, disciplinary action in schools, tenant applications, and assignment of counseling resources.¹⁰⁸

The harsh punitive consequences of being classified under criminal intelligence databases, both within and outside of the carceral system, holds important insights into the role and function of SDS. The legal and administrative rules, as well as the bureaucratic modes of these classification systems, can often obscure the range of punitive, and often violent, consequences they trigger. Indeed, it is the looming threat of harm that accompanies inclusion in the database and is meant to discipline the individuals and groups that it targets through these labels. While these systems are often justified for their ability to predict crime and identify budding criminality, this predictive quality creates a feedback loop. Categorization into an SDS triggers hyper-surveillance, more police contacts, and a range of penal and social consequences for individuals and groups that make it difficult, if not impossible, to evade arrests, indictments, and other forms of punishment. In doing so, they also strengthen the correlation between criminality or deviance and the factors often associated with such classifications and identity markers (e.g., poverty and membership of a marginalized community) that are used to justify other forms of disproportionate targeting and surveillance.¹⁰⁹

The third legal feature of SDS is the lack of robust legal oversight and constitutional scrutiny of these databases, leaving few avenues to challenge their operation or introduce structural safeguards. Most criminal justice constitutional doctrines focus on the abuse of due process at the level of individual action or actors. Such legal frameworks are ill-suited for the broad-based targeting and harm that stem from database uses, which typically involve several institutional actors and multiple intersecting laws. The stage at which databases are used in the criminal investigation process is typically considered within the discretionary power of police. ¹¹⁰ In the U.S., such uses occur before constitutional scrutiny, primarily via the 4th amendment, is triggered.¹¹¹

Legal scholar K. Babe Howell has argued that these databases are actively employed to avoid constitutional requirements of due process.¹¹² In Gang Policing: The Post Stop-and-Frisk Justification for Profile-Based Policing, she details how in 2013 the NYPD increased use of its gang database and covert surveillance to advance a new policy priority of policing “crews,” a law enforcement term for gangs that are loosely organized, neighborhood-based and primarily composed of young people.¹¹³ She argues that this increased reliance on gang databases was intentional because their use was less likely to be subject to review or legally challenged because they are not governed by constitutional or statutory requirements like the NYPD’s recently invalidated stop and frisk regime. ¹¹⁴ The lack of serious constitutional challenges to these legal regimes can also be explained by the lack of transparency and visibility into databasing practices that would be necessary to build or foreground such a challenge. These systems are typically decentralized (explored in Section d), vary greatly across different states and municipalities, and there is typically no public access to these records and systems due to sector-specific data confidentiality policies.

C. Political Economy Features

This category highlights common political economy features that emerge and are produced by SDS use. These features explore and illuminate the relationships between modes of economic production, governance, customs and practices, and the vested interests of private enterprise and political institutions. Political economy features are necessary for understanding the drivers of SDS development and use, as well as their disproportionate impact on particular communities.

The first political economy feature of SDSs is that they are a modern form of the commodification of security and risk management, where private-public partnerships and other arrangements are created to extract profit from the development and use of these public administrative databases and actuarial practices. The 1970s mark an important period where economic and political conditions engendered the development of this feature. During this period, the U.S., U.K., and India experienced economic volatility and declines due to deindustrialization, foreign economic policies, and global stagflation.¹¹⁵ This period is also when each country dramatically shifted penological policymaking and practices towards more punitive and “law and order” approaches that emphasized categorical and group-based risk management methods as more efficient.¹¹⁶ As a result, each government enacted harsh criminal justice policies or practices that legitimized and thinly disguised the targeting of certain social groups. As inequality in access to security grew (especially in marginalized communities), a burgeoning private market in security, risk management, and information technology developed.¹¹⁷ Over time this resulted in the creation and growth of government administrative databases and actuarial practices that often required collaboration with private enterprises that financially benefited from the arrangement.

The U.K. enacted national policy reforms that required intelligence systems, crime audits, and statistically based performance monitoring and systematic analysis of police crime-fighting efficiency.¹¹⁸ While most of the information technology infrastructure was created and used by the government, these policy shifts produced a market for private security and insurance (whose policies deterred victims from reporting crime).¹¹⁹ In the U.S., large urban police departments like the NYPD and CPD engaged in public-private partnerships with large technology companies to develop or procure computerized information systems that could manage and statistically manipulate a growing body of police administrative data.¹²⁰ It is notable that during this period, the securities and technology sectors experienced significant growth compared to rest of the labor market.¹²¹ In India, the Crime and Criminal Tracking Network System (CCTNS) was created to digitize criminal administrative data, and is now reportedly being opened up to commercial background check services to generate revenue.¹²²

The second SDS political economy feature is that these systems facilitate administrative opacity, which shields private and public actors from dissent, democratic inquiries, and various other appeals for transparency, access, due process, and equity. “Administrative opacity is a deliberate strategy to manage regulatory environments” by controlling access to information regarding technical systems.¹²³ Sociologist Tressie McMillian Cottom argues that a prominent technique of administrative opacity is obfuscation through privatization, where “data that would have previously been public, publicly available or legally discoverable” is either privatized or made practically inaccessible.¹²⁴ SDSs, or the information within, can be privatized and made inaccessible when private companies that develop the systems assert trade secrets claims or withhold documentation,¹²⁵ and when public agencies claim that the information within or the system itself is classified or meets exceptions to public transparency mandates.¹²⁶ SDSs are made practically inaccessible through various means, but one notable strategy is the use of reasonable, yet ethically questionable, rationales to quell inquiries regarding the underlying policies or logics that undergird SDS design or use. Regardless of which approach to administrative opacity is used, the consequences are the same—they hamper public inquiries and contestations.

Evidence of the second SDS political economy feature varies in each jurisdiction. In the U.K., critics of the Gangs Matrix argue that it is overinclusive and question whether the “right people” are targeted by or included in the system.¹²⁷ Critics rightly assert that gang membership in and of itself is not a crime and can be mutually exclusive of criminal activity.¹²⁸ Yet the Gangs Matrix remains administratively opaque in two ways. First, an Enforcement Notice regarding violations of data protection laws revealed that inconsistencies in the Gangs Matrix’s operating policies, along with informal practices by police, distorted public perception about how the systems was used and possibly concealed legal violations.¹²⁹ Second, the government employed a reasonable, but controversial, rationale to quell scrutiny of the Gang Matrix’s net-widening effects. The government reasoned that it adopted a preventative public health model of violence interruption to justify why the Gangs Matrix included individuals with no record of violence, though this rationale and practice were challenged in the aforementioned Enforcement Notice.¹³⁰ In the United States, police engage in several tactics to keep details regarding gang databases opaque. Indeed, a coalition letter to the NYPD’s Inspector General and testimony at legislative hearings on the lack of transparency regarding the NYPD’s gang database revealed that individuals in the database never receive notice of this fact, and that this information is not included in their criminal history or rap sheet, documents that are publicly accessible.¹³¹ In India, various local police manuals maintain that the existence of history sheets and other surveillance databases is confidential and public Right to Information requests have been denied.¹³²

The third political economy feature is that SDSs facilitate and serve a carceral economy¹³³ of cheap labor or other exploitative labor dynamics by accelerating criminalization and incarceration, in addition to serving as a mechanism that maintains a high correlation between poverty, local social hierarchies, and criminalization. This feature may seem more apparent because several criminal justice policies, practices, and technologies facilitate similar outcomes, and research demonstrates that changes in incarceration and the carceral economy are linked to social and economic structural changes, rather than actual crime rates.¹³⁴ SDS use accelerates criminalization and incarceration, which creates a surplus labor force that can be subjected to exploitive labor arrangements both within and outside of carceral institutions. Individuals with database designations like “gang member” or “habitual offender” are subjected to heightened law enforcement scrutiny and increased punitive outcomes with potentially fewer constitutional protections due to issues discussed in Section B and poor data quality practices.¹³⁵ For example, in the U.S., the NYPD invested in the development of criminal intelligence databases and other data-driven administrative systems, like Compstat, to enable greater data sharing and “decentralize decision making,” which altered policing from “a logic of functional specialization (detectives, forensics, youth officers) to one of territorial specialization (census blocks, hot spots, street segments).”¹³⁶ This provided data-based pretext to conceal otherwise biased targeting and micromanaging of certain groups and communities that politicians blamed for social disorder. This also helped justify government divestment that significantly reduced economic opportunities for those communities.¹³⁷

In U.S. prisons, incarcerated workers are subjected to low pay and exploitative work conditions and practices; such labor arrangements are constitutionally permissible via the Thirteenth Amendment and beyond the reach of labor laws.¹³⁸ In society, markers of involvement in the criminal justice system, whether criminal records or criminal intelligence designations, can also diminish job opportunities relegating those individuals to low-wage and exploitative jobs.¹³⁹

SDSs also ensure that specific forms of marginalization, particularly poverty, race, and caste, are targeted by criminalization process, which in turn results in these marginalized groups being disproportionately represented in the criminal justice system. This is because criminal intelligence databases reinforce and rationalize differential law enforcement practices and treatment, and groups disproportionately targeted by database-enabled law enforcement often have fewer resources and less power to contest their treatment and outcomes.¹⁴⁰ For instance, India’s Criminal Procedure Code allows magistrates to issue bonds to individuals with a history sheet record as an assurance for good behavior.¹⁴¹ Yet, because history sheets are indiscriminately used against poor and marginalized communities, they are often unable to pay the bonds, this failure to pay is then used to legitimize their inclusion in the registers and subsequent differential treatment.¹⁴²

The development, use, and impact of criminal intelligence databases lie in the intersection of political economy interests and concerns. The features of this SDS category provide several facets of analysis that can uncover often opaque interests and drivers of SDSs development and use, which can aid identification of unique points for policy intervention and public contestation. This analysis can also deepen understanding of the structural implications of SDS use and the outcomes they produce.

D. Organizational Features

This category examines the organizational structures, practices, and decisions (e.g., deferential actions) that are relevant to understanding how SDSs are operationalized by and within institutions. Though SDSs are demonstrably technological in character, they are still created by parts of the criminal justice process and this is best understood through an examination of the organizational and operational features of the institutions designing and using SDSs.¹⁴³

Police departments, like other government agencies whose missions are oriented around addressing social issues, are hierarchical bureaucracies. Yet because police departments are increasingly becoming the primary or at least initial government response to a myriad of social problems,¹⁴⁴ their outwardly facing organizational structure distorts perceptions of how policing actually operates. This distorted view is amplified with the use of criminal intelligence databases. The public views these agencies as operating under formal rules with oversight because of their hierarchical organizational structure.¹⁴⁵ But in reality, the majority of police work consist of hyper-local, unsupervised, and discretionary practices of front-line or patrol level police officers, which shapes and sustains criminal intelligence database use and outcomes.¹⁴⁶ Indeed, some surveillance scholars have argued that criminal intelligence databases and other information technologies have facilitated the growth of police work as information work,¹⁴⁷ and that the integration of these technologies further distorts policing practices and decision-making to seem more impartial.¹⁴⁸ Thus, the first organizational feature of SDS is a hierarchical and “top-down” organizational structure with formal rules and policies that practically operate as minimally supervised, highly discretionary, local units of front-line workers.

This first SDS organizational feature means that despite the existence of formal policies and leadership structures that should presumably shape and oversee SDS use, it is actually the superficially supervised, discretionary practices of front-line police officers that give meaning to and concretize SDS contents, logics, uses, and outcomes, which are often noncompliant with local legal rules or requirements. In several states in India, there are police manuals and “standing orders” that provide executive instructions regarding the use of “history sheets” for habitual offenders. However, these policies include a range of arbitrary and subjective standards that are often at odds with constitutional and statutory standards and give police officers complete discretion regarding who can be added and removed from the database. For example, one standing order permits police to open a suspect history sheet for any person who is convicted of any crime in the Indian Penal Code, even if it is their sole offense.¹⁴⁹ Although individuals with history sheets are considered habitual offenders when they’ve been charged more than once, High Courts have found that the commission of an isolated crime cannot be characterized as a habitual act.¹⁵⁰ The practical implications of such discretionary policing and database practices mean that database inclusion and overall composition reflect police biases.¹⁵¹ In one case, an Indian court highlighted a police database record suggesting that a woman might commit theft and should be subjected to surveillance because she was unmarried and “is always found fashionably dressed without any income and therefore, her activities should be watched.”¹⁵² Similar observations were made in other jurisdictions examined. One example is CalGang, which provides database guidelines to individual police officers to ideally reduce discretion.¹⁵³ Despite these guidelines, a 2016 audit of CalGang noted that an “inadequate leadership structure” resulted in the database not complying with several legal requirements governing its use.¹⁵⁴

A second organizational feature is that while SDS are developed and appear to centralize information and oversight, in practice, they are diffused with some localities having duplicative, informal systems because of the decentralized organizational structure of law enforcement. In the U.K., the Gangs Matrix is overseen by the Trident Gang Command, a unit of a Metropolitan Police Service, but it is locally managed by the police and local governments in each of London’s 32 boroughs.¹⁵⁵ Although the Gangs Matrix is considered a central database, police officers in a given borough only have access to information in their jurisdiction, so if an individual in the database moves to another borough, the local agencies are encouraged to pass information along rather than access it directly through the database.¹⁵⁶ The centralization of information is also hampered by the fact that practices for adding and removing individuals to the database vary by borough so “[e]ach borough effectively has its local matrix.”¹⁵⁷ Similar challenges were found in California’s CalGang, which is considered a “pointer system,” where records within the database “point or refer” to external source documents that provide the basis for official police actions.¹⁵⁸ The CalGang data entry process is sorted through local and regional agency nodes so that all data uploaded to the central database is viewable by all user agencies but the information can only be added through specific regional nodes.¹⁵⁹ While this filtered process exists to provide more supervision and oversight, police departments still use duplicative, informal systems and practices locally, which can threaten the integrity of the central database.¹⁶⁰ Indeed, a 2020 CalGang audit revealed significant misuses by the Los Angeles Police Department (LAPD), which resulted in the department’s permanent withdrawal from the program and the state’s Attorney General’s revocation of statewide access to the department’s records, which makes up nearly 25 percent of the databases.¹⁶¹ Notably, less than a year before this finding of significant misuse, the LAPD performed its first department audit of CalGang, which found the LAPD had a 95 percent compliance rate with most CalGang policy objectives.¹⁶² This revelation is significant because it highlights how the decentralized organizational structure of policing can impede centralized oversight of SDSs.

The third SDS organizational feature is that SDS enable compliance-based administrative practices that serve to justify law enforcement’s suspect profiling practices.¹⁶³ Compliance-based policing is a risk management practice that is executed and articulated through the administrative function of policing.¹⁶⁴ Scholars have distinguished this model of policing from deterrence-based models because it is more interested in regulating undesirable activity rather than individual acts.¹⁶⁵ Criminal intelligence databases systematize this model by encoding current law enforcement profiling practices and policies, as well as constraining law enforcement’s understanding and response to a predetermined social problem. This process is made simple because police administrative practices are primarily retrospective, in that they document and, in many ways, justify what has happened.¹⁶⁶ When this data is entered and organized in criminal intelligence databases, the data entry requirements, labels, and other classification schemes shape how information is entered and presented, which in turn structures and legitimizes how police view their prior actions and subsequent investigations.¹⁶⁷ Thus, criminal intelligence databases can prospectively shape the nature of a law enforcement inquiry and therefore what is subsequently entered.¹⁶⁸ If police are familiar with the data entry requirements of a database, they may standardize the sorts of questions they ask or evidence they look for in anticipation of expected administrative duties. Data in criminal intelligence databases can also unnecessarily limit law enforcement investigations, both “supplant[ing]” traditional investigative methods and serving as pretext for subjective and politically influenced policing practices (e.g., meeting “quotas”).¹⁶⁹

This third SDS organizational feature manifested in various ways among the cases reviewed. In the U.K., this feature influenced practices and decisions made by several criminal justice actors. In a speech regarding his concerns with the British criminal justice system, British Parliament member David Lammy noted how information in the Gangs Matrix is used in decision-making without verification:

[T]he Gangs Matrix features information provided by the police to the Crown Prosecution Service (CPS) at the point when the CPS makes charging decisions. The inclusion of this information suggests that prosecutors regard it as pertinent to whether defendants are charged, or what they are charged with. If cases make it as far as court, the Gangs Matrix could than [sic] [be] used by the prosecution in cases involving Joint Enterprise. It is deployed to substantiate claims that individuals are part of a gang and therefore played their part in a crime.¹⁷⁰

Similar practices have been observed in the U.S. and India. For example, an investigative report on the NYPD’s gang database practices, Gang Takedowns in the De Blasio Era: The Dangers of Precision Policing, revealed that gang database designations are used to justify heightened police harassment.¹⁷¹ The report highlights several incidents where gang designation instigated hyper-policing of minor offenses like jaywalking, and how data entered from these hyper-policing practices were used to substantiate fabricated gangs and justify serious criminal charges.¹⁷² Moreover, case law and testimony from criminal lawyers across Indian states suggest that the surveillance registers are used to meet and thus justify politically motivated law enforcement goals or targets.¹⁷³ Ethnographic accounts have observed that local politicians regularly put pressure on police departments to use the registers to ‘solve’ cases.¹⁷⁴ Similarly, a 2020 Madras High Court decision noted that the creation of a history sheet against the plaintiff without cause represented a “shortcut method” to meet police targets.¹⁷⁵

SDS organizational features help clarify a number of misrepresentations regarding policing practices, including the use of information technologies like SDS. The analysis enabled through SDS organizational features excavates pertinent, yet under-examined, aspects of SDS development, use, and outcomes, which can also function as sites for institutional reform. Such reforms can both mitigate abuses of SDS and their negative outcomes as well as address broader systemic concerns.

E. Social Features

This category explores conditions and features within society that structure the development, use, and outcomes of SDSs. Some social features are historical, so they precede and influence the development and use of databases, while others are contemporaneous with database use and thus constrain or influence how databases are operationalized and their broader social impact.

The first social feature is that SDSs are developed or enacted after major social change or crisis, which often include demographic, political, and economic shifts. In India, the practices and policies that shaped current surveillance and database practices stem from colonial police governance. Following independence, the Parliament replaced the Criminal Tribes Act with the Habitual Offenders Act, which established the rules that still govern surveillance databases in several states.¹⁷⁶ As scholar Mark Brown discusses, there was a paradigm shift from the group criminality model of classification (“grounded in concepts of tribes, castes, or gangs”) to the individual model (“based upon the idea of habituality”) that functions to normalize these discriminatory practices as part of “ordinary criminal law”.¹⁷⁷ Individualizing the category of the habitual offender could not overcome the group classification that remains embedded in a society ordered by a rigid and discriminatory caste system. In fact, in 2007 the UN Committee on Elimination of Racial Discrimination (CERD) asked the Indian government to repeal the Habitual Offenders Act, concluding that “the so-called denotified and nomadic, which are listed for their alleged ‘criminal tendencies’ under the former Criminal Tribes Act (1871), continue to be stigmatized under the Habitual Offenders Act.”¹⁷⁸ In the U.S., increased investments in gang suppression tactics and information technologies have occurred after mass migrations and immigration of racial and ethnic minorities that fundamentally altered the demographic makeup of the country and its major cities.¹⁷⁹ These demographic changes also coincided with major economic declines, high unemployment rates induced by deindustrialization, and severe austerity measures encouraged by the financial sector as a condition of post-deindustrialization bailouts.¹⁸⁰ National and local crime rates rose as a consequence of government divestment and economic insecurity, but politicians employed racialized and inflammatory rhetoric to scapegoat the victims of their social policies as drivers of crime and social disorder.¹⁸¹ These same politicians then lobbied for technological upgrades to target and suppress gang violence and other criminal justice priorities.¹⁸² In the U.K., the Gangs Matrix was created following nationwide civil riots that the government labeled as gang crime, though 81 percent of those arrested were not identified as gang members.¹⁸³ The government used this crisis to announce new anti-gang strategies that would involve more intelligence creation and sharing to identify and target known gang members and individuals at risk of violence.¹⁸⁴

The second social feature is that SDS development and use is normalized through media amplification and dramatization of crime-related moral panic, which in turn shapes public opinion and encourages public acquiescence to related government practices and policies. Law enforcement officials and politicians often use vague but sensationalized rhetoric and terms like “gangs” and “habitual” or “prolific” offenders to suggest that their criminal intelligence database classifications are justified because these terms imply that their targets have extensive or violent criminal records.¹⁸⁵ Yet these suggestive labels are misnomers since many individuals in the systems reviewed have no criminal record or fail to meet the criteria for inclusion.¹⁸⁶ These mischaracterizations also allow the government to avoid public scrutiny because they focus attention on symbols of disorder and evade interrogation of how the government contributes to the outcomes they profess to correct via policing.¹⁸⁷ Despite these mischaracterizations and fallacies, the media (e.g. journalism, scripted fiction, true crime) is drawn to the sensational nature of the subject and tends to overemphasize and amplify the problem, which then distorts public perceptions about actual crime rates and activity.¹⁸⁸ For example, research shows that there are fewer gangs today despite narratives of growing gang activity.¹⁸⁹ In the U.S. and the U.K., Black, Indigenous, and People of Color (BIPOC) and Black, Asian, and minority ethnic (BAME) communities¹⁹⁰ are the public faces of the gang problem because they are overrepresented in these databases, although white individuals make up a considerable portion of gang and organized crime activity.¹⁹¹ Similarly, fictional crime dramas frequently mention or depict law enforcement use of SDSs in ways that suggest law enforcement use is always valid and lawful, and that these technologies are both effective and harmless, which normalizes and distorts public perceptions regarding these tools. Episode seven of the canceled crime drama Tommy portrayed a fictional account of the murder of Los Angeles rapper Nipsey Hussle, including the posthumous revelation that the rapper was listed in CalGang.¹⁹² Towards the end of the episode the main character, Tommy, who is also the first female police chief of the LAPD, starts to clear the LAPD’s gang database of individuals with no actual gang affiliation. This episode is one example of how crime dramas distort public perceptions of SDSs because it suggests that law enforcement officials actively audit and maintain these systems, undermines real-world incidents of LAPD abuse of gang databases, and reinforces myths that “good cops” can fix systemic issues.¹⁹³

The third social feature is that SDSs encourage and rely on public participation in databasing efforts to legitimize its goals. This third feature is related to the second SDS social feature because public participation and reactions to perceptions of social problems are influenced by government and media representations. Who people know can be limited by neighborhoods and social networks, which are shaped and constrained by segregation.¹⁹⁴ Thus, public participation in databasing efforts often reproduces systemic and societal disparities. The proliferation of neighborhood communication systems like Nextdoor, Amazon’s Ring Neighbors app, and municipal 311 apps represent an emerging and direct form of public participation in profiling and surveillance practices that can lead to biased SDS inclusion.¹⁹⁵ Moreover, informant arrangements remain a longstanding and prominent form of public participation in criminal justice investigations and practices. These arrangements are often coercive or incentivized, “exacerbat[ing] some of the most problematic features of the criminal justice process,” particularly its racial and caste disproportionality.¹⁹⁶ For instance, in some informant arrangements police will ignore or tolerate criminal activity if certain objectives are met.¹⁹⁷ This under-enforcement and tolerance of criminality, especially in historically overpoliced areas, skews criminal justice data and distorts public perceptions of who is a criminal or other symbolic terms that communicate criminality or some form of social risk (e.g. “gang” or “rowdie”).¹⁹⁸ One example from the U.S. is the NYPD’s Five Point Plan to prevent gangs from re-establishing a foothold in communities. Under this plan, the department notifies elected officials and community members about arrests and activity to effectively solicit community support and involvement in suppression efforts.¹⁹⁹ Similarly, in India, interviews with legal practitioners in multiple states revealed that the informant system is critical to the police’s process for determining who is included in surveillance databases.²⁰⁰ For example, in the state of Madhya Pradesh, the informant system colloquially known as “mukhbiiri” (word-of-mouth) typically involves individuals from targeted communities being coerced into becoming informants, but this cooperation does not preclude these informants from being targeted by police in the future.²⁰¹ This menacing dynamic also contributes to the same communities being disproportionately represented in these systems and the range of punitive consequences that follow.

SDS social features highlight important contexts and dynamics that influence both how SDSs are developed and used, and how their use or impact is perceived publicly. These features also clarify why database use remains opaque to the public and why their outcomes are not subjected to the same level of scrutiny as other controversial technologies, such as predictive policing. The analysis enabled through this category and the broader framework should inform policy advocacy agendas and academic scholarship by broadening issue analysis beyond current practices and consequences.

II. Suspect Development Systems Beyond
the Criminal Justice System

While criminal intelligence databases are a paradigmatic example of an SDS, our categorical definition and analytical framework have value beyond the criminal justice system. The framework and analysis presented in Section I offers a systematic way to interrogate, evaluate, and eventually inform policy proposals to address database and related information technologies being used across government and private spheres. The key components of the definition of an SDS that we have put forth in this Article are (a) the use of information technologies to (b) manage social risk by (c) targeting individuals or groups to greater suspicion, differential treatment, and more punitive and exclusionary outcomes. There is a range of non-criminal databases that present these characteristics, especially as governments across the world turn to punitive measures to respond to a range of socio-economic problems in domains like social welfare, education, and immigration.²⁰² As “technologies, discourses, and metaphors of crime and criminal justice”²⁰³ have crept into multiple spheres of governance, we show how the category and framework of SDS can be employed helpfully across various domains.

In this Section, we identify three key categories of databases across jurisdictional contexts (social welfare, national ID, and citizenship) and argue that revisiting them through the SDS framework offers tactical benefits and nuances to our understanding of how these technologies can amplify structural inequities and cumulative disadvantage. For example, despite the origins of welfare distribution as a socially inclusive and non-punitive area of governance, the use of databases in this sphere has coincided with a global shift to a welfare state characterized by assumed scarcity, and an increased focus on eliminating “risky” people and behaviors.²⁰⁴ In this context, welfare databases present characteristics of an SDS and we analyze them in Part A of this Section through the features and categories distilled in Section I of this Article. Similarly, national biometric ID systems that are promoted across the Global South under an inclusive development mandate have had disproportionately exclusionary outcomes and legitimize differential treatment of those who are already subject to intersecting forms of marginality.²⁰⁵ Here, the act of naming these systems as SDS can offer tactical benefits to civil society’s resistance to these projects.²⁰⁶ Finally, the fact that immigration policies and practices are increasingly focused on securitization and criminalization is well-studied as a global phenomenon,²⁰⁷ and in Part C of this Section, we use the SDS framework to explore how recent citizenship database projects facilitate broad-based suspicion, exclusionary outcomes, and new forms of precarity for those who are deemed suspect.

A. Welfare System Databases

Welfare state databases are uniquely ripe for SDS analysis because “welfare state development involves defining the boundaries of national membership,” particularly “who is entitled to the benefits and social protection that the state will offer[.]”²⁰⁸ The welfare state was initially conceived as an inclusive policy regime²⁰⁹ that could engender social solidarity, but the persistent political commitment to austerity economics has left governments with fewer resources and has recast the welfare state as an exclusive policy regime²¹⁰ that is more suspicious and punitive in practice and focused on eliminating people and behaviors that are considered risky or deviant.²¹¹ Databases are a historical and contemporary feature of welfare administration because welfare policies have increasingly necessitated large data collection and use. Because these databases have become digitized, in whole or part, during the aforementioned policy regime shift, these “systems of social protection and assistance are increasingly driven by digital data and technologies that are used to automate, predict, identify, surveil, detect, target and punish” in countries that have adopted them.²¹²

Digitized welfare databases and systems are becoming more prevalent globally, but their uses can vary. Some systems are used to implement new welfare policies and administrative practices, while others are used to detect noncompliance and undesirable behavior. India (“Aadhaar”) and Kenya (“Huduma Namba”) have developed national biometric identification databases (discussed in detail in Section II(b)) that are used for verifying welfare recipients and to distribute benefits, services, and subsidies.²¹³ The U.K. introduced its Universal Credit system as a “digital-by-default” social welfare reform intended to integrate and automate the administration of six “legacy” welfare benefits through a singular platform where benefits were consolidated and allocated monthly.²¹⁴ Unlike previous policies, Universal Credit emphasizes the individual responsibility of benefits claimants through the imposition of additional requirements to receive benefits and harsh sanctions.²¹⁵ In contrast, the Netherlands implemented the System Risk Indication (SyRI) system to predict individuals most likely to commit benefits fraud,²¹⁶ though a national district court recently found the use of the system unlawful because it violated the right to privacy.²¹⁷ Digitized welfare systems have also been a source of legal challenges in the U.S. Systems used to determine and allocate benefits and automated fraud detection and benefits disqualification systems have been met with litigation.²¹⁸

Critical analysis by the U.N. Special Rapporteur on Extreme Poverty and Human Rights, Philip Alston, and successful legal challenges have illuminated significant concerns regarding the development and use of digital welfare state systems. Understanding these systems as SDSs, however, can expand public understanding overall and better inform holistic policy interventions. Here, we demonstrate how the five SDS framework categories can broaden our collective understanding of welfare databases as SDSs.

The SDS technical features demonstrate that database design exists on a spectrum and digitized welfare systems can exist in various forms. In the U.S., the government of Idaho used a formula in Microsoft Excel to determine and terminate Medicaid benefits for individuals with developmental disabilities,²¹⁹ whereas the Netherlands’ SyRI system used deep learning and data mining methods for fraud detection.²²⁰ The SDS legal features broaden our understanding of the legal and social consequences of welfare databases. In particular, SDS legal features reveal that these systems are often put into place without robust legal oversight or constitutional scrutiny, which means that their overall legality is questionable. Furthermore, a lack of meaningful legal safeguards may enable new modes of governance and social exclusion. Kenya’s Huduma Namba system amplifies decades of social policy aimed to exclude certain ethnic groups, which was previously found unconstitutional, and the failure to produce suitable government identification can lead to an automatic denial of government services, which is out of line with constitutional principles.²²¹ Similarly, the U.K.’s Universal Credit system impedes appeal procedures, particularly those regarding disability assessments.²²² Prior to the Universal Credit system, citizens could appeal decisions directly to an independent tribunal, but now under the system, appeals are a two-stage process that introduces a variety of impediments, including the system losing appeal requests and dissuasion by government officials.²²³ Thus, the SDS legal features help reveal that in practice, welfare databases can operate as a new mode of governance or social exclusion. The lack of scrutiny such databases received upon their introduction has resulted in them lacking meaningful legal safeguards, which means that these databases effectively determine welfare rights and the allocation of resources without providing avenues for legal recourse.²²⁴

The SDS political economy features demonstrate how welfare databases are another form of the commodification of risk management for private-sector financial benefit. In the U.S., many state welfare systems are developed by private companies. For example, Indiana entered a $1.3 billion contract with IBM to privatize and automate the state’s welfare eligibility process.²²⁵ IBM also sells a customizable, off-the-shelf software package to automate eligibility decisions.²²⁶ In Kenya, there are allegations that banks associated with the Kenyan President’s family could benefit from a credit lending platform linked to the Huduma Namba.²²⁷ The SDS organizational features reveal that while welfare systems are often digitized to centralize information and overall governance, in practice, these systems facilitate greater diffusion and delegation. Indeed, the U.K.’s Universal Credit system included delegation of unanticipated government responsibilities as well as automation of some system functions, which ultimately led to further diffusion. The government failed to anticipate the additional administrative burdens that would accompany a “digital by default” policy change, so digital assistance for digitally excluded and digitally illiterate citizens was “outsourced to public libraries and civil society organizations.”²²⁸ The Universal Credit system also requires automated calculations of benefits via the Real-Time Information system, but this automation was error-ridden and subsequently required a team of fifty full-time civil servants to rectify 2% of millions of incorrect transactions every month.²²⁹

Finally, SDS social features aid our understanding of how politicians and media employ and amplify sensational rhetoric to engender public acquiescence or support of these controversial systems and their disproportionate effects.²³⁰ Government officials and media invoke tropes like, “ghost beneficiaries” and “fraudsters”²³¹ to suggest that certain individuals or communities are undeserving of government benefits and to justify exclusionary practices and effects. The Netherlands’ SyRI system was promoted as an intervention to identify potential “fraudsters,” but the system was exclusively used in municipalities with “a high proportion of low-income residents, migrants, and ethnic minorities.”²³² This not only meant that the harsher consequences associated with the system would disproportionately affect more vulnerable communities,²³³ but also that it increased the likelihood of socially-biased and false conclusions that these communities have a greater propensity for crime or are undeserving of government benefits.

B. National Biometric ID Databases

National biometric identity (ID) systems that generate a unique identifier for each person in the country and create a corresponding biometric database are being rolled out at “a dizzying pace across the developing world.”²³⁴ The Indian ID system, known as “Aadhaar,” was launched in 2009. With more than one billion enrollments, it is currently the largest biometric database in the world.²³⁵ The “Aadhaar model” is also being promoted in West Africa²³⁶ and the Middle East.²³⁷ Along with demographic information, these systems typically collect biometrics like fingerprints, iris scans, or facial scans that are used for one-time enrollment into an ID database and in many cases, as a continuing means of authentication.²³⁸ These projects have been promoted by traditional development organizations like the World Bank as a “universal identity gateway” for all of a person’s interactions with state and private bodies and touted as a solution to the problems of corruption and fraud in service delivery in the Global South.²³⁹ Yet, across countries these projects are met with stiff civil society resistance and multiple legal challenges on grounds of hardship and exclusion experienced by vulnerable people due to errors in implementation, unwarranted bodily surveillance, privacy, and data security risks.²⁴⁰ Judicial responses to these systems are mixed. After years of public resistance and strategic litigation against the Aadhaar project, in 2018 the Indian Supreme Court found the project to be constitutional and put several limits on the use of the database by private companies.²⁴¹ In 2019, the Jamaican Supreme Court struck down Jamaica’s centralized, mandatory biometric ID system.²⁴² In January 2020, the Kenyan High Court also paused enrollment into their biometric ID system that proposed to collect a range of biometric information including face, voice sample, and even DNA.²⁴³

Despite heated public and legal resistance, debates around ID databases remain largely siloed from broader Western-dominated discourse on data-driven systems like AI or ADS.²⁴⁴ In this context, the SDS categorical definition and framework for conceptualizing and evaluating these systems can offer two strategic benefits. First, the analysis helps interrogate and challenge the narratives of universality and inclusion that have legitimized these projects in the Global South by focusing on the ways they can further entrench marginality and lead to the exclusion of certain groups from the vision of the nation state. Second, the analysis creates linkages with other SDSs across the world that can bridge counterproductive siloes between the Global South and Global North and prevents these developments from being relegated to the margins of mainstream discourse on AI and other so-called “advanced” systems.

As demonstrated by the technical features of the SDS framework, databases can be structured across a range of technical affordances. National biometric ID systems in different countries have varying technical designs, often distinguished based on the degree of centralization of the database, the use of biometrics (either one time at enrollment or as a continuing means of enrollment), and the degree to which the identifier allows linkages between different government (and even private) databases.²⁴⁵ In many countries, these systems have led to the digital transformation of legacy, paper-based civil registration systems,²⁴⁶ and the impact of these shifts should be explored. The SDS framework discourages the creation of technical thresholds and instead analyzes the ways in which these varying features and technological legacies shape the logic and outcomes of these databases.

The SDS legal analysis illuminates the complex relationship of law and regulation to national ID systems. The authorizing legislation for many of these projects has been characterized by vague and open-ended definitions of the purpose of these systems, the types of biometric data to be collected, and implementation protocols on the ground.²⁴⁷ As the first legal feature of our framework explains, this definitional ambiguity works to frustrate the aims of due process and rule-of-law that legal and policy frameworks purport to fulfill. The third legal feature speaks to the lack of any meaningful form of legal redress, given that SDS require structural reforms rather than procedural safeguards at the individual level. When governments like India and Kenya were faced with constitutional challenges that posed existential threats to their respective ID projects, they argued that data protection regulation would be sufficient to address any perceived risk.²⁴⁸ But this argument ignores broader structural challenges to these biometric ID databases including the exclusion of individuals from basic entitlements, the hyper-surveillance enabled by these systems, and the potential to facilitate undemocratic forms of governance.²⁴⁹ As we explore in Section I(b), legal frameworks that apply to SDS are often cosmetic and obscure or even legitimize more structural concerns (e.g. discrimination and privacy intrusions).

The SDS organizational features reveal that while biometric ID systems are touted as centralized systems with clearly defined national protocols, in practice these rules did not always govern the behavior of those operating the biometric systems on the ground, and there were no oversight safeguards to monitor non-compliance. For example, the Indian government attempted a system of “manual overrides” to address the issue of biometric errors that lead to exclusion from government benefits and systems.²⁵⁰ However, managers of these systems often failed to exercise this option and refused people access to services because of lack of interest in overcoming technical errors.²⁵¹ As Anthropologist Ursula Rao has noted, these “mega-projects . . . aim to reduce the space for digression in decision-making. Yet, in practice, they cannot be implemented and do not function without the creative improvisation of intermediaries who adapt the system . . . sometimes having to break the rules in the process.”²⁵²

The first political economy feature of SDS guides exploration of how a range of private interests played a covert, yet foundational, role in shaping the design, development, and current use of these government projects. While the Indian government and development agencies promoted Aadhaar primarily as a means of government service delivery, the project was designed from the start to facilitate new forms of market activity and commercial surveillance,²⁵³ and make the crucial shift of turning “citizens” into “customers.”²⁵⁴ Private companies (e.g. banks, telecom companies, background check services) were using the Aadhaar databases for authentication and tracking of individuals with little scrutiny or safeguards until the Indian Supreme Court intervened in 2018.²⁵⁵ Meanwhile, in Ghana, Rwanda, Tunisia, Uganda, and Zimbabwe, ID databases have facilitated “citizen scoring” exercises like credit reference bureaus to emerge at scale.²⁵⁶ Finally, the social features of SDS enable an exploration of the narratives that accompany and promote these projects. Faced with growing criticism and multiple legal challenges, the head of the Aadhaar project would regularly publish opinion pieces in the country’s most respected newspapers stating that privacy concerns were a “Luddite paranoia” and numerous advertisements on television channels portrayed wealthy upper-caste Indians vouching for the fact that Aadhaar is the most trusted ID in India.²⁵⁷ Meanwhile, in Kenya, political leadership infamously announced that the Huduma Namba would be the “single source of truth” about every Kenyan.²⁵⁸ Paying close attention to these popular narratives is critical to an understanding of how SDSs are justified and legitimized through the use of strategic imagery and rhetoric.

C. Citizenship Databases

In this Section, we analyze certain kinds of citizenship databases using the SDS framework. These government databases aid immigration policy enforcement and can be used to record and verify citizenship status or maintain information relevant to immigration enforcement priorities (e.g. deportation of individuals with criminal charges or convictions).²⁵⁹ Citizenship databases represent a burgeoning global use case, as many of these systems are in different stages of development and use.²⁶⁰ Here, we review India’s National Register of Citizens and the United States’ Homeland Advanced Recognition Technology System (HART). Because these databases are not fully implemented, this Section demonstrates how the SDS framework can be used to forecast analysis of concerns and harms before they occur based on available information and local historical precedent.

1. National Register of Citizens (NRC)

In December 2019, massive nationwide protests erupted across India with protestors holding signs that read “freedom from databases”²⁶¹ and “freedom from the [National Register of Citizens].”²⁶² The protestors were responding to the Home Affairs Minister’s announcement of the intention to create a National Register of Citizens (NRC).²⁶³ The NRC would require every person who self-identifies as a citizen of the country to furnish documentary evidence that confirms them as Indians, culminating in a computerized database that would be the definitive determinant of citizenship status.²⁶⁴ The National Population Register (NPR), a parallel civil registration project, was also announced as means to identify the “doubtful” citizens that might be subject to deregistration in the NRC.²⁶⁵ The announcement provoked memories of the horrors of the NRC that had been conducted in the Indian state of Assam starting in 2005, a process that was eventually shelved, but marked by “[k]afkaesque bureaucracy,”²⁶⁶ lengthy tribunal processes for those appealing exclusions, and eventually, four million people rendered stateless.²⁶⁷

The consequence of not being included in the NRC in Assam was mass disenfranchisement, and many believe a nationwide NRC will inevitably replicate this effect at a frightening scale. Across India, the poor, illiterate, women, and members of marginalized communities (like Dalits) lack access to documentary evidence of citizenship, which will leave them disproportionately excluded from the final database.²⁶⁸ The national NRC also had an explicitly discriminatory stance against members of the Muslim community in India within an atmosphere of growing Hindu majoritarianism.²⁶⁹ The announcement of the NRC was accompanied by the swift enactment of the Citizenship Amendment Act (CAA)²⁷⁰ which specifically offered citizenship to Hindus, Christians, Jains, Sikhs, Buddhists, and Parsis from neighboring countries in South Asia.²⁷¹ Muslims were categorically left out of this declaration.

The NRC functions as an SDS, both in terms of its final consequences for those excluded from the database, as well as for the process which is designed to cultivate suspicion and distrust against a much broader network of people.²⁷² The SDS framework categories provide a systematic way to unravel and evaluate the multifaceted forms of marginality that such databasing creates and reinforces. The timeline for implementation of the NRC has still not been announced, and with the COVID-19 pandemic it is likely to be delayed considerably. This provides a window of opportunity where the SDS framework can also be used to strengthen advocacy against the project before its harms are realized.²⁷³

The SDS technical category aids inquiries into how the computerized form of the NRC²⁷⁴ interacts with legacy paper-based citizenship records and expands the scope of its consequences. For example, the networked database form of the NRC would allow inter-and intra-government sharing, which will likely make the consequences of exclusion more immediate, broader, and definitive.²⁷⁵ The SDS legal category elucidates how the blurred line between colloquial (and often socially charged) phrases and legal categories can create the conditions for the discriminatory and discretionary implementation of SDS. In election manifestos and public speeches, the ruling party in India (the BJP) repeatedly used the term “persecuted minorities” for the groups that it plans to grant citizenship under the NRC process.²⁷⁶ Yet the CAA law itself has no mention of definition of the word persecuted, leaving it to the discretion of street-level bureaucrats to determine whether persecution will be a prerequisite,²⁷⁷ and how it will be defined. Similarly, the category of “doubtful citizens” in the NPR is left undefined and is up to the discretion of the “[l]ocal [r]egistrar who can decide that someone is doubtful’ even if they do possess the ‘particulars’ required.”²⁷⁸

The SDS organizational category decenters the analysis from national-level policy frameworks and instead examines the organizational practices of the front-line officers who, through performing their respective administrative functions, give meaning to and concretize the logics and outcomes of SDS. With the NRC, it will be crucial to study how a historical culture of suspicion regarding claims of citizens (and a particular mistrust of paper documents) influences how bureaucrats implement these systems and the discriminatory outcomes it produces.²⁷⁹ Indeed, the third SDS organizational feature speaks to how organizational priorities drive biased implementation and the Assam NRC experience suggests that similar problems will manifest in the national NRC. Accounts explain how the extreme pressure on bureaucracy to meet the centrally dictated deadlines of the Assam NRC translated into even more arbitrary and ad-hoc practices.²⁸⁰ Applying the SDS political economy analysis to the NRC shows that this system reflects a common SDS feature regarding the commodification of security and risk management via private-public partnerships because the database system is to be designed and implemented by large private technology vendors.²⁸¹ More generally, it may be useful to examine how the NRC interacts with broader economic trends towards nationalism and economic self-reliance.²⁸²

Finally, the SDS social features reveal how sensationalized and often misleading rhetoric is used to engender political support for the NRC, and in particular the framing of the NRC as a system that would not take away citizenship but instead present the “privilege of Indian citizenship” to many.²⁸³ Political leadership highlighted the granting of citizenship to Hindus and other religious groups while obscuring the fact that Muslims, the country’s largest minority group, were categorically left out of this inclusive paradigm.²⁸⁴ This, too, needs to be situated in the historical context of the “partition” of British-ruled India in 1947 that resulted in the creation of a separate state of Pakistan that was meant to be a “homeland for Muslims.”²⁸⁵ The partition was accompanied by gruesome communal violence, and the Muslims who remained in newly independent India have faced systematic discrimination, prejudice, and violence since, despite constitutional protections.²⁸⁶

2. Homeland Advance Recognition Technology System (HART)

The Homeland Advanced Recognition Technology System (HART) is a multimodal identity management system for U.S. citizens and foreign nationals that is developed and managed by the Office of Biometric Identity Management (OBIM) in the U.S. Department of Homeland Security (DHS) along with “mission partners” including other internal DHS components, other federal government agencies, and international partners.²⁸⁷ The system is being developed to replace and modernize DHS’ decades-old legacy biometric database, IDENT, which is at risk of failure and requires significant investments to remain operational.²⁸⁸ This modernization project will allow DHS and partner institutions to integrate new biometric technology capabilities, expand the types of information collected, and better identify relationships between individuals and groups.

HART is expected to store and process biometric data (fingerprints, face, iris, voice, etc.), contextualizing data (demographic data or camera quality), biographic data (names, signature, physical identifying details), derogatory information (warrants, suspected terrorist designation, immigration violations), encounter history (metadata associated with the collection of biometric or biographic data), and other associated data from disparate sources, which then links each entry with OBIM unique enumerators.²⁸⁹ Mission partners may be authorized users, data providers, or both, and authorized users can either query the system and upload data or function as “search only” users.²⁹⁰ While HART will purportedly offer many services and modalities that aid law enforcement and investigatory activities by a range of national and international institutions, existing documentation suggests that users will most likely use the system to identify individuals and verify the identity of known individuals.

HART is currently being developed in four incremental phases²⁹¹ to minimize impact on its institutional users. The first phase (Increment 1), which is currently underway, focuses on technical and infrastructural upgrades (e.g. migration of the database to cloud services and the integration of biometric matching capabilities²⁹²) that are necessary to operate HART.²⁹³ The subsequent phases will focus on increasing interoperability, providing new interface capabilities, and improving analysis and reporting tools.²⁹⁴ It is projected that by 2022, HART will include information on at least 259 million unique identities, making the system the world’s second-largest biometric database.^[296]However, whether the system will meet these projections and include anticipated capabilities is questionable, especially since a 2021 U.S. Government Accountability Office report revealed that this program is in breach of its cost and schedule goals, and failed to begin its transition from IDENT as originally planned.²⁹⁵

The primary functions and expected uses of HART make it a more apparent SDS because it is used to profile individuals and groups for immigration enforcement actions that are punitive and exclusionary in effect, but analysis of the SDS framework can still help broaden understandings of its potential risks and impact. While HART’s development is incremental, the early phases emphasize the technical affordances illuminated by the SDS technical features. The first two phases of HART development focus on increasing interoperability, matching capabilities, expanding capacity for additional modalities, and other improvements that add functionality and facilitate greater data sharing.²⁹⁶

Analysis of the SDS legal features indicates that vague criteria or faults in data governance practices make systems susceptible to inaccuracies thus increasing the risk of biased and unlawful outcomes. There are several bias and accuracy concerns regarding HART,²⁹⁷ but the lack of rigorous standards regarding data entered and stored in HART is notable under the SDS legal features. HART is exempt from accuracy and other data quality requirements mandated by the Privacy Act of 1974 (Privacy Act), and OBIM only recommends data providers follow certain standards rather than systematically verifying the accuracy and quality of data comprising HART.²⁹⁸ Absent and low standards for data entry and storage can increase the likelihood of misidentification in HART query results, which the World Privacy Forum notes “is remarkable for a system . . . that will have a high impact on individuals’ civil liberties.”²⁹⁹ SDS legal feature analysis also brings to light that there are few legal remedies available due to a lack of robust legal oversight or other safeguards. This is because HART’s legal governance structure is fragmented. Operationally, OBIM considers itself HART’s “system owner and data steward,” which means that the partner institutions own and are legally responsible for the data they enter into the system, and OBIM only exerts ownership over the unique enumerators³⁰⁰ used to link data within HART.³⁰¹ As a result, DHS seeks to exempt the database that populates HART from elements of the Privacy Act, individual Privacy Impact Assessments, Systems of Records Notice, and information sharing access agreements that govern the data collection, processing, and storage rights of HART, DHS components, and partner agencies.³⁰²

HART also reflects a common SDS political economy feature regarding the commodification of security and risk management via private-public partnerships. HART is being developed and maintained by private companies through significant government grants. The aerospace and defense company Northrop Grumman was awarded a $95 million contract to develop and integrate the first two phases of HART, and the system is hosted on Amazon Web Services GovCloud, though the costs of such services are undisclosed and not standardized.³⁰³

The SDS organizational features clarify that, despite formal rules and policies, a system’s use can be minimally supervised and highly discretionary. While HART has several formal rules regarding use and operation of the system, use by its various and fairly autonomous users lacks oversight, which enables discretion and intensifies inaccuracy issues. As previously mentioned, each mission partner is responsible for the documentation of their collection and processing of data in HART via their own SORNs and PIAs and this also means that each entity is responsible for notification requirements to individuals in the system.³⁰⁴ As a result, OBIM has no obligation to oversee or ensure compliance. When inaccurate or flawed data is entered and stored in HART, timely correction or updating of information is dependent on the prerogative of mission partners.³⁰⁵ This lack of oversight combined with the exemption of HART components from elements of the Privacy Act means that individuals in the system can face significant legal consequences (e.g., false arrest or deportation) with few means of redress.³⁰⁶

Finally, the SDS social features can aid understanding of the impetus for developing HART. Like other SDSs, motivations to develop HART followed several converging social dynamics and changes. HART’s development was brought about by the intersection of growing social and political hostility regarding immigration enforcement and reform,³⁰⁷ significant advancements and declining costs in biometric technology development (driven by exponential economic growth in the technology sector), and the government’s growing interest in technology-enabled surveillance practices. In 2015, OBIM began planning the creation of a more robust and efficient biometric database that could take advantage of and integrate new technological advancements to support DHS core missions.³⁰⁸ Since biometric technology development requires enormous start-up costs, the sweeping advancements that OBIM aimed to integrate into this newly conceived system were due to billions of dollars of investments via government subsidies, grants, and private-public partnerships.³⁰⁹ Indeed, early-stage investments funded the creation of IDENT and other fledgling government biometric systems.³¹⁰ Over time, the costs of development decreased, the variety of biometric applications expanded, and their performance in controlled settings improved.³¹¹ At the same time, the U.S. government’s interest in automation, which can be significantly enhanced by biometric technologies,³¹² spurred interest to reinvest in new interagency databases, like HART, and integrate use of these technologies into immigration and migration enforcement practices and policies, like “biometric exit” programs.³¹³

Conclusion

Digitization and databases are often thought of as the raw material for algorithmically-determined futures, even as they are already doing this kind of prognostic work in database form. SDS, both as a categorical term and analytical framework, resolves this misconception by elucidating the varied contexts, features, policies, and practices that must be evaluated to understand how these seemingly amorphous technologies function in practice, the ways they are used, and the outcomes they produce. While our framework is not exhaustive, throughout this Article we have demonstrated the analytical utility of the framework as a whole and its distinct constitutive parts in bringing forth necessary and novel analysis. For instance, the perspicacity enabled through this definition and framework allows one to recognize SDS design, use, and impact as a new and less obvious form of predatory inclusion practices in the technology sector, where marginalized groups are included in ostensibly democratic schemes on extractive terms. While there is emergent scholarship examining these practices in the gig economy platforms and credit scoring techniques,³¹⁴ our SDS framework demonstrates that an important characteristic of the power of SDSs is that through inclusion or exclusion in the databases one is practically excluded from full democratic participation, as a direct result or collateral consequence. In the database context, exploitative processes surrounding inclusion or exclusion are obscured and legitimized because of their primary use in state administrative functions or law enforcement.

In the criminal justice context, inclusion in any of the criminal intelligence databases reviewed individuals to increased police contacts, harassment, detrimental inferences, and other serious punitive outcomes, particularly when information in these databases is used to inform government decision-making like sentencing decisions or public housing eligibility determinations. These consequences are made worse by the fact that the databases or information within is shared with other government and non-governmental actors, the lack of consistent notification practices means individuals are often unaware of the cause of their differential treatment, and existing means for redress are hollow or futile.³¹⁵ In the welfare context, inclusion or exclusion from these databases can have deleterious effects. Inclusion in the United Kingdom’s Universal Credit system subjected individuals to onerous paperwork and documentation requirements, and flaws in the system resulted in extreme cuts to and fluctuations in monthly benefit payments which forced some people into debt.³¹⁶ Exclusion from universal databases that are used to administer welfare programs like India’s Aadhaar and Kenya’s Huduma Namba, can lead to further marginalization by making participation in the database a precondition for carrying out business or the exercise of democratic rights, such as the right to vote.³¹⁷

In addition to clarifying the full spectrum of uses, risks, and consequences of SDS, our definition and framework can inform evaluations of relevant policy proposals and interventions. A common axiom in technology policy is that technological innovation outpaces the law, but this is only true if one takes a myopic view of which laws and regulations are relevant to a given technology. This axiom ignores the fact that SDS are multifarious assemblages and, as the analysis in Section I demonstrates, how the legal, political economy, organizational, and social features of these systems are equally constitutive as technical features. Therefore, one must examine the policies, practices, and conditions that precede SDS development to understand their design, use, and impact. This analysis can also aid assessments of technological solutions for complex social policy concerns, specific technology policy proposals, and can help identify alternative opportunities or locations for interventions that would produce similar outcomes as new legal or regulatory frameworks.

First, our analysis demonstrates that SDSs are often not an appropriate solution to complex social issues. While certain efficiencies can be gained through digitized databases and data-driven analysis, these purported benefits must be evaluated and balanced with the risks and costs to society, along with whether their use or outcomes undermine governance goals or missions. In the criminal justice context, gang and other criminal intelligence databases sow community distrust, particularly in marginalized communities that bear an outsized burden of databasing practices.³¹⁸ This distrust means individuals are less likely to cooperate with police investigations, report crime, and some consider leaving their communities to avoid police encounters.³¹⁹ All of these outcomes ultimately make communities less safe, which is antithetical to the law enforcement mission of public safety.³²⁰ Outside of the criminal justice context, universal ID systems are promoted under the developmental goal of inclusion. In practice, however, their operation facilitates the exclusions of historically marginalized groups and creates suspect categories for citizens in different sectors (i.e., social welfare and housing).

While we focus our case studies on database systems, the SDS framework can also bring useful insights to technology policy debates around a broader range of data-driven tools, like the use of pretrial risk assessments as a prevailing solution to cash bail. Pretrial risk assessments attempt to statistically measure a defendant’s dangerousness, flight risk, or risk of recidivism to determine whether they should be detained before their trial. These tools can be categorized as an SDS because they are information technologies that exist in various technical forms but are primarily used to measure vague social risks to justify heightened scrutiny and punitive outcomes like pre-trial detention or electronic monitoring.³²¹ These actuarial tools have been used in criminal justice institutions for decades,³²² but as they have become digitized or automated they are increasingly proposed as the alternative to cash bail.³²³ For example, a 2020 California ballot initiative, Proposition 25, sought to uphold contested legislation (Senate Bill 10) that would replace the state’s cash bail system with a pretrial risk assessment.³²⁴ When attempting to balance the purported benefits of pre-trial risk assessments with the risks and costs to justice-involved communities and governance integrity, analysis of our SDS framework can offer incisive observations. One example is the political economy category, which can illuminate the entanglement of government and private interests in SDS design, use, and outcomes. In California, only one pretrial risk assessment tool (developed by the Arnold Ventures philanthropy, an arm of the Laura & John Arnold Foundation) met the validation requirements mandated in the controversial Senate Bill 10.³²⁵

Second, the SDS definition and framework can aid evaluations or implementation of prevailing policy proposals. The definition and framework expand public conceptions of what technologies warrant scrutiny and should be included in existing AI and ADS accountability or oversight mechanisms. Our analysis can also broaden the scope and substance of current or emergent government assessment practices of technologies, such as cost-benefit analysis or algorithmic impact assessments (AIA). Analysis of the SDS framework can illuminate assumptions and oversights typically made in these evaluations. For instance, SDS organizational features can bring attention to inefficiencies in government operations or practices that undermine the objectives of using a given technology or will reduce its overall efficacy. Similarly, SDS political economy features can expand conventional audits by revealing how the startup costs of some technologies are balanced by exploitative or extractive labor practices, which distort the overall costs to society.³²⁶ Moreover, as demonstrated in Section II(C), the SDS framework can be utilized for proactive examinations of concerning technologies before they are fully developed or implemented. This temporal expansion is important since AIA evaluations tend to occur after the technology is developed, procured, or in use.

Our analysis can also inform policy agendas seeking to mitigate and redress the harms of SDS. Given the opacity and secrecy regarding SDS use, some advocates call for transparency as an intervention.³²⁷ Transparency can be a helpful diagnostic tool for understanding the ecosystem of technologies used for governance, but knowledge regarding the existence of these systems does not forestall their harmful effects or empower communities seeking redress.³²⁸ Several categories and features of the SDS framework reveal that SDS are brought about by specific or gradual social policy changes; our analysis can redirect advocacy towards structural reforms of core social policies as complementary or alternative solutions.³²⁹

We encourage technology developers, policymakers, legal practitioners, government officials, scholars, advocates, and community members to use our SDS definition and analytical framework to inform and actively change the trajectory of SDS design, use, and outcomes. As demonstrated in Section II, our analysis can be used to examine important features and contexts that are pertinent to identifying and evaluating the legal and social implications of such technologies in addition to the development of meaningful legislative and regulatory interventions. Thus, the SDS framework can and should evolve as it is used to evaluate emerging or under-examined technologies in sensitive social domains, such as education and child welfare, where the use of databases and information technologies are both evolving and expanding.³³⁰

While SDS use is global and our cross-jurisdictional analysis demonstrates common challenges, there is no universal panacea. Still, the SDS framework can help advance policy discourse and reforms internationally and locally. Since there is a growing global discourse regarding banning or significantly limiting the use of some AI technologies that are demonstrably harmful (e.g., autonomous weapon systems or facial recognition technologies),³³¹ the SDS framework can aid these conversations in creating international norms that can inform domestic policies. However, for solutions to be effective and address the myriad of issues reviewed in this Article, they must be designed and implemented locally while centering the needs and interests of the individuals and communities harmed. Our SDS framework analysis makes clear that SDS development, use, and outcomes are context-driven; therefore, the development of interventions and solutions will require critical analysis of such contexts to ensure the goals of policy reforms are not merely focused on efficiency or short-term wins.