♪ ♪ SARAH BRAYNE: We live in this era where we leave digital traces throughout the course of our everyday lives.

ANDY CLARNO: What is this data, how is it collected, how is it being used?

NARRATOR: One way it's being used is to make predictions NARRATOR: One way it's being used is to make predictions about who might commit a crime... Hey, give me all your money, man!

NARRATOR: ...and who should get bail.

JUDGE: On count one, you're charged with felony intimidation... ANDREW FERGUSON: The idea is that if you look at past crimes, you might be able to predict the future.

WILLIAM ISAAC: We want safer communities, we want societies that are less incarcerated.

NARRATOR: But is that what we're getting?

Are the predictions reliable?

CATHY O'NEIL: I think algorithms can, in many cases, be better than people.

But, of course, algorithms don't have consciousness.

The algorithm only knows what it's been fed.

RUHA BENJAMIN: Because it's technology, we don't question them as much as we might a racist judge or a racist officer.

They're behind this veneer of neutrality.

ISAAC: We need to know who's accountable when systems harm the communities that they're designed to serve.

NARRATOR: Can we trust the justice of predictive algorithms?

NARRATOR: Can we trust the justice of predictive algorithms?

And should we?

"Computers Vs. Crime," right now, on "NOVA."

(computers booting up) ♪ ♪ NARRATOR: We live in a world of big data, where computers look for patterns in vast collections of information in order to predict the future.

And we depend on their accuracy.

And we depend on their accuracy.

Is it a good morning for jogging?

Will this become cancer?

What movie should I choose?

The best way to beat traffic?

Your computer can tell you.

Similar computer programs, called predictive algorithms, are mining big data to make predictions about crime and punishment-- reinventing how our criminal legal system works.

Policing agencies have used these computer algorithms in an effort to predict where the next crime will occur in an effort to predict where the next crime will occur and even who the perpetrator will be.

ASSISTANT DISTRICT ATTORNEY: Here, the state is recommending... NARRATOR: Judges use them to determine who should get bail and who shouldn't.

JUDGE: If you fail to appear next time, you get no bond.

NARRATOR: It may sound like the police of the future in the movie "Minority Report."

I'm placing you under arrest for the future murder of Sarah Marks.

NARRATOR: But fiction it's not.

How do these predictions actually work?

How do these predictions actually work?

Can computer algorithms make our criminal legal system more equitable?

make our criminal legal system more equitable?

Are these algorithms truly fair and free of human bias?

ANDREW PAPACHRISTOS: I grew up in Chicago in the 1980s and early 1990s.

♪ ♪ My dad was an immigrant from Greece, we worked in my family's restaurant, called KaMar's.

NARRATOR: Andrew Papachristos was a 16-year-old kid NARRATOR: Andrew Papachristos was a 16-year-old kid in the North Side of Chicago in the 1990s.

I spent a lot of my formative years busing tables, serving people hamburgers and gyros.

It kind of was a whole family affair.

NARRATOR: Young Papachristos was aware the streets could be dangerous, NARRATOR: Young Papachristos was aware the streets could be dangerous, but never imagined the violence would touch him or his family.

but never imagined the violence would touch him or his family.

REPORTER: Two more gang-related murders Monday night.

PAPACHRISTOS: And of course, you know, the '80s and '90s in Chicago was some of the historically most violent periods in Chicago.

Street corner drug markets, street organizations.

Street corner drug markets, street organizations.

And then like a lot of other businesses on our, on our block and in our neighborhood, local gangs tried to extort my family and the business.

local gangs tried to extort my family and the business.

And my dad had been running KaMar's for 30 years And my dad had been running KaMar's for 30 years and kind of just said no.

♪ ♪ (sirens blaring) (sirens blaring) NARRATOR: Then, one night, the family restaurant burned to the ground.

NARRATOR: Then, one night, the family restaurant burned to the ground.

Police suspected arson.

PAPACHRISTOS: It was quite a shock to our family, 'cause everybody in the neighborhood worked in the restaurant at one point in their life.

And my parents lost 30 years of their lives.

And my parents lost 30 years of their lives.

That was really one of the events that made me want to understand violence.

Like, how could this happen?

♪ ♪ NARRATOR: About a decade later, Papachristos was a graduate student searching for answers.

Papachristos was a graduate student searching for answers.

PAPACHRISTOS: In graduate school, I was working on a violence prevention program I was working on a violence prevention program that brought together community members, including street outreach workers.

including street outreach workers.

And we were sitting at a table, and one of these outreach workers asked me, the university student, "Who's next?

Who's going to get shot next?"

And where that led was me sitting down with stacks of shooting and, and homicide files with a red pen and a legal pad, by hand creating these network images of, this person shot this person, and this person was involved with this group and this event, and creating a web of these relationships.

And then I learned that there's this whole science about networks.

I didn't have to invent anything.

♪ ♪ NARRATOR: Social network analysis was already influencing popular culture.

"Six Degrees of Separation" was a play on Broadway.

"Six Degrees of Separation" was a play on Broadway.

And then, there was Six Degrees of Kevin Bacon.

PAPACHRISTOS: The idea was, you would play this game, and whoever got the shortest distance to Kevin Bacon would win.

So Robert De Niro was in a movie with so-and-so, who was in a movie with Kevin Bacon.

It was creating, essentially, a series of ties among movies and actors.

And in fact, there's a mathematics behind that principle.

It's actually old mathematical graph theory, right?

It's actually old mathematical graph theory, right?

That goes back to 1900s mathematics.

And lots of scientists started seeing that there were And lots of scientists started seeing that there were mathematical principles, and computational resources-- computers, data-- and computational resources-- computers, data-- were at a point that you could test those things.

So it was in a very exciting time.

We looked at arrest records, at police stops, and we looked at victimization records.

Who was the victim of a homicide or a non-fatal shooting?

♪ ♪ The statistical model starts by creating the social networks of, The statistical model starts by creating the social networks of, say, everybody who may have been arrested in a, in a particular neighborhood.

So Person A and Person B were in a robbery together, they have a tie, and then Person B and Person C were, were stopped by the police in another instance.

And it creates networks of thousands of people.

Understanding that events are connected, places are connected.

That there are old things, like disputes between crews, which actually drive behavior for generations.

which actually drive behavior for generations.

What we saw was striking.

(snaps): And you could see it immediately, and you could see it a mile away.

Which was, gunshot victims clumped together.

You, you very rarely see one victim.

You see two, three, four.

Sometimes they string across time and space.

And then the model predicts, what's the probability And then the model predicts, what's the probability that this is going to lead to a shooting on the same pathway in the future?

(gun firing, people shouting) REPORTER: Another young man lies dead.

NARRATOR: In Boston, Papachristos found that 85% of all gunshot injuries occurred within a single social network.

Individuals in this network were less than five handshakes away from the victim of a gun homicide or non-fatal shooting.

The closer a person was connected to a gunshot victim, he found, the greater the probability that that person would be shot.

the greater the probability that that person would be shot.

Around 2011, when Papachristos was presenting his groundbreaking work on social networks and gang violence, the Chicago Police Department wanted to know more.

PAPACHRISTOS: We were at a conference.

The then-superintendent of the police department, he was asking me a bunch of questions.

He had clearly read the paper.

NARRATOR: The Chicago Police Department was working on its own predictive policing program to fight crime.

They were convinced that Papachristos's model could make their new policing model even more effective.

could make their new policing model even more effective.

LOGAN KOEPKE: Predictive policing involves looking to historical crime data to predict future events, looking to historical crime data to predict future events, either where police believe crime may occur or who might be involved in certain crimes.

♪ ♪ So it's the use of historical data to forecast a future event.

NARRATOR: At the core of these programs is software, NARRATOR: At the core of these programs is software, which, like all computer programs, is built around an algorithm.

So, think of an algorithm like a recipe.

♪ ♪ ♪ ♪ You have inputs, which are your ingredients, you have the algorithm, which is the steps.

♪ ♪ ♪ ♪ And then there's the output, which is hopefully the delicious cake you're making.

GROUP: Happy birthday!

So one way to think about algorithms is to think about the hiring process.

In fact, recruiters have been studied for a hundred years.

And it turns out many human recruiters have a standard algorithm when they're looking at a risumi.

So they start with your name, and then they look to see where you went to school, and then finally, they look at what your last job was.

If they don't see the pattern they're looking for... (bell dings) ...that's all the time you get.

And in a sense, that's exactly what artificial intelligence is doing, as well, in a very basic level.

It's recognizing sets of patterns and using that to decide what the next step in its decision process would be.

to decide what the next step in its decision process would be.

♪ ♪ ♪ ♪ NARRATOR: What is commonly referred to as artificial intelligence, or A.I., is a process called machine learning, where a computer algorithm will adjust on its own, without human instructions, in response to the patterns it finds in the data.

in response to the patterns it finds in the data.

These powerful processes can analyze more data than any person can, and find patterns never recognized before.

and find patterns never recognized before.

The principles for machine learning were invented in the 1950s, but began proliferating only after about 2010. but began proliferating only after about 2010.

What we consider machine learning today came about because hard drives became very cheap.

came about because hard drives became very cheap.

So it was really easy to get a lot of data on everyone in every aspect of life.

And the question is, what can we do with all of that data?

Those new uses are things like predictive policing, they are things like deciding whether or not a person's going to get a job or not, or be invited for a job interview.

NARRATOR: So how does such a powerful tool like machine learning work?

NARRATOR: So how does such a powerful tool like machine learning work?

Take the case of a hiring algorithm.

First, a computer needs to understand the objective.

Here, the objective is identifying the best candidate for the job.

The algorithm looks at risumis of former job candidates and searches for keywords in risumis of successful hires.

and searches for keywords in risumis of successful hires.

The risumis are what's called training data.

The algorithm assigns values to each keyword.

Words that appear more frequently in the risumis of successful candidates are given more value.

The system learns from past risumis the patterns of qualities that are associated with successful hires.

Then it makes its predictions by identifying these same patterns from the risumis of potential candidates.

♪ ♪ In a similar way, the Chicago police wanted to find patterns in crime reports and arrest records to predict who would be connected to violence in the future.

They thought Papachristos's model could help.

They thought Papachristos's model could help.

Obviously we wanted to, and tried, and framed and wrote all the caveats and made our recommendations to say, "This research should be in this public health space."

"This research should be in this public health space."

But once the math is out there, once the statistics are out there, people can also take it and do what they want with it.

NARRATOR: While Papachristos saw the model as a tool to identify future victims of gun violence, CPD saw the chance to identify not only future victims, CPD saw the chance to identify not only future victims, but future criminals.

First it took me, you know, by, by surprise, and then it got me worried.

What is it gonna do?

Who is it gonna harm?

♪ ♪ NARRATOR: What the police wanted to predict was who was at risk for being involved in future violence.

Gimme all your money, man.

NARRATOR: Training on hundreds of thousands of arrest records, the computer algorithm looks for patterns or factors associated with violent crime to calculate the risk that an individual will be connected to future violence.

will be connected to future violence.

Using social network analysis, arrest records of associates are also included in that calculation.

The program was called the Strategic Subject List, or SSL.

The program was called the Strategic Subject List, or SSL.

It would be one of the most controversial in Chicago policing history.

ANDY CLARNO: The idea behind the Strategic Subjects List, or the SSL, was to try to identify the people who would be most likely to become involved as what they called most likely to become involved as what they called a "party to violence," either as a shooter or a victim.

♪ ♪ NARRATOR: Chicago police would use Papachristos's research to evaluate what was called an individual's "co-arrest network."

And the way that the Chicago Police Department calculated an individual's network was through kind of two degrees of removal.

Anybody that I'd been arrested with and anybody that they would, had been arrested with counted as people who were within my network.

So my risk score would be based on my individual history of arrest and victimization, as well as the histories of arrest and victimization of people within that two-degree network of mine.

It was colloquially known as the "heat list."

If you were hot, you were on it.

And they gave you literally a risk score.

At one time, it was zero to 500-plus.

If you're 500-plus, you are a high-risk person.

♪ ♪ And if you made this heat list, you might find a detective knocking on your front door.

you might find a detective knocking on your front door.

♪ ♪ ♪ ♪ NARRATOR: Trying to predict future criminal activity is not a new idea.

future criminal activity is not a new idea.

Scotland Yard in London began using this approach by mapping crime events in the 1930s.

But in the 1990s, it was New York City Police Commissioner William Bratton it was New York City Police Commissioner William Bratton who took crime mapping to another level.

BRATTON: I run the New York City Police Department.

My competition is the criminal element.

NARRATOR: Bratton convinced policing agencies across the country that data-driven policing was the key to successful policing strategies.

Part of this is to prevent crime in the first place.

♪ ♪ ♪ ♪ NARRATOR: Bratton was inspired by the work of his own New York City Transit Police.

by the work of his own New York City Transit Police.

As you see all those, uh, dots on the map, that's our opponents.

NARRATOR: It was called Charts of the Future, and credited with cutting subway felonies by 27% and credited with cutting subway felonies by 27% and robberies by a third.

Bratton saw potential.

Bratton saw potential.

He ordered all New York City precincts to systematically map crime, collect data, find patterns, report back.

collect data, find patterns, report back.

The new approach was called CompStat.

BRAYNE: CompStat, I think, in a way, is kind of a precessor of predictive policing, in the sense that many of the same principles there-- in the sense that many of the same principles there-- you know, using data tracking, year-to-dates, identifying places where law enforcement interventions could be effective, et cetera-- really laid the groundwork for predictive policing.

♪ ♪ NARRATOR: By the early 2000s, as computational power increased, criminologists were convinced this new data trove could be used in machine learning to create models could be used in machine learning to create models that predict when and where crime would happen in the future.

♪ ♪ REPORTER: L.A. police now say the gunmen opened fire with a semi-automatic weapon.

NARRATOR: In 2008, now as chief of the Los Angeles Police Department, Bratton joined with academics at U.C.L.A.

to help launch a predictive policing system called PredPol, powered by a machine learning algorithm.

♪ ♪ ISAAC: PredPol started as a spin-off of a set of, like, government contracts that were related to military work.

They were developing a form of an algorithm that was used to predict I.E.Ds.

(device explodes) And it was a technique that was used to also detect aftershocks and seismographic activity.

to also detect aftershocks and seismographic activity.

(dogs barking and whining, objects clattering) And after those contracts ended, the company decided they wanted to apply this in the domain of, of policing domestically in the United States.

(radio beeping) NARRATOR: The PredPol model relies on three types of historical data: type of crime, crime location, and time of crime, type of crime, crime location, and time of crime, going back two to five years.

The algorithm is looking for patterns to identify locations where crime is most likely to occur.

to identify locations where crime is most likely to occur.

As new crime incidents are reported, they get folded into the calculation.

they get folded into the calculation.

The predictions are displayed on a map as 500 x 500 foot areas that officers are then as 500 x 500 foot areas that officers are then directed to patrol.

ISAAC: And then from there, the algorithm says, "Okay, based on what we know about the kind of "very recent history, "where is likely that we'll see crime in the next day or the next hour?"

♪ ♪ BRAYNE: One of the key reasons that police start using these tools is the efficient and even, to a certain extent, like in their logic, more fair, um, and, and justifiable allocation of their police resources.

♪ ♪ NARRATOR: By 2013, in addition to PredPol, predictive policing systems developed by companies like HunchLab, IBM, and Palantir were in use across the country.

(radios running in background) And computer algorithms were also being adopted in courtrooms.

BAILIFF: 21CF3810, State of Wisconsin versus Chantille... BAILIFF: 21CF3810, State of Wisconsin versus Chantille... KATHERINE FORREST: These tools are used in pretrial determinations, they're used in sentencing determinations, and they're used in housing determinations.

They're also used, importantly, in the plea bargaining phase.

They're also used, importantly, in the plea bargaining phase.

They're used really throughout the entire process to try to do what judges have been doing, to try to do what judges have been doing, which is the very, very difficult task of trying to understand and predict what will a human being do tomorrow, or the next day, what will a human being do tomorrow, or the next day, or next month, or three years from now.

ASSISTANT DISTRICT ATTORNEY: Bail forfeited.

He failed to appear 12/13/21.

Didn't even make it to preliminary hearing.

The software tools are an attempt to try to predict it better than humans can.

MICHELLE HAVAS: On count one, you're charged with felony intimidation of a victim.

SWEENEY: So, in the United States, you're innocent until you've been proven guilty, but you've been arrested.

Now that you've been arrested, a judge has to decide whether or not you get out on bail, or how high or low that bail should be.

You're charged with driving on a suspended license.

I've set that bond at $1,000.

No insurance, I've set that bond at $1,000.

ALISON SHAMES: One of the problems is, judges often are relying on money bond or financial conditions of release.

JUDGE: So I'm going to lower his fine to make it a bit more reasonable.

So instead of $250,000 cash, surety is $100,000.

SHAMES: It allows people who have access to money to be released.

If you are poor, you are often being detained pretrial.

Approximately 70% of the people in jail are there on pretrial.

Approximately 70% of the people in jail are there on pretrial.

These are people who are presumed innocent, but are detained during the pretrial stage of their case.

NARRATOR: Many jurisdictions use pretrial assessment algorithms NARRATOR: Many jurisdictions use pretrial assessment algorithms with a goal to reduce jail populations and decrease the impact of judicial bias.

SHAMES: The use of a tool like this takes historical data SHAMES: The use of a tool like this takes historical data and assesses, based on research, associates factors that are predictive of the two outcomes associates factors that are predictive of the two outcomes that the judge is concerned with.

That's community safety and whether that person will appear back in court during the pretrial period.

♪ ♪ NARRATOR: Many of these algorithms are based on a concept called a regression model.

The earliest, called linear regression, The earliest, called linear regression, dates back to 19th-century mathematics.

dates back to 19th-century mathematics.

O'NEIL: At the end of the day, machine learning algorithms do exactly what linear regression does, which is predict-- based on the initial conditions, the situation they're seeing-- predict what will happen in the future, whether that's, like, in the next one minute or the next four years.

NARRATOR: Throughout the United States, over 60 jurisdictions use predictive algorithms as part of the legal process.

use predictive algorithms as part of the legal process.

One of the most widely used is COMPAS.

The COMPAS algorithm weighs factors, including a defendant's answers to a questionnaire, to provide a risk assessment score.

to provide a risk assessment score.

These scores are used every day by judges to guide decisions These scores are used every day by judges to guide decisions about pretrial detention, bail, and even sentencing.

about pretrial detention, bail, and even sentencing.

But the reliability of the COMPAS algorithm has been questioned.

In 2016, ProPublica published an investigative report In 2016, ProPublica published an investigative report on the COMPAS risk assessment tool.

BENJAMIN: Investigators wanted to see if the scores were accurate BENJAMIN: Investigators wanted to see if the scores were accurate in predicting whether these individuals would commit a future crime.

And they found two things that were interesting.

One was that the score was remarkably unreliable One was that the score was remarkably unreliable in predicting who would commit a, a crime in the future over this two-year period.

But then the other thing that ProPublica investigators found But then the other thing that ProPublica investigators found was that Black people were much more likely to be deemed was that Black people were much more likely to be deemed high risk and white people low risk.

NARRATOR: This was true even in cases when the Black person was arrested for a minor offense and the white person in question was arrested for a more serious crime.

in question was arrested for a more serious crime.

BENJAMIN: This ProPublica study was one of the first to begin BENJAMIN: This ProPublica study was one of the first to begin to burst the bubble of technology as somehow objective and neutral.

NARRATOR: The article created a national controversy.

NARRATOR: The article created a national controversy.

But at Dartmouth, a student convinced her professor they should both be more than stunned.

HANY FARID: As it turns out, one of my students, Julia Dressel, HANY FARID: As it turns out, one of my students, Julia Dressel, reads the same article and said, "This is terrible.

We should do something about it."

(chuckles) This is the difference between an awesome idealistic student and a jaded, uh, professor.

And I thought, "I think you're right."

And as we were sort of struggling to understand the underlying roots of the bias in the algorithms, we asked ourselves a really simple question: are the algorithms today, are they doing better than humans?

Because presumably, that's why you have these algorithms, is that they eliminate some of the bias and the prejudices, either implicit or explicit, in the human judgment.

NARRATOR: To analyze COMPAS's risk assessment accuracy, they used the crowdsourcing platform Mechanical Turk.

Their online study included 400 participants Their online study included 400 participants who evaluated 1,000 defendants.

FARID: We asked participants to read a very short paragraph about an actual defendant.

How old they were, whether they were male or female, what their prior juvenile conviction record was, and their prior adult conviction record.

And, importantly, we didn't tell people their race.

And then we ask a very simple question, "Do you think this person will commit a crime in the next two years?"

Yes, no.

And again, these are non-experts.

These are people being paid a couple of bucks online to answer a survey.

No criminal justice experience, don't know anything about the defendants.

They were as accurate as the commercial software being used in the courts today, one particular piece of software.

That was really surprising.

We would've expected a little bit of improvement.

After all, the algorithm has access to huge amounts of training data.

NARRATOR: And something else puzzled the researchers.

The MTurk workers' answers to questions about who would commit crimes in the future and who wouldn't showed a surprising pattern of racial bias, even though race wasn't indicated in any of the profiles.

They were more likely to say a person of color will be high risk when they weren't, and they were more likely to say that a white person would not be high risk when in fact they were.

And this made no sense to us at all.

You don't know the race of the person.

How is it possible that you're biased against them?

(radios running in background) In this country, if you are a person of color, you are significantly more likely, historically, to be arrested, to be charged, and to be convicted of a crime.

So in fact, prior convictions is a proxy for your race.

Not a perfect proxy, but it is correlated, because of the historical inequities in the criminal justice system and policing in this country.

(siren blaring) MAN: It's my car, bro, come on, what are y'all doing?

Like, this, this is racial profiling.

NARRATOR: Research indicates a Black person is five times more likely to be stopped without cause than a white person.

Black people are at least twice as likely as white people to be arrested for drug offenses, even though Black and white people use drugs at the same rate.

Black people are also about 12 times more likely to be wrongly convicted of drug crimes.

FORREST: Historically, Black men have been arrested at higher levels FORREST: Historically, Black men have been arrested at higher levels than other populations.

Therefore, the tool predicts that a Black man, for instance, Therefore, the tool predicts that a Black man, for instance, will be arrested at a rate and recidivate at a rate that is higher than a white individual.

FARID: And so what was happening is, you know, the big data, the big machine learning folks are saying, "Look, we're not giving it race-- it can't be racist."

But that is spectacularly naive, because we know that other things correlate with race.

In this case, number of prior convictions.

And so when you train an algorithm on historical data, well, guess what.

It's going to reproduce history-- of course it will.

NARRATOR: Compounding the problem is the fact that predictive algorithms can't be put on the witness stand and interrogated about their decision-making processes.

and interrogated about their decision-making processes.

FORREST: Many defendants have had difficulty getting access to the underlying information getting access to the underlying information that tells them, what was the data set that was used to assess me?

What were the inputs that were used?

How were those inputs weighted?

So you've got what can be, these days, increasingly, a black box.

A lack of transparency.

NARRATOR: Some black box algorithms get their name from a lack of transparency about the code and data inputs they use, which can be deemed proprietary.

But that's not the only kind of black box.

A black box is any system which is so complicated that you can see what goes in and you can see what comes out, but it's impossible to understand what's going on inside it.

All of those steps in the algorithm are hidden inside phenomenally complex math are hidden inside phenomenally complex math and processes.

FARID: And I would argue that when you are using algorithms in mission-critical applications, like criminal justice system, we should not be deploying black box algorithms.

NARRATOR: PredPol, like many predictive platforms, claimed a proven record for crime reduction.

In 2015, PredPol published its algorithm In 2015, PredPol published its algorithm in a peer-reviewed journal.

William Isaac and Kristian Lum, research scientists who investigate predictive policing platforms, analyzed the algorithm.

ISAAC: We just kind of saw the algorithm as going back to the same one or two blocks every single time.

And that's kind of strange, because if you had a truly predictive policing system, you wouldn't necessarily see it going to the same locations over and over again.

NARRATOR: For their experiment, Isaac and Lum used a different data set, public health data, to map illicit drug use in Oakland.

public health data, to map illicit drug use in Oakland.

ISAAC: So, a good chunk of the city was kind of evenly distributed in terms of where potential illicit drug use might be.

But the police predictions were clustering around areas where police had, you know, historically found incidents of illicit drug use.

Specifically, we saw significant numbers of neighborhoods that were predominantly non-white and lower-income being deliberate targets of the predictions.

being deliberate targets of the predictions.

NARRATOR: Even though illicit drug use was a citywide problem, the algorithm focused its predictions on low-income neighborhoods and communities of color.

ISAAC: The reason why is actually really important.

It's very hard to divorce these predictions from those histories and legacies of over-policing.

As a result of that, they manifest themselves in the data.

As a result of that, they manifest themselves in the data.

NARRATOR: In an area where there is more police presence, more crime is uncovered.

The crime data indicates to the algorithm that the heavily policed neighborhood is where future crime will be found, even though there may be other neighborhoods where crimes are being committed at the same or higher rate.

ISAAC: Every new prediction that you generate is going to be increasingly dependent on the behavior of the algorithm in the past.

So, you know, if you go ten days, 20 days, 30 days into the future, right, after using an algorithm, all of those predictions have changed the behavior of the police department and are now being folded back into the next day's prediction.

NARRATOR: The result can be a feedback loop that reinforces historical policing practices.

SWEENEY: All of these different types of machine learning algorithms are all trying to help us figure out, are there some patterns in this data?

It's up to us to then figure out, are those legitimate patterns, do they, are they useful patterns?

Because the computer has no idea.

It didn't make a logical association.

It just made it, made a correlation.

MING: My favorite definition of artificial intelligence is, it's any autonomous system that can make decisions under uncertainty.

You can't make decisions under uncertainty without bias.

In fact, it's impossible to escape from having bias.

It's a mathematical reality about any intelligent system, even us.

about any intelligent system, even us.

(siren blaring in distance) NARRATOR: And even if the goal is to get rid of prejudice, bias in the historical data can undermine that objective.

Amazon discovered this when they began a search for top talent with a hiring algorithm whose training data depended on hiring successes from the past.

depended on hiring successes from the past.

MING: Amazon, somewhat famously within the A.I.

industry, MING: Amazon, somewhat famously within the A.I.

industry, they tried to build a hiring algorithm.

they tried to build a hiring algorithm.

They had a massive data set.

They had all the right answers, because they knew literally who got hired and who got that promotion in their first year.

(typing) NARRATOR: The company created multiple models to review past candidates' risumis and identify some 50,000 key terms.

and identify some 50,000 key terms.

MING: What Amazon actually wanted to achieve was to diversify their hiring.

Amazon, just like every other tech company, and a lot of other companies, as well, has enormous bias built into its hiring history.

has enormous bias built into its hiring history.

It was always biased, strongly biased, in favor of men, It was always biased, strongly biased, in favor of men, in favor, generally, of white or sometimes Asian men.

Well, they went and built a hiring algorithm.

And sure enough, this thing was the most sexist recruiter you could imagine.

the most sexist recruiter you could imagine.

If you said the word "women's" in your risumi, then it wouldn't hire you.

If you went to a women's college, it didn't want to hire you.

So they take out all the gender markers, and all of the women's colleges-- all the things that explicitly says, "This is a man," and, "This is a woman," or even the ones that, obviously, implicitly say it.

So they did that.

And then they trained up their new deep neural network to decide who Amazon would hire.

And it did something amazing.

It did something no human could do.

It figured out who was a woman and it wouldn't hire them.

It was able to look through all of the correlations that existed in that massive data set and figure out which ones most strongly correlated and figure out which ones most strongly correlated with someone getting a promotion.

And the single biggest correlate of getting a promotion was being a man.

And it figured those patterns out and didn't hire women.

NARRATOR: Amazon abandoned its hiring algorithm in 2017.

NARRATOR: Amazon abandoned its hiring algorithm in 2017.

Remember the way machine learning works, right?

It's like a student who doesn't really understand the material in the class.

They got a bunch of questions, they got a bunch of answers.

They got a bunch of questions, they got a bunch of answers.

And now they're trying to pattern match for a new question and say, "Oh, wait.

"Let me find an answer that looks pretty much like the questions and answers I saw before."

The algorithm only worked because someone has said, "Oh, this person whose data you have, "they were a good employee.

This other person was a bad employee," or, "This person performed well," or, "This person did not perform well."

O'NEIL: Because algorithms don't just look for patterns, they look for patterns of success, however it's defined.

But the definition of success is really critically important to what that end up, ends up being.

And a lot of, a lot of opinion is embedded in, what, what does success look like?

NARRATOR: In the case of algorithms, human choices play a critical role.

O'NEIL: The data itself was curated.

Someone decided what data to collect.

Somebody decided what data was not relevant, right?

And they don't exclude it necessarily intentionally-- they could be blind spots.

NARRATOR: The need to identify such oversights becomes more urgent as technology takes on more decision making.

as technology takes on more decision making.

♪ ♪ Consider facial recognition technology, used by law enforcement in cities around the world for surveillance.

In Detroit, 2018, law enforcement looked to facial recognition technology when $3,800 worth of watches were stolen from an upscale boutique.

Police ran a still frame from the shop's surveillance video through their facial recognition system to find a match.

through their facial recognition system to find a match.

How do I turn a face into numbers that equations can act with?

You turn the individual pixels in the picture of that face into values.

What it's really looking for are complex patterns across those pixels.

The sequence of taking a pattern of numbers and transforming it into little edges and angles, then transforming that into eyes and cheekbones and mustaches.

NARRATOR: To find that match, the system can be trained on billions of photographs.

Facial recognition uses a class of machine learning called deep learning.

The models built by deep learning techniques are called neural networks.

are called neural networks.

VENKATASUBRAMANIAN: A neural network is, you know, stylized as, you know, trying to model how neural pathways work in the brain.

You can think of a neural network as a collection of neurons.

as a collection of neurons.

So you put some values into a neuron, and if they're, sufficiently, they add up to some number, or they cross some threshold, this one will fire and send off a new number to the next neuron.

this one will fire and send off a new number to the next neuron.

NARRATOR: At a certain threshold, the neuron will fire to the next neuron.

If it's below the threshold, the neuron doesn't fire.

If it's below the threshold, the neuron doesn't fire.

This process repeats and repeats across hundreds, possibly thousands of layers, making connections like the neurons in our brain.

♪ ♪ The output is a predictive match.

Based on a facial recognition match, in January 2020, the police arrested Robert Williams in January 2020, the police arrested Robert Williams for the theft of the watches.

The next day, he was released.

Not only did Williams have an alibi, but it wasn't his face.

MING: To be very blunt about it, these algorithms are probably dramatically over-trained on white faces.

♪ ♪ ♪ ♪ So, of course, algorithms that start out bad can be improved, in general.

The Gender Shades project found that certain facial recognition technology, when they actually tested it on Black women, it was 65% accurate, whereas for white men, it was 99% accurate.

How did they improve it?

Because they did.

They built an algorithm that was trained on more diverse data.

So I don't think it's completely a lost cause to improve algorithms to be better.

MAN (in ad voiceover): I used to think my job was all about arrests.

MAN (in ad voiceover): I used to think my job was all about arrests.

NEDY: LESLIE KEN There was a commercial a few years ago that showed a police officer going to a gas station and then waiting for the criminal to show up.

MAN: We analyze crime data, spot patterns, and figure out where to send patrols.

They said, "Well, our algorithm will tell you exactly where the crime, the next crime is going to take place."

Well, that's just silly, uh, it, that's not how it works.

MAN: By stopping it before it happens.

(sighs) MAN: Let's build a smarter planet.

♪ ♪ JOEL CAPLAN: Understanding what it is about these places that enable crime problems to emerge and/or persist.

NARRATOR: At Rutgers University, the researchers who invented the crime mapping platform called Risk Terrain Modeling, or RTM, bristle at the term "predictive policing."

CAPLAN (voiceover): We don't want to predict, we want to prevent.

I worked as a police officer a long time ago, in the early 2000s.

Police collected data for as long as police have existed.

Police collected data for as long as police have existed.

Now there's a greater recognition that data can have value.

But it's not just about the data.

It's about how you analyze it, how you use those results.

There's only two data sets that risk terrain modeling uses.

These data sets are local, current information about crime incidents within a given area current information about crime incidents within a given area and information about environmental features that exist in that landscape, such as bars, fast food restaurants, convenience stores, schools, parks, alleyways.

KENNEDY: The algorithm is basically the relationship between these environmental features and the, the outcome data, which in this case is crime.

The algorithm provides you with a map of the distribution of the risk values.

ALEJANDRO GIME NEZ-SANTANA: This is the highest-risk area, on this commercial corridor on Bloomfield Avenue.

NARRATOR: But the algorithm isn't intended for use just by police.

NARRATOR: But the algorithm isn't intended for use just by police.

Criminologist Alejandro Giminez-Santana leads the Newark Public Safety Collaborative, a collection of 40 community organizations.

They use RTM as a diagnostic tool to understand not just where crime may happen next, to understand not just where crime may happen next, but why.

GIME NEZ-SANTANA: Through RTM, we identify this commercial corridor on Bloomfield Avenue, which is where we are right now, as a risky area for auto theft due to car idling.

So why is this space particularly problematic when it comes to auto theft?

One is because we're in a commercial corridor, where there's high density of people who go to the beauty salon or to go to a restaurant.

Uber delivery and Uber Eats, delivery people who come to grab orders that also, and leave their cars running create the conditions for this crime to be concentrated in this particular area.

What the data showed us was, there was a tremendous rise in auto vehicle thefts.

But we convinced the police department to take a more social service approach.

NARRATOR: Community organizers convinced police not to ticket idling cars, and let organizers create an effective public awareness poster campaign instead.

And we put it out to the Newark students to submit in this flyer campaign, and have their artwork on the actual flyer.

GIME NEZ-SANTANA: As you can see, this is the commercial corridor on Bloomfield Avenue.

The site score shows a six, which means that we are at the highest risk of auto theft in this particular location.

And as I move closer to the end of the commercial corridor, the site risk score is coming down.

NARRATOR: This is the first time in Newark that police data for crime occurrences have been shared widely with community members.

ELVIS PEREZ: The kind of data we share is incident-related data-- sort of time, location, that sort of information.

We don't discuss any private arrest information.

We're trying to avoid a crime.

NARRATOR: In 2019, Caplan and Kennedy formed a start-up at Rutgers to meet the rising demand for their technology.

Despite the many possible applications for RTM, from tracking public health issues to understanding vehicle crashes, law enforcement continues to be its principal application.

law enforcement continues to be its principal application.

Like any other technology, risk terrain modeling can be used for the public good when people use it wisely.

♪ ♪ ♪ ♪ We as academics and scientists, we actually need to be critical, because it could be the best model in the world, it could be very good predictions, but how you use those predictions matters, in some ways, even more.

REPORTER: The police department had revised the SSL numerous times... NARRATOR: In 2019, Chicago's inspector general contracted the RAND Corporation Chicago's inspector general contracted the RAND Corporation to evaluate the Strategic Subject List, the predictive policing platform that incorporated Papachristos's research on social networks.

that incorporated Papachristos's research on social networks.

PAPACHRISTOS: I never wanted to go down this path of who was the person that was the potential suspect.

And that problem is not necessarily with the statistical model, it's the fact that someone took victim and made him an offender.

You've criminalized someone who is at risk, that you should be prioritizing saving their life.

NARRATOR: It turned out that some 400,000 people were included on the SSL.

NARRATOR: It turned out that some 400,000 people were included on the SSL.

Of those, 77% were Black or Hispanic.

The inspector general's audit revealed that SSL scores were unreliable.

The Rand Corporation found the program had no impact on homicide or victimization rates.

on homicide or victimization rates.

(protesters chanting) NARRATOR: The program was shut down.

But data collection continues to be essential to law enforcement.

♪ ♪ O'NEIL: There are things about us that we might not even be aware of that are sort of being collected by the data brokers and will be held against us for the rest of our lives-- held against people forever, digitally.

NARRATOR: Data is produced and collected.

But is it accurate?

And can the data be properly vetted?

PAPACHRISTOS: And that was one of the critiques of not just the Strategic Subjects List, but the gang database in Chicago.

Any data source that treats data as a stagnant, forever condition Any data source that treats data as a stagnant, forever condition is a problem.

WOMAN: The gang database has been around for four years.

It'll be five in January.

We want to get rid of surveillance in Black and brown communities.

BENJAMIN: In places like Chicago, in places like L.A., where I grew up, there are gang databases with tens of thousands of people listed, their names listed in these databases.

Just by simply having a certain name and coming from a certain ZIP code could land you in these databases.

Do you all feel safe in Chicago?

DARRELL DACRES: The cops pulled up out of nowhere.

Didn't ask any questions, just immediately start beating on us.

Didn't ask any questions, just immediately start beating on us.

And basically were saying, like, what are, what are we doing over here, you know, like, in this, in this gangbang area?

I was already labeled as a gangbanger from that area because of where I lived.

I, I just happened to live there.

NARRATOR: The Chicago gang database is shared with hundreds of law enforcement agencies.

Even if someone is wrongly included, there is no mechanism to have their name removed.

there is no mechanism to have their name removed.

If you try to apply for an apartment, or if you try to apply for a job or a college, or even in a, um, a house, it will show or even in a, um, a house, it will show that you are in this record of a gang database.

I was arrested for peacefully protesting.

And they told me that, "Well, you're in the gang database."

But I was never in no gang.

MAN: Because you have a gang designation, you're a security threat group, right?

NARRATOR: Researchers and activists have been instrumental in dismantling some of these systems.

And so we continue to push back.

I mean, the fight is not going to finish until we get rid of the database.

♪ ♪ FERGUSON: I think what we're seeing now is not a move away from data.

It's just a move away from this term "predictive policing."

But we're seeing big companies, big tech, enter the policing space.

We're seeing the reality that almost all policing now is data-driven.

You're seeing these same police departments invest heavily in the technology, including other forms of surveillance technology, including other forms of databases to sort of manage policing.

(chanting): We want you out!

NARRATOR: More citizens are calling for regulations to audit algorithms and guarantee they're accomplishing what they promise without harm.

BRAYNE: Ironically, there is very little data on police use of big data.

And there is no systematic data at a national level on how these tools are used.

The deployment of these tools so far outpaces legal and regulatory responses to them.

What you have happening is essentially this regulatory Wild West.

O'NEIL: And we're, like, "Well, it's an algorithm, let's, let's just throw it into production."

Without testing it to whether it "works" sufficiently, um, at all.

NARRATOR: Multiple requests for comment from police agencies and law enforcement officials in several cities, including Chicago and New York, in several cities, including Chicago and New York, were either declined or went unanswered.

were either declined or went unanswered.

♪ ♪ ♪ ♪ Artificial intelligence must serve people, and therefore artificial intelligence must always comply with people's rights.

NARRATOR: The European Union is preparing to implement legislation NARRATOR: The European Union is preparing to implement legislation to regulate artificial intelligence.

In 2021, bills to regulate data science algorithms In 2021, bills to regulate data science algorithms were introduced in 17 states, and enacted in Alabama, Colorado, Illinois, and Mississippi.

SWEENEY: If you look carefully on electrical devices, you'll see "U.L.," for Underwriters Laboratory.

That's a process that came about so that things, when you plugged them in, didn't blow up in your hand.

That's the same kind of idea that we need in these algorithms.

O'NEIL: We can adjust it to make it better than the past, and we can do it carefully, and we can do it with, with precision in an ongoing conversation about what it means to us that it is, it's biased in the right way.

I don't think you remove bias, but you get to a bias that you can live with, that you, you think is moral.

To be clear, like, I, I think we can do better, but often doing better would look like we don't use this at all.

(radio running) FARID: There's nothing fundamentally wrong with trying to predict the future, as long as you understand how the algorithms are working, how are they being deployed.

What is the consequence of getting it right?

And most importantly is, what is the consequence of getting it wrong?

OFFICER: Keep your hands on the steering wheel!

MAN: My hands haven't moved off the steering wheel!

MAN 2: Are you gonna arrest me?

MAN 1: Officer, what are we here for?

OFFICER: We just want to talk with... ♪ ♪ ANNOUNCER: This program is available with PBS Passport and on Amazon Prime Video.

♪ ♪ ♪ ♪