Judge’s footnote on immigration agents using AI raises accuracy and privacy concerns

The Department of Homeland Security did not respond to requests for comment, and it was unclear if the agency had guidelines or policies on the use of AI by agents. The body camera footage cited in the order has not yet been released.

Adams said few departments have put policies in place, but those that have often prohibit the use of predictive AI when writing reports justifying law enforcement decisions, especially use-of-force reports. Courts have established a standard referred to as objective reasonableness when considering whether a use of force was justified, relying heavily on the perspective of the specific officer in that specific scenario.

“We need the specific articulated events of that event and the specific thoughts of that specific officer to let us know if this was a justified use of force,” Adams said. “That is the worst case scenario, other than explicitly telling it to make up facts, because you’re begging it to make up facts in this high-stakes situation.”

Private information and evidence

Besides raising concerns about an AI-generated report inaccurately characterizing what happened, the use of AI also raises potential privacy concerns.

Katie Kinsey, chief of staff and tech policy counsel at the Policing Project at NYU School of Law, said if the agent in the order was using a public ChatGPT version, he probably didn’t understand he lost control of the images the moment he uploaded them, allowing them to be part of the public domain and potentially used by bad actors.

Kinsey said from a technology standpoint most departments are building the plane as it’s being flown when it comes to AI. She said it’s often a pattern in law enforcement to wait until new technologies are already being used and in some cases mistakes being made to then talk about putting guidelines or policies in place.

“You would rather do things the other way around, where you understand the risks and develop guardrails around the risks,” Kinsey said. “Even if they aren’t studying best practices, there’s some lower hanging fruit that could help. We can start from transparency.”

Kinsey said while federal law enforcement considers how the technology should be used or not used, it could adopt a policy like those put in place in Utah or California recently, where police reports or communications written using AI have to be labeled.

Careful use of new tools

The photographs the officer used to generate a narrative also caused accuracy concerns for some experts.

Well-known tech companies like Axon have begun offering AI components with their body cameras to assist in writing incident reports. Those AI programs marketed to police operate on a closed system and largely limit themselves to using audio from body cameras to produce narratives because the companies have said programs that attempt to use visuals are not effective enough for use.

“There are many different ways to describe a color, or a facial expression or any visual component. You could ask any AI expert and they would tell you prompts return very different results between different AI applications, and that gets complicated with a visual component,” said Andrew Guthrie Ferguson, a law professor at George Washington University Law School.

“There’s also a professionalism questions. Are we OK with police officers using predictive analytics?” he added. “It’s about what the model thinks should have happened, but might not be what actually happened. You don’t want it to be what ends up in court, to justify your actions.”