Yikes! NYC's Automated Employment Decision Tool Law Hailed and Criticized

Thursday - July 13 2023, 15:22 UTC - 1 year ago

New York City has implemented a law which requires employers to tell candidates if they use AI in the hiring process. Employers will also have to submit to annual independent audits to prove that their systems are not biased towards race, gender, or ability. The regulation has been both hailed and criticized by various stakeholders, including business, civil rights groups, and public interest groups.

The use of AI in hiring has been criticized for the way it automates and entrenches existing racial and gender biases. AI systems that evaluate candidates’ facial expressions and language have been shown to prioritize white, male, and abled-bodied candidates. The problem is massive, and many companies use AI at least once during the hiring process. US Equal Employment Opportunity Commission chair Charlotte Burrows said in a meeting in January that as many as four out of five companies use automation to make employment decisions.

NYC’s Automated Employment Decision Tool law, which came into force on Wednesday, 14 July 2023, says that employers who use AI in hiring have to tell candidates they are doing so. They will also have to submit to annual independent audits to prove that their systems are not racist or sexist. Candidates will be able to request information from potential employers about what data is collected and analyzed by the technology. Violations will result in fines of up to $1,500.

Proponents of the law say that it’s a good start toward regulating AI and mitigating some of the harms and risks around its use, even if it’s not perfect. It requires that companies better understand the algorithms they use and whether the technology unfairly discriminates against women or people of color. It’s also a fairly rare regulatory success when it comes to AI policy in the US, and we’re likely to see more of these specific, local regulations. Sounds sort of promising, right? .

But the law has been met with significant controversy. Public interest groups and civil rights advocates say it isn’t enforceable and extensive enough, while businesses that will have to comply with it argue that it’s impractical and burdensome.Groups like the Center for Democracy & Technology, the Surveillance Technology Oversight Project (S.T.O.P.), the NAACP Legal Defense and Educational Fund, and the New York Civil Liberties Union argue that the law is "underinclusive" and risks leaving out many uses of automated systems in hiring, including systems in which AI is used to screen thousands of candidates.

What’s more, it’s not clear exactly what independent auditing will achieve, as the auditing industry is currently so immature. BSA, an influential tech trade group whose members include Adobe, Microsoft, and IBM, filed comments to the city in January criticizing the law, arguing that third-party audits are "not feasible." .

"There’s a lot of questions about what type of access an auditor would get to a company’s information, and how much they would really be able to interrogate about the way it operates," says Albert Fox Cahn, executive director of S.T.O.P. "It would be like if we had financial auditors, but we didn’t have generally accepted accounting principles, let alone a tax code and auditing rules." .

Cahn argues that the law could produce a false sense of security and safety about AI and hiring. "This is a fig leave held up as proof of protection from these systems when in practice, I don’t think a single company is going to be held accountable because this was put into law," he says.

Importantly, the mandated audits will have to evaluate “all aspects” of the potential AI, including data, algorithms, and outputs. This means that companies will have to open up their inner workings and give insight into how they are selecting job applicants.