Hypervisible: “UW researchers found that ChatGPT consistently ranked resumes with disability-related honors and credentials — such as the ‘Tom Wilson Disability Leadership Award’ — lower than the same resumes without those honors and credentials.”

“UW researchers found that ChatGPT consistently ranked resumes with disability-related honors and credentials — such as the ‘Tom Wilson Disability Leadership Award’ — lower than the same resumes without those honors and credentials.”

ChatGPT is biased against resumes with credentials that imply a disability — but it can improvewww.washington.edu UW researchers found that ChatGPT consistently ranked resumes with disability-related honors and credentials — such as the “Tom Wilson Disability Leadership Award” — lower than the same...

why did they write, "but it can improve." they should ask experts. this is journalistic malpractice.

Because it’s not a fatal flaw for AI when it comes to resume reviewing. It can and hopefully will improve. That’s not journalistic malpractice, that’s a big piece of the story.

'ai' can't "review resumes", so let's start there.

lol. Yes it can. I’ve been a recruiter for ten years and there are all sorts of tools available depending on how much you want AI to make decisions about resumes. aspenhr.com/ai-resume-sc...

Using AI For Resume Screening: What HR Teams Should Knowaspenhr.com AI resume screening offers many benefits but also comes with problems. Review pros and cons, and learn how to use AI for resume screening while avoiding the pitfalls.

if the computer is making the decisions you're not not the recruiter anymore.

bsky.app/profile/symb...

We'd all be better off if we strictly adhered to the truth of this:

A computer makes a decision when you answer whether you’re eligible to work in the country you applied to work in.

This is precisely why hiring is such a mess right now. ATS that try to make the lone recruiter handling 100s of reqs more productive, doing vastly more with less. AI is a marketing term nearly devoid of meaning now. Some tools can classify resumes using ML LLMs are not one. It's not how they work

The issue isn’t necessarily more reqs, it’s that you get way more applicants today. Hell, you can go on The Ladders and sign up for a tool that will apply to jobs for you. We have reached out to applicants who had no idea they applied for our job.

It can also get better at making it's bias harder to spot.

Second paragraph is the tell. ATS' are over 20yrs old and have had AI from the start for screening and reasoning. AI isn't a new additive in the space.

And ATS’ are one of the worst things to happen to the hiring process and keep getting worse by the year 🙃

ATS has not had “AI” for 20 years—it’s had biased, discriminatory statistics. Current chatbots do not have “AI”—they have language models, which are biased, discriminatory, and operate as a black box so you can’t know their full harm till they do it.

Maybe it's better in your mind to offshore some of the work, but as a degreed worker with technical skills, I can assure you humans making the decision are just as bad at hiring people with visible disability. Now they can just blame the algorithm which is clearly just as biased.

I’m literally just arguing that these tools exist. It’s bizarre that you’re speculating about what’s going on in my mind other than the argument I’m making.

ChatGPT isn't a "hiring AI", which are machine learning models trained on actual variables in resumes and outcomes. It is a LLM meant to complete a document. It actually scares the shit out of me that you can't tell the difference.

Also uh, if you do this you're feeding PII into the model which is an extreme breach of privacy.

Out of curiosity, is "the software ranked you *here*" a liability backstop when you get sued?

Guess what !?

Good question. The article clearly states people had to program the corrections. "But it can be improved" gives a more accurate take. Underscores the need for robust frequent auditing, which SHRM recommended a few yrs ago and moves towards the AI Bill of Rights. www.whitehouse.gov/ostp/ai-bill...

Blueprint for an AI Bill of Rights | OSTP | The White Housewww.whitehouse.gov Among the great challenges posed to democracy today is the use of technology, data, and automated systems in ways that threaten the rights of the American public. Too often, these tools are used to li...

Every govt intervention on the 'ai' menace has been actively unhelpful, written by the capitalists who created the problem. The correct response—we have to set fire to it all. shut down these systems of automated oppression so hard that they can never come back: ali-alkhatib.com/blog/fuck-up...

Destroy AIali-alkhatib.com I’ve been struggling to articulate this idea, and maybe the problem is that it’s actually kind of simple once you put it out there, and there’s really no good reason to unpack a whole case for it once...

Many (Schumer's SAFE) are born of corruption, others are corrupted by reg capture. One needs a broad def of capitalist to fit Alondra & Suresh. He's not wrong. "AI" is just the latest stage in the war on work, which in the West begins w/the Greeks, & that's fundamentally an assault on humanity.

I understand it is the latest stage in the war on workers. Firstly, it destroys all forms of education. It has to be eliminated.

Yes, a war on workers, but 1st it's a war on work - it's nature, how we relate to it, it's place/role in social structures, the language around it. It's so deeply ingrained in culture it's readily weaponized over centuries and generations without question and often with complicity of those harmed.

Guess What !?

Interesting and important work. I'm not convinced that asking LLMs to explain themselves provides actual insight into their decision-making process, but this is a second-order quibble.

It's like any other interaction with an LLM - what you get is the model's best guess at what a plausible answer would look like, with no checks that it's actually true. More a smooshed average of human explanations for similar behavior than any real self knowledge.

yeah at that point you're just asking the model to make up a reason someone might rank it lower. Understanding why LLMs do what they do is a famously hard problem.

Yeah, I’d be much more interested in the saliencey of the terms used in the making of the recommendations. What is the model looking at?

Using generative AI to scan resumes….More ways for companies to discriminate in hiring.

But now with no paper trail and no one to sue

Amazon was already sued for something very similar. They had an algorithm trained off successful resumes, but because the resumes they trained it on were all white men, the algorithm only accepted resumes that at least looked like they'd been written by white men.

If I recall correctly, it had 100% hiring rates for anyone who played Lacrosse or was named Jared

mitsloan.mit.edu/ideas-made-t...

Unmasking the bias in facial recognition algorithms | MIT Sloanmitsloan.mit.edu In her new book, computer scientist Joy Buolamwini examines the “power shadows” lurking in the datasets used to train artificial intelligence.

www.reuters.com/article/worl...

Insight - Amazon scraps secret AI recruiting tool that showed bias against womenwww.reuters.com Amazon.com Inc's <AMZN.O> machine-learning specialists uncovered a big problem: their new recruiting engine did not like women.

GDPR creates a right for someone subject to automated decision making to be given "meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject" and the right to obtain human intervention

Difficult to see how someone using an off the shelf LLM for sorting candidates can give meaningful information about the logic involved given that those things are basically black boxes

And if you ask a LLM to explain its reasoning you just get an answer-shaped response, not any real information about its processing because that is not only inscrutable but meaningless to the model

Americans do not have that and probably never will.

“yr resume is disabled, get well soon”

"not that we'll lift a finger to help"

But it can improve! (Yeah, and I can fly to the moon if I flap my arms hard enough)

Actually it's interesting to look at the advanced research techniques that got it to improve: "They instructed this chatbot to not exhibit ableist biases and instead work with disability justice and DEI principles." Huh. It's almost like that should be the default or something.

And yet here we have companies openly saying they will not apply those principles and a Supreme Court who will no longer be interested in allowing the government to pursue those few cases they bother to take up.

You just need more electricity, possibly more than could be supplied by a dedicated nuclear reactor

Post