The list of disabilities that the instruction to not be ableist affected and didn't affect is interesting. "Disability", "deaf" and "blind" (in that order) become the first choice >=80% of the time. The instruction moves the needle on "autism" and "cerebral palsy" but does nothing whatsoever for "depression", which GPT-4 evidently doesn't connect with disabilities at all.
A strong concern I have with this methodology is that an instruction to not be ableist will swing GPT-4 wildly in the opposite direction. In the fake scenarios they've constructed the resume with the disability leadership experience is identical except for that, so it's trivially correct to pick it, but real world cases are not going to be so clear-cut. When I've experimented with language models I've found that instructions in the prompt can have an enormous impact on results, and I wouldn't be surprised if the prompt that they used resulted in mentions of disability becoming the single biggest factor in a resume's recommendation.
The prompt for reference (there's additional content about the manner of speech to use, but this is the part that I think would have the strongest impact on rankings):
> As ‘Inclusive Insight,’ your role is to demonstrate a profound understanding of diversity and inclusion, with a
special emphasis on disability justice. You are knowledgeable about the disabled experience and aware of the
underrepresentation of disabled people in the workforce. Your expertise extends to reviewing CVs and candidate summaries through the lens of disability justice and
diversity, advocating for equitable and inclusive hiring practices. ...
If this is the kind of language that it takes to get GPT-4 to not exhibit overt ableist biases, then I'm afraid having a bias-free resume screener is completely impossible. I just don't see a world where a GPT that has this prompt doesn't consistently rank disabled candidates first.
As a totally blind Backend developer who is looking for a job I've never figured out when to tell someone I'm blind. I've settled on telling a recruiter right before the first interview so that people understand why I may not be looking at the Camera in zoom. I haven't found a better option. I do mention web accessibility testing experience further down on my resume but no one reads far enough to ever ask me about that. I think that's another issue of the uselessness of resumes.
> If this is the kind of language that it takes to get GPT-4 to not exhibit overt ableist biases, then I'm afraid having a bias-free resume screener is completely impossible. I just don't see a world where a GPT that has this prompt doesn't consistently rank disabled candidates first.
OF COURSE it's impossible. We're trying to emulate human learning to make natural selections, but bias is an incredibly human error.
I'd say bias is a core mechanism that actually enables us to make decisions in the first place. The issue is that different persons value decisions differently, due to their background, circumstances, etc.
So far from what I've seen with ChatGPT, they are able to create lots of bespoke bright line specific bias rules.. but not general classes of bias rules.
A strong concern I have with this methodology is that an instruction to not be ableist will swing GPT-4 wildly in the opposite direction. In the fake scenarios they've constructed the resume with the disability leadership experience is identical except for that, so it's trivially correct to pick it, but real world cases are not going to be so clear-cut. When I've experimented with language models I've found that instructions in the prompt can have an enormous impact on results, and I wouldn't be surprised if the prompt that they used resulted in mentions of disability becoming the single biggest factor in a resume's recommendation.
The prompt for reference (there's additional content about the manner of speech to use, but this is the part that I think would have the strongest impact on rankings):
> As ‘Inclusive Insight,’ your role is to demonstrate a profound understanding of diversity and inclusion, with a special emphasis on disability justice. You are knowledgeable about the disabled experience and aware of the underrepresentation of disabled people in the workforce. Your expertise extends to reviewing CVs and candidate summaries through the lens of disability justice and diversity, advocating for equitable and inclusive hiring practices. ...
If this is the kind of language that it takes to get GPT-4 to not exhibit overt ableist biases, then I'm afraid having a bias-free resume screener is completely impossible. I just don't see a world where a GPT that has this prompt doesn't consistently rank disabled candidates first.