[music]

Brigid Bergin: You're listening to The Takeaway. I'm Brigid Bergin in for Melissa Harris-Perry. Child welfare agencies across the country are notoriously underfunded, understaffed, and overburdened. The pandemic has only made things worse with workers leaving their jobs in droves. In Kentucky, Governor Andy Beshear raised salaries by 10% in order to try to retain workers when some 600 left the system last year, many were higher-paying lower stress jobs.

Marta Miranda-Straub: These folks can work for target and Chick-fil-A and McDonald's for double what the state pays them. When they have to have a bachelor's degree or a master's degree, go in people's homes, had a gun put to their head, take away kids who are burned from a family who's so desperate that that's how they're acting out.

Brigid Bergin: That was Marta Miranda-Straub, commissioner of Kentucky's Department of Community-based Services speaking with WLKY. Some states have adopted algorithmic screening tools to help identify cases that are likely to be the highest risk using data to try to help prioritize cases for workers and systems that are stretched to the breaking point. Agencies in at least 26 states plus DC have considered using algorithms and at least 11 states are currently using them. Developers say such tools can help welfare officials with large caseloads save children from neglect and even combat racism and bias in human decision-making, but many say it's doing exactly the opposite.

Allegheny County, Pennsylvania adopted a tool that was designed to predict the risk a child would be placed in foster care in the two years after they were investigated. It used historic data to calculate a child's potential risk of abuse or neglect and assigned that child a score from 1 to 20. The higher the score, the higher the risk. Research from a team at Carnegie Mellon first reported by the AP earlier this year found that these algorithmic decisions are simply reinforcing the structural inequities they were adopted to address.

The tool in Allegheny County was actually flagging two-thirds of black children for investigation, as opposed to just one-half of all other children. The county's social workers themselves disagreed with the algorithm scores at least one-third of the time. Joining us is Anjana Samanth, senior attorney for the ACLU Women's Rights Project. She previously served as assistant attorney general in the Civil Rights Bureau of the New York State Attorney General's office. Thank you so much for joining us.

Anjana Samantha: Thank you for having me.

Brigid Bergin: How does bias end up showing up in these systems?

Anjana Samantha: Well, in the simplest sense, what these tools are doing, the screening tools, is generating a risk score for complaints that are received. The way the tool is created is that the tool designers looked at an agency's historical cases, and then look to identify the characteristics that keep coming up again, again, in those past cases. Identify then among those characteristics which ones are associated with different outcomes and which characteristics may show up more often than others, and then basically try to predict what the odds of some future event is based on how often those characteristics are now coming in or are appearing in complaints.

I think one of the things to really keep in mind is that even though they're called risk assessment tools or risk prediction tools, one question to ask is risk of what? Because no one can predict the future. No math equation or computer program can actually tell us when violence or death is going to happen. While the tools are described as predicting the risk of harm to children or risk of maltreatment, what they're really measuring is the relative odds of something happening that is being used as a proxy for harm.

In the case of a lot of the call screening tools, the proxy, the something that the tool is predicting is actually the odds that a child in the complaint is going to be removed from their parents by the agency within two years. Here's where one of the problems lies. Removing a child from the home is not a clear or even necessarily likely indicator that the child has been abused. According to the most recent federal compilation of data, 61% of all cases in which agencies substantiated maltreatment allegations involved only neglect. While the definition of neglect varies from one place to the next, at minimum, it's basically something along the lines of an act or failure to act that creates an imminent risk of serious harm.

Brigid Bergin: There's so much to unpack here. Just slow us down a bit. It sounds like when you think about the types of data that are being gathered, that this algorithm is likely to target a pretty specific group of people, is that fair?

Anjana Samantha: The tools are created using a data that the government has in its possession. They will pull on to find what characteristics kept recurring in agencies' historical files. They'll look not only at the child welfare agencies' demographic or breakdowns of those files, but they'll also then cross-reference that with the county or state's criminal justice data, so juvenile probation, jail records. They may also cross-reference it with behavioral health system data.

People who are using public services snap, some places have looked at use of public benefits. People who are basically coming into contact with the state for reliance on the social network, everyone's whose data the government is going to have more readily at their fingertips. These are people who are disproportionately poor, experiencing poverty, experiencing housing instability, and certainly are predominantly disproportionately Black, depending on what parts of the country you are in. Also, either Latinx or Native American.

Brigid Bergin: When the algorithm determines a child's case to be high risk, what happens next? What does the agency do?

Anjana Samantha: When a higher-end risk score comes up, the call screen worker then takes that number into account, at least this is how it's supposed to be used, to decide whether or not the complaint should be referred into the agency for some investigation or safety check, or if they should just close out the complaint. It's not replacing human decision-making. Although in some places where these scores are used, if you fall on, let's say, the highest end of the score spectrum, then you're supposed to automatically get screened in for some response in order to override that mandatory screening. Supervisor has to give some explanation.

Brigid Bergin: As agencies are using these tools, they say these algorithms aren't about making decisions. As you said, they're simply tools to provide overworked case workers with guidance. They say they're being trained to recognize these biases. What do you say to that?

Anjana Samantha: One of the problems with the tools for everyone, whether it's case workers or the general public is just the lack of information we have about what actually is the tool measuring? What factors is it taking into account? Are there any factors that it's prohibited from using in trying to predict the odds that a child is going to be removed? There's really no way to insulate for that bias entirely.

I think the real concern is because this tool is going through historical records, and we know for a fact that both within the child welfare system statistically Black families, Native American families in particular are disproportionately overrepresented and have been then white counterparts, for instance. These files and these statistics, these risk scores are faking in that disproportionality. They're assuming that that baseline of skewed participation is neutral or natural.

Brigid Bergin: How did these tools get put into place? What was Allegheny County trying to fix here?

Anjana Samantha: I'd say, as you started out at the top, one of the issues that I think a lot of the jurisdictions that have adopted these tools are talking about is we want to do a better job of sifting through the complaints that come in to find "real cries" for help or the real abuse as opposed to situations where agency intervention may not be required. I think the problem with that is, again, the risk that's being measured by the tool is not one of harm to the child, but it's actually what are the odds that this kid is going to fit the profile of the type of situation where we the agency would have intervened in the past? It becomes a cycle.

It's not necessarily that these state interventions happen every time a child has been abused or harmed. Rather, again, children are often removed because of things like a child doesn't have a safe place to stay, maybe they're living in a shelter, maybe they don't have access to regular meals, or their family isn't able to get to do laundry every day, and then they get flagged for not being able to provide and meet the basic needs of their child. Again, what this tool is measuring is how closely does a complaint match situations in which the agency has intervened in the past.

Brigid Bergin: The recent reporting on these systems has prompted some agencies to reconsider their use. Oregon announced at the beginning of June that it will stop using its algorithm system but Allegheny County has so far not stopping its use and has even implemented a new surveillance program recently. Why do you think we're seeing these different responses?

Anjana Samantha: I think it's a variety of factors. One definitely, in some places is stakeholder opposition. Where people and communities, particularly communities who are impacted by it are informed and are voicing opposition, that's one reason to certainly reassess whether you want to be using this tool, to begin with. If the families that are directly impacted are telling you, "Look, refining whether or not I get called in isn't the problem.

Maybe actually, the interventions aren't useful." I think another reason is Illinois, for instance, used to use a risk assessment tool that they ended up dropping about five years ago. In public statements, they stated that we felt that the tool was no better at predicting harm than our caseworkers. That was part of the reason that they dropped the tool. It's also is can be expensive to use these tools properly. This isn't a one and done, installed some software, and then you can move on.

These tools are they are "machine learning tools", so if you want to keep them up to date on what do the current case files look like, you do have to constantly be looking at your more recent case files and feeding that information to tool designers or into the tools so that I can see if the constellation of factors that flag befores continues to be the constellation of factors today. I'm sorry. I was going to say that's where you end up with something that people have called a feedback loop, and that's the concern. You're going to keep feeding it this closed universe of information with the same demographics, the same criteria, and that's just going to compound itself.

Brigid Bergin: We're going to have to leave it there. Anjana Samant is senior attorney for the ACLU Women's Rights Project. Thanks so much for joining us today.

Anjana Samantha: Thank you for having me.

Brigid Bergin: We reached out to Allegheny County's Department of Human Services for comment. A statement they provided said in part, the algorithm was "designed not to replace the worker, but rather to provide additional data that supplements other data they receive to help them screen referrals. We stand by our efforts to be transparent and accountable.

We will continue to seek disconfirming views and criticism as opportunities to improve outcomes for vulnerable children and their families." Joining us now to explore this further is Nico'Lee Biddle, a licensed clinical social worker and child welfare reform advocate and lived experience expert based in Pittsburgh, Pennsylvania. Nico'Lee currently works for the Centre for Study of Social Policy as a senior program analyst on their System's Change team. Welcome to The Takeaway, Nico'Lee.

Nico'Lee Biddle: Hello, Bridget. Thanks for having me.

Brigid Bergin: Prior to your work at the Centre for the Study of Social Policy, you worked in Family Services. Can you tell us how families were identified as at-risk prior to the implementation of any kind of algorithms?

Nico'Lee Biddle: Sure. I worked in various child welfare private agencies in Pittsburgh, Pennsylvania. Basically, those are the agencies that provide the foster care and adoption services for Allegheny County,. How its families would be identified would be pretty typical. Most times, a hotline report would be sent or made via phone or online about concerns about a child and family, and then the county receives that report, and then they decide on their own to do an investigation or not based on whatever the allegation was.

Brigid Bergin: What happens to children and their families once they're flagged for an investigation.

Nico'Lee Biddle: Well, in Pennsylvania, it depends on what the allegation is, whether they get tracked into like a general protective services, or a child protective services, based on that report. Child protective services would be very, very severe neglect or allegations of child abuse. Then the general protective services is a track that's more focused on allegations of neglect, truancy, stuff like that, from my understanding.

Brigid Bergin: We're talking about these algorithms this morning. Can you talk us through the pros and/or cons of using these algorithms to support social workers in the field?

Nico'Lee Biddle: Sure. I mean, the pros are that, from what I've been told, is that they can save a lot of time for workers and that they can also provide the workers with additional information in order for them to make a more informed decision. That's really the argument that Allegheny County uses for using the algorithm that they do, is that it provides their intake workers or the workers who are sifting through those hotline calls more information in order to make a better decision on whether or not that child line report needs investigated.

Cons with that would be that it feels very minority reportish. What I mean by that is that in Allegheny County, they have a really unique and I think good system in that everything is under this one umbrella of the Department of Human Services. That means it includes things like child protective services, behavioral health, their child care programs. All of these things are under one umbrella agency which helps to keep things more connected, but the caveat whenever using something like this is that this algorithm is able to pull all of this data from all of these different DHS systems.

With that, they're using those to make some type of determination about whether involvement in those various systems and services means that that's how is more likely to experience child abuse or neglect and come in contact with the child welfare system throughout their lifetime. That's really where the problems come in because that's where there's a lot of secrecy. That's where there are a lot of judgments and biases, because although it's an algorithm evaluating this information, these are algorithms designed by real people, and these are also algorithms pulling from data, that source from systems that are not without bias. It gets very sketchy and the consequences can be really dire for families who fall victim to being investigated.

Brigid Bergin: We know that this is certainly an imperfect system. You experienced the foster care system as a teenager, can you tell us about your own experience?

Nico'Lee Biddle: I did experience foster care as a teenager. I was in the system for about seven years before aging out. This was years ago at this point and so this was before the algorithm or anything like that. The thing that concerns me, especially now as a person who I've been able to acquire a lot of privilege. I have a good education and a good job and the house and a car and all those good things that we don't necessarily see for a lot of former foster youth. What I'm concerned of is that there's data in that DHS warehouse on me and on my time of being in foster care and the services that I had to use as I transitioned out of care. Let's say that there's a childlike report made against me that, I don't know, my child showed up to school dirty or something.  I'm using a really simplistic example here. Let's say that that happened and that the call hotline worker that they received this algorithm score that says, "Oh, this person has a higher score because they were in foster care or because they were on Medicaid when they were 23 or just whatever it says in there. We don't know what the impact that is on the score. For me as a former foster youth is the system rating that I'm at a higher risk of having my child in the system because of that.

Brigid Bergin: It sounds families aren't even aware that their data is being collected. Is that right?

Nico'Lee Biddle: I think in a broad sense, yes. I think that, generally speaking, people know that they fill out forms and they have to jump through a lot of hoops to get some of these other services. I think what they don't realize is how that data is then being used. In some ways, it might be being used biased against them.

Brigid Bergin: Do you think families will be less likely to seek support if they know that this information might be used against them in the future?

Nico'Lee Biddle: Oh, absolutely. We encourage people to use the services that are there, but the reality is that in cases like these, where we're not sure if that access to that service is going to count against them because they are a lower income family, for example, and that means that they're going to be more likely to be involved in child welfare in Allegheny County's eyes or any other system size, that's really problematic. People think that it is really hard to have your children removed from you. In a lot of cases that's simply not true.

Brigid Bergin: What are some of the structural and systemic issues that could drive these child services departments to seek out and use these kinds of automated tracking systems?

Nico'Lee Biddle: The child welfare systems really across the board tend to be underfunded. They tend to be understaffed in that even if they have a full amount of staff, that staff keeps turning over. They're really trying to make fewer resources stretch, and they're trying to look for some way to save some type of time and some type of stress on their workers. They're also looking for a way-- child welfare, historically again, across the board is a lot of times, they're lacking that true data analysis and data that can really help inform decisions.

Trying to build data and trying to make sure you're collecting a lot of data so that you can improve the system is not a bad thing, but more data in and of itself is also not inherently a good thing depending on how that data is being used. They're using these things to try to save some time so that they can hopefully make their resources stretch a bit further.

Brigid Bergin: Nico’Lee Biddle is the senior program analyst at the Center for the Study of Social Policy. Thank you so much for joining us today.

Nico'Lee Biddle: Thank you.

Copyright © 2022 New York Public Radio. All rights reserved. Visit our website terms of use at www.wnyc.org for further information.

New York Public Radio transcripts are created on a rush deadline, often by contractors. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of New York Public Radio’s programming is the audio record.