When Social Sentinel representatives pitched their service to Florida’s Gulf Coast State College in 2018, they billed it as an innovative way to find threats of suicides and shootings posted online. But for the next two years, the service found nothing dangerous.
One tweet notified the school about a nearby fishing tournament: “Check out the picture of some of the prizes you can win – like the spear fishing gun.”
Another quoted the lyrics from a hit pop song from 2010: “Can we pretend that airplanes in the night sky are like shooting stars? I could really use a wish right now.”
As police and administrators fielded a flood of alerts about posts that seemed to pose no threat, the company told the school in emails that it had eliminated more than half of all irrelevant alerts. Months later, they said the number had decreased by 80%. By January 2019, the company told schools its service flagged 90% fewer irrelevant posts.
But at Gulf Coast, the problem continued.
One alert from March 2019 read, “Hamburger Helper only works if the hamburger is ready to accept that it needs help.”
“Nothing ever came up there that was actionable on our end,” David Thomasee, the executive director of operations at Gulf Coast, said in an interview earlier this year. The college stopped using the service in April 2021.
Gulf Coast was not the only college inundated with irrelevant alerts. Officials from 12 other colleges raised concerns about the performance of Social Sentinel in interviews and emails obtained by The Dallas Morning News and the Investigative Reporting Program at UC Berkeley’s Graduate School of Journalism.
Only two of the 13, North Central Texas College and the University of W Connecticut, still use the service.
As schools and universities confront a worsening mental health crisis and an epidemic of mass shootings, Social Sentinel offers an attractive and low-cost way to keep students safe. But experts say the service also raises questions about whether the potential benefits are worth the tradeoffs on privacy.
Records show Social Sentinel has been used by at least 38 colleges in the past seven years, including four in North Texas. The total number is likely far higher — The company’s co-founder wrote in an email that hundreds of colleges in 36 states used Social Sentinel.
The News also analyzed more than 4,200 posts flagged by the service to four colleges from November 2015 to March 2019. None seem to contain any imminent, serious threat of violence or self-harm, according to a News
analysis, which included all of the posts obtained through public records requests.Some schools contacted by The News said the service alerted them to students struggling with mental health issues. Those potential success stories were outweighed by complaints that the service flagged too many irrelevant tweets, interviews and emails between officials show. None of the schools could point to a student whose life was saved because of the service.
[…]
For one former Social Sentinel employee, it only took three days before they had serious doubts about the effectiveness of the service.
The worker estimated that 99.9% of the flagged posts sent to clients were not threatening. The service often crashed because it flagged too many posts. At least 40% of clients dropped the service every year, the employee said.
Over the course of several months, the employee repeatedly raised concerns with supervisors and fellow employees about flaws in the system, but those complaints were often ignored, the worker said.
The employee, who asked not to be named for fear of retribution, said problems with the service were an open secret at the company, and described it as “snake oil” and “smoke and mirrors.”
The News also contacted more than two dozen other former company employees, who either did not respond or said they had signed nondisclosure agreements preventing them from speaking publicly about their time at the company.
At the University of Texas at Dallas, which started using the service in 2018, campus police officers in charge of the service also grew increasingly skeptical of its performance, emails obtained through a records request show.
“Does the company have any data (not anecdotal) to show its success rate in mitigating harm or disaster through its alert system?” UT Dallas Police Lieutenant Adam Perry asked his chief in an email obtained by The News. The chief forwarded the email to a company employee who didn’t answer the question.
Perry said that while the school used the service, the technology never alerted police to legitimate threats of suicide or shootings.
“I think in concept, it’s not a bad program,” Perry said. “I just think they need to work on distinguishing what a real threat is.” UT Dallas ended its use of the service last year.
Ed Reynolds, police chief at the University of North Texas, defended the system, but also estimated that “99.9 percent (of the alerts) were messages we didn’t need to do anything with.” After using the service for about three years, UNT ended its contract with the company in November 2018.
As noted before, the Uvalde school district was among the ISDs in Texas that have used Social Sentinel. Putting my cybersecurity hat on for a minute, there are similar services in that space that do provide good value, but they have been around longer, there’s far more data on cyber threats, and it’s much easier to configure alerts for these services to very specific things, which greatly reduces the noise factor. I do think a service like this could be useful, but what we have now is not mature enough. More data and more analysis to help eliminate likely false positives before they show up in a customer’s alert feed are needed. Even with that, it’s still likely to be noisy and to require fulltime human analysis to get value out of it. For now, the best use of this is probably for academics. After they’ve had some time with it, then school districts and colleges might make use of it.