Imagine your search terms, key-strokes, private chats and photographs are being monitored every time they are sent. Millions of students across the country don’t have to imagine this deep surveillance of their most private communications: it’s a reality that comes with their school districts’ decision to install AI-powered monitoring software such as Gaggle and GoGuardian on students’ school-issued machines and accounts. As we demonstrated with our own Red Flag Machine, however, this software flags and blocks websites for spurious reasons and often disproportionately targets disadvantaged, minority and LGBTQ youth.

The companies making the software claim it’s all done for the sake of student safety: preventing self-harm, suicide, violence, and drug and alcohol abuse. While a noble goal, given that suicide is the second highest cause of death among American youth 10-14 years old, no comprehensive or independent studies have shown an increase in student safety linked to the usage of this software. Quite to the contrary: a recent comprehensive RAND research study shows that such AI monitoring software may cause more harm than good.

That study also found that how to respond to alerts is left to the discretion of the school districts themselves. Due to a lack of resources to deal with mental health, schools often refer these alerts to law enforcement officers who are not trained and ill-equipped to deal with youth mental crises. When police respond to youth who are having such episodes, the resulting encounters can lead to disastrous results. So why are schools still using the software–when a congressional investigation found a need for “federal action to protect students’ civil rights, safety, and privacy”? Why are they trading in their students’ privacy for a dubious-at-best marketing claim of safety?

Experts suggest it’s because these supposed technical solutions are easier to implement than the effective social measures that schools often lack resources to implement. I spoke with Isabelle Barbour, a public health consultant who has experience working with schools to implement mental health supports. She pointed out that there are considerable barriers to families, kids, and youth accessing health care and mental health supports at a community level. There is also a lack of investment in supporting schools to effectively address student health and well-being. This leads to a situation where many students come to school with needs that have been unmet and these needs impact the ability of students to learn. Although there are clear and proven measures that work to address the burdens youth face, schools often need support (time, mental health expertise, community partners, and a budget) to implement these measures. Edtech companies market largely unproven plug-and-play products to educational professionals who are stretched thin and seeking a path forward to help kids. Is it any wonder why schools sign contracts which are easy to point to when questioned about what they are doing with regard to the youth mental health epidemic?

One example: Gaggle in marketing to school districts claims to have saved 5,790 student lives between 2018 and 2023, according to shaky metrics they themselves designed. All the while they keep the inner-workings of their AI monitoring secret, making it difficult for outsiders to scrutinize and measure its effectiveness.

We give Gaggle an “F”

Reports of the errors and inability of the AI flagging to understand context keep popping up. When the Lawrence, Kansas school district signed a $162,000 contract with Gaggle, no one batted an eye: It joined a growing number of school districts (currently ~1,500) nation-wide using the software. Then, school administrators called in nearly an entire class to explain photographs Gaggle’s AI had labeled as “nudity” because the software wouldn’t tell them:

“Yet all students involved maintain that none of their photos had nudity in them. Some were even able to determine which images were deleted by comparing backup storage systems to what remained on their school accounts. Still, the photos were deleted from school accounts, so there is no way to verify what Gaggle detected. Even school administrators can’t see the images it flags.”

Young journalists within the school district raised concerns about how Gaggle’s surveillance of students impacted their privacy and free speech rights. As journalist Max McCoy points out in his article for the Kansas Reflector, “newsgathering is a constitutionally protected activity and those in authority shouldn’t have access to a journalist’s notes, photos and other unpublished work.” Despite having renewed Gaggle’s contract, the district removed the surveillance software from the devices of student journalists. Here, a successful awareness campaign resulted in a tangible win for some of the students affected. While ad-hoc protections for journalists are helpful, more is needed to honor all students’ fundamental right to privacy against this new front of technological invasions.

Tips for Students to Reclaim their Privacy

Students struggling with the invasiveness of school surveillance AI may find some reprieve by taking measures and forming habits to avoid monitoring. Some considerations:

• Consider any school-issued device a spying tool. 
• Don’t try to hack or remove the monitoring software unless specifically allowed by your school: it may result in significant consequences from your school or law enforcement. 
• Instead, turn school-issued devices completely off when they aren’t being used, especially while at home. This will prevent the devices from activating the camera, microphone, and surveillance software.
• If not needed, consider leaving school-issued devices in your school locker: this will avoid depending on these devices to log in to personal accounts, which will keep data from those accounts safe from prying eyes.
• Don’t log in to personal accounts on a school-issued device (if you can avoid it – we understand sometimes a school-issued device is the only computer some students have access to). Rather, use a personal device for all personal communications and accounts (e.g., email, social media). Maybe your personal phone is the only device you have to log in to social media and chat with friends. That’s okay: keeping separate devices for separate purposes will reduce the risk that your data is leaked or surveilled. 
• Don’t log in to school-controlled accounts or apps on your personal device: that can be monitored, too. 
• Instead, create another email address on a service the school doesn’t control which is just for personal communications. Tell your friends to contact you on that email outside of school.

Finally, voice your concern and discomfort with such software being installed on devices you rely on. There are plenty of resources to point to, many linked to in this post, when raising concerns about these technologies. As the young journalists at Lawrence High School have shown, writing about it can be an effective avenue to bring up these issues with school administrators. At the very least, it will send a signal to those in charge that students are uncomfortable trading their right to privacy for an elusive promise of security.

Schools Can Do Better to Protect Students Safety and Privacy

It’s not only the students who are concerned about AI spying in the classroom and beyond. Parents are often unaware of the spyware deployed on school-issued laptops their children bring home. And when using a privately-owned shared computer logged into a school-issued Google Workspace or Microsoft account, a parent’s web search will be available to the monitoring AI as well.

New studies have uncovered some of the mental detriments that surveillance causes. Despite this and the array of First Amendment questions these student surveillance technologies raise, schools have rushed to adopt these unproven and invasive technologies. As Barbour put it: 

“While ballooning class sizes and the elimination of school positions are considerable challenges, we know that a positive school climate helps kids feel safe and supported. This allows kids to talk about what they need with caring adults. Adults can then work with others to identify supports. This type of environment helps not only kids who are suffering with mental health problems, it helps everyone.”

We urge schools to focus on creating that environment, rather than subjecting students to ever-increasing scrutiny through school surveillance AI.

Reposted from the EFF’s Deeplinks blog.

In an indictment against Defend the Atlanta Forest activists in Georgia, state prosecutors are citing use of encrypted communications to fearmonger. Alleging the defendants—which include journalists and lawyers, in addition to activists—in the indictment were responsible for a number of crimes related to the Stop Cop City campaign, the state Attorney General’s prosecutors cast suspicion on the defendants’ use of Signal, Telegram, Tor, and other everyday data-protecting technologies.

“Indeed, communication among the Defend the Atlanta Forest members is often cloaked in secrecy using sophisticated technology aimed at preventing law enforcement from viewing their communication and preventing recovery of the information” the indictment reads. “Members often use the dark web via Tor, use end-to-end encrypted messaging app Signal or Telegram.”

The secure messaging app Signal is used by tens of millions of people, and has hundreds of millions of global downloads. In 2021, users moved to the nonprofit-run private messenger en masse as concerns were raised about the data-hungry business models of big tech. In January of that year, former world’s richest man Elon Musk tweeted simply “Use Signal.” And world-famous NSA whistle-blower Edward Snowden tweeted in 2016 what in information security circles would become a meme and truism: “Use Tor. Use Signal.”

Despite what the bombastic language would have readers believe, installing and using Signal and Tor is not an initiation rite into a dark cult of lawbreaking. The “sophisticated technology” being used here are apps that are free, popular, openly distributed, and widely accessible by anyone with an internet connection. Going further, the indictment ascribes the intentions of those using the apps as simply to obstruct law enforcement surveillance. Taking this assertion at face value, any judge or reporter reading the indictment is led to believe everyone using the app simply wants to evade the police. The fact that these apps make it harder for law enforcement to access communications is exactly because the encryption protocol protects messages from everyone not intended to receive them—including the users’ ISP, local network hackers, or the Signal nonprofit itself.

Elsewhere, the indictment hones in on the use of anti-surveillance techniques to further its tenuous attempts to malign the defendants: “Most ‘Forest Defenders’ are aware that they are preparing to break the law, and this is demonstrated by premeditation of attacks.” Among a laundry list of other techniques, the preparation is supposedly marked by “using technology avoidance devices such as Faraday bags and burner phones.” Stoking fears around the use of anti-surveillance technologies sets a dangerous precedent for all people who simply don’t want to be tracked wherever they go. In protest situations, carrying a prepaid disposable phone can be a powerful defense against being persecuted for participating in first-amendment protected activities. Vilifying such activities as the acts of wrongdoers would befit totalitarian societies, not ones in which speech is allegedly a universal right.

To be clear, prosecutors have apparently not sought to use court orders to compel either the defendants or the companies named to enter passwords or otherwise open devices or apps. But vilifying the defendants’ use of common sense encryption is a dangerous step in cases that the Dekalb County District Attorney has already dropped out of, citing “different prosecutorial philosophies.”

Using messengers which protect user communications, browsers which protect user anonymity, and employing anti-surveillance techniques when out and about are all useful strategies in a range of situations. Whether you’re looking into a sensitive medical condition, visiting a reproductive health clinic with the option of terminating a pregnancy, protecting trade secrets from a competitor, wish to avoid stalkers or abusive domestic partners, protecting attorney-client exchanges, or simply want to keep your communications, browsing, and location history private, these techniques can come in handy. It is their very effectiveness which has led to the widespread adoption of privacy-protective technologies and techniques. When state prosecutors spread fear around the use of these powerful techniques, this sets us down a dangerous path where citizens are more vulnerable and at risk.

Republished from the EFF’s Deeplinks blog.

On February 8, François-Philippe Champagne, the Canadian Minister of Innovation, Science and Industry, announced Canada would ban devices used in keyless car theft. The only device mentioned by name was the Flipper Zero—the multitool device that can be used to test, explore, and debug different wireless protocols such as RFID, NFC, infrared, and Bluetooth.

While it is useful as a penetration testing device, Flipper Zero is impractical in comparison to other, more specialized devices for car theft. It’s possible social media hype around the Flipper Zero has led people to believe that this device offers easier hacking opportunities for car thieves*. But government officials are also consuming such hype. That leads to policies that don’t secure systems, but rather impedes important research that exposes potential vulnerabilities the industry should fix. Even with Canada walking back on the original statement outright banning the devices, restricting devices and sales to “move forward with measures to restrict the use of such devices to legitimate actors only” is troublesome for security researchers.

This is not the first government seeking to limit access to Flipper Zero, and we have explained before why this approach is not only harmful to security researchers but also leaves the general population more vulnerable to attacks. Security researchers may not have the specialized tools car thieves use at their disposal, so more general tools come in handy for catching and protecting against vulnerabilities. Broad purpose devices such as the Flipper have a wide range of uses: penetration testing to facilitate hardening of a home network or organizational infrastructure, hardware research, security research, protocol development, use by radio hobbyists, and many more. Restricting access to these devices will hamper development of strong, secure technologies.

When Brazil’s national telecoms regulator Anatel refused to certify the Flipper Zero and as a result prevented the national postal service from delivering the devices, they were responding to media hype. With a display and controls reminiscent of portable video game consoles, the compact form-factor and range of hardware (including an infrared transceiver, RFID reader/emulator, SDR and Bluetooth LE module) made the device an easy target to demonize. While conjuring imagery of point-and-click car theft was easy, citing examples of this actually occurring proved impossible. Over a year later, you’d be hard-pressed to find a single instance of a car being stolen with the device. The number of cars stolen with the Flipper seems to amount to, well, zero (pun intended). It is the same media hype and pure speculation that has led Canadian regulators to err in their judgment to ban these devices.

Still worse, law enforcement in other countries have signaled their own intentions to place owners of the device under greater scrutiny. The Brisbane Times quotes police in Queensland, Australia: “We’re aware it can be used for criminal means, so if you’re caught with this device we’ll be asking some serious questions about why you have this device and what you are using it for.” We assume other tools with similar capabilities, as well as Swiss Army Knives and Sharpie markers, all of which “can be used for criminal means,” will not face this same level of scrutiny. Just owning this device, whether as a hobbyist or professional—or even just as a curious customer—should not make one the subject of overzealous police suspicions.

It wasn’t too long ago that proficiency with the command line was seen as a dangerous skill that warranted intervention by authorities. And just as with those fears of decades past, the small grain of truth embedded in the hype and fears gives it an outsized power. Can the command line be used to do bad things? Of course. Can the Flipper Zero assist criminal activity? Yes. Can it be used to steal cars? Not nearly as well as many other (and better, from the criminals’ perspective) tools. Does that mean it should be banned, and that those with this device should be placed under criminal suspicion? Absolutely not.

We hope Canada wises up to this logic, and comes to view the device as just one of many in the toolbox that can be used for good or evil, but mostly for good.

*Though concerns have been raised about Flipper Devices’ connection to the Russian state apparatus, no unexpected data has been observed escaping to Flipper Devices’ servers, and much of the dedicated security and pen-testing hardware which hasn’t been banned also suffers from similar problems.

Originally posted to the EFF Deeplinks blog.