Yes, Facebook Scans Users' Private Conversations Looking For Sexual Predators and Child Porn

A report from Reuters this week has the world in a tizzy over the fact that Facebook has algorithms to detect whether sexual predators are trying to pick up kids on the site. This wasn’t news to me. Facebook’s chief of security Joe Sullivan and I discussed it when I profiled him for the magazine earlier this year. From that report:

In another incident, proactive policing by Facebook’s security team led to a potential pedophile being fingered. The site employs algorithms to detect suspicious behavior and bring it to the attention of Sullivan’s group. “We found that a youth pastor and children’s sports coach in Indiana was using fake accounts to try to engage with kids on our site,” he says. “So we called the FBI in Indiana and sent them his information.”
While a youth pastor reaching out to young people doesn’t seem particularly nefarious, Sullivan suggests that his use of fake accounts to do so, as well as the content of his communications, was disturbing enough to warrant police involvement.
Users may often forget they are constantly watched on the site, if not by actual people, then by algorithms. Last year Facebook adopted a Microsoft program called PhotoDNA, which scans every picture uploaded to the site to see if it matches known child porn images compiled by the FBI’s National Crime Information Center. “Our list of child porn images is actually much longer than the FBI’s,” says Sullivan. “Every time we find something new—through a user report or flagging on a keyword—we manually review the user album to see if there are other images that should be added to the list, and then we add them to our library. We’re exploring how to share our library with others.”
For years the site has had back-end algorithms to weed out fake spam accounts and monitor kid-adult interactions. Lotharios, beware: “If you’re sending friend requests that trend to 80% female, that’s a red flag, or if you change your birth date a lot—under and above the 18 threshold,” says Sullivan. “Our site integrity team has built engines to feed in characteristics, and they start hunting people down. When you have single concrete rules, it’s easy for people to figure them out, but with machine learning, it’s evolving all the time.”
via Facebook’s Top Cop: Joe Sullivan – Forbes.

The Reuters report further details the meaning of “machine learning:” Facebook “uses the records of convicted pedophiles’ online chats to teach the software what to seek out.”
The Constitution protects us against unreasonable searches by the feds, requiring them, for example, to get a search warrant from a judge to invade our digital homes (just as they do for our physical ones). But when it comes to our privacy rights from the companies that store our data? That’s far more complicated.