In Screening for Suicide Risk, Facebook Takes On Tricky Public Health Role
Facebook uses its apps to track users it thinks could threaten employees and offices
- Facebook maintains a list of individuals that its security guards must "be on lookout" for that is comprised of users who've made threatening statements against the company on its social network as well as numerous former employees.
- The company's information security team is capable of tracking these individuals' whereabouts using the location data they provide through Facebook's apps and websites.
- More than a dozen former Facebook security employees described the company's tactics to CNBC, with several questioning the ethics of the company's practices.
CNBC.com
In early 2018, a Facebook user made a public threat on the social network against one of the company's offices in Europe.
Facebook picked up the threat, pulled the user's data and determined he was in the same country as the office he was targeting. The company informed the authorities about the threat and directed its security officers to be on the lookout for the user.
"He made a veiled threat that 'Tomorrow everyone is going to pay' or something to that effect," a former Facebook security employee told CNBC.
The incident is representative of the steps Facebook takes to keep its offices, executives and employees protected, according to more than a dozen former Facebook employees who spoke with CNBC. The company mines its social network for threatening comments, and in some cases uses its products to track the location of people it believes present a credible threat.
Several of the former employees questioned the ethics of Facebook's security strategies, with one of them calling the tactics "very Big Brother-esque."
Other former employees argue these security measures are justified by Facebook's reach and the intense emotions it can inspire. The company has 2.7 billion users across its services. That means that if just 0.01 percent of users make a threat, Facebook is still dealing with 270,000 potential security risks.
"Our physical security team exists to keep Facebook employees safe," a Facebook spokesman said in a statement. "They use industry-standard measures to assess and address credible threats of violence against our employees and our company, and refer these threats to law enforcement when necessary. We have strict processes designed to protect people's privacy and adhere to all data privacy laws and Facebook's terms of service. Any suggestion our onsite physical security team has overstepped is absolutely false."
Facebook is unique in the way it uses its own product to mine data for threats and locations of potentially dangerous individuals, said Tim Bradley, senior consultant with Incident Management Group, a corporate security consulting firm that deals with employee safety issues. However, the Occupational Safety and Health Administration's general duty clause says that companies have to provide their employees with a workplace free of hazards that could cause death or serious physical harm, Bradley said.
"If they know there's a threat against them, they have to take steps," Bradley said. "How they got the information is secondary to the fact that they have a duty to protect employees."
Facebook picked up the threat, pulled the user's data and determined he was in the same country as the office he was targeting. The company informed the authorities about the threat and directed its security officers to be on the lookout for the user.
"He made a veiled threat that 'Tomorrow everyone is going to pay' or something to that effect," a former Facebook security employee told CNBC.
The incident is representative of the steps Facebook takes to keep its offices, executives and employees protected, according to more than a dozen former Facebook employees who spoke with CNBC. The company mines its social network for threatening comments, and in some cases uses its products to track the location of people it believes present a credible threat.
Several of the former employees questioned the ethics of Facebook's security strategies, with one of them calling the tactics "very Big Brother-esque."
Other former employees argue these security measures are justified by Facebook's reach and the intense emotions it can inspire. The company has 2.7 billion users across its services. That means that if just 0.01 percent of users make a threat, Facebook is still dealing with 270,000 potential security risks.
"Our physical security team exists to keep Facebook employees safe," a Facebook spokesman said in a statement. "They use industry-standard measures to assess and address credible threats of violence against our employees and our company, and refer these threats to law enforcement when necessary. We have strict processes designed to protect people's privacy and adhere to all data privacy laws and Facebook's terms of service. Any suggestion our onsite physical security team has overstepped is absolutely false."
Facebook is unique in the way it uses its own product to mine data for threats and locations of potentially dangerous individuals, said Tim Bradley, senior consultant with Incident Management Group, a corporate security consulting firm that deals with employee safety issues. However, the Occupational Safety and Health Administration's general duty clause says that companies have to provide their employees with a workplace free of hazards that could cause death or serious physical harm, Bradley said.
"If they know there's a threat against them, they have to take steps," Bradley said. "How they got the information is secondary to the fact that they have a duty to protect employees."
Making the list
One of the tools Facebook uses to monitor threats
is a "be on lookout" or "BOLO" list, which is updated approximately once
a week. The list was created in 2008, an early employee in Facebook's
physical security group told CNBC. It now contains hundreds of people,
according to four former Facebook security employees who have left the
company since 2016.
Facebook notifies its security professionals anytime a new person is added to the BOLO list, sending out a report that includes information about the person, such as their name, photo, their general location and a short description of why they were added.
Facebook notifies its security professionals anytime a new person is added to the BOLO list, sending out a report that includes information about the person, such as their name, photo, their general location and a short description of why they were added.
In recent years, the security team even had a large
monitor that displayed the faces of people on the list, according to a
photo CNBC has seen and two people familiar, although Facebook says it
no longer operates this monitor.
Other companies keep similar lists of threats, Bradley and other sources said. But Facebook is unique because it can use its own products to identify these threats and track the location of people on the list.
Users who publicly threaten the company, its offices or employees — including posting threatening comments in response to posts from executives like CEO Mark Zuckerberg and COO Sheryl Sandberg — are often added to the list. These users are typically described as making "improper communication" or "threatening communication," according to former employees.
The bar can be pretty low. While some users end up on the list after repeated appearances on company property or long email threats, others might find themselves on the BOLO list for saying something as simple as "F--- you, Mark," "F--- Facebook" or "I'm gonna go kick your a--," according to a former employee who worked with the executive protection team. A different former employee who was on the company's security team said there were no clearly communicated standards to determine what kinds of actions could land somebody on the list, and that decisions were often made on a case-by-case basis.
The Facebook spokesman disputed this, saying that people were only added after a "rigorous review to determine the validity of the threat."
Other companies keep similar lists of threats, Bradley and other sources said. But Facebook is unique because it can use its own products to identify these threats and track the location of people on the list.
Users who publicly threaten the company, its offices or employees — including posting threatening comments in response to posts from executives like CEO Mark Zuckerberg and COO Sheryl Sandberg — are often added to the list. These users are typically described as making "improper communication" or "threatening communication," according to former employees.
The bar can be pretty low. While some users end up on the list after repeated appearances on company property or long email threats, others might find themselves on the BOLO list for saying something as simple as "F--- you, Mark," "F--- Facebook" or "I'm gonna go kick your a--," according to a former employee who worked with the executive protection team. A different former employee who was on the company's security team said there were no clearly communicated standards to determine what kinds of actions could land somebody on the list, and that decisions were often made on a case-by-case basis.
The Facebook spokesman disputed this, saying that people were only added after a "rigorous review to determine the validity of the threat."
Awkward situations
Most people on the list do not know they're on it. This sometimes leads to tense situations.
Several years ago, one Facebook user discovered he was on the BOLO list when he showed up to Facebook's Menlo Park campus for lunch with a friend who worked there, according to a former employee who witnessed the incident.
The user checked in with security to register as a guest. His name popped up right away, alerting security. He was on the list. His issue had to do with messages he had sent to Zuckerberg, according to a person familiar with the circumstances.
Soon, more security guards showed up in the entrance area where the guest had tried to register. No one grabbed the individual, but security guards stood at his sides and at each of the doors leading in and out of that entrance area.
Eventually, the employee showed up mad and demanded that his friend be removed from the BOLO list. After the employee met with Facebook's global security intelligence and investigations team, the friend was removed from the list — a rare occurrence.
"No person would be on BOLO without credible cause," the Facebook spokesman said in regard to this incident.
Several years ago, one Facebook user discovered he was on the BOLO list when he showed up to Facebook's Menlo Park campus for lunch with a friend who worked there, according to a former employee who witnessed the incident.
The user checked in with security to register as a guest. His name popped up right away, alerting security. He was on the list. His issue had to do with messages he had sent to Zuckerberg, according to a person familiar with the circumstances.
Soon, more security guards showed up in the entrance area where the guest had tried to register. No one grabbed the individual, but security guards stood at his sides and at each of the doors leading in and out of that entrance area.
Eventually, the employee showed up mad and demanded that his friend be removed from the BOLO list. After the employee met with Facebook's global security intelligence and investigations team, the friend was removed from the list — a rare occurrence.
"No person would be on BOLO without credible cause," the Facebook spokesman said in regard to this incident.
It's not just users who find themselves on
Facebook's BOLO list. Many of the people on the list are former Facebook
employees and contractors, whose colleagues ask to add them when they
leave the company.
Some former employees are listed for having a track record of poor behavior, such as stealing company equipment. But in many cases, there is no reason listed on the BOLO description. Three people familiar said that almost every Facebook employee who gets fired is added to the list, and one called the process "really subjective." Another said that contractors are added if they get emotional when their contracts are not extended.
The Facebook spokesman countered that the process is more rigorous than these people claim. "Former employees are only added under very specific circumstances, after review by legal and HR, including threats of violence or harassment."
The practice of adding former employees to the BOLO list has occasionally created awkward situations for the company's recruiters, who often reach out to former employees to fill openings. Ex-employees have showed up for job interviews only to find out that they couldn't enter because they were on the BOLO list, said a former security employee who left the company last year.
"It becomes a whole big embarrassing situation," this person said.
Some former employees are listed for having a track record of poor behavior, such as stealing company equipment. But in many cases, there is no reason listed on the BOLO description. Three people familiar said that almost every Facebook employee who gets fired is added to the list, and one called the process "really subjective." Another said that contractors are added if they get emotional when their contracts are not extended.
The Facebook spokesman countered that the process is more rigorous than these people claim. "Former employees are only added under very specific circumstances, after review by legal and HR, including threats of violence or harassment."
The practice of adding former employees to the BOLO list has occasionally created awkward situations for the company's recruiters, who often reach out to former employees to fill openings. Ex-employees have showed up for job interviews only to find out that they couldn't enter because they were on the BOLO list, said a former security employee who left the company last year.
"It becomes a whole big embarrassing situation," this person said.
Tracked by special request
Facebook has the capability to track BOLO users'
whereabouts by using their smartphone's location data collected through
the Facebook app, or their IP address collected through the company's
website.
Facebook only tracks BOLO-listed users when their threats are deemed credible, according to a former employee with firsthand knowledge of the company's security procedures. This could include a detailed threat with an exact location and timing of an attack, or a threat from an individual who makes a habit of attending company events, such as the Facebook shareholders' meeting. This former employee emphasized Facebook could not look up users' locations without cause.
When a credible threat is detected, the global security operations center and the global security intelligence and investigations units make a special request to the company's information security team, which has the capabilities to track users' location information. In some cases, the tracking doesn't go very far -- for instance, if a BOLO user made a threat about a specific location but their current location shows them nowhere close, the tracking might end there.
But if the BOLO user is nearby, the information security team can continue to monitor their location periodically and keep other security teams on alert.
Depending on the threat, Facebook's security teams can take other actions, such as stationing security guards, escorting a BOLO user off campus or alerting law enforcement.
Facebook only tracks BOLO-listed users when their threats are deemed credible, according to a former employee with firsthand knowledge of the company's security procedures. This could include a detailed threat with an exact location and timing of an attack, or a threat from an individual who makes a habit of attending company events, such as the Facebook shareholders' meeting. This former employee emphasized Facebook could not look up users' locations without cause.
When a credible threat is detected, the global security operations center and the global security intelligence and investigations units make a special request to the company's information security team, which has the capabilities to track users' location information. In some cases, the tracking doesn't go very far -- for instance, if a BOLO user made a threat about a specific location but their current location shows them nowhere close, the tracking might end there.
But if the BOLO user is nearby, the information security team can continue to monitor their location periodically and keep other security teams on alert.
Depending on the threat, Facebook's security teams can take other actions, such as stationing security guards, escorting a BOLO user off campus or alerting law enforcement.
Facebook's information security team has tracked users' locations in other safety-related instances, too.
In 2017, a Facebook manager alerted the company's security teams when a group of interns she was managing did not log into the company's systems to work from home. They had been on a camping trip, according to a former Facebook security employee, and the manager was concerned about their safety.
Facebook's information security team became involved in the situation and used the interns' location data to try and find out if they were safe. "They call it 'pinging them', pinging their Facebook accounts," the former security employee recalled.
After the location data did not turn up anything useful, the information security team then kept digging and learned that the interns had exchanged messages suggesting they never intended to come into work that day — essentially, they had lied to the manager. The information security team gave the manager a summary of what they had found.
"There was legit concern about the safety of these individuals," the Facebook spokesman said. "In each isolated case, these employees were unresponsive on all communication channels. There's a set of protocols guiding when and how we access employee data when an employee goes missing."
In 2017, a Facebook manager alerted the company's security teams when a group of interns she was managing did not log into the company's systems to work from home. They had been on a camping trip, according to a former Facebook security employee, and the manager was concerned about their safety.
Facebook's information security team became involved in the situation and used the interns' location data to try and find out if they were safe. "They call it 'pinging them', pinging their Facebook accounts," the former security employee recalled.
After the location data did not turn up anything useful, the information security team then kept digging and learned that the interns had exchanged messages suggesting they never intended to come into work that day — essentially, they had lied to the manager. The information security team gave the manager a summary of what they had found.
"There was legit concern about the safety of these individuals," the Facebook spokesman said. "In each isolated case, these employees were unresponsive on all communication channels. There's a set of protocols guiding when and how we access employee data when an employee goes missing."
Safety first
While the company is aggressive about dealing with
potential threats, the risks are real. Just in recent weeks, Facebook
had to deal with a with bomb threat against the company's Menlo Park
campus and with an employee getting "swatted" -- that's when an attacker
calls in a false emergency to get police to send an armed SWAT team to
somebody's home, a prank with potentially fatal results.
One person pointed to an incident in 2015 where the BOLO list was essential. Facebook's security teams recognized the license plate of a suspicious car that was loitering on the company's campus, said a former Facebook physical security employee who left the company in 2016.
The Facebook security guards kept watch on the individual until Menlo Park Police Department officers showed up, the former employee said.
They eventually arrested the driver on charges of indecent exposure for public masturbation, according to a public records request confirming the incident.
WATCH: Here's how to see which apps have access to your Facebook data — and cut them off
One person pointed to an incident in 2015 where the BOLO list was essential. Facebook's security teams recognized the license plate of a suspicious car that was loitering on the company's campus, said a former Facebook physical security employee who left the company in 2016.
The Facebook security guards kept watch on the individual until Menlo Park Police Department officers showed up, the former employee said.
They eventually arrested the driver on charges of indecent exposure for public masturbation, according to a public records request confirming the incident.
WATCH: Here's how to see which apps have access to your Facebook data — and cut them off
Salvador RodriguezTech Reporter for CNBC.com
Facebook prefers to recruit from Google over Twitter employees
Searching publicly available information across the web has revealed some interesting insights about Facebook’s workforce.
The U.S. government and Facebook are negotiating a record, multibillion-dollar fine for the company’s privacy lapses
The
Federal Trade Commission and Facebook are negotiating over a
multi-billion dollar fine that would settle the agency’s investigation
into the social media giant’s privacy practices, according to two people
familiar with the probe.
The fine would be the
largest the agency has ever imposed on a technology company, but the two
sides have not yet agreed on an exact amount. Facebook has expressed
initial concern with the FTC’s demands, one of the people said. If talks
break down, the FTC could take the matter to court in what would likely
be a bruising legal fight.
Facebook confirmed
it is in discussions with the agency but declined to comment further.
The FTC declined to comment. The two people familiar with the probe
spoke on the condition of anonymity because they were not authorized to
discuss the private talks.
A
multi-billion dollar fine would amount to a reckoning for Facebook in
the United States after a series of privacy lapses that may have put the
personal information of its users at risk. Lawmakers have faulted the
company for mishandling that data while failing to crack down on other
digital ills, including the rise of online hate speech and the spread of
disinformation from Russian operatives and other foreign actors.
“Facebook
faces a moment of reckoning and the only way it will come is through an
FTC order with severe penalties and other sanctions that stop this kind
of privacy misconduct going forward,” said Democratic Sen Richard
Blumenthal (Conn.).
For the FTC, a significant
punishment levied against Facebook could represent a new era of scrutiny
for Silicon Valley companies after years of privacy missteps. To date,
the largest fine the FTC has imposed on a tech giant for breaking an
agreement with the government to safeguard consumers’ data was a $22.5 million penalty that Google paid to settle a probe over in 2012.
“It
is an open question at this moment in time whether the Federal Trade
Commission is an effective privacy agency, and it is also an open
question as to whether the FTC is willing to use its current authority
to safeguard consumer privacy in the United States,” said Marc
Rotenberg, the executive director of the Electronic Privacy Information
Center.
With a steep fine and other penalties, Rotenberg said it “would indicate the FTC is now prepared to enforce its consent orders.”
The FTC’s probe of Facebook began in March
of last year in response to reports about the social giant’s
entanglement with Cambridge Analytica, a political consultancy that
improperly accessed data on 87 million of the social site’s users. The
agency’s inquiry focuses on whether Facebook’s conduct — along with a
series of additional privacy mishaps made public in recent months —
amount to violations of a 2011 agreement Facebook brokered with the FTC
to improve its privacy practices. Facebook has maintained it did not
breach that accord.
The FTC agreement stipulated that Facebook had to
be more transparent and notify users in a clearer way before it shares
personal data with third parties. The order also barred Facebook from
deceiving users about its privacy practices, and it instituted regular
checkups on the way it uses data. Under FTC rules, the agency can seek
steep fines determined in part by the number of times a company violates such an order.
Facebook
could broker a deal with the U.S. government by agreeing to pay a fine
and make some changes to its business practices. That settlement would
then have to be approved by a judge. The FTC’s punishment could include a
new order that could force the tech giant to submit to tougher checkups
to ensure it is complying with the settlement, according to two other
people familiar with the probe but not authorized to discuss it
publicly.
Alternatively, Facebook could choose
to fight the federal agency over its findings and proposed punishments.
If that battle lands in federal court, the move could prove bruising to
both sides, analysts say, by putting Facebook’s top executives on a
witness stand while subjecting the agency’s authority over tech giants
to high-profile judicial review.
But Facebook
could face significant reputation risk if it decided to fight the FTC
fine. “They’re hemorrhaging users, they’re hemorrhaging trust, and I
think this would only exacerbate the problem,” said Justin Brookman, the
director of consumer privacy and technology policy for Consumer
Reports.
Last
year, Facebook said it would contest a small fine levied by regulators
in the United Kingdom last year over its entanglement with Cambridge
Analytica. The social-media giant is also battling back a lawsuit filed
by the attorney general of the District of Columbia that contends
Facebook misled its users about its data-collection practices. A slew of
other attorneys general in states including New York, Pennsylvania and
California have previously said they are investigating Facebook.
Adding
to the pressure in Washington, a collection of consumer advocates urged
the FTC last month to penalize Facebook aggressively with “substantial
fines,” perhaps exceeding $2 billion, along with an order that limits
how and when Facebook collects data about its users
“The
company’s business practices have imposed enormous costs on the privacy
and security of Americans, children and communities of color, and the
health of democratic institutions in the United States and around the
world,” wrote groups led by EPIC, which filed the original complaint
leading to the FTC’s 2011 settlement.
Lawmakers also have pressed the FTC to speed up its work
and penalize Facebook nearly a year after it first announced its
investigation. “When Americans’ privacy is breached, they deserve a
speedy and effective response,” wrote Blumenthal and Sen. Edward J.
Markey (D-Mass.) in a letter in January.
Facebook Settlement With Law Enforcement Officers Could Run Into the Billions
By Natasha Singer
Earlier
that day, a local woman wrote a Facebook post saying she was walking
home and intended to kill herself when she got there, according to a
police report on the case. Facebook called to warn the Police Department
about the suicide threat.
The
officer who took the call quickly located the woman, but she denied
having suicidal thoughts, the police report said. Even so, the officer
believed she might harm herself and told the woman that she must go to a
hospital — either voluntarily or in police custody. He ultimately drove
her to a hospital for a mental health work-up, an evaluation prompted
by Facebook’s intervention. (The New York Times withheld some details of
the case for privacy reasons.)
Police stations from Massachusetts to Mumbai
have received similar alerts from Facebook over the last 18 months as
part of what is most likely the world’s largest suicide threat screening
and alert program. The social network ramped up the effort after
several people live-streamed their suicides on Facebook Live in early
2017. It now utilizes both algorithms and user reports to flag possible
suicide threats.
Facebook’s rise as a
global arbiter of mental distress puts the social network in a tricky
position at a time when it is under investigation for privacy lapses by
regulators in the United States, Canada and the European Union — as well
as facing heightened scrutiny for failing to respond quickly to election interference and ethnic hatred campaigns on its site. Even as Facebook’s chief executive, Mark Zuckerberg, has apologized for improper harvesting of user data, the company grappled last month with fresh revelations about special data-sharing deals with tech companies.
Advertisement
The
anti-suicide campaign gives Facebook an opportunity to frame its work
as a good news story. Suicide is the second-leading cause of death among
people ages 15 to 29 worldwide, according to the World Health Organization.
Some mental health experts and police officials said Facebook had aided
officers in locating and stopping people who were clearly about to harm
themselves.
Facebook has computer
algorithms that scan the posts, comments and videos of users in the
United States and other countries for indications of immediate suicide
risk. When a post is flagged, by the technology or a concerned user, it
moves to human reviewers at the company, who are empowered to call local
law enforcement.
“In
the last year, we’ve helped first responders quickly reach around 3,500
people globally who needed help,” Mr. Zuckerberg wrote in a November post about the efforts.
But
other mental health experts said Facebook’s calls to the police could
also cause harm — such as unintentionally precipitating suicide,
compelling nonsuicidal people to undergo psychiatric evaluations, or
prompting arrests or shootings.
WASHINGTON
— Facebook and the Federal Trade Commission are discussing a settlement
over privacy violations that could amount to a record,
multibillion-dollar fine, according to three people with knowledge of
the talks.
The company and the
F.T.C.’s consumer protection and enforcement staff have been in
negotiations over a financial penalty for claims that Facebook violated a
2011 privacy consent decree with the agency, said the people, who spoke
on the condition of anonymity because the investigation is private.
In
2011, Facebook promised a series of measures to protect user privacy
after an investigation found it had harmed consumers with its handling
of user data.
The current talks have
not yet reached the F.T.C.’s five commissioners for a vote and it is
unclear how close the two sides are to wrapping up the nearly 11-month
investigation. The commissioners met in mid-December and were updated by
staff members that they had at that point found considerable evidence
of violations of the 2011 consent decree.
The
F.T.C. began its investigation into Facebook’s mishandling of data
after The New York Times reported in March 2018 that the information of
87 million users had been harvested by a British political consulting firm, Cambridge Analytica, without their permission.
Facebook confirmed the negotiations with the F.T.C., which could still break down and lead to litigation. The discussions were first reported by The Washington Post.
You have 4 free articles remaining.
Subscribe to The Times
Facebook’s
2011 consent decree requires the company to seek permission from users
for plans to share their data with third parties. The trade commission
also requires Facebook to notify it when third parties misuse this data.
Some
F.T.C. officials have pressed for maximum penalties because of several
new reports of potential privacy breaches since the start of the
investigation. The agency can seek up to $41,000 for each violation
found by the agency. In the case of Cambridge Analytica, 87 million
people were affected.
The highest
financial penalty imposed by the F.T.C. so far was a $22.5 million fine
on Google for violating an agreement to protect consumer data.
Continued
news reports of data violations, including a Times report about data
sharing between Facebook and partner tech companies in December, have
raised concerns among F.T.C. officials of setting the fine too low,
according to a person familiar with the discussions.
The
agency is under pressure to show teeth in its Facebook investigation,
and the company’s missteps over the past two years have made it a target
for lawmakers. Members of Congress have complained that the F.T.C. is
lagging behind European regulators that have taken a tougher stand with
internet companies.
No comments:
Post a Comment