Facebook Screws Up Again

from the please-make-it-stop dept

Another day, another Facebook privacy scandal.

This time around, a "senior Facebook employee" has informed security expert Brian Krebs that Facebook has been storing the passwords of "hundreds of millions" of Facebook (and Instagram) users in plain text (aka unencrypted). This is a fundamental security error that no company should ever make, yet it's been a pretty common occurrence for tech companies where security and privacy are commonly seen as an afterthought. According to Krebs, the passwords were accessible to around 20,000 Facebook employees for the better part of the last decade:

"The Facebook source said the investigation so far indicates between 200 million and 600 million Facebook users may have had their account passwords stored in plain text and searchable by more than 20,000 Facebook employees. The source said Facebook is still trying to determine how many passwords were exposed and for how long, but so far the inquiry has uncovered archives with plain text user passwords in them dating back to 2012.

My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords."

On the "plus side," this latest scandal is slightly less terrible than past scandals like the Cambridge Analytica fracas. In those instances, the scandals made it clear Facebook routinely viewed consumer privacy as a distant afterthought as it looked to monetize every brain fart of its userbase. In this case, insiders told Motherboard that this does appear to have been a bug, and that the majority of Facebook passwords are usually encrypted:

"A current Facebook employee told Motherboard that "it sucks."

"Obviously we don’t store them in plaintext ‘normally,’" the employee, who has a technical role, told Motherboard. "Logged in plaintext in some unique weird cases we found and fixed and are talking about." Motherboard granted multiple sources in this story anonymity to speak more candidly about a security incident.

"It should’ve never happened," they said.

Still, given Facebook's resources and the volume of security talent they have on staff, the fact that it happened at all is grossly embarrassing. The scandal comes right on the heels of Facebook's other recent scandals -- like its cavalier sharing of user health and real estate data -- and is only compounding a scandal-ridden 2018 for the company. Krebs stated that as many as 600 million of the company's 2.7 billion users could be affected by the company's latest screw up, though, thus far, Facebook has yet to notify any of the impacted users.

Facebook was quick to issue a blog post amusingly entitled "keeping passwords secure," before confirming that Facebook failed to do precisely that. Throughout the post Facebook's Pedro Canahuati downplays the scope of the threat, while remaining somewhat murky on how many people were actually impacted:

"To be clear, these passwords were never visible to anyone outside of Facebook and we have found no evidence to date that anyone internally abused or improperly accessed them. We estimate that we will notify hundreds of millions of Facebook Lite users, tens of millions of other Facebook users, and tens of thousands of Instagram users. Facebook Lite is a version of Facebook predominantly used by people in regions with lower connectivity."

Given that this data was available to 20,000 employees over a period of roughly seven years, the claim that they've found "no evidence" of abuse should be of cold comfort. The company, meanwhile, continued to insist that consumer privacy is among its top priorities:

"In the course of our review, we have been looking at the ways we store certain other categories of information — like access tokens — and have fixed problems as we’ve discovered them. There is nothing more important to us than protecting people’s information, and we will continue making improvements as part of our ongoing security efforts at Facebook."

At this point it's fundamentally obvious that has never actually been true. And while that may be true now that the company is staring at looming regulation and mammoth fines all around the globe, at this point Facebook would need to be able to go a week without a major privacy scandal before any sentient being would take those claims at face value. In the interim, if you're not using a decent password manager and unique passwords on every website you visit, you might just want to get on that.

Filed Under: passwords, plaintext, privacy
Companies: facebook


Reader Comments

Subscribe: RSS

View by: Time | Thread


  • identicon
    Federico, 22 Mar 2019 @ 6:23am

    Password logging

    The "plain text" file was presumably some debugging or webserver log. Elsevier recently made a similar mistake with Kibana, even making all such recorded passwords public.
    https://news.ycombinator.com/item?id=19423770

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 22 Mar 2019 @ 9:25am

      Re: Password logging

      From the linked article: "Facebook is probing a series of security failures in which employees built applications that logged unencrypted password data for Facebook users and stored it in plain text"

      Karl is likely wrong about this part:

      In this case, insiders told Motherboard that this does appear to have been a bug, and that the majority of Facebook passwords are usually encrypted:

      The quote does not support that statement. It says they don't normally store them in plaintext. If they're at all competent, which remains to be seen, they'd be stored hashed, not encrypted. The important difference is that Facebook cannot reverse a hash to learn someone's password.

      reply to this | link to this | view in chronology ]

  • icon
    Mason Wheeler (profile), 22 Mar 2019 @ 7:16am

    Hmm... let's see.

    The Facebook source said the investigation so far indicates between 200 million and 600 million Facebook users may have had their account passwords stored in plain text

    "Obviously we don’t store them in plaintext ‘normally,’" the employee, who has a technical role, told Motherboard.

    How many people are on Facebook? Google says around 1 billion. That's 1,000 million, of which between 20-60 percent had this happen to them. That sure sounds like "normally" to me. (Especially when you consider the first rule of data breaches: it's always bigger than they realize at first!)

    My Facebook insider said access logs showed some 2,000 engineers or developers made approximately nine million internal queries for data elements that contained plain text user passwords.

    And yet Facebook officially claims that:

    We have found no evidence to date that anyone internally abused or improperly accessed them.

    War is peace! Freedom is slavery! Ignorance is strength!

    reply to this | link to this | view in chronology ]

    • identicon
      Dan, 22 Mar 2019 @ 7:48am

      Re:

      War is peace! Freedom is slavery! Ignorance is strength!

      I don't think that's necessarily what's going on here, though I'm far from sure that Facebook deserves the benefit of the doubt. What it sounds like, based on the anonymous dev who's quoted, is that passwords were written in cleartext to an error log file, presumably along with lots of other information about the user's environment and what they were doing. Devs (2000 of them, apparently) queried those log files, and the results of those queries included the passwords. But there is (allegedly) no evidence that those passwords were misused. Believe it or not, it's at least facially plausible, and it's reasonably consistent.

      reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 22 Mar 2019 @ 12:06pm

      Re:

      it's always bigger than they admit at first!

      Fixed that for you.

      reply to this | link to this | view in chronology ]

      • icon
        Mason Wheeler (profile), 25 Mar 2019 @ 7:15am

        Re: Re:

        Yes, this is also true. But I meant it the way I wrote it: it's very common for the damage to be larger than they actually realize at first.

        reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 22 Mar 2019 @ 8:01pm

      Re:

      If this isn't enough reason to scuttle fb and have zuck plucked from his poshness and arrested, then it'll probably never happen.

      reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 22 Mar 2019 @ 7:26am

    Password encryption vs. hashing

    Small nitpick: passwords aren't usually encrypted, but hashed, so that the password can still be compare to the one the user signed up with, but the original password can't be extracted out of it. (Or at least not in an easy way.)

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 22 Mar 2019 @ 8:32am

      Re: Password encryption vs. hashing

      too bad "hashing" isn't just a one way version or another word for "encryption" where the user still has access to the public key portion to encrypt and compare words to the original (without being able to decode the original)... Glad nobody is being pendantic...

      reply to this | link to this | view in chronology ]

      • icon
        Mason Wheeler (profile), 22 Mar 2019 @ 8:44am

        Re: Re: Password encryption vs. hashing

        Umm... it isn't. Encryption is reversible, while hashing is not. The use of the term "public key" implies the existence of a private key, but there is no private key with a hash.

        With the way hashes work, reducing an input of arbitrary size to an output of a fixed size, (the input is usually larger than the output, though of course with passwords this isn't always the case,) it can be trivially shown via the Pigeonhole Principle that there exist an infinite number of inputs that will hash to the same output. While it's true that the purpose of cryptographic hash design is making it as difficult as possible to find such "colliding" inputs, they are mathematically required to exist. And therefore, if there are infinite different inputs that can yield the same hash output, it is impossible to "decrypt" a hash and say "this is the (singular) input that it came from."

        Hashing is not in any way "another word for encryption," and claiming that it is makes people who actually understand the principles involved cringe at your ignorance.

        reply to this | link to this | view in chronology ]

  • icon
    PaulT (profile), 22 Mar 2019 @ 7:29am

    "Still, given Facebook's resources and the volume of security talent they have on staff, the fact that it happened at all is grossly embarrassing"

    There's a couple of things I can think of here, though. First, Facebook is very much a developer-led organisation that has experienced incredible, unprecedented growth at various points in its short history. It would not be unusual for security to have taken a backseat in such an environment. If security is less a part of the corporate culture than adding new features and trying to find ways to monetise users, they probably won't have been going back to patch holes opened by bad process or design until a breach is noticed. That's just the way a lot of these companies work.

    Second, unless I'm mistaken the lion's share of the issue is reported to be affecting Facebook Lite, a version of the app directly targeted at people in developing countries. So, reading between the lines, the issue was probably introduced during some push to expand global reach, but then not fixed because a lower per-user income metric meant they weren't a priority. Meaning that it's likely that certain parts of the company weren't getting the resources and talent you assume they did.

    I could be wrong with all of the above, but having worked for larger companies I can certainly say that it's the norm for boring things like security and painful things like best practices to get the most corner cutting, especially in departments not considered the main revenue generators.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonmylous, 22 Mar 2019 @ 7:58am

    True story bro

    There is nothing more important to us than protecting people’s information..."

    This is a 100% true statement. Yet this:

    The company, meanwhile, continued to insist that consumer privacy is among its top priorities:

    Is a misinterpretation of that statement. Facebook wants to protect the data they harvest because it is valuable, user privacy is barely a tertiary concern because FB never releases information, they sell it. You cannot sell what has been given away for free through malfeasance or blunder or intentionally. They on count us being gullible idiots to misinterpret their words, then pat themselves on the back when we do.

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 22 Mar 2019 @ 8:27am

    but sockpuppets...

    But how else could Facebook manipulate and deceive the majority of users if the weren't able to steal individual users identities to post 'sock puppet' type information...

    Maybe it's just an off color remark or a leading subject (how about those Swasticas?), maybe it's an ad for an opposing candidate or side.

    How can Facebook keep the 'discussions' flowing if they aren't allowed to inject their own opinions (under their users names/accounts... but I'm SURE this has NEVER EVER EVER happened, we pinkey swear)...

    reply to this | link to this | view in chronology ]

    • icon
      PaulT (profile), 22 Mar 2019 @ 9:33am

      Re: but sockpuppets...

      "the weren't able to steal individual users identities to post 'sock puppet' type information"

      Do you have any actual evidence this has happened?

      I mean, it's dumb to think that the people who have root access to the servers and databases themselves would need a user password in the first place, but I'm sure if you're going to accuse them of such things you must know it happened, right?

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 22 Mar 2019 @ 8:11pm

        Re: Re: but sockpuppets...

        It makes sense that fb would want employees to be able to access user accounts for whatever purpose fb wants. It does not seem that this was an erroneous error, but calculated access.

        reply to this | link to this | view in chronology ]

        • identicon
          Anonymous Coward, 22 Mar 2019 @ 8:21pm

          Re: Re: Re: but sockpuppets...

          Maybe fb would take a block of these accounts and offer for enticement to procure sale of data. Their history of mishandling their own users' data would make anyone suspicious of this. You let enough rain fall on a company that put off protecting itself with paint and it'll start rusting fast.

          reply to this | link to this | view in chronology ]

          • icon
            PaulT (profile), 23 Mar 2019 @ 1:42am

            Re: Re: Re: Re: but sockpuppets...

            Maybe a lot of things, you can make up whatever shit you want at this point. But the fact remains you don't have to make up wild stories about how people get into accounts.

            Every single organisation you deal with, no matter how big or small, has somebody who can access the data collected by that company at a root level. If this surprises you, you are exceeding uninformed about how things work. No conspiracy theory is required - whatever company you have dealt with someone has a means by which they could have accessed your data. Some companies are better than others at reducing the number of people for whom this is possible, but someone always has the potential.

            reply to this | link to this | view in chronology ]

  • icon
    Jeffrey Nonken (profile), 22 Mar 2019 @ 8:42am

    El Reg's take on this in its usual snarky manner.

    reply to this | link to this | view in chronology ]

  • icon
    Bamboo Harvester (profile), 22 Mar 2019 @ 9:37am

    Alternatively Titled...

    ...Another Day Ending in "Y"...

    reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 22 Mar 2019 @ 10:32am

    Oh look, Masnick shilling for Facebook again...

    ... wait, where'd all the usual trolls go?

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 22 Mar 2019 @ 12:10pm

      Re: Oh look, Masnick shilling for Facebook again...

      Masnick didn't write this.

      If he had, you can be sure it would have been littered with excuses, downplaying of the incident, or something about how GDPR somehow caused this.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 26 Mar 2019 @ 1:14am

        Re: Re: Oh look, Masnick shilling for Facebook again...

        Found the troll!

        Seriously, we get it, the thought that Masnick lives makes you piss in your own oatmeal every morning...

        reply to this | link to this | view in chronology ]

  • identicon
    bobob, 22 Mar 2019 @ 12:02pm

    Why did facebook ever have the passwords to begin withto begin with? The usual way of storing psswords is to encrypt a block of 0's using the password as the key (at least on unix and I'm pretty sure on VMS). the only thing that is stored is the encrypted block of zeroes. The password used to encrypt it never gets stored anywhere. When someone logs in and enters a password, the password then encrypts a block of zeroes, and THAT is what gets compared against the stored password.

    This has been common as far as I can remember, which like 30 years, so why did facebook ever have the passwords to begin with? Not only should they not have the passwords stored in plaintext, they should not have the passwords stored in an encrypted form that they can decrypt.

    reply to this | link to this | view in chronology ]

    • identicon
      Anonymous Coward, 22 Mar 2019 @ 12:26pm

      Re:

      The passwords at issue were probably captured in plaintext from within password handling code via an internal development mechanism (such as a debugging/trace log, for instance). As a result, the plaintext store the article refers to was not their "Master Password Table", but some debug.log file dumped onto a server's hard disk.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 22 Mar 2019 @ 4:26pm

        Re: Re:

        Facebook should never have had the plaintext passwords even for debugging, since that would not be needed for any debugging. There is simply no excuse at all for ever having anyone's password stored anywhere in any way.

        reply to this | link to this | view in chronology ]

  • identicon
    Anonymous Coward, 22 Mar 2019 @ 2:08pm

    So Facebook internal employees had access to all those accounts, with everything in them. This would have to be including those accounts that were set to Private/Friends only, and not subject to sharing, so if some of those photos ended up being publicly exposed (we know how people are about such things), what recourse do end-users have? Civil suit for privacy violation?

    reply to this | link to this | view in chronology ]

    • identicon
      bobob, 22 Mar 2019 @ 4:36pm

      Re: So Facebook internal employees had access to all those accou

      I hate to be the bearer of bad news, but regardless of the platform you are using, there will be people that have access to everything. Aside from the planet9 operating system, I know of no other operating system that doesn't require some form of admin or superuser that has total access to everything, for maintenence reasons, if nothing else. And, how else do you think they obtain files requested by law enforcement?

      So, if you have an account on any platform anywhere, you should assume that some people other than you can access anything you upload or post. Whether or not they are allowed to is a different question, but lack of permission never has been much of an impediment.

      reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 22 Mar 2019 @ 8:35pm

        Re: Re: So Facebook internal employees had access to all those a

        OOOOOr.. what if there is some internal grand scheme here by a very lofty employee who has been secretly selling data to very questionable types, but very wealthy questionable types and that employee was setting up buyers with passwords to a certain amount of users' data in order to make a tidy sum under the table? Could it be?

        reply to this | link to this | view in chronology ]

        • icon
          PaulT (profile), 23 Mar 2019 @ 1:34am

          Re: Re: Re: So Facebook internal employees had access to all tho

          It could be. But, there are a great many other stories that make a lot more sense and have a lot more evidence to support them.

          reply to this | link to this | view in chronology ]

      • identicon
        Anonymous Coward, 23 Mar 2019 @ 7:27am

        Re: Re: So Facebook internal employees had access to all those a

        "I hate to be the bearer of bad news, but regardless of the platform you are using, there will be people that have access to everything."

        Yes, this is true.
        The sys admin will have access to the shadow password file in which hashed passwords are kept. The sys admin will not have access to any passwords in the clear because there are none on the system.

        reply to this | link to this | view in chronology ]

        • icon
          PaulT (profile), 23 Mar 2019 @ 8:29am

          Re: Re: Re: So Facebook internal employees had access to all tho

          That is true, in a well designed system with security built in from the ground up, at least. Although that doesn't preclude him from changing a user password or bypassing security in other ways.

          Now, what about people with greater access than he does?

          reply to this | link to this | view in chronology ]

    • icon
      PaulT (profile), 23 Mar 2019 @ 1:33am

      Re:

      "So Facebook internal employees had access to all those accounts, with everything in them"

      Yes, Facebook employees have the root passwords for all server and databases they control, meaning that someone in the organisation has access to everything at a fundamental base level.

      If this scares you, you need to stop dealing with any company that has an IT staff.

      reply to this | link to this | view in chronology ]


Add Your Comment

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Remember name/email/url (set a cookie)

Close

Add A Reply

Have a Techdirt Account? Sign in now. Want one? Register here



Subscribe to the Techdirt Daily newsletter




Comment Options:

  • Use markdown. Use plain text.
  • Remember name/email/url (set a cookie)

Follow Techdirt
Techdirt Gear
Shop Now: Techdirt Logo Gear
Advertisement
Report this ad  |  Hide Techdirt ads
Essential Reading
Techdirt Deals
Report this ad  |  Hide Techdirt ads
Techdirt Insider Chat
Advertisement
Report this ad  |  Hide Techdirt ads
Recent Stories
Advertisement
Report this ad  |  Hide Techdirt ads

Close

Email This

This feature is only available to registered users. Register or sign in to use it.