Dependencies: Both Technological And Human, On Display In The Story Of A Developer Who Deleted Code Being Used By ICE

from the our-interconnected-world dept

Three years ago, we had a pretty fascinating story about how a developer, after getting an ambiguously threatening note from a company about how a bit of his code might violate the trademark of another company, deleted all of his code from NPM (Node Package Manager), a key repository for node.js code. One of the bits that the developer deleted (totally unrelated to the potential trademark dispute) was simple code that tons of websites relied on — leading many of them to break in response. The story raised all sorts of interesting questions not just about trademark, but namespaces, who controls code, dependencies, and much more. Indeed, the story was so interesting to me that I (very loosely) used it as inspiration for a science fiction story I recently wrote that will be released very soon (more on that very soon as well!)

Having been thinking a lot about all of that lately thanks to the story I was working on, I was surprised to see a similar situation pop up last week, with slightly different issues. This one involved an IT automation company, Chef, that helps lots of organizations better manage the configuration of various physical and virtual servers. The story kicked off with some controversy as someone noticed that Chef had signed a contract with ICE. Lots of people got (reasonably) angry about this, following on a pattern that has been playing out in the tech sector over the last few years.

Chef’s CEO put out a pretty lame email and blog post, basically saying “but we signed this deal under the previous administration,” which (among other things) fails to recognize that ICE was pretty fucking terrible during the previous administration as well.

But here’s where the story gets a lot more interesting. A former Chef employee named Seth Vargo, who had created a bit of open source software called Chef Sugar, got quite reasonably upset to learn that ICE was using his code to more efficiently detain children.

“I was having trouble sleeping at night knowing that software?code that I personally authored?was being sold to and used by such a vile organization,” he told Motherboard in an online chat. “I could not be complicit in enabling what I consider to be acts of evil and violations of our most basic human rights.”

Vargo asked the company to explain this (prior to that awful blog post mentioned above) and then, after a few days went by without a response, took down his code from two key repositories: Github and RubyGems. As he wrote:

I have removed my code from the Chef ecosystem. I have a moral and ethical obligation to prevent my source from being used for evil.

Of course, because no one has learned anything, multiple other systems depended on that code being in those repositories, and those systems started breaking as well. Even more fascinating, some of the people who this caused problems for still supported Vargo’s decision:

This certainly started getting much wider attention — leading Chef’s CEO to issue an update on Friday, which first seemed to unnecessarily attack Vargo:

On Thursday, September 19th an action was performed by a trusted community member in violation of the standards of open source software (OSS) development. The individual yanked several RubyGems that they authored while employed by Chef. In order to remove the gems, they first removed the other owners and took unilateral action to yank the gems, violating established processes for making OSS changes and improperly removing property which Chef owned. This ownership has been established through the Github history of commits, licenses, etc. The individual did not have Chef?s permission to remove these items from the RubyGems site.

So, obviously, some may point out that since Vargo’s work was initially done on Chef’s dime as an employee, he has less ground to stand on. But, again, as an act of protest, it’s pretty fascinating.

Also, it turned out to be incredibly effective. By Monday, Chef had completely reversed its position and said that it would not renew its work with ICE:

As many of you know, we began our work with the U.S. Government in earnest in 2014 and 2015. This included DHS and its various departments under a different set of circumstances than exists today. The overarching goal was to help them modernize their computing infrastructure and create a cooperative community of IT professionals inside the government that could share practices and approaches in a similar way to many open source communities. Policies such as family separation and detention did not yet exist.

While I and others privately opposed this and various other related policies, we did not take a position despite the recommendation of many of our employees. I apologize for this. I had hoped that traditional political checks and balances would provide remedy and that our relationship with our various government customers could avoid getting intermingled with these policies. However, it is clear that checks and balances have not provided relief to the fundamental issues of the policies in question. Chef, as well as other companies, can take stronger positions against these policies that violate basic human rights. Over the past year, many of our employees have constructively advocated for a change in our position, and I want to thank them.

After deep introspection and dialog within Chef, we will not renew our current contracts with ICE and CBP when they expire over the next year. Chef will fulfill our full obligations under the current contracts.

The company also promised to donate the equivalent revenue that it had received from the contracts to charities helping people impacted by ICE’s family separation policy.

This whole story is quite interesting on multiple levels. Seeing tech workers recognize that they have some moral stake in how tech they develop is used is quite amazing — especially given the exaggerated (and incorrect) stereotype that Silicon Valley never cares or thinks through these things. That’s never been true, but it’s especially interesting to see people taking some element of ownership over how what they’ve developed is eventually used. Second, it’s another interesting example of how interdependence on code hosted elsewhere is creating a somewhat fragile web in certain places. I’m almost surprised that we haven’t seen this as an attack vector — gaining control over repositories and doing something with them that impacts lots of other services.

Either way, it’s a representation of how interconnected the entire world is — at both a technological and human level.

Filed Under: , , , ,
Companies: chef

Rate this comment as insightful
Rate this comment as funny
You have rated this comment as insightful
You have rated this comment as funny
Flag this comment as abusive/trolling/spam
You have flagged this comment
The first word has already been claimed
The last word has already been claimed
Insightful Lightbulb icon Funny Laughing icon Abusive/trolling/spam Flag icon Insightful badge Lightbulb icon Funny badge Laughing icon Comments icon

Comments on “Dependencies: Both Technological And Human, On Display In The Story Of A Developer Who Deleted Code Being Used By ICE”

Subscribe: RSS Leave a comment
53 Comments
PaulT (profile) says:

Re: Re: Delusional

It’s simple logic – he hates Mike for whatever pathological reason, he’s been told by his right-wing echo chambers that "socialism" is a label to apply to everything he hates whether relevant or not, therefore Mike writing an article about code library sharing means he’s the same as Stalin.

It’s easy once you forget those pesky things like facts, logic and sanity that this guy happens to be immune to.

PaulT (profile) says:

OK, although I work with devs and GitHub to some degree this confuses me, possibly because I don’t really get involve with Ruby on the coding end. Maybe someone can help clarify here?

"In order to remove the gems, they first removed the other owners and took unilateral action to yank the gems… The individual did not have Chef’s permission to remove these items from the RubyGems site."

So, how did he remove them? Unless I’m mistaken about the way gems operate, surely that would require some admin level access to the project? Did someone at Chef mess up and leave him as an admin, did he retain some sort of access due to some Google involvement in the project, or am I missing something?

Ven says:

Re: Re:

RubyGems (the server infrastructure) has basically 2 permissions levels for each gem, the general public that can download the gem, and the gem’s owner that can do anything with the gem. You can add and remove owners via their command line or api by providing an email address.

It’s fairly common in open source projects to allow any trusted developer to have fairly broad rights and different projects have different standards for trust.

I know few developers that created an addon for a larger project, they get contracted by the company that owns the larger project to add some features to the addon, or develop some related addon, and then after the contract expires they continue to maintain their addons. Because there was a contract some or all of the addon code it could be owned by the company, but because of the prior relationship it’s in everyone’s interest that the developer can still work on the code.

Thad (profile) says:

Re: Re:

Did someone at Chef mess up and leave him as an admin

More or less. While Chef owns the copyright on the code, Vargo owns the accounts that provided it. This was a chain-of-custody problem; Chef should have hosted the code in its own repositories, not relied on a third party to control them (not even when he was an employee, and certainly not after he was no longer with the company).

PaulT (profile) says:

Re: Re: Re:

"While Chef owns the copyright on the code, Vargo owns the accounts that provided it."

Yeah, there’s your problem. I’ve worked for more than one company where the entire organisation was in danger of going down the tubes because they didn’t care much for access control and ownership until the day they realise they don’t own their own domain name, or the head dev’s mate who was controlling their AWS account decided not to pay the bill when something went south on a personal level…

Such is the way of things at the moment, I fear – rush to get things out the door, then forget to make sure you have everything locked up behind you.

Anonymous Coward says:

Re: Re: Re:

With git, all developers, and quite a few users, will have a clone of the master git repository, git being designed to be used with multiple copies of the code, and so it is easy to duplicate the Github version under a new name. Ownership of the public repository is more a matter of convenience than someone controlling the existence of the code.

Bruce C. says:

I can understand the ICE hate...

But does it need to reach this level? Its methods are frequently deplorable, but ICE does serve an important mission of the US government.

If the court of public opinion has decided that hamstringing ICE at every opportunity is the best way to make it reform its practices in regard to immigration and detention, we should be discussing and preparing for the unwanted side effects that are likely to occur as a result. These boycotts also affect necessary enforcement functions against things like drug smuggling and sex trafficking.

PaulT (profile) says:

Re: I can understand the ICE hate...

"ICE does serve an important mission of the US government"

The mission isn’t the problem, it’s the inhumanity committed in the name of doing it. It might be important to keep streets free of litter, but you’ll still be criticised if standard procedure were to tase and jail anyone spotted dropping a plastic bag.

btr1701 (profile) says:

Re: Re: I can understand the ICE hate...

LOL!

If you think the US government is ever going to just walk away from the nation’s borders and throw them open to anyone who wants to walk/fly in, you’re nuts.

Abolish ICE and another agency will be created immediately to do what ICE does. It’ll just have a different name.

Stephen T. Stone (profile) says:

Re:

ICE does serve an important mission of the US government

And that is a mission the U.S. government was carrying out well before ICE and the Department of Homeland Security were ever a thing. The U.S. doesn’t need ICE or DHS to handle immigration and customs enforcement — and we don’t need their apparent inhumanity towards anyone who isn’t White.

btr1701 (profile) says:

Re: Re: I can understand the ICE hate...

When most people say "Abolish ICE" they mean roll back to the pre-9/11 state of things.

I’m not going to quibble over the word ‘most’ but a significant number of people who advocate abolishing ICE literally don’t want any border enforcement whatsoever. They don’t believe there even should be borders in the first place, let alone enforcing them.

ECA (profile) says:

Re: I can understand the ICE hate...

Bruce.
Something many have not seen yet, and its on the Wiki..
Look up Gov. Police agencies.
ICE and DHS are now in charge of over 40 of the policing agencies.
They were HIRED, to do a certain job, and are going above and beyond that. ALSO, its interesting that, as I understand it…They WERE HIRED.. anyone remember the Names they have used??
blackwater?

Anonymous Coward says:

Re: I can understand the ICE hate...

These boycotts also affect necessary enforcement functions against things like drug smuggling

It’s definitely not necessary for drug smuggling to be illegal, and may not even be helpful. People die due to high American drug prices; others import from Canada in amounts that are technically illegal.

Anonymous Coward says:

I will laugh if ICE sues everyone involved in the violations

I assume a contract states what is and isn’t allowed to happen that might cause problems. A former worker signing in, changing lots of settings and then deleting work done while under contract seems like it could cause millions of dollars worth of liability and damages. If the hacker (since they were not authorized to access the work this way any longer, it is a CFAA violation) isn’t made an example, it will happen again and again. Good luck moron. You just screwed up your life over a morally questionable action.

Anonymous Coward says:

got quite reasonably upset to learn that ICE was using his code to more efficiently detain children.

So by deleting his code, he’s causing ICE to be less efficient at detaining children. Therefore, it’s somehow a win that ICE keeps right on doing the things he doesn’t like, but now causes even greater harm by its reduced efficiency? Which is more likely: removing this code causes ICE to stop detaining children, removing this code causes ICE to process detainees less efficiently (thereby keeping them detained even longer, possibly under even worse conditions since the reduced efficiency may increase overcrowding), or removing this code is a purely symbolic gesture that hurts a bunch of unrelated organizations and doesn’t impact ICE’s detention practices at all?

Yes, I saw that Chef will not be renewing their contract. ICE is a big organization, and there’s nothing here that suggests Chef’s non-renewal will affect ICE’s family detention subunit in the slightest. It might do that, or it might only impact other ICE missions that enjoy broad support.

Wouldn’t it be funny if the only things this hurts are the naturalization/visa issuance divisions, groups that help immigrants become lawful residents/visitors?

The Cosmic Comic-Consciousness says:

Hey, Maz, ever heard of CHINA? Have any of its products?

First, don’t forget the suicide nets at Foxconn factories making your precious APPLE phones. That wasn’t enough to give you pause, just another PR problem.

Experts Call on UN to Investigate China’s Killing of Religious Dissidents for Their Organs

https://www.theepochtimes.com/experts-call-on-un-to-investigate-chinas-killing-of-religious-dissidents-for-organ-harvesting_3094703.html

https://www.theepochtimes.com/c-organ-harvesting-in-china

Now, as the only true humanist here who’s actually concerned about even furriners, isn’t secretly a rabid militarist in favor of "humanitarian bombing" for Oil and Empire nor a corporatist intending to destroy the US of A precisely because of our freedoms, I’ve been railing at trading with murderous Communists and especially the worst, China for longer than Maz has been alive.

YOU are betraying your own and American principles, Masnick. You advocate for Chinese-model total surveillance / "social credit" score society here, all done efficiently by unlimited untaxed corporations which decide who has a public voice.

You are so inconsistent that are a total fraud.

The Cosmic Comic-Consciousness says:

Re: Hey, Maz, ever heard of CHINA? Have any of its products?

I bet you don’t give away your supposed science-fiction novel! You’ll try to extract money or email address up-front, not risk AFTER people have read it.

I further bet that you don’t actually EVER release it! You may have showed it around to your pals who will of course say nothing even less than fullsome praise, because they’re all frauds too!

But to EXPOSE yourself (WAY more than your dry vaguely factual basis here) and skills to people not highly biased in your favor? HA. You are not a good writer to start with, and your notions are childish ones from the covers of lurid 50s magazines, like Elon Musk.

Anonymous Coward says:

To me this is more about stupid business practices.

I do not want to take away from the moral choices being made by the developer with what I’m about to say.

To me, this issue speaks volumes about how commercial companies are abusing open source tools for monetary benefit and the stupid decisions they are making.

In this particular case we have a commercial entity with their product, Chef, relying on the personal source code repository of an individual developer for key functionality of their product.

Why would any clear thinking business owner decide to rely on the resources and services of an outsider, provided for free by that outsider, to supply a crucial resource needed for their business?

We just saw the result of that decision.

If you are going to run a business based on providing reliable services to support open source software why would you not make sure that you can continue to provide reliable service in the event that the projects that you rely on for the software shut down or remove themselves from public access?

PaulT (profile) says:

Re: To me this is more about stupid business practices.

In my experience, it probably wasn’t that at all. Devs sometimes decide to take the path to least resistance at the moment they need something, not really considering consequences long term outside of their code. Many a time I’ve seen some random personal accounts being used to host mission critical code because the developer needed to bypass some restrictions that they felt were slowing down development work.

In other words, it’s not greed, it’s people not keeping tabs on what others in their team are doing until they need to work out a problem. Fine for early dev work, but not something that belongs in production – but someone has to remind them, I think.

Anonymous Coward says:

Re: To me this is more about stupid business practices.

Git based development results in multiple copies of the code repositories, where a developers copy only differs from the ‘master’ in the changes they have made, and which have not been pulled into the master repository, and other copies updated to copy the changes. So while that deletion made a statement, it should not have been more than a minor inconvenience to chef, as they likely have several copies on machines within the company.

Thad (profile) says:

Chef says:

On Thursday, September 19th an action was performed by a trusted community member in violation of the standards of open source software (OSS) development.

That’s an inaccurate simplification.

The Open Source Definition bans discrimination against persons, groups, or fields of endeavor. That means that, for example, a license that says "this software may not be used by ICE" is not an open-source license. There are such licenses, and Bruce Perens has discussed why they violate the principles of open source, even if they’re well-intentioned:

When Licenses Discriminate

Sorry, Ms. Ehmke, The “Hippocratic License” Can’t Work

However, there’s nothing in the Open Source Definition that says you have to keep code you’ve previously worked on available, on your own private accounts, if you no longer wish to support it. The Open Source Definition says that if you distribute a program, you have to make its source code available. It doesn’t say that you have to continue distributing that program. And, not for nothin’, Vargo is neither the owner nor the legal author of the programs in question; the company that employed him when he wrote them, Chef, is.

Chef relied on a third party, who no longer worked for the company, to provide access and distribution of code that it owns. That was unwise. Even if you believe that what Vargo did was unethical (and I don’t), it wouldn’t have impacted Chef if Chef had taken proper custody of its own codebase.

Anonymous Coward says:

First:

a bit of his code might violate the trademark of another company

How, exactly, can source code violate a trademark? A copyright I can understand but a trademark? Source code is never seen by the public and is never exposed as trade dress so it couldn’t possibly cause any kind of brand confusion.

And second:

it’s another interesting example of how interdependence on code hosted elsewhere is creating a somewhat fragile web in certain places. I’m almost surprised that we haven’t seen this as an attack vector — gaining control over repositories and doing something with them that impacts lots of other services.

The choice for a software house always boils down to "build or borrow?". They can either invest a lot of time, effort and money into building a tool, widget or feature OR they can use an appropriately licensed open (or sometimes closed) source solution. The foreign code rarely behaves exactly as you would have written it but it’s "good enough" and get incorporated into your project to save time and money. This is and has been happening globally for a lot of years. It’s not a new phenomena and it been consistently causing problems, sometimes as this article describes, since it began.

This practice has also long been recognized, though not nearly as broadly as it should be, as a security issue. Vulnerabilities are often found in such code and every outfit using it needs to update to the latest version or find some other workaround to the security hole. The worse situation is that an "owner" of the foreign code could intentionally introduce a vulnerability or even get his account hacked resulting in vulnerabilities added to the code. None of this is a new story either and has happened many times. It has to be considered in the cost vs benefit analysis of using external code.

The only bit of any of this that is new is why the code was taken down, not that it was taken down.

Anonymous Coward says:

Re: Re:

How, exactly, can source code violate a trademark? A copyright I can understand but a trademark? Source code is never seen by the public and is never exposed as trade dress so it couldn’t possibly cause any kind of brand confusion.

I’ve trademarked my company name, which is "Error."

If your program displays any sort of dialog box which uses that word, your code is infringing upon my trademark.

Anonymous Coward says:

I have no opinion on whether he should have done this, but as someone who has fought numerous battles trying to get people not to have build systems depend on being able to just go download from the Internet in the middle of a build, my biggest gripe with this is that it didn’t cause wider fallout. People need to learn to cache dependencies locally, and having things suddenly break due to a dispute between parties you don’t know is a nice motivator for that.

Add Your Comment

Your email address will not be published. Required fields are marked *

Have a Techdirt Account? Sign in now. Want one? Register here

Comment Options:

Make this the or (get credits or sign in to see balance) what's this?

What's this?

Techdirt community members with Techdirt Credits can spotlight a comment as either the "First Word" or "Last Word" on a particular comment thread. Credits can be purchased at the Techdirt Insider Shop »

Follow Techdirt

Techdirt Daily Newsletter

Ctrl-Alt-Speech

A weekly news podcast from
Mike Masnick & Ben Whitelaw

Subscribe now to Ctrl-Alt-Speech »
Techdirt Deals
Techdirt Insider Discord
The latest chatter on the Techdirt Insider Discord channel...
Loading...