So I have been doing open-source for a little while now. I've actually learnt how to code by contributing to an open-source e-commerce solution. Later I started contributing to Prometheus while in college and my contributions snagged me an internship at CoreOS which let me do Prometheus full-time for a summer. I kept contributing, travelling internationally for my first-time and several times after because of Prometheus! I even got my current job because of my contributions. I continue to work on Prometheus and related OSS projects as part of my job, and even co-authored a popular OSS log aggregation solution.

As you can see, OSS is an important part of my life, and now I'm also an early employee of a fast growing company based on OSS software. I've been thinking about a few things around OSS lately, especially with a ton of OSS projects changing licenses and blocking cloud providers from providing a hosted solution.

Where I come from

So I contributed to Prometheus because it was super fun. The community was kind and they were patient and they taught me tons of stuff. But more importantly, I loved what I was doing and the impact I was having. Today, literally every single company I speak to, large or small is either running Prometheus or considering running it! My own friends are now dealing with Prometheus everyday in their jobs, and hey a decent part of it is code I wrote in college while studying with them :) It felt great that my code was being deployed to production not once but 1000s of times!

I contributed full-time as an intern at CoreOS in the summer and when I returned to school, school sucked. So much, that I decided to take a semester off to continue my contributions. I'm in India, and this is unheard of! Most people thought it was the dumbest thing I did, and I literally had no upside, at least, it wasn't visible then. I did it because I loved working on Prometheus and wanted to explore a way to monetize my work so I can continue to do it. I didn't even try too hard to monetize, just focused on having fun :) I've also traveled as much as I could, speaking at conferences, meeting all these folks in the community and they were so amazing! Down to earth, friendly and kind that I felt right at home with them. Prometheus became an important part of my life then and I decided any potential future job must contain my maintainership as part of its description.

All of this made me care about Prometheus and about the work I was doing. I wasn't doing it because it was a job, but because I actually care! Because I was passionate about Prometheus.

Where Prometheus comes from

Prometheus is a very dominant project in the multi-billion dollar monitoring space, but its history is very interesting. It was initially started at SoundCloud because they couldn't find a decent monitoring system that was open-source and fit the kind of modern dynamic computing environment that SC already started moving to ahead of almost anyone else. It was always OSS, but they didn't talk about it during the first couple of years. They apparently even actively discouraged making noise while they were iterating on it. Once they were happy enough with it, after running it in Production on large systems internally, they started talking about it and invited outside contributions. And slowly the community and project grew to its now skyrocketing adoption.

But here is an interesting thing, not a lot of people were paid to do Prometheus full-time for a long time. It started out with Julius and Matt working on it in their free time and then as their 20% time project, before Julius and a few others spent significant portion of their time on it. But it reduced over time as things stabilised internally at SC. When I started using Prometheus around 0.10, it had maybe a couple of people paid to work on it full-time, and that number remained very small for a very long time. And the project slowly kept on growing, with people who care about it pouring their 20% and then some into it!

Today, there are several companies that have hired Prometheus maintainers to do solely Prometheus stuff and the pace of development is quickening, and it is one of the fastest growing projects in the CNCF. Yet, we're still a very lean operation, and still a lot of maintainers try to do it in their free or 20% time, but we have a community that just keeps chugging on!

I think this was only possible because of the truly open nature of Prometheus. All discussions are public, we have no closed features and anybody can contribute anything as long as it is inline with the Prometheus design goals! Hell, I even think it would have failed if we had a company controlling it.

COSS?

Now there is a new breed of companies that are gaining prominence, termed as Commercial Open Source (COSS) companies. These companies have an open-source project at their center that they lead. To sustain the development costs, they have some premium offering on top, like support or an enterprise version. I work for one. A lot of these companies treat OSS as a user-acquisition/product-marketing strategy, they build a really useful and successful open-source project used by 1000s of people! Kaboom, there are your users, acquired via your OSS project. Now they aim to monetize a small percentage, as little as 1% or less to sustain all the development that goes into the project and then make some profit.

Now, a few things are typical with these kinda OSS projects:

1. Majority of the maintainers work for the company, if not all of them.
2. Anyone who is contributing to the project eventually gets hired by the company.
3. The development is never truly open. There is an internal Slack where the maintainers discuss project related matters that the OSS community has no access to.
4. There are some closed-source features.

I don't have anything against the COSS companies, quite the contrary. The impact and value that COSS companies generate is magnitudes higher than that of comparable closed-source companies and in fact, I want to build a COSS company someday.

Okay so what?

Now that we've established the above, I want to talk about the dynamics between the project, the company and the contributors. I think the projects that don't have a company backing them are better for a bunch of very personal reasons. In fact so personal, that in the general case, these are some of the reasons COSS projects are better ;)

1. It is very hard to be passionate about a project owned by a $$$ corp. Would I have taken a gap semester to work on InfluxDB with no promise of an upside? Nope!
2. I don't think I could have contributed meaningfully to a commercially backed project while still in college. Because everyone was working on their free-time, I could keep up with the pace and also make meaningful contributions while burning the midnight oil after finishing up my assignments. Commercially backed projects have 5 people or (usually tons) more working on things full-time and anything important and meaningful is picked up by them. They cannot afford to wait for a college student to fix things next weekend. Oh wait forget all this, and the discussion happens in an internal Slack anyways.
3. Some deny the interesting features. Part of the passionate contributions is the freedom to work on what is interesting to me. And usually the closed source features like clustering are what are interesting to me.

#2 is super interesting. Isn't more people working full-time better for the project? Won’t having to write up everything in design docs or waiting for review from a community member doing contributions in their free time slow things down? Yes, but Prometheus is an interesting project to consider here. When I started contributing the pace was definitely slower, much slower than what it could have been if a commercial entity was building it. But now, Prometheus and it's ecosystem is being developed much faster than potentially any other monitoring system out there! Maybe you need to go slower to go faster?

What is stopping COSS projects from being great?

So this going slow before you go fast is not suitable for a company that needs to turn a profit. In fact it goes against logic and is a hugeass risk. Why will anyone do it when they can afford to speed things up? But if you don’t engage the community and make them equal stakeholders, you won’t have a community that will work for you. No 20% time for ya. So please make sure all discussions and design happen in public and be patient if someone is really interested but can only spend a few hours every week.

You also need to give up control. It's easier when you cannot afford to have control lol. Sometimes the maintainers had let me pick issues/features that they themselves wanted to work on simply because they didn't have the time to do it themselves. I know because I've done it too :) This means the community can pick up really impactful stuff to work on! And sometimes they implement things in a different way or wants to do something you disapprove of, and it's okay. This is what grows the community.

But it is very tricky, sometimes you need to say a strong no, and before you really figured out your project, engaging the community too much might actually be detrimental. And giving up control is always hard. With a commercial entity, you don't need to, so you won't.

And you need to allow competition. If you really want to grow your community, you need to encourage the competition. Companies rarely put full-time devs on projects unless they are benefiting from it monetarily and that will cannibalize some of the original company's profits and nobody is okay with that :) If AWS is going to launch a hosted service, you're going to gain 1000s of users very quickly, you're not going to make any money off them, but the ecosystem of users will be better off!

(Also, in related news, I think Elastic saying "the community will benefit from blocking AWS Elasticsearch" is ridiculous! The user community will benefit immensely if AWS Elasticsearch is encouraged, more companies would be adopting it because its integrated into the stack and they could use their AWS commit money to pay for it. Further, how much should AWS contribute before it is allowed to host Elasticsearch? 1%, 10%? I think the answer is more like, "it doesn't matter" and turns out, Amazon does contribute to Lucene, the engine powering Elasticsearch).

Need to set expectations

Despite all this dragging, I think the new age of COSS is an absolute boon to OSS. There's more open-source (OS) software being created now than ever before, and companies are giving serious thought to OS and starting to outright prefer OS software over closed. Even the bigger and older companies are starting to make their own code open-source! I don't think it would have been possible without all these new age startups and unicorns! Finally, COSS companies are driving adoption in the enterprise. A lot of the large Fortune 500 companies I am dealing with for Prometheus are reaching out to Grafana Labs for support. In these enterprises, it is required that any OSS project they use has a support contract attached from a long-term durable company. They are not going to adopt OS software from a 10 person company because the risk that the company shuts down is high. So a fast growing, large vendor supporting an OSS project is critical for its adoption in the enterprise and this usually ends up being the COSS company behind the OSS project. Now, you might ask, do we really care about these large enterprises? YES. They have the $$$ required to fund the project development and they need a company to give it to.

But I think the COSS can do better, be more open, and be better for the communities in the long run. First figure out what your first priority is, is it to build a profitable business, or is it to grow the community. Both of them can be a priority but which one is more important will define a lot of the decisions. It will define if you'll be okay sharing the pie with other companies (AWS!) or not. And then be clear about that to the community. There is a lot of confusion around this, and sometimes the community feels blind-sided when licenses are suddenly changed because you realised that you can't be fast growing silicon valley company if you gave away half your pie to AWS.

Oh Goutham, you hopeless idealist

You know whats funny. With this outlook, it's hard to find COSS companies that are willing to give away the pie for the community. Only example I can think of is Buoyant with Linkerd, they're not doing it too well, but they're atleast trying hard to make sure other companies are also interested in the pie. But I am sure I am missing some of the smaller companies that don't have VC backed marketing budgets.

Please let me know on Twitter @putadent if you find more, I'd love to see more examples! Specially if you know of indie companies that are bootstrapped. I am asking because that is the kind of company I want to build in the future, but again I am an idealist and not very practical ;) Knowing that companies like that exist is going to be very encouraging to me.

Cortex and Loki

So I am deeply involved in two different projects, Cortex and Grafana Loki. The difference should be clear by how I referenced them. Cortex is part of the vendor neutral CNCF which doesn't care which vendor makes how much, and Loki is backed by Grafana Labs. Grafana Labs cares about the success of both and is monetizing on top of both projects. We're now seeing more and more vendors contributing to Cortex and even looking to monetize the project building SaaS offerings. But with Loki, it is being run like a typical COSS project. Loki does have a governance and we're seeing more and more adoption and contributions, one highlight being RedHat. Both the projects are thriving and have a bright future ahead and I am very curious to see how things evolve in the future!

Conclusion

Overall, I think the new spurt in COSS companies is a net positive for open-source but we need to recognise that the community dynamics and the kind of contributors that are involved are going to be very different. COSS companies should not disguise themselves as being community first and should be clear about the priorities. And finally, I think all of this is very evolving and even companies like AWS are changing too. It would be very interesting to see how the next 10 years would change the COSS landscape.

Finally, I wrote this in Jan 2020 and I got a lot of great feedback from Nikhita Raghunath, Björn Rabenstein, Julius Volz and others I can't recall :( Thanks everyone for spending the time to read this, let me know if I've shared this with you before and I'll add credit!