Is AWS the meteor that kills OSS companies?

Elastic changed the license for Elasticsearch from Apache 2.0 to SSPL. They are following the lead from other similar companies like MongoDB, Confluent, Cockroach Labs, Redis Labs, Timescale and others.  I've written about why I think it is a bad decision here: Why I dislike Elastic's license change. Sadly, after several conversations with people, I also come to think of it as inevitable.

Why is Elastic doing this?

(A lot of this was pieced together with light Googling so the specifics might be off. But I think this is true in principle. I’d love to correct myself if I’m wrong, so please reach out!).

While I don't know exactly why they changed the licensing, it's not hard to venture a guess. Elastic is a public company with $427Mil in revenue and it needs to maximise the value for its shareholders. I'm not an expert in reading the public reports, but from what I could see, Elastic Cloud is bringing in a large portion of its revenue.

But it is not the only company that has a hosted Elasticsearch product, several other companies have one as well. Of these companies, the most notable one is AWS. Elastic is losing a significant amount of revenue to AWS and they want that revenue to flow to them instead.

But now it's time to take a step back and look at the history. Elasticsearch was released as an Open-Source project under the Apache 2.0 license. As the adoption and community grew the project creators started a company around the project to fund further development. After all, good engineers cost lots of money! The company and community grew really fast and Elasticsearch adoption skyrocketed.

But seeing the opportunity, many companies started offering a hosted Elasticsearch product and these products competed with Elastic's own managed service. To fend off the competition, Elastic went the Open-Core route where they kept the already useful Elasticsearch under Apache 2.0, and built X-Pack, a suite of closed-source "enterprise" features on top. This worked great, Elastic had the best hosted solution out there, but other companies could also offer a hosted Elasticsearch solution that was useful but not as featureful.

I think this part of the community is crucial. I do want Elastic, the company, to make money, but the pricing is not affordable for everyone, especially small teams and startups. Of course, they could always host it themselves, but managing Elasticsearch clusters is no easy task. To cater to these users looking for cheaper managed hosting, many companies launched hosted Elasticsearch products and they are quite successful with 100s of customers. The companies and competition are a net positive for the community and helped companies that couldn't afford Elastic Cloud adopt Elasticsearch successfully.

But at some point in this process AWS entered. AWS is a massive beast that kills startups. If you're doing a hosted "ProductX" and AWS launches a managed ProductX, people will still go to AWS even if AWS' offering is 3x inferior and 3x pricier. This is because the really big companies already have agreements in place and budgets allocated for AWS. They don't want to deal with the procurement process with a new company which could take months! The AWS service would also be tightly coupled with the IAM, SLAs, and other features in AWS and in general companies would want to consolidate everything in AWS rather than have a provider per service. This makes competing with them very difficult.

Now AWS launching a managed Elasticsearch is bad for Elastic but likely not as bad for the smaller competitors. If a customer can afford AWS ES, they could as well have paid for Elastic Cloud and all its features. The smaller companies will still lose some customers, but I think lots of them will still stick.

As AWS ES grew, its customers started demanding more of the closed source features Elastic was building and AWS started working on its distribution which took the opensource Elasticsearch code and bundled it along with other opensource plugins which replicated the features of X-Pack. I guess this move forced Elastic to move the entire code base to SSPL. The differentiation they've built as part of X-Pack is no longer a good enough moat for them to fend off AWS.

Wait, can't they happy with what they make?

Lets take a step back. Elastic is a public company, but this is happening with startups too! Good engineers cost money. Lots of it. And the startups behind the OSS projects hire lots of great engineers to make the OSS project awesome. They raise millions of dollars from investors to be able to do so. But they also promise the investors a really high return on their investment. The companies expect to grow quickly and maybe IPO in a few years.

But with AWS in the mix, it is hard to hit all those goals. The IPO timelines need to be stretched. Succeeding as a company is already hard, but when you're competing with AWS, its worse. When this happens, investors are hesitant to invest in the company and as a result, the investment into the project itself is reduced. It makes little business sense to invest as much you'd like to when the returns are not what you're looking for. And some argue that without the VC backed investments, the projects wouldn't have been able to gain traction or become as successful.

An analogy given to me was about Meteors and Dinosaurs. And AWS launching a managed service is likely a meteor event for the startup. They need to do their best to break it up before it kills them.

It is the same with Elastic. They're investing a lot of money into Elasticsearch. They're already making a lot of money yes, but lots of members invested in Elastic hoping that it'd make more. Elastic is simply choosing to maximise the returns to its investors while also justifying further investment into the Elasticsearch project.

How can we fix this?

tbh, I don't know! Until AWS starts making meaningful partnerships and revenue share agreements with OSS companies, I don't think we will see any changes in the situation. And ofcourse it doesn't make business sense for AWS to do so. But also, I might be idealistic but hear me out, if AWS starts making these agreements, it will allow more OSS companies to thrive and thereby give AWS more things to offer to its customers (via partnerships). So it might even be a net positive for AWS in the long-term.

But can there be a truly OSS company?

I think that if you want to build a successful OSS company, you could also build with a different definition for success. Ghost, a company that I deeply admire is a non-profit company. From their about page:

We set Ghost up as non-profit foundation so that it would always be true to its users, rather than shareholders or investors.

This is an interesting take. Can you build a company where you priortise the community and users over profitability? You don't need to be a non-profit, but you could be a bootstrapped, slow growing company that would rejoice if AWS starts offering a managed service. It would be a net positive for the community and users. Ofcourse the revenues won't be high, hence the company would likely never be a public company or attract investment, but it could still be a profitable company that employs 50-100 people in total and slowly grows that number.

One really cool thing about a small company is that it can't productise or build every single use-case and this gap opens up space for other companies and individuals to innovate. There is now an ecosystem of themes, plugins and other features around Ghost. And people in the ecosystem are empowered to build around Ghost, including hosting services. I'm sure the Ghost founders celebrate the competition rather than feel threatened by it.

I'm constantly learning!

While this is quite interesting, I don't think this model of building a business is not for everyone. I'm not even sure if it is for me :) Given that, I think we will see lesser open source projects in the future. Never say never though, there are companies with Open Source licenses that are doing quite well in our space, like Hashicorp. I keep wondering what their secret sauce is.

Overall I am very interested in this topic and I'd love to chat about it if you have opinions around this topic. When AWS approached us about AMG and AMP, I had lots of thoughts, but one of them is that I'll see the whole OSS dilemma play out. We're currently betting on feature differentiation with GME. I will try to share what I can from Grafana Labs' business and other interesting things I come across in this space here!