Collin: 0:00

Hey everybody, welcome to the podcast. Today, in addition to, uh, myself and Joel, we have a guest André Arko, who is the head of open source at Ruby central. Andre, how, how are you doing?

Andre: 0:14

I'm doing great. I'm just really excited to talk about the, the open source work that Ruby Central has been, you know, trying to set up recently.

Collin: 0:22

yeah. And we're, we're really excited to have you. Um, Joel, uh, is also here. Hey Joel.

Joel: 0:28

Hey.

Collin: 0:30

Hey, so, uh, so let's just start off, like, what, what is Ruby Central and what are they doing with, you're the first Ruby Central person we've had on. So maybe give us a little intro of Ruby Central and how it relates to the open source work that you're doing.

Andre: 0:44

absolutely. Not a problem. So, uh, Ruby Central is the main, uh, nonprofit, um, that backs the Ruby community, um, It's, uh, kind of like the American counterpart to the Japanese Ruby Society. Um, and the, as, as Ruby Central is the American Ruby non profit. It runs the American Ruby Conferences that you've probably heard of. RubyConf and RailsConf every year. Um, Ruby Central also, uh, acts as kind of like the, Uh, sort of main backing organization for a lot of the community work that benefits kind of like the entire Ruby community worldwide. So, uh, this is kind of all of an, all an outgrowth of, uh, work that I, I personally did actually starting a nonprofit that you may be heard of in the past called Ruby Together. Um, that was specifically around open source and eventually, Ruby Central and Ruby together realized that the work we were doing was starting to overlap more and more. And so we joined together to be a single organization where I personally am focusing on the open source, but Ruby Central as a whole is focusing on kind of like the high level Ruby community as a whole. So that includes conferences, that includes open source, that includes trying to figure out how we can. Help meet up organizers. That includes trying to figure out how we can do other things that sort of like benefit the community and move things forward.

Collin: 2:16

Yeah, that, that all makes a lot of sense. So how did you get involved with Ruby and decided you wanted to start this original nonprofit, uh, Ruby together?

Andre: 2:25

Yeah, absolutely. Um, I, I kind of love talking about this question because of the different things it shows about the Ruby community. Um, so, uh, I actually got involved in Ruby the language really early on. Um, I was an undergrad in, uh, the like mid 2000s, and I was... I was very bored of my CS class that was C only, and I ran across this tiny language that only had two books about it written in English. Uh, the, the Pickaxe Programming Ruby book and the Ruby Way. They were the only books in English at the time in like 2003 or something like that. And I wound up just like falling in love with the, the language Ruby. Like, I really, it worked really well for my brain. I loved it. And I loved kind of like using it to write programs and it felt really good. And then, uh, shortly after I had decided that I really liked Ruby and wanted to figure out how to use it more. Uh, I saw this, this email on the RubyTalk mailing list that said, Hey, I made something to help you build web applications. You should check it out. And that was the announcement for I think 0. 9 of Rails, maybe 0. 10. Um, And so I wound up starting to use Rails and just kind of like being blown away by the, uh, speed at which you could do things, right? I feel like that's like a big component to why people like using Ruby and why people like using Rails is you can kind of like imagine something and then you can try writing it and then find out if it worked or not. And it takes basically that long. Um, And so, uh, I wound up going to a really early RubyConf, um, actually right around the time that RubyCentral was getting formed, um, and then, uh, went to the first RailsConf in 2006 and ran into some people that I became friends with and wound up going to work at their company. Uh, and, uh, by a few years later, uh, I had a job at Engine Yard, which was one of the, like, very early, kind of, like, big Ruby companies that employed a bunch of people doing open source. At the time, they were paying, um, a couple people to work on Rails, a couple people to work on, uh, Rubinius, R I P Rubinius, the Ruby implemented in Ruby, and paying some people to work on JRuby. And, uh, and as a dev there... I was very excited about trying Rails 3. This is, we've, we've now transitioned to the part where it's about how I got involved in the open source. And so, uh, at Engine Yard, I was very excited about trying the Rails 3 beta. And so one night after work, I went to try it, and immediately, like, it was like, Oh, you have to use Bundler now? Okay, guess I better install Bundler. Uh, okay, you run bundle install. And it just, like, exploded with, uh, exception. And I was like, well. That's not super helpful. Wait, this is a, you know, no method on nil. Maybe there's just a missing object that's obvious that I can fix. And there was, and I had it fixed in like, I don't know, half an hour. And then the next day I went to, uh, it was probably Yehuda Katz, although I'm not sure, one of my Engine Yard coworkers who was working on the open source stuff. And I was like, Hey, I sent you a pull request. Could you like fly the pull request today so that I can try this tonight, uh, and see what all the Rails 3 hype is about. And the reply I got was something along the lines of, I'm too busy, but. Uh, you can merge the pull request. I've given you full admin access to the repo. And so then I, I mean, one, I was a little bit stunned, but, uh, but two, I was like, okay, I can do this. And I went and merged my PR. And then I saw that someone else had opened an issue with the same problem. And so I said, oh, Hey, I fixed your problem. And then I saw someone else had opened a new issue about a slightly different, but similar problem. And I was like, oh, I bet I could fix this one too. And then I fixed that one. And then I. You know, updated the issue to say that I'd fixed that one. And I don't know, fast forward, maybe three months. And I was basically fixing Bundler issues every night. And at that point, the other folks on the Bundler team said, Hey, we're already giving talks at RailsConf about Rails. Can you give the talk about Bundler? And that is how I wound up a member of the Bundler core team. Uh, just by showing up with a PR and then not going away. That's yeah, I don't know I feel like open source has stories that are all equally I feel like they're equal parts, like, totally banal, like, mind numbingly boring, and just kind of like, wow, I can't believe that that was actually possible. So that's the story of how I got into Bundler.

Collin: 7:19

Yeah. Wow. That's, uh, that's, that's pretty crazy. And now I guess you've been doing, uh, so you've been mostly involved just in doing open source now for, I guess, at least over a decade, right?

Andre: 7:31

Yep, definitely. Um, so, uh, so we, we shipped the first version of Bundler in, uh, 2009 or 2010 or something like that. Um, and a few years later. Uh, I kinda like looked around and I noticed that I was the last person from the Bundler team who was still, like, working on it. Everybody else had gotten new jobs or gotten new hobbies or gotten, uh, more of an idea of what they wanted to do with their life. And, uh, and, and then I was like, wait, this is, this is gonna be a problem. Cause like, if I get a job and then stop working on this. Like, Bundler won't have any maintainers, and I think that's a problem because Bundler is now the single most used tool of anything written in Ruby. Uh, because every project now uses it. It seems really problematic if Bundler has no maintainers. So, I should try to do something about that. And, uh, the, in a, a long story with many ups and downs short. Uh, I wound up founding a non profit specifically to solve that problem, or with the hope that it would solve that problem. Um, so RubyTogether was a 501c6 non profit, which is a special kind of non profit that, uh, is slightly different from the more general, like, if you hear about a non profit, you probably automatically think of 501c3, like the World Wildlife Foundation, or, or actually even Ruby Central, which was founded, uh, Longer ago, and for a kind of broader purpose. Um, for open source projects in particular, the IRS doesn't let you form this kind of broad kind of non profit, what's technically called a charity. Turns out the IRS does not view open source projects as charities. Uh, although, maybe at some point in the past, someone went to the IRS and said, We're giving away software! And the IRS was like, Wow, that's like a charity! They don't do that anymore. Uh, and so, the 501c6 structure is what's called a trade association. And you've probably heard of some trade associations. They're like, uh, the American Dairy Farmers Association that buys GotMilk ads, or the Linux Foundation that pays, uh, Linus Torvald's salary. Uh, those are all trade associations where there's a group of companies, they all have a shared common interest, and they all kick money into a non profit together, and that non profit... Uh, does something that benefits everyone in that line of work. And so, what RubyTogether did was it collected membership funds from Ruby using developers and Ruby using companies, and we paid developers to work on Bundler and RubyGems, and we paid for kind of like the infrastructure that So when we had more money, that sometimes meant that we could pay more devs, or it meant that we could support work on like the Ruby toolbox, a very helpful gem index website. Uh, but fundamentally what it came down to was funding the developer work that keeps Bundler and Ruby gems functioning on a day to day basis.

Collin: 10:42

So you formed this organization essentially to, uh, to maintain and develop Bundler and RubyGems and then I guess realized that you and, uh, RubyCentral had sort of a, you know, um, there, there was a lot in common with like the goals and

Andre: 11:01

Exactly, exactly. Yep. And so the, the story there is that. So, uh, a long time ago, like a good 10 years ago, uh, I had even, uh, talked to the Ruby Central folks about whether they would be interested in starting a program like what Ruby Together became. And at the time, they were a very small, very lean nonprofit, and they were kind of like laser focused on the conferences. They wanted to make sure that RubyConf and RailsConf happened every year. They didn't really have any, like. Uh, Slack or Buffer left over after that, and they wanted to be really sure that the conferences wouldn't, you know, get, uh, you know, there wouldn't be any distractions for them or there wouldn't be anything that could potentially interfere with them. And I said, that's totally fair. I've, I have a completely untested idea that no one has ever done before. I can see why you wouldn't want to like chain it to the conferences. Uh, but by the time RubyTogether had been running for maybe five years. Uh, and Ruby central had also kind of like come to realize that they didn't want to be laser focused on just conferences, that they wanted to like support the whole Ruby community, getting more interested in, uh, re helping regional conferences, helping meetups, helping, you know, kind of like, uh, work with the ecosystem of people using the Ruby language. Uh, that was when we started to realize that as Ruby Together wanted to do more stuff that wasn't just. maintain Bundler and RubyGems, and as Ruby Central wanted to do more stuff that wasn't just run RubyConf and RailsConf, we were like starting to overlap more and more. Uh, and so at that point, the boards of directors at both non profits got together and said, hey, like we have this idea where we could be a future organization that sort of like is effective at this broader range of things for the whole Ruby community. And we could, like, execute on both of our respective visions to keep the programming ecosystem for Ruby alive, to keep the meetups ecosystem for Ruby alive, to keep the conference ecosystem for Ruby alive. And that was when we realized that we, we pretty much wanted the same thing. Um, and so that's, that's actually pretty recent. We've, we merged together. Uh, I want to say, uh, 2022, um, maybe, maybe like the very beginning of 2022, something like that. Um, and so we've basically been working to figure out what it means to be a single organization. And now we're working, uh, on what it means to sort of like actively do outreach to the Ruby community and figure out how we can work with them and support them and make sure that. Uh, all of the things that we're doing are getting support from the Ruby community and are providing support to the Ruby community in return.

Collin: 13:52

So in the scope of open source, right, you mentioned Bundler, you mentioned RubyGems, um, does it, so is most of your work focused on the development of those? You mentioned, like, meetups and conferences and things, but are you also doing, like, outreach and trying to support open source projects or what does that look like?

Andre: 14:11

totally. Absolutely. So Ruby Central as an organization has kind of like a high level mandate to do conferences and they're working on expanding the conferences to include more community oriented events and figuring out how to support meetup people who are running meetups. I personally, as kind of like an outgrowth of my role for Ruby Together, And like really focused on the open source stuff in particular. And so, uh, head of open source ultimately ends up meaning, uh, negotiating with companies that want to support open source and negotiating with open source projects that have kind of like an ecosystem wide impact about how we can support them. Um, and so, uh, today the really big, obvious examples of that, that are kind of no brainers are things like. Bundler, where it's a tool that every single Ruby project uses. If we, you know, if we fix a bug in Bundler, that's like a positive impact on everyone using the Ruby language, and so it like just makes sense to do that. Um, the other kind of like super easy no brainer is the rubygems. org infrastructure, which is actually very large and very involved and soaks up a significant chunk of our kind of like, uh, open source Ruby ecosystem focused time. Um, I have some impressively large stats that I can talk to you about RubyGems. org, uh, when we, when we get to that. Uh, and then kind of like the, the third aspect of supporting sort of community aimed open source projects is figuring out things like, uh, is there a better way to surface documentation? Or like, working with a project called RubyAPI. I think the website's RubyAPI. org? It's like a, trying to be a successor to kind of like the venerable RubyDoc. org and uh, you know, like these other documentation sites that were all kind of scattered and kind of haven't been updated in like a good 10 years. Um, there's the Ruby Toolbox, which I mentioned before, which is kind of like a hand curated selection of gems. in categories where you might be like, if I want to do this kind of thing, what are my options in terms of gems? And, uh, kind of like, what's the activity like on those gems? When have they been released? How, what are their GitHub issues look like? How's the activity in their commits? Uh, you know, like, is it still maintained, uh, and the Ruby toolbox is like set out to figure out those things by looking at GitHub stats, looking at rubygems. org stats, looking at these various other indicators about each gem. And so it's like, uh, it's absolutely like an invaluable resource if you want to like figure out. Whether there's a Ruby library to do a thing, like the Ruby Toolbox is always the first place that I go. That's been one of the projects that we sometimes try to support. Um, and then, uh, just kind of like, very broadly, trying to be a good open source ecosystem participant, right? Sometimes that means, uh, up streaming into other open source projects, changes that we make because of Bundler, or changes that we make because of RubyGems. Sometimes that means participating in Uh, kind of like community wide efforts to migrate from one, you know, one old deprecated version of a gem to another. Sometimes that means working on, uh, improving security. RubyGems. org has historically been, you know, kind of like one of the richest targets for security vulnerabilities in either Ruby or in Rails. Um, and in fact, one of the motivating incidents way back when for me to think about starting Ruby together was... Uh, I want to say it was around 2012 or 2013, there was a security vulnerability in Rails and it, I guess not the exact vulnerability in Rails, but kind of like a next door vulnerability in RubyGems. meant that the RubyGems. org server itself got hacked. And someone posted the password file to Pastebin, And we just had to like, take the entire server offline because it wasn't secure anymore.

Collin: 18:27

Oh, Jesus.

Andre: 18:28

it was yeah it was really bad. Like all of RubyGems. org was down for three or four days while we checksummed every single file in S3 to make sure that no one had... Kind of like sneakily replaced any of the gems with malware.

Joel: 18:43

Wow.,

Andre: 18:43

Um, it was, it was a big deal. And it was definitely one of those cases where like, at the time, everyone working on it was a volunteer. And we even knew about this potential problem. And we had said, oh, this might be a problem. And we should have someone look into it. Oh, all of us are busy. Okay, whoever has free time at the end of the work week, you know. So, I'm going to the chat if you're the first person to have free time to look at this. But, we'll have to wait, you know, until the rest of us finish our day jobs, because we've got to pay bills. And, of course, that led to the entire server going, you know, the whole service going down. And hundreds of people saying, I literally cannot do my job because rubygems. org doesn't exist right now. Uh, and, you know, all of those people told us, like, my boss has said that I can do, You know, whatever, instead of my job, because I can't do my job to help you guys fix this. And of course, when the problem is security has been compromised, and you need to very carefully verify that everything is still secure, uh, giving random people access to help is not actually going to solve that problem. And so, uh, the, the really disappointing thing for me personally, and that led directly to kind of like trying to create this situation with memberships and with a dedicated, paid team, was that as soon as we got it back up, those hundreds of people who had all said, my boss said that I can do nothing but try to work on this, uh, all had to go back to their jobs. Right? And, and so two weeks after this giant disaster, we were back to 100% volunteers working on RubyGems. org. And if we'd had the same security problem happen again, we would have had the same issue where none of us, like, we would have had to choose between PTO And, uh, you know, being able to fix this security problem. And that's just like, not, it's not a good way to run an open source ecosystem, even if it is a community project. Right. And so the underlying pitch behind Ruby central's current open source plan is. Uh, all of the people and companies that need RubyGems to stay up and operation, you know, like to be working so that they can do their jobs, so that they're, so that they can deploy, so that their business can operate. Uh, we need contributions from them so that we can pay the developers who keep RubyGems up so that they can all do their jobs. Um, that's, that's ultimately the, uh, the membership program at Ruby Central in a nutshell. Uh, you know, kind of like the, the core thing that we're... Trying to do initially with

Collin: 21:30

Wow, that's, that's a crazy story. So for three or four days there, I guess we're saying is if somebody hit like bundle install or whatever, it would say

Andre: 21:39

It literally did not work. There was nothing there People tried to deploy by you know Like maybe they had a copy of their gems installed somewhere that they could scrape together and then shove into their deploy Artifact and keep deploying but like changing gems not a thing for those three or four days Yeah, it was wild.

Collin: 21:59

It's sort of amazing that something that was that crucial did not have full time support on it

Andre: 22:06

Absolutely right, absolutely and uh And to just kind of, like, give you, uh, an idea of, like, the scale of the issue, right? So, like, uh, I am proud to report that ever since I started this program with the memberships, uh, we have managed to keep rubygems. org from going completely down again. Ever. Uh, you know, sometimes it gets like kind of sketchy and you see an error or two, but we've kept the site from going a hundred percent down, uh, which was kind of like the, the goal that we set out to do. So that part has been a success. Uh, the part that hasn't been as much of a success is like you were saying. It's really hard to imagine that a project of this magnitude doesn't have full time developers working on it. And the answer is it still doesn't have full time developers working on it. A decade into this project, we have. a small handful of developers who work on it part time, either as their single part time job or as a paid side gig on top of their actual full time job. But we have not yet been able to put together the budget to hire a single full time developer to work on RubyGems. And that's something that I think both we and everyone who uses Ruby would really benefit from, and so that's one of the things that we're actively trying to work on now.

Collin: 23:34

That blows my mind. Uh, cause this is a, this is a project that like publicly traded companies worth billions of dollars, right? Like

Andre: 23:41

Absolutely. Yes. And if those publicly traded companies worth billions of dollars, uh, I guess apparently can't deploy sometimes if it were gone. Uh, at least that's what happened ten years ago. So,

Collin: 23:55

Yeah. It seems.

Andre: 23:57

yeah. Seems, seems mind boggling to me.

Joel: 24:00

It's, it's not just that they can't deploy if it goes down, whenever they do deploy, they're depending on that infrastructure being like secure and that the gems that they're getting.

Andre: 24:13

Oh

Joel: 24:13

been tampered with, right? So it's not just like, they need to be worried about like, maybe I can't deploy. Literally every time they are deploying, you know, they, they brag about deploying every 10 minutes.

Andre: 24:26

Uh

Joel: 24:26

You know, they're, they're using your infrastructure and trusting it, so.

Andre: 24:29

that is also true and, and in fact, one of the things that, that handful of part-time volunteers does is notice anytime someone uploads a gem that is trying to pretend to be a different gem with a typo in the name or, uh, notices anytime someone has gotten their Gmail hacked and there's a new version of a gem that has malware in it. Or, uh, yeah, like that's, that's definitely one of the things that we actually do on a daily basis is, uh, is like secure kind of like the gems themselves against malicious actors.

Collin: 25:09

I imagine there's a lot of challenges, but would you say like the biggest challenge then has just been around having the support, right? To be able to put, you know, uh, like full time effort into this?

Andre: 25:21

Yeah, absolutely. I would say the biggest challenge has always been getting the support. Um, you know, in, in, in some ways, uh, Because this is such a, kind of like, grassroots, ground up, community funded project, and because the Ruby community as a whole is like, against the idea of, you know, kind of like, a for profit commercial company taking this over, not that that's a viable, you know, plan, but like, the community ethos is very much against even attempting that. And the kind of, like, core challenge there is, is in some ways kind of like a tragedy of the commons where, uh, there is a RubyGem server and a company shows up and says, Oh, I can just use this. Okay. And then they do. And then we say, Hey, like, you know, you're doing thousands of requests a second worth of traffic and we have... You know, an entire segment of our AWS build dedicated to you personally, and they say, Oh, okay, we'll try to do less of that, but like, you know, actually helping us run the infrastructure, not so much. Um, so that's, that's definitely been the really difficult challenge.

Collin: 26:37

So you're actually paying for the infrastructure that's not donated?

Andre: 26:41

Uh, so it, it has depended on, you know, from like month to month and year to year. Um, right now, I am happy to report that the actual servers themselves are donated by AWS and the, uh, kind of like the global distribution is donated by Fastly. Um, there, there was a period of, uh, several months sometime. I want to say it was six months ago, maybe it was 12 months ago. There was a little bit there where we, our, uh, donation agreement with AWS lapsed. And so for a few months we were paying about 10, 000 a month in server bills until we got that back online. Um, but for the time being, the actual server bills are being covered by Amazon, and the actual bandwidth and CDN bills are being covered by Fastly. And it's a really good thing that they are, because the scale of each of those things is truly mind boggling. Like, uh, the last time I went back and looked at the stats, we served 100, 000 unique client IPs. Like, we don't know exactly users, right? IPs is the closest analog that we've got. 100, 000 different users, and we served 2 billion gems in that month to the 100, 000 users. Um, and 2 billion gems is a lot of gems. Um, like. Uh, I, I went, I went back and hand calculated it. And if we hadn't gotten donated infrastructure from Fastly and from Amazon, our server bills would have amounted to something like 500, 000 a month.

Collin: 28:29

Yeah, that's, that's, that's a number. Um, yeah, I was imagining I'm like just, you know, uh, the number of people, you know, doing deploys. Using this service, like, yeah, that must be like extremely provisioned, right?

Andre: 28:44

Definitely.

Collin: 28:45

are there a lot of challenges with that? Just in like, you know, uh, Like, performance wise, whatever, anything, like, technically, like,

Andre: 28:53

Yeah, absolutely. It's, it's been a super interesting adventure to work on this in the long run. Um, one of the biggest, uh, one of the biggest drivers technically of various changes that we've made to the way RubyGems works and the way that Bundler talks to RubyGems has actually been, you know, uh, ultimately in the end kind of like user facing latency, right? Um, so like, uh, in around 2012, I think, we built a new API just for Bundler to talk to RubyGems because the original way that Bundler fetched information about gems was very inefficient, it wasn't optimized for Bundler's needs, um, and so sometimes it would take like, Sometimes it would take like a full minute to run a bundle install, and we realized that with a specialized API that was optimized for bundlers needs, uh, we could get that down to like maybe 10 seconds, which was this huge, amazing, fantastic improvement. And then I went to a Ruby conference in South Africa, and they revealed that bundle installs had gone from taking five minutes to instead taking merely five minutes. Uh, and as we kind of like dug in, we realized that this much more optimized way of fetching information still had a large number of round trips.

Joel: 30:19

Mm

Andre: 30:20

And so if you were very close to a server, and by a server, I mean the single server that could authoritatively answer this optimized API query dynamically in Virginia, in the U. S., if you were close to it. You could install in one minute, you know, in 10 seconds instead of 60 seconds. But if you were so far away that it took you five minutes to bundle install before this new API, it actually still needed so many round trips halfway across the world that it wasn't any faster to use the new API than the old API. Um, and so one of the big things that we did when we re we, we did yet again a new way of Bundler to fetch information about gems based on the problems we were seeing with that. Uh, and we rolled that out, I want to say, in 2016. Um, it's called the Compact Index Format. Um, and the, one of the biggest criteria that we had for that format was that it be plain text, flat files that were incredibly easy to cache, that you could update just by, like, fetching whatever was new since the last time you fetched it. And that you didn't need to, like, do many round trips that all had to complete one at a time. Um, instead you could do one round trip once, and then you could fire off as many follow up requests as you needed simultaneously in parallel. And so, ideally, you would only need to wait for maybe two round trips, rather than potentially having to wait for dozens, which was the, the old setup. Um, so, uh, the, the compact index is the standard today. Um, we, we actually weirdly noticed this trend over the last few years of anytime the site got unstable and started occasionally returning errors, it was because we were getting slammed on the bad old... API that was very optimized for Bundler, but very not optimized for the server. It, it constructed this gigantic, extremely gnarly Postgres query, and it was basically uncacheable. And so if you hit our server enough times simultaneously on the old API, it would like... Um, and so that was actually something that we finally managed to take care of this year where we announced a deprecation period, we started doing brownouts, um, and what we discovered is there were some people using that old API a Um, one of, this is actually, uh, maybe yet another kind of like mind blowing story where. We discovered as we turned off that API, that sometimes we were getting retries or fallbacks to the even older, uh, way of talking to RubyGems in the tens of thousands per second,

Collin: 33:20

Wow.

Andre: 33:21

by the time, so, so the absolute worst one when we were first starting to do these brownouts where we would turn it, turn it off for a little while and then turn it back on to like, kind of like let people know, Hey, this is going to go away soon. Like you shouldn't be depending on it. And so we did one 24 hour, uh, brownout. And during that time, we actually saw 200, 000 requests per second because of all of the, like, hard coded retries and all of the fallback to more, you know, like, more requests, uh, slower API that was older because no one had upgraded past the version of Bundler that only knew how to talk to this old API.

Collin: 34:03

And so, it's just spinning. It's

Andre: 34:04

yeah, and so it was just, like, frantically making requests to RubyGems. Like 200, 000 requests per second is like the tier that Shopify brags about during Black Friday season level of requests, right? Like, uh, it's just like unbelievably high numbers of requests, right? Uh, we eventually discovered that the vast majority of that traffic was coming from fleets of VMs that were all running. An old version of Chef running an even older version of Bundler that literally had not been upgraded since we made the compact index in 2016. So like, I don't know if they installed them in 2016 or if they installed an older Chef later, but It meant that there were thousands upon thousands of machines that every 30 minutes would throw away everything Bundler had done previously, and then start a new bundle install as a prerequisite to do a chef run, which was a, like, 30 minute cron job. And so we, it was just, it was incredible.

Collin: 35:11

So, so, how, how did this resolve? I'm,

Andre: 35:14

So we, we hammered at it from every way that we could figure out how to hammer at it. Some of it was directly talking to the most impacted fleets of VMs. So like the very largest single user was like one IP address doing 25, 000 requests per second to RubyGems by itself. Uh, we eventually managed to get a hold of the folks who were running that fleet. Um, and, uh. It was the egress proxy for some, I don't know if it was every single server, but for some significant percentage of every server run by CrowdStrike, the security vendor. Um, and I guess they just didn't realize that they had all of these VMs running this very old version of Bundler, and we were able to work with them to get it upgraded so that it could use the compact index so that it wouldn't frantically retry every few seconds. Um, we also wound up working with, uh, Amazon. They had a, a service called Ops something that I now forget the name of. Uh, and it apparently, uh, it was like a, an AMI vending machine where you could say, I want... A VM that's controlled by Chef that, you know, is like configured in these ways and then it would just like hand you the VM for you to run. And so, they didn't actually, it was OpsWorks, that was the name of the product. Um, but they didn't have a way to like upgrade Chef themselves. So they had these customers who would get mad at them if the VM stopped working, but they didn't have a way to like Replace software that was in a VM that they had given to customers Potentially years ago and that those customers were now running on EC2 because like they'd handed off the VM to the customer Long ago, and so it became this like hairy kind of situation of like, we're gonna set up more brownouts We're gonna figure out how to do messaging to the, you know Like we're gonna help them figure out a way to message to those customers what they need to upgrade and then we're gonna like actively Turn it off, you know every other day for several weeks Just to like make sure that no one who is depending on this is gonna think that it will keep working in the future Stuff like that. It's uh, it's Continuously a challenge is how I would describe it

Collin: 37:40

Yeah, that sounds, I, I assumed it was challenging, and that sounds more challenging. That's, sounds like a lot.

Andre: 37:48

Totally. It is definitely a lot.

Joel: 37:51

Scale of this is incredible. I had no idea that RubyGems had that, that kind of scale. I'm really impressed that you managed to contact CrowdStrike. Was it, um, like, how did you figure out from an IP address who it was?

Andre: 38:07

so, uh, that is also a funny story. We spent a few weeks trying to backtrack through their ISP. Like we reached out to the, the, like the, you know, the ISP that serves their data center where the egress was located. And we were like, Hey, we're getting a lot of like traffic from this IP. Could you put us in contact? We literally never heard back. Um, so the, the magic solution. was to write a Fastly rule that had a hard coded response, if the IP address is this, return a JSON payload with an error that says you are murdering us. Please send an email to support at rubygems. org and we will try to help you get back online, but you just can't. I'm sorry. Um, and that actually worked. We, we heard from them like the next day after we put up. A like IP focused error message.

Collin: 38:59

that is some seriously creative problem solving.

Andre: 39:02

Yeah, definitely.

Collin: 39:04

Um, well, geez, this is, this is all really interesting. So we've talked a lot about what, you know, all of this RubyGems, Bundler, um, your history with all of that, RubyCentral. Uh, so what are you focused on in the future as far as, like, improvements you can make or

Andre: 39:22

Absolutely. Um, so, uh, one of the benefits of having worked on this for the last 10 years, especially in a kind of like scrappy shoestring budget under resourced kind of way, is that we have a really long list of stuff that we are sure would make things better. Um, like, uh, user requests or ideas that we've had or things that we've started to work on and have kind of like proved out the prototype of, but like, ultimately they're all things where we've, you know, kind of like built up a roadmap and said, wow, it would be really great if we were able to, you know, do some of these things. Um, and, uh, that roadmap has, you know, just kind of like, uh, An enormous amount of stuff on it. I can, I can kind of like give you a tour through some of the highlights, maybe if you want, um, in the direction of like, uh, let's see, how would I describe this? So there's a few different categories of things that we really want to work on. Um, and I would, I would describe the high level categories as like, uh, making Ruby gems more secure for everyone. And that includes kind of like the, uh, the somewhat buzzy projects like, uh, um, SigStore or the Update Framework. But it also includes things like, uh, getting support for passkeys as an alternative to passwords. So far, we've implemented passkeys as an alternative for 2FA so that you can, you know, replace like a one time... code with a passkey instead. We definitely want to keep pushing that forward and offer passkeys as an alternative to passwords entirely. Some of the other stuff that we're working on at this exact moment is like getting full checksums written into the lock file and checksum verification every time that you download a new gem file. That's like a A way to proactively prevent the kind of, like, disaster that I was describing a decade ago where we had to checksum every gem because we weren't sure if they'd been replaced. Well, if your lockfile has a list of the checksums in it, you can be really confident that no one silently swapped out that gem while you weren't looking. Because it's the same one you installed last time. Um, we also have kind of like, uh, security focused projects that include things like, uh, getting, uh, building, it's, it's, uh, technically it's using OIDC to authenticate gem pushes. Um, but in a, in kind of like more layman's terms, what it ends up meaning is, uh, publishing that's as secure as 2FA, but that doesn't require you to copy and paste a token into GitHub Actions to work. Um, so we can get automated, verifiable, uh, you know, kind of like, uh, consistent builds that anyone can see, yes, I built the same package from the same source code, and that's the same package that's on RubyGems. And the checksums all line up, and the tokens that were used to push them were all temporary tokens. And trying to, trying to push the security of the packages themselves and the ecosystem as a whole forward as much as we can. Um, another class of stuff that we're really excited about working on is making it better for users of gems, right? Like... Uh, you have probably noticed, if you go to rubygems. org, that there is technically a webpage for the gem, but it doesn't really contain that much useful information unless you want to know, like, what versions were pushed on what days. Uh, but that's basically all the information that's there. Our roadmap for that includes, uh, pulling more information from GitHub, pulling information from the Ruby toolbox, adding the ability to see the project's README, adding the ability to, like... Browse through what's inside the gem that we have on rubygems. org. Um, being able to view diffs ultimately, right? Like we'd love to be able to show you, uh, here's, you know, this gem that we got pushed and here's the diff to this other gem that you're thinking about upgrading to, you can see it right here. And this is the actual contents of the gem rather than the contents of the repo that eventually created the gem, right? Uh, repos have like strong correspondence to gems. But what's actually inside the gem isn't exactly what was inside the repo, it's like a built artifact. Someone ran a build process and created a, you know, a tar tarballed up dot gem file and gave that to us. And so we want to give users more ability to see what's inside those and, uh, you know, like be able to... access information to answer questions that they have about them. yeah, yeah. Um, I, I could, I could go on for a long time, but I also want to be, uh. Mindful of the fact that we only have so many minutes for the podcast.

Collin: 44:45

yeah, absolutely. I think, yeah, I think that's a great place to, uh, to wrap it up. If, uh, Joel, if you don't have any more, more questions or anything, um, but, uh, yeah, so it seems like you have a lot of, uh, improvements you'd like to make in the future and you just need one of these, uh, a couple of these billion dollar companies to pony up a little bit and, uh, and it'll be easier.

Andre: 45:09

That would be amazing. Um, absolutely yeah, there's there's a bunch of stuff that we would love to do that would make make things better for the community as a whole and And yeah, we're actively working on on sponsors to get us to the point where we cannot just you know Kind of like keep it up and keep it running and keep it not getting hacked But also, you know, like actively make it better We're really excited about that idea.

Collin: 45:35

Yeah. Mm hmm. Yeah, that's incredible. So, maybe tell people where they can find you, or anything you'd like to point them to, and

Andre: 45:43

absolutely. Um, so, uh, the Ruby Central website is at rubycentral. org. And if you're interested in becoming a supporting member to help. With the budget for all of this exciting open source work, you can do that at rubycentral. org. Um, and if you're interested in getting involved in Bundler or RubyGems itself, the GitHub organization is at github. com slash rubygems, and the repos for rubygems. org and Bundler and RubyGems are all in there. Um, and if you want to reach out to me directly for any reason, my website is mylastname. net, A R K O dot net. And you can find me on all of the internet services or just send me an email from there.

Collin: 46:26

That all sounds amazing, and people should do that. In the meantime, uh, Thanks for listening. If you like the show, please make sure to tell your friends and family, uh, if your family is full of Ruby developers and we will see you next week.