We doubled engineering productivity at eBay, but couldn't change culture transcript

This is my first lead dev. So, I'm so excited. I'm going to tell you today about the achievement that I'm most proud of in my entire 38-year professional career, and then I'm going to tell you why I got fired for it. So, this talk is about uh I've been at eBay twice in my career. First, for 7 years as an individual contributor back in the early 2000s, and then most recently as chief architect and VP of platform engineering between 2020 and 2022.

And that's the story that I'm going to tell right now. So, title is we doubled engineering productivity, which we totally did, and that's the thing I'm really proud about. But, the thing that got me fired, I guess, is that we weren't able to change the pathological organizational culture of the company. Uh okay. So, uh let's see. Who's heard of eBay? Raise your hand. Keep your hand up if you have bought or sold something on eBay in the last year.

That is the problem. Uh okay. So, in the last 20 year I mean, eBay's been around since '95, but in the last 20 years, it's basically been a flat business. So, this the sum total of goods and services transacted through eBay, what's called the gross merchandise volume, was 50 billion US dollars 20 years ago, and it is now just under 75 billion US dollars. You're like, "Okay, well that that grew by uh 50%." Well, Randy, what happened in a flight to inflation uh during those 20 years?

Well, inflation grew 61%. So, actually, if you if you spent a dollar on eBay's market, which you kind of couldn't, uh and then you waited 20 years, you would get 91 cents back. And then you'd say, "Well, gosh, Randy, didn't US e-commerce grow a lot during that period?" Well, as as a matter of fact, it did. It grew 9x or 6x in constant dollars. So, if you compare the two in constant dollars, so let's hold the you know, hold the cost constant uh independent of inflation. uh A dollar bet on US e-commerce would return $6 20 years later.

That's a decent return. eBay would return a loss of 9 cents. This was known >> [laughter] >> when I was back there at 2020. And these were the words that the CTO told me. He and I worked together in my previous stint there. We both had different roles. But he said this. He said, "Look, Randy, you you went out and you did Google and you did Stitch Fix and you did some of these other things. I need you to come back as chief architect and help bring eBay into the modern world of software.

And that's a pretty big challenge. So, I was I was up for it. And here's what we were looking at. So, eBay had about 5,000 engineers then. It's roughly the same now. 3,000 in what they called core product. So, like customer-facing and all the back office stuff. And then 2,000 engineers what they called core technology, which is essentially all the platform and infrastructure. And eBay runs and operates its own data center.

So, like that's why there's quite a number of engineers that are associated with that. What we were seeing was multi-quarter initiatives that typically and commonly would involve 50 or more teams. We had 4,500 applications and services active in production. On average, the deployment frequency of those applications was once or twice a month. And the lead time for change was 10 days. So, what's lead time? That's the time from when a developer commits her code to when it actually shows up in production serving value to customers.

Okay, so there's a lot of work to do here. Especially if you're going to wear the chief architect hat. So, we started this initiative that we called velocity, which we wanted you know, we wanted to speed up in a good direction. And that's velocity. I'll tell you the end story first and then I'll fill in how we got here. So, the end result is that we were able to double engineering productivity at the company. So, what is doubling engineering productivity mean?

You can think of it as the bug fix features and bug fixes per unit time. So, a team before we work with them would be able to produce n features and bug fixes per week or per month, and after we work with them they were able to produce 2n. That feels pretty good. Uh along the way the DORA metrics improved quite a lot. So, the deployment frequency improved 10x. So, we went from once or twice a month to daily deployments for most of the applications.

Lead time improved 5x from 10 days to 2 days, and we weren't even focused on the stability metrics of change failure rate and MTTR, but as the Nicole Forsgren's research in the accelerate book and in the frictionless book would tell you, um by improving the speed metrics we also improved the stability metrics as well. So, if you want to look at the accelerate or DORA metrics, we were a very solid medium performer uh before we started this work.

That's 35th percentile. 35th percentile, and we were able to move the company over a number of years to 75th percentile as high performers. So, again, I'm very proud about this, and it wasn't me. It was an entire uh team, and we worked with all the teams at eBay over successively over time, and I feel really really proud about what we did. How did we do it? Thank you for asking. Um I did not have any novel ideas at all.

No novel ideas at all. We were simply executing a very standard DevOps transformation playbook, where the mental model is go and talk to the teams that are actually doing the work, ask them what if their bottlenecks are in order to be able to go faster, fix those bottlenecks, right? So, the the cycle was simply "Hi, what's your current reason that you can't go faster?" And there are legitimate reasons. Let's work together on releasing that bottleneck.

Then, as somebody once said, "Once you solve problem number one, problem number two gets a promotion." Uh so, we focused on identifying and removing bottlenecks. So, many of the teams were bottlenecked on build time, test time, startup time for the applications, obviously PR validation times, right? Code reviews. Um that's a that's a big chunk of everybody's time, particularly now. Um we invested heavily in the common staging environment.

Uh we automated everything we could automate. So, we automated upgrades from of, you know, dependent software. We upgrade we automated testing, as I hope you're all are doing now. We automated the deployment process, which had been like partially automated, but still required a human to click buttons to move things from stage A to stage B to stage C. And we automated uh the measurement of what we call site speed. So, that's like user experience latency, right?

Like so, you're using one of the one of the pages and automate the collection of data about uh the latency that the users experience. Um and then it wasn't just technical changes. In fact, some of the most meaningful changes were changes in how teams operated. Like, uh all right, every day after the morning standup, like do all the code reviews that are open, you know. Uh streamline the inner right relations between teams.

I produce a library or a uh service that other teams use, instead of coordinating back and forth individually with my customer teams about, "Can you test with my new version?" We did some more automation around that, so on. Uh how we worked. This was new. This was new to eBay. We collaborated. Uh Uh so, I came in I worked I didn't say, but I worked on the platform side. And so, you know, our customers were those other 3,000 engineers that served essentially all of you.

Um and this is not unique to eBay at all, but there had been a uh let's call a fractured relationship between the platform area and the and their customers. So, the platform people are like, "Well, we keep releasing new versions of the framework, and nobody uses them, and why not?" And the customer teams are like, "They keep producing uh new versions of the framework forcing us to upgrade, but they're not solving the actual problems that we have.

And so, lo and behold, we had this crazy idea that like we should take the people that want the software and the people that produce the software and put them in a room and talk about it. And it turns out like both sides are kind of nice. And and those and those conversations go really well. Uh, we did that also at the leadership level. So, I didn't by myself lead the lead this initiative. I had another co-conspirator, partner in crime who was on the that product side.

And he and I met together every single day to unblock each other and help each other out to make sure that we were entirely aligned. We did what I call an embedding model where we brought really senior individual contributors into my central platform team. And then in my phrase, I would send them out into the provinces to like go help pair with teams you know, on the product side, help them help to understand their problems in in situ, bring those thoughts back to the platform team so that we could solve them, solve them in place when that's possible to do, etc.

And one of my very favorite actually the very best individual contributor engineer that I've ever worked with is in the audience here. His name is Justin Abrams. He was the best guy of one of those embedders. And again, platform and product engineering working together. So again, daily leadership stand-ups at the at the like VP level. We had a weekly team of teams meeting where like a representative for all of the teams that we were working with any moment would come and basically be a stand-up.

So they'd say, what did you what did we do last week? What are we planning to do this week? What are we blocked by? And that engendered a lot of collaboration. So, you know, one team said, "Hey, well, we put this process in place in our team and we, you know, made our code reviews a lot easier because we did them all at the same time or whatever." And other teams were inspired by that. And "Oh, I built a little tool to, you know, hassle people when you're waiting on code review." And so, "Hey, can I have that tool?

So, it was very collaborative between product and the platform side, but also laterally among the product teams and among the platform teams. And then we did a monthly operating review with executive leadership. So, how we worked, as I was mentioning before, is a very standard cycle. This is called the Deming cycle. W. Edwards Deming was in the 1950s, and he was the American after World War that went over to the to Japan and helped out um Japanese manufacturing and auto stuff in particular to like rebuild and be the absolute best manufacturers of things on the planet.

So, we use the DORA metrics as the outcome metrics. So, those were the things we were trying to drive. The input metrics were a bunch of things that Google has identified as what's called developer friction. So, like we instrumented the entire deployment and development process in a very detailed way, and we could then see give a take a global view of what all 3,000 engineers were struggling with, and we would look at how we could make improvements in those bottlenecks.

And we had a dashboard that every showed the DORA metrics and other associated metrics for every application, rolled up to every team, rolled up to every organization, rolled up to the company. So, great visibility for everybody to see, be inspired by teams that are doing really well, and then offering help to teams that were still struggling with legacy and various other things. And again, the iteration loop is simply what is the next thing we might thing we might bottleneck we might want to remove.

We try something as an experiment. If it's good, we continue to do more of it. If it doesn't work, we back up and we try something else. A very simple plan, do, check, act. This is the Deming cycle. Okay. Here was the conversation that I had as chief architect. I would say, "Hi, I'm your friendly neighborhood chief architect, and I see that you're deploying your application you know, once or twice a month. Okay, great.

Imagine that I told you have to deploy your application every single day. Tell me all the reasons you can't. And they're like, "Okay, uh, how long do you have?" And do you want the whole list, Randy, or just the top 150 items? Uh, and that was great. I mean, and as soon as we started talking with more than a handful of teams, we started hearing the same things over and over and over again. And as the owner or the leader of the platform uh, team, I was able to say, "Great.

Those impediments that you told me are exactly my team's backlog." Your impediments are my backlog. Cool. And we had so much fun. >> [laughter] >> We genuinely had so much fun. It's fun to win, right? It's fun to make regular progress. Teams were inspiring each other as I was mentioning. Uh, and again, teams often the the mental model was platform teams produce tools, product engineering teams consume tools, but hey, turns out product engineers are engineers, too.

And they're capable of writing scripts and tools and all sorts of other stuff. Of course, they are. Um, and so teams would often solve their own problems and then laterally share those solutions to other people. Uh, one that one thing that's critical in the eBay culture was making it safe even for teams to say that they were struggling. So, psychological safety is very important as an individual human in your team, as an individual human in your family, and in your community, but it's also equally important sort of at the team level within the organization if that makes sense.

Um, and the other aspect of partnership that was super important is uh, in the organization that I was in, were lots of teams that most other people would see as as bottlenecks, right? So, like, "Oh, I'd go faster, but except for the security team." Okay, great. Let me go see if we can automate some of those security reviews. "Well, I'd go a lot faster, except for the compliance team." Oh, great. Let's automate SOX compliance.

Let's automate segregation of duties management, blah, blah blah. I'd go faster except for the accessibility checks. Okay, great. Let's do the accessibility checks in parallel or make them make them cheaper. I'd go faster except localization translation bottlenecks me. Okay, great. Let's do those translations in parallel. Get the English out first, bring the other ones in later. You know, so just thinking holistically and a system-wide level about taking a bottleneck that's slowing us down and seeing how we can make it go away.

And then for at least most of the two years that I was there, we had strong executive support. The CEO would continually say at all hands of the entire company, "This is the most important project that we're working on. Randy, can you and your team please go faster?" Okay, no pressure. We also presented my partner and I to the board of directors as well and it was mentioned in some of the quarterly earnings calls. We how we did it is we didn't start with everybody all at once.

Instead, we started with a set of 10% or so pilot teams. It started out with 10 teams. Other people heard that it was pretty cool. So, five more teams shoved their way in. So, we had started with 15 pilot teams. We worked with them throughout the entire year of 2021. We felt we felt like we had a playbook that was working really well. We'd automated a bunch of things that helped everybody immediately. And then we started adding quarterly cohorts.

So, we went to 25% of teams after whatever Q1 and Q2 of 2022. We kind of had quarterly cohorts from then on. And now as of at least a year, maybe even two ago, every team had been touched. Again, strong focus on automation. So, regularly deploying every app even if the code hasn't changed cuz you need to check your deployment process. Dependencies have may have changed, etc. etc. We also automated what we call the patch pipeline.

So, updating infrastructure libraries, again, updating things for CVEs and vulnerabilities and so on. Just a regular cadence of making updates to software that is in production. Uh and then since I can't have a talk in 2026 without talking about AI, um this was after I was there, um but they did a ton of AI. So like again, looking end-to-end at the software development process, looking at bottlenecks, and seeing how we can use AI to help remove those.

So obvious things are, you know, code generation and test generation. That's obvious. Legacy code migrations, that's also pretty obvious. But also, you know, summarizing PRs, what was the point of this thing? Automated code review ahead of humans reviewing it. Um uh cycle of looking at pipeline failures in the CI system, diagnosing them, producing an RCA, fixing them automatically when that's possible. So just really leveraging AI as much as we possibly can to remove bottlenecks and streamline the process.

The other thing we did that was super proud that made me super proud was we went from when I arrived in 2020, eBay's iOS and Android apps were being released once a month, and now they're released once a week, but we could do them in 1 day. Uh and there was a lot of work associated with re-architecting things, uh working on the release process. And uh one of the the most skeptical person about this transformation was my mobile release manager.

Wonderful guy. Uh quite a character, and uh he said, you know, I said, "Look, let's try to get to weekly releases by the end of this year." And he's like, "Randy, can't be done." I'm like, "Uh let's give it a go." Uh and then when I connected with him recently, uh he said this. Uh cuz we did we were able to make it go. Uh he said, "Randy, you were kind enough to help me adapt and see the light through air cover and rational small tests of the process." Um so a thing that he told me was impossible to do in a year, we actually successfully did within 7 months.

And it was the team that did it, not me forcing. So, we went for monthly releases. We tried a bunch of bi-weekly releases. Like, "Wow, half the number of changes, half the number of problems. Like, this is great." They tried one weekly release, you know, like 6 months or 7 months into it, and they were like, "This is awesome." Like again, so many fewer changes, so many fewer iterations back and forth with the teams, and since that time, it's been weekly all the way along.

And then the other thing that really warms my heart is I keep hearing from my my former team there that they're constantly asking themselves, "What would Randy do?" So, you know, leaving a legacy is is kind of a thing that you like to do as a parent and and as an older person. So, yeah. All right, so we're all done. Thank you very much. Goodbye. Uh-oh. >> [laughter] >> Uh-oh. We still didn't change any of this. Okay, well, so what did we even do it for?

Or more precisely, why in the world does this I mean, we doubled engineering productivity, right? Like, that's amazing. 3,000 people, and why didn't we save the company? Okay, three reasons. First is strategy and planning, second is execution and delivery, and third is a pathological organizational culture at the company. So, starting with strategy and planning, um eBay is a suffers a very common thing, which is called Innovator's Dilemma.

So, Clay Christensen posited this in maybe the 1980s, actually, talking about new disruptor somebody who initially disrupts a space and takes it over, has an excellent business model, and they're making tons of money. And what happens is they get comfortable with that, and it is very difficult both psychologically and also frankly financially to see the next disruption, and even if you see it to take advantage of it, right?

So, Kodak invented digital cameras, and digital cameras put them out of business, right? Blockbuster was the first company to do streaming over the internet, but Netflix put them out of business, right? So, Innovator's Dilemma. eBay has this in spades. Uh by the way, if you wanted to disrupt eBay, don't try to be the eBay of the world, try to be the eBay in the UK of, for example, used clothing. Which I think exists as a V Vintage.

Uh Vinted, right? There you go. Uh or you could be um you could be a uh musical instrument reseller in the United States. That one's called Reverb. You could be an electronics reseller in the United States. That one's called StockX. So, that's the way you disrupt eBay, in case you're wondering. Uh the other the other aspect is uh what I call learned helplessness. So, again, 15 or 20 years of an entirely flat business.

And so, what is the adaptive response of humans within of a um within a flat organization? The actually correct adaptive response when you're in that environment is to be very risk-averse. See why that's true, right? That's not cuz people are like evil, it's actually adaptive. Um and the other thing and the and they had very What's I say this? Highly risk-averse and they had lots of evidence for the aversion. Um when I was there the first time, and this is still true, frankly, um eBay has a very fractured relationship with its seller community.

Anybody here who sell who sold on eBay will know exactly what I'm talking about. They always seem to be at odds in this way that seems like it doesn't have to be, but it is factually. And so, every time we did a user-facing change, it felt like the sellers would revolt. And the the phrase I used to use in my head was like, "It's the seller straightjacket." Like, it makes it impossible for us to move forward. So, one quick anecdote from when I was there the first time, I worked on the search engine.

And every time we made improvements to the search engine, we would disrupt somebody's business model. So, simple things like we would do spelling corrections, right? That's just very standard search stuff. So, someone types in i f o n e, we we respell that to i p h o n e. But, by doing that, we disrupted somebody's arbitrage model because somebody already had the model of searching for misspelled iPhones, buying them for cheap, re- changing the spelling, uh and selling them again and making making money on the on the float there.

Um and every literally every time we make changes to the search engine was like that. We improved uh the relevance and the ranking function. Like, I spent a ton of time doing machine learning in that area 20 years ago. Uh and we would piss off sellers every time we made an improvement, they were mad because they used to be able to find things that now the system could find automatically. Uh centralized waterfall planning.

So, 20 years ago and also today, uh eBay tip will do a multi-month entire company, what are we going to do for the next year? I don't work there anymore, but I suspect they've already locked in their 2027 plan today in June of 2026. Uh and then success in quotes is, "Okay, we have this plan and I'm going to execute to that plan and as an executive, I get praise and incentive and bonus for executing to a plan that we we agreed to 18 months ago." Um one of the other aspects of the centralized waterfall planning is small changes are very difficult to get funded in this really weird way.

So, let me explain. Work can only happen if it's approved by the executive team. Work can only get in front of the executive team if it's big enough to get in front of the executive team. So, smaller projects can only like survive in this ecosystem by like tacking themselves on as a rider to one of the other things. Like, you can think of like a remora on a shark or like, I don't know, some gnat that like attaches itself to the bottom of an elephant or something like that, right?

That's the way that you get smaller projects. And when I say smaller projects, I mean not hundreds of people. Uh so, that's a challenge. The other thing is execution and delivery. So, uh part part and parcel of the massive uh top-down planning is massive coordinated releases. So, cycle time, meaning time from start to finish, uh was often measured in quarters and often years. Uh commonly would involve 50 or more teams.

A thing that was touted internally as as a success was the 3-year * 2,000 people eBay managed payments. It cost 1.5 billion US dollars in personnel costs and gave sellers a worse experience than they had with PayPal. Uh it's a feature factory. So, again, in a flat business, uh I as an as an executive, I really don't want to tie my bonus to improving the revenue line. Instead, I'd like to tie my bonus to I told you I was going to execute these 10 things and I did execute these 10 things.

Uh last and most importantly, organizational culture. So, anybody who attended um uh Nicole Forsgren's talk uh yesterday evening would have heard her mention the Accelerate book. She's the primary author. Very briefly, there are three types of organizational cultures. Terrible, okay, and great. eBay is in the terrible, called called pathological, fear, threat, [snorts] risk aversion. If something bad happens, you look for something to somebody to blame rather than something to fix.

Uh and the science behind this book and also behind her book Frictionless shows that if you tell me what somebody's organizational culture is, I can predict their software delivery and or engineering performance, and I can also predict their organizational and business performance. A direct predictive line between terrible culture, terrible engineering results, terrible business results. So, culture of fear, very highly predictable, uh highly apolitical rather, and then as I was mentioning, the executive team, it's like a Game of Thrones, right?

Like I'm trying to like it's a zero-sum game, so like I'm trying to edge you out and like take over air you know, more people, more scope, and anything that I get comes away from you. It's uh it's quite a uh quite pathological. Right in the name. Uh eBay thinks of itself as exceptional, and in many ways it is, but the reality is, particularly in technology, like eBay's data centers are just like everybody else's data centers, eBay's software is just like everybody else's software, um but there's this long-time long sort of internal resistance to things that aren't invented here.

Again, that top-down waterfall thing also execute also has impacts in the culture. So, again, if you have this 18-month, you know, plan that has been uh decided on 18 months ago, then almost by definition, as you're going along and iterating, you can't take advantage of those learnings. Does it make sense? Like, well, I'm going to execute this plan. Well, like what if the plan wasn't great? Like we know more things now than we did, uh you know, 18 months ago.

Um and we had a fantastic experimentation system from a statistical perspective, but the experiments were mostly used to confirm earlier biases and earlier uh predictions, not to uh learn and change. Uh the reason why I'm not there is because I had a run-in with uh somebody who'd been there for a long time. I will never name this person, it's fine. Uh in that person's organization, it was not a culture of fear, it was a culture of terror.

Um this person accumulated an empire of 700 employees and contractors to more or less build the product page. Um and the thing that was my naivete, uh, I even as old as I am, is I ended up saying to this person's boss, because he asked me, uh, why is this re-architecture taking project taking so long? And I said, cuz it's waterfall. >> [laughter] >> And I have proof of it that it's waterfall. And the fact that we saw that it was waterfall meant, great, now I can help.

But again, in this pathological culture, calling somebody out is bad. In fact, it's so bad that you want to make that person that called you out go away, which exactly happened. Uh, and then that person was let go 6 months later. So, karma happens. Uh, what did I learn? And I'm sorry I'm going a little bit over, but we'll be done. Uh, just like Silicon Valley, the TV show, uh, change comes from the top down. And also, successful change needs to be top-down and bottom-up.

And also, middle out. And what do I mean by middle out? The opportunity that I have, if I needed to do this again, was getting more buy-in from my peer VP leaders, right? I had great buy-in from the CEO and the executive team, move faster, and I had great buy-in from the grassroots teams, thank goodness, finally you're removing all of my bottlenecks. But the fear and pathological executive culture was something I I should have, in retrospect, learned more about and done more with that.

Uh, just like the internet, route around resistance instead of going right into it. Again, that's not usually what I do, but uh, that's fine. See the whole board. So, that is systems thinking, really understand end-to-end what the problems are. If you focus in some micro area, you might get make a micro improvement, but if you see the whole board, you can make a systemic improvement. And then finally, you know, it's kind of hard to transform a 5,000 person organization, which is why I currently work at one that has 100 people.

Thank you very much. >> [music] >> Mhm. Mhm.