399: Scaling Code Ownership and Accountability

Stephanie experienced bike camping. Joël describes his experience during a week when he's in between projects. Stephanie and Joël discuss the concept of code ownership, the mechanisms to enforce it, and the balance between bureaucracy and collaboration. They highlight the challenges and benefits of these systems in large codebases and emphasize that scaling a team is as much a social challenge as it is a technical one. Out Our Front Door Conway’s Law Transcript: JOËL: Hello and welcome to another episode of The Bike Shed, a weekly podcast from your friends at thoughtbot about developing great software. I'm Joël Quenneville. STEPHANIE: And I'm Stephanie Minn. And together, we're here to share a bit of what we've learned along the way. JOËL: So, Stephanie, what's new in your world? STEPHANIE: This weekend, I went bike camping for the first time. So, it was my turn to try out the padded bike shorts and go out on a long ride, combining two things that I really enjoy: biking and camping. It was so awesome. We did about 30 miles outside of the city of Chicago to close to the Indiana border. And we were at a campground that's owned by the forest preserves where I'm at. It was so much fun. I packed all my stuff, including my tent and sleeping bags. And it was something that I never really imagined myself doing, but I'm really glad I did because I think it'll be something that I want to kind of do more of in the future, maybe even do multi-day bike camping trips. JOËL: So, what's your verdict on the bike shorts? STEPHANIE: Definitely a big help. Instead of feeling a little bit sore an hour or so along the bike ride, it kind of helped me stay comfortable quite a bit longer, which was really nice. JOËL: Would you do this kind of trip again? STEPHANIE: I think I would do it again. I think the next step for me is maybe to go even farther, maybe do multiple stops. Yeah, I was talking to my partner about it who came along with me, and he was saying, like, "Yeah, now that you've done that many miles in one day and, you know, camped overnight, you can really go anywhere. [laughs] You can go as far as you want." And I thought that was pretty cool because, yeah, he's kind of right, where I can just pack up and go and, you know, who knows where I'll end up? Not that I would actually do that because of my need to plan. [laughs] I'm not that go-with-the-flow. But there was definitely something really special about being able to get from A to B with just, like, my physical body and not relying on any other kind of transportation. JOËL: Yeah, there's a certain freedom to that spontaneity that's really nice. STEPHANIE: Yeah. And I actually went with a group called Out Our Front Door. And if anyone is in Chicago and is interested in doing this kind of thing, they do group bike camping adventures, and they make it really accessible. So, it's a very easy pace. You are with a group, so it's just really fun. They make it really safe. And I had a really great time. There was about 60 of us actually at camp, and they had rented out the entire campground, so it was just our group. And they even had a live reggae band come out and play music for us while we had dinner. And that was a really nice way for me to do it as a first-timer because there was stuff already planned for me, like meals. And I didn't have to worry about that because I was already, you know, just worrying about making sure I got there with all of my stuff. So, if that sounds interesting to you and you're in Chicagoland, definitely check them out. JOËL: That's a great way to bring in newer people to say, let's have a semi-organized thing, where all you have to focus on is the skill itself, you know, can I bike the 30 miles? Rather than planning all the logistics around it. STEPHANIE: Yeah, exactly. Joël, what's new in your world? JOËL: So, speaking of planning and logistics, this week, I'm in between projects at thoughtbot. And on the Boost team that I'm on, we've introduced a kind of special rotation for people who are in between projects where we have an internal project, where just internal strategic initiatives that we want to push forward. Whoever is unbooked gets to work on that. So, it's a small team with a very high churn. And one of the things that we do is every week; we have somebody act as the project manager for that team. And I was in between projects this week. I was assigned that project manager job. And so, I've been doing that in addition to some of the tickets myself, and that's been really interesting. STEPHANIE: Cool. I really like how that role is rotated among team members. JOËL: Yes. And the whole team itself is very high churn. So, somebody might be on that for just one week and then rotate off, and a new person comes in; maybe someone's on there a couple of weeks while we're waiting to find them a project. But we're always looking to prioritize booking people onto new client projects. And so, whoever is on that team, typically, is there for a short period of time. And so that means that the project manager role has to rotate a lot. But also, just in general, as you're managing tickets, you have to deal with the fact that people are not going to be on this project long-term. This is just, they're here for a few days, and they get some things done, and then they're moving off. And I think that presents some unique challenges in terms of the project management side of things. STEPHANIE: Yeah. What kind of challenges did you find interesting in this role for the week that you were on it? JOËL: So, in particular, I think making sure that outstanding work from the previous week gets done, especially when the people who were working on that ticket are no longer working on it. So, they may have done some partial work and then moved on to something else. And then, you have to ascertain the state of the ticket. Has it been completed? If it's only partial, what parts have and haven't? Can this be passed on to somebody else? Is there some unique knowledge that the previous person had? Has the code been pushed up? That kind of thing. STEPHANIE: So, that reminds me of something I heard about the idea of being expendable. You know, there are certain industries where anyone else with that skill set can kind of step in and take over for another worker without a lot of issues, and they can continue on doing that work. So, I'm thinking about, you know, maybe doctors or pharmacists where they have that, like, shared skill set, and everything is documented enough so that they can just take whatever their case is. And if someone is out, it's not a big deal because people can just step in. And I'm curious about if this is something that could work for software development. JOËL: I think it is important to have a team where nobody is irreplaceable. When it comes down to individual tickets, one of the things that I've been pushing for is that, at the end of the week, I would like to not see any tickets remain in the in-progress column. We're using a Kanban-style board. So, ideally, all work either moves to the done column or moves to the to-be-done column for next week. And it's no longer owned by anyone, so people have removed their faces from it. Ideally, though, if you pick up a ticket during the week, you get it to completion. So, one thing that I've been really pushing with our team this week is splitting tickets up. If this feels like it's bigger than a few days, then it needs to be split up, and part of it gets done moves to the done column. Part of it might be some work that somebody else is going to pick up next week; move that to the to-do column. And so, that way, at the end of the week, we have, ideally, a column full of things that were pushed over the finish line that are done. And then, we have a column of things to be done next week that nobody has kind of called dibs on yet. So that then next week, when we have a new group of people coming in, you don't just look at this column of to-do things. It's like, well, all of these have someone's face on them. I'm not able to pick up anything, so now what do I do? By having those all kind of fresh and available to be picked up, you make it easy for the next batch of people to hit the ground running on Monday morning. STEPHANIE: That's really interesting. You said you were doing this Kanban style, but it almost kind of sounds like one-week sprints in a way. JOËL: Kind of, because the way we book people onto clients is typically on a per-week basis. And so, if there's going to be a gap between clients, typically, it's in increments of a week. Because they're on the project for a week, it doesn't necessarily mean that we're tracking the tickets on a per-week thing. So, it's not like, oh, we're committing to doing all of these tickets by the end of a particular time or anything like that. We are working in a more Kanban style where there's a backlog, and you pull tickets, and whatever gets done gets done. What we do try to do, though, is not have individual tickets hang in the in-progress column over a week boundary. So, there's a nuance there. I guess there's some ways in which maybe it feels a little bit sprint-like. But I think we are running in much more of a Kanban-style workflow. STEPHANIE: Yeah, that makes sense. JOËL: It's really to deal with that churn and the idea that even though the ticket might stick around for a while or maybe it gets split up into multiple small tickets, the people are switching constantly. And so, making the workflow play nicely with the fact that the team is churning on a weekly basis kind of adds an extra, you know, a little bit of spice to the project management side of things. STEPHANIE: Did you find yourself being the one to break down tickets to make sure that they weren't larger than a week's worth of work? Or did you work with the developer themselves to find opportunities to break out what they were working on if we got to the mid-week and progress wasn't looking like it would be completed by the end? JOËL: I've left this up to individual developers. This is more of a broad conversation I had with our team, kind of saying, "Hey, here's our goal. We want to get some things done by the end of the week. If we don't think we can get them done, here are some strategies I recommend. I'm available to pair if people want it." But I didn't go through and estimate all the tickets and split them up. I did a little bit of, like, grooming ahead of time. So, I had a sense of when we started the week if tickets felt roughly sized correctly. But oftentimes, you know, that kind of thing, you start working on it, and then you realize, wait a minute, this is a bigger ticket than I thought. STEPHANIE: Yeah. I think even just having someone check in and be like, "Hey, how is progress? Can I support you in making sure that you're able to get to somewhere that feels completed by the end of the week so that the rest of the work is set up for someone new to take on?" That seems really valuable to me. Because as an individual, I'm like, yeah, I don't know, I'm maybe heads down just deep and trying to get my thing done, but maybe not so aware of progress and relative to how much time I've spent on it. And having just someone prompt me on that could help kind of pull myself out a little bit, you know, come out for some air and be like, oh, actually, you know, this is a good spot for me to break this down. Do you have any insights into this week that you might be bringing with you into client work or anything like that? JOËL: And I think this has just given me an even deeper appreciation for breaking tickets down. Because of that arbitrary end-of-the-week deadline, I think that forces more tickets to break down in a way that I might say, oh, well, I picked up a ticket on Thursday for a client. It can totally bleed into next week; that's fine. It's still a fairly short ticket, just, you know, I started the work later. And so, trying to make sure that tickets get scoped down really tightly, I think, is an area where I could probably benefit from that discipline on client projects as well. You know, even if I'm not doing it to the extreme, I'm doing it this week. STEPHANIE: Yeah. I would be really curious to find out if next week the folks who are on this project feel like they're in a good spot to, you know, keep on making forward momentum because they can just pull from the backlog and not have to go and do that knowledge transfer. JOËL: Right. I will see with all of this, right? Maybe even with all the conversations and things, maybe we'll end the week, and I'll have 10 cards in the in-progress column. And it'll be like, okay, we tried a thing this week, mixed success. How do we want to iterate on that idea next week? Potentially with a different team. STEPHANIE: Right, exactly. JOËL: I feel like one way to maybe summarize the type of work that I was doing this week is that it's a kind of a scaling challenge. But over time, the team itself is small, but it's constantly churning. And I think you've been working on a team where it's kind of had a similar problem but in a different dimension. You're scaling over team size, actually a massively large team, and seeing some of the challenges there. What are some of the things that you've been facing? STEPHANIE: Yeah. So, my current client project, I'm working on a codebase where there are hundreds of developers also working and committing to this codebase daily. And this codebase is really massive. There is so much stuff going on. And I've really only explored the world of the particular team that I'm on. But I recently had to do a little bit of work in some code that is owned by a different team. And I actually really appreciated the way that we were able to collaborate across, I guess, ownership boundaries. And I was really interested in talking about some of the different ways that we've seen the idea of, like, who is owning, and, like, who is accountable for areas of the codebase once you reach a certain size. So, what was really convenient about the way that I was working was that in my pull request, there was an automated step that told me I needed specific owner approval on the code that I was writing because I was touching some files that were owned by a different team. And it gave me all of the handles for the people on that team. So, I knew who to go talk to. And it ended up being that that team had a public Slack channel specifically for people outside to ask them questions about their domain. And they had a rotating ambassador system. And so, in the Slack channel, in the channel topic, it said who was the ambassador for that week. So, you know, I saw who it was. I got to @ them and say, like, "Hey, like, I'm working on some of these files for this feature for my team. And, like, here's my pull request. Could you give me a review?" JOËL: The more you're describing this, the more this is feeling very large team, almost bureaucratic systems. I'm hearing public Slack channels, which implies that teams have private Slack channels. I'm hearing, like, a rotating ambassador. One word that you mentioned that I'd like to dig into a little bit is the idea of ownership because I think that the concept of ownership is present on probably most teams, but it probably means wildly varying things. And it sounds like on your team, it's a very kind of codified thing. So, what does ownership look like on your project? STEPHANIE: Yeah, I love that you asked that question because you're right; it is codified, literally, in the codebase. There are ownership files that are in the repo itself where they've specified, like, all of the models that a team owns, you know, down to the names of the files themselves, or maybe a namespace. It has the team name and all of the team members' handles. So, that's how it was able to tell me in an automated way, like, hey, reach out to these people. It was really interesting because it was pretty frictionless on my end, where all I had to do was see that, you know, I couldn't submit my pull request until I got that approval. But it was enough friction to be, like, well, you can't just, you know, change files in this domain without someone with extra context taking a look. JOËL: This reminds me a little bit of a system that GitHub has where you have this CODEOWNERS file that you can add to a repo. Have you messed around with that at all, or kind of seen how that looks? STEPHANIE: I have a little bit. I think I've only seen it in the context of being notified that someone is wanting to submit a pull request, but I'm not sure if it does gate merging based on ownership. Do you know if that's the case? JOËL: I don't. I think you can set it up to automatically request reviews from owners. And on a large repo, the owner could be...I assume this is based maybe on directories, or it might be a regex pattern. I forget the exact details. But you can have owners for partial parts of the code instead of owners of the entire repository. So, then, if you make a change to a particular part of the code, it would ping the correct person automatically to review your code, which sounds like a really nice feature. STEPHANIE: Yeah, absolutely. I think for the project that I'm working on, this definitely seemed like a custom process that they, at one point, decided to enforce. I'm not really sure about the history of how this came to be. But I found it actually quite a good way to meet people who are working in other parts of the codebase. The person who happened to be ambassador that I pinged was so helpful in just, you know, making sure that I kind of understood the parts of the code that they owned that were honestly, like, quite complex. Like, I would not have felt confident just going ahead and making those changes necessarily myself because this is a pretty legacy codebase. There are quite a few gotchas, and they were able to point some [laughs] of them out to me. Yeah, having that extra confidence was helpful for the particular feature I was working on. But it did also kind of give me a little pause because I've not worked at such a scale where there was so much uncertainty about the domain and that being so diffused across like I mentioned, hundreds of people. JOËL: Do you think that this ownership system that's in place helps manage the complexity of scaling up to a team of hundreds of developers? Or does it feel like it kind of just adds a lot of process that gets in your way? STEPHANIE: Ooh, that's a good question. It seems like kind of a chicken-and-the-egg situation because I felt better with someone else's input, right? Like, with someone else with more domain knowledge than me about what I was touching to be, like, "Yeah, like, this looks good to me," giving the plus one. Whereas if that didn't exist, maybe I would have tried to seek it out on my own. But I would not have known where to start, right? I would have to ask around and be like, "Hey, like, who has worked in this directory before?" or whatever. Or I could have just went ahead and merged my code and hope my lack of context didn't really cause any huge problems, like outside of what was covered by the tests. But this is helpful for, like, where the codebase is at, you know, and the size it has grown. JOËL: Do you think requiring an owner to review the code puts maybe an undue burden on the person who's the owner and that they might end up spending a lot of time reviewing code because they now kind of manage that part of the app? STEPHANIE: Yeah, that's a really good question. I think it can. But I also feel a little better that that role is rotated, that everyone on the team gets the opportunity to really, like, focus on that. And I'm pretty sure the way it works is that that is their main focus for the sprint, or the week, or whatever, and that they're not assigned any other feature work but to prioritize being that ambassador. So, in some ways, that is a lot of process, right? And there is that trade-off of having to allocate someone specific to answer people's questions. But at least from what I've seen, it does seem like a necessity because people do have questions, right? And I think they have figured out a system where it's very clear who you're supposed to talk to, and that accountability aspect of it has been met. Because I've also, like, worked on teams where that role is not well defined. People don't want to do it. And it's almost kind of a bystander effect where someone asks a question, but no one is specifically responsible for answering it, and so no one answers it. JOËL: Oh yeah. Yes. And then you get kind of the cost of the bureaucracy without the benefits of kind of diffusing that knowledge. So, we've been talking a lot about how this kind of ownership system can be really beneficial despite the overhead for a team that's 200 or 300 developers and where nobody knows all of the code or all of the nuances. And kind of at the other extreme, it's absolutely not worth it for a team of two or three developers where everybody knows the code, and there's kind of shared ownership of the project. Somewhere in between, there is where you start having maybe some of those conversations about scaling the team, and do we need to introduce more process? In your experience, where do you think introducing some sort of ownership system like this starts becoming valuable? Or maybe what are some of the questions that a team should ask themselves to gauge, like, at the size we're at right now, would we get value from an ownership system? STEPHANIE: Yeah, that's a really good point because as you were saying that, I was just starting to think of, yeah, I've certainly worked on projects where I have reviewed every piece of code that is to be merged, right? And then, at some point, that starts to change where, like, I can't do that anymore. And that transition has always been really interesting to me. And then, I think there is, like you mentioned, another one where it's like, okay, now we aren't able to review everything, but, like, how do we trust that the code that is being merged, even if we don't all share that same context, is up to the quality we want it or is bug-free? Because without that context, there's always the opportunity that something might be missed. I think I've seen that on teams, you know, really look like more bugs than usual, right? And maybe there is, like, actually, like, a big problem, and the site is down. And maybe there is, like, a post-mortem or something to discuss, like, why this happened. And, you know, it turns out that the siloing or, like, the lack of context sharing was partly involved. And so, I do think there are definitely symptoms when we're starting to firefight a little more [chuckles] that might be kind of an indicator that the app has grown to a point where some context is being lost, and there are not guardrails in place to do our best to, like, share it, like, when we can and not when it's too late. JOËL: Would it be fair to say that your recommendation is the team should not have an ownership system and kind of stick to everybody reviews all the code as they grow until they start hitting actual pain points, such as real bugs caused and, at that point, let that pain or maybe even the post-mortem be the thing that triggers the introduction of an ownership system? STEPHANIE: I think so. I have not seen something like that proactively introduced. I would be curious if anyone has experienced something like that. But, you know, I think it's okay for change to be a little painful, right? And that's part of the growing pains of becoming a larger team, or organization, or codebase and continuing to reevaluate. Though, I guess I would be a little cautious about, you know, jumping straight to introducing processes or policies, right? Because those can be really hard to undo if they end up not being actually helpful for the root cause of the problem. But, like, how you experimented with making sure that, you know, we didn't have any in-progress tickets for that project. The idea of just trying something and seeing how it works and kind of getting the team's feedback that is really valuable to me, at least as an IC. And, yeah, just making sure, like, you know, hearing from all of your team members on how those processes are changing the way they work and if it's feeling good or not. Like I said, I enjoyed the process on my client project because it helped me feel more confident that the code that I was changing...because I can't possibly gain all of the knowledge that the owners of that area of the code have. It's just not going to happen. But also, I can imagine it being maybe not so good for someone else, right? It kind of being a barrier or being frustrating because, oh no, they really need to merge the code. And maybe they made the smallest change in a file owned by another team, right? And having to jump through that hoop. JOËL: Yeah, that has absolutely never happened to me. STEPHANIE: Really? Because it sounds like it has. [laughter] JOËL: Yes, it absolutely has. We've been kind of throwing around the idea of ownership the idea of team almost interchangeably as if they're one-to-one. And I want to lean a little bit into this idea of the team. Because I think there's an implicit assumption here that, within a team, there's enough knowledge sharing that happens, enough shared context, that everybody can kind of understand all of the code for their team, and that knowledge is just shared around, so you don't need these extra processes within teams. But maybe once you start having multiple teams as part of your engineering department, then there starts to be some friction or some lost context that needs some mechanism to get around. Does that sound about right to you? STEPHANIE: Ooh, I think that depends because even within my team, we are working on different projects, and I am definitely not on top of, you know, what some other folks are working on. And even within our team, there are silos. The difference there, though, is that I know who they are. Like I still am in contact with them in our daily syncs, that the barrier to finding someone who has the right information is much lower. So, yeah, I think that is definitely a part of it, too, if, like...I think just the social barrier, even of, like, reaching out to someone you don't know and being like, "Hey, like, can you review my code?" that is [laughs] kind of...can be a little scary. And the dynamics definitely feel different within a team and between teams. JOËL: Yeah, and definitely just the idea of, like, someone you see every day for your daily sync, you're going to feel much more comfortable reaching out to them for help or for a quick review than to a total stranger. So, it's interesting that you mentioned the social aspects of things. I don't know if you're familiar with Conway's Law, the idea that the technical structures of our code, over time, end up reflecting the social structures of our teams. STEPHANIE: Yeah, that is something that makes a lot of sense on the project that I'm working on now, where the boundaries, like I mentioned, between teams and between different namespaces are semi-rigid, I suppose, right? Rigid enough that, you know, there is a process but not so high that it becomes a burden, at least in my opinion. But for another feature that I worked on, I actually had to interact with an external system that's owned more by the parent company of my current client. And that process was definitely more rigid. And I had to figure out who to email and had to, you know, look up this person's profile in the company directory to make sure that, you know, I was talking to the right person who had information that was relevant to me. And then, you know, even, like, the technical aspect of talking to this external service had a lot of various barriers and, you know, special authorization and configuration that I needed to set up. So, definitely felt that in terms of the different levels of ease and talking to systems owned by different parties. JOËL: So, the fact that there's, like, an actual, like, departmental or even, like, corporate boundary, definitely showed up in, like, a very hard boundary in the code as well. STEPHANIE: Yeah, absolutely. JOËL: And I think taking this to an extreme, I've seen this happen when teams want to introduce microservices. And oftentimes, the boundaries of those microservices are not necessarily driven entirely by technical reasons, but they're often by social reasons. So, we can say, hey, this team is going to own this service, and everybody else only needs to interact with a public API. And we can make all sorts of changes internally, and you never need to know that. They will never break your code. And also, we don't need to bother each other or feel the need to fully deeply understand the internals of each system. STEPHANIE: Yeah. Once you're introducing APIs that are accessible to a certain group of people and having to navigate, you know, making changes to the API or aligning in the expected structure of the, you know, communication that you're sending between them, that is definitely a pretty rigid boundary [laughs] and ends up being a lot of overhead to talk to those systems. And I certainly have been in the position of trying to communicate with the people who built and designed those systems and figuring out how to get on the same page. And even just recently, I was accidentally sending something as an array, and they were expecting it as a string. And that caused all these problems of making the request happen, you know, successfully. And we didn't even realize it until someone pulled out the doc that had the API schema and pointed out that there was some miscommunication along the way. JOËL: And that can be such a hard boundary around even, like, the idea of ownership. So, you were talking about how, earlier, when you were working in code that's maybe owned by another team, they might want to review it before it gets merged. So, there's a bit of a gatekeeping there. When a team transitions fully to microservices, I've seen it go almost, like, more extreme where it's even, like, you don't even change the code. You submit a ticket into our system. We will prioritize it, and then eventually, we will build your feature. But you don't even get to make a change to the code and have us approve it. We're going to make all that because we own it. So, it kind of feels like taking that ownership idea and then just really running to a full extreme. STEPHANIE: Yeah, right. That makes a lot of sense in the lens of Conway's Law if those are the processes they have in place for navigating cross-team collaboration or communication. Because, at some point, maybe they just reached a level where it had to be enforced that way because maybe things were getting dropped, or more casual lower barrier connection was too overwhelming or just not working for the organization. JOËL: I think what I've been hearing just now and then just more broadly throughout the episode is that while there's a lot of interesting technical solutions that can make things better, at its root, scaling a team is a social problem. And it's all about how your teams communicate with each other so that you can scale smoothly and that the system doesn't suffer from adding more people. STEPHANIE: Yeah. I think this is an area where I would love to hear any thoughts from our listeners about how their organizations handle something similar because I find all of this really interesting. And, you know, it ends up impacting my day-to-day work in a very real way. And so, if other places have figured out how that scaling and, you know, social and technical boundaries work in a way that feels good, I would love to know. JOËL: On that note, shall we wrap up? STEPHANIE: Let's wrap up. Show notes for this episode can be found at bikeshed.fm. JOËL: This show has been produced and edited by Mandy Moore. STEPHANIE: If you enjoyed listening, one really easy way to support the show is to leave us a quick rating or even a review in iTunes. It really helps other folks find the show. JOËL: If you have any feedback for this or any of our other episodes, you can reach us @_bikeshed, or you can reach me @joelquen on Twitter. STEPHANIE: Or reach both of us at [email protected] via email. JOËL: Thanks so much for listening to The Bike Shed, and we'll see you next week. ALL: Byeeeeeeee!!! ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com.Support The Bike Shed