I am WAY not smart enough for this. Anybody want to distill down the info and ELI5?
If you look carefully, I've linked a lot of the technical terms in the article to definitions and explanations. The sort of tl;dr is that we've built a lot of automation to help the team work more efficiently. To use a car analogy, this article is about the factory full of robots that build cars, except instead of cars it's a video game.
Let's say I'm not very savvy on this kindof stuff and only understand ~10% of this article on a first pass with zero context
Hypothetically, what would you want a player like me to know?
If you could explain it like you would explain advanced biomechanics to a 5 year old I'd (hypothetically ofc, obviously I get it and am the most smartest) appreciate that.
For players, all I'd really want you to know is that we work really hard to provide yall with the best experience we possibly can. I don't directly work on the content in the game itself, but I do everything I can to help the people who do make all that amazing content.
This article's primary target audience is actually people in my role but on other games. There isn't that much literature out there about how games-as-a-service are built, tested, and shipped, and I'm hoping to encourage more sharing and collaboration in this field :)
A lot of this stuff is pretty esoteric tech that isn't especially relevant in day to day life. But it's critical in modern software development.
When you write new code, you want to make sure it doesn't break any of your old code.
To make sure you didn't break anything, you run a bunch of automated tests on the new code.
It helps to run these tests yourself, but you want to guarantee that all the tests get run every single time anyone adds new code.
So, what you do is set up a robot that looks at the master copy of the code, and any time someone wants to make a change, the robot runs all the tests on the new code before it's allowed to become the "real" code.
Once the robot has made sure that the new code is safe, you want to put that code to work! This is usually called "deploying," and it happens much more often than you might expect. You might get an update to your game client every other week and have to download some new stuff, but the LoR game servers probably get updated code every night!
This is another thing that you could do by hand if you wanted to, but it's much easier and safer to set up another robot to do it automatically.
So after the first robot runs all your tests and verifies that the new code is safe, the second robot comes in and puts that new, safe code in all the places that it needs to be updated.
This is great! One clarification - we don't update the live game servers every day unless we've got a lot of surprise bugs to fix. However we do deploy to our internal test environments many times per day.
Really enjoyed this article! I recently had my first internship as a DevOps Engineer so it was really cool relating all the concepts I worked on with my favorite hobby
That's really not what he was suggesting; he was saying it’s widely considered to be challenging, and even he makes errors all the time:
Git is hard. I’ve been using Git for years as a software engineer and I still make mistakes all the time. For game designers, artists, and other folks who are probably only using Git because we’re forcing them to use it, Git can be intimidating, unfriendly, and unforgiving.
A bit of both :) anything specific to a particular field can be considered esoteric. Git and other topics in the article are normal things for software engineers, but for anyone else it might be as meaningless as reading a paper about quantum mechanics. But even for software engineers, git can be tricky.
Though, to be honest, I’m fairly certain it would be impossible to get a game of this calibre without serious tooling like this to support it.
Great point! I consider a CI pipeline to be essential for software of this complexity. The fascinating part is in comparing the implementation details :)
Windows machines: Unity can build on Linux, but that support is relatively recent. Given time we might well switch to primarily Linux build machines - I definitely prefer Linux as a CI/CD environment. Managing Windows with Ansible can be tricky. Note that at least currently, our game client builds and game asset builds are not containerized - they run on persistent workspaces on plain old EC2 VMs.
Hey, fellow engineer here. You mentioned early in the article that LoR has at least some microservices, although they aren't in your monorepo.
Can you expand on those services and how monolithic or microservicey the rest of your monorepo is?
I know one of the major sell points of microservices is enabling separate development and CI/CD pipelines for each, hopefully reducing the complexity and burden of any single piece to a small and manageable amount. Seems very relevant for a system with hour long build times. I start looking for ways to cheat when our builds exceed 10 minutes
I can tell you a little, though our services are mostly owned and operated by one of our sibling teams, so I'm definitely not a subject matter expert. There's a single services monorepo with all the services in it. I believe they each have their own subfolder. They share a common service framework they're built on. They have their own Jenkins multibranch pipeline that builds all of the services in parallel in one run (also optimized to only build services that have code changes in order to avoid wasted builds). And the services builds are all containerized (and are much, much faster than our game builds).
Like u/mvndrstl mentioned as well, it is surprising to see such a big project not end up broken into more pieces. Its very impressive thats its both remained together and that you have made it work well! You should be proud of the consistency of your release schedule and the support youve given to your devs.
In your ideal world where you have infinite time and money, would you break the project down into pieces? Or do you feel that the monolithic nature is something you prefer?
Specifically in a game dev context, I do personally prefer monorepos - it makes all this complex custom tooling so much easier. We've actually talked about moving our services into the game monorepo, though it's not a priority to figure that out.
In my experience, choices like this are actually mostly irrelevant. What's important is to find out how your users like to work, and tailor your tools to their needs and preferences. For example - personally, I love test-DSL-based automation frameworks like Robot Framework or Cucumber. But the game engineers I work with asked to write tests in just plain Python, so that's what we built for them, and they're happy with it. When I joined LoR, the team was using a monorepo, and they liked it and it worked well, so we just kept using it. It's of course possible to implement a monorepo poorly or implement a many-repos system poorly.
I work not in games but on an IT webapp that uses a lot of the same tech you guys are using (at least for tools and on the server side). I had a bunch of questions that popped up while I was reading but they were mostly answered by later points in the article.
How long do iteration builds take?
Is there any way to build and run the game locally or does it have to be built and deployed by the pipeline?
How long do iteration builds take?
Depends on what changed in a dev's commit. Small changes to card scripts can be under ten minutes from build start to playable on your local work PC. Changes to assets could still take 45 minutes.
Is there any way to build and run the game locally or does it have to be built and deployed by the pipeline?
Yep, there's a local build system. The main limitation currently is that we don't have a good way to create sharable playtestable builds from a local build, because we've just focused on the CI/CD pipeline as a core workflow. Making it easier to share and playtest local builds is in our backlog.
This is some cool reading, and I'll do it! I've just graduated and will be heading into a career in tech recruitment, so this is super valuable.
I just want to say that your love for what you do is so clear; you know your stuff and you obviously live & breathe it. Wishing you all the best, Guy, and look forward to the next blog!
Yeah that makes a lot of sense - I work in the space domain so I'm sure I miss out on some of the intricacies and quirks of the game development domain
Thanks for the responses and articles with your insights from your devops career - keep up the great work!
Space as in rockets and satellites and such? That sounds super cool.
Any reason y’all call it PE:STAB instead of just devops?
Personally, I think of “devops” as a philosophy and not a team. Our services developers are also the ones who operate the live systems - we don’t have a separate ops team, although we do have many supporting infrastructure teams at various levels in the company.
Production Engineering is the initiative that our team and a few others are included under. PE or SRE (site reliability engineering) are the terms that seem to be getting industry traction for this kind of work. I think you can do devops the philosophy as a company even if you have an SRE team on the front lines, but it’s definitely different from a web startup where engineers handle everything from design to release.
I really appreciate the insight. This gave me a clear picture of what a working day would be like at Riot.
As a scripter, I wasn't able to determine my main question though: after making script changes, how long does it take to be able to test those changes from your workstation? Is that the 2-3 minute window or the 60 minute window? Was the longer period to integrate the changes into the build?
If you’re talking about only card script changes, those can be tested locally basically instantly. If you want to playtest them with someone else, you’re looking at the 60 minute build and deploy.
Once you understand that git is a directed acyclic graph managed using simple text based references to commit sha hashes, doing things like dealing with detached heads becomes easy.
Yes, git is easy right up until it isn’t and you spend an hour thinking about how to get back to what you wanted.
I am an SDET for web and mobile UIs and backend REST APIs. All of this made sense to me and is really cool to see. I had no idea that clients were Chromium based. I am not a fan of Cucumber at all but I digress.
How does the developer find the selectors for the elements? Do they have access to Chrome developer tools like you would in the regular browser?
How big is your test suite?
I am assuming you can only test up into the point a match has started? Is there an entirely separate team that tests the gameplay?
I did, but I'm still curious if you wouldn't mind.
Oh, sure. I can tell you what I remember, though keep in mind I haven't worked on the League Client in some four years now, so some of the tech may well have changed. Yes, there's access to Chrome developer tools just like in a regular browser. The test suite included tons of unit tests across the client as well as a few hundred functional tests. We did have game loop tests that could go from launching the client, to matchmaking, to running the game and doing some simple game functional tests, and then to validating the end-of-game screen. But yeah, you'd feel right at home writing automated tests for the League Client, it's very similar to what you do.
As a fellow game dev in the mobile space, I loved the technical specificity in this article, especially how you go about performing headless unit tests and automatic log scanning.
I'm curious though, have you guys considered splitting the Unity build pipeline between asset only and data only build jobs so very basic iterations like a simple image change or audio swap only require a subset of Unity's build process? Is there a way to avoid Apples / Google rigorous review process by overriding a client's asset manifest or server data hash without having to upload a new client? Curious what Riots philosophy is about pushing balance changes without having to upload a client at a faster pace? Of course, fixing bugs I'd imagine would require a new client but that could be reserved for the two week client update cadence.
have you guys considered splitting the Unity build pipeline between asset only and data only
We do in fact do this! And yes, we do have the ability to do data-only deploys that don't require a client redeploy, depending on the scope of any given change.
Can you help us understand what hurdles prevent the team from shipping more regular updates?
We hear a lot about e.g. balance patches requiring an approval by Apple for the ios release. My understanding is that the card's logic lives server side. I'm guessing some balance changes could be done entirely server side without any update to clients?
That's assuming things like health and mana costs are also driven by the server and not assumed by the client.
Edit: thank you for writing this article. I've worked with a lot of CI /CD pipelines but never one for a video game! There are some interesting challenges there for sure.
One aspect of it is just the sheer amount of manual work involved in validating changes. Video games are so complicated that even with a bunch of test automation, we still need to do a ton of manual QA work, and that can take a lot of time and energy.
What would you say has been the biggest technical challenge of making Legends of Runeterra so far?
The hardest part for me personally has been balancing short term fixes vs long term sustainability improvements. There's constantly stuff breaking, because anytime you change software, you will break it. I could spend every waking moment just fixing stuff and still have more to do. There's also a lot we can do that has less immediate impact, but makes a bigger difference in our efficiency and success later. But if we always focus on the long term stuff, then we stop getting anything done and end up blocking ourselves. So there's some sort of in-between balance to find, but it's really hard to be sure we're making the right priority choices.
When you have bugs in the game that are consistently reproducible, something that cropped up after a patch, either visual or non-visual (a card cost reduction not happening for example). Do you find that it was the lack of a test for that scenario that should’ve been there before release? Or is it usually crazy edge cases that were never considered, and do you then add new assertions for that crazy edge case? There’s so many potential combinations in a card game I imagine it’s tough to exhaustively test everything. Curious Software Engineer asking here.
I think it's sort of all of the above. And yeah, it's pretty much impossible to exhaustively test every possible interaction in the game. We do have some fuzz tests that run continuously just doing random stuff, but that's best for finding rare crashes, since it's sort of hard to automatically validate what behavior is correct in random situations like that.
Yeah I can imagine! I love what you've shared with your python lib that allows Rioters to write simple and elegant game and server tests. I'm confident you guys will keep increasing that test coverage and reduce the dependency on manual tests!
The way I like to think of it is that automation doesn't replace manual testing, it augments it. By automating the boring kinds of testing, we free up our manual testers to do more creative exploratory testing to reach a higher quality standard.
Hey! Really surprising post to find in my favorite video game subreddit :)
As a fellow DevOps engineer, really curious why you chose Artifactory for your LFS storage and not the built in GitHub support?
Our Artifactory is backed by S3, so it basically (kind of) has infinite capacity, while our GitHub is running on a VM with fixed capacity.
I'm a somewhat tenured software developer and I've recently started entertaining the idea of starting to work towards publishing a small game. This is similar to pipelines I've used in the past, but I know those pipelines grow about as quickly as trees (okay maybe a little faster than that), so I wonder what you think is a good place to start - What kind of tooling can someone really benefit from when starting to develop a game?
Version control is a must
A simple CI pipeline that builds every push
If possible, run the same build process locally and in CI
Auto-deploys to make playtesting easier can be really nice to have
Keep as much configuration as possible in version control. Minimize manual configuration of your CI/CD infrastructure and automation.
Like u/mvndrstl mentioned as well, it is surprising to see such a big project not end up broken into more pieces. Its very impressive thats its both remained together and that you have made it work well! You should be proud of the consistency of your release schedule and the support youve given to your devs.
In your ideal world where you have infinite time and money, would you break the project down into pieces? Or do you feel that the monolithic nature is something you prefer?
One of the biggest things to consider when going with a "mono-repo" approach for something like a set of micro-services is the ability to have a unified commit history. I.E. When all your services are in a single repo, any single code commit to that repo creates a new "version", which you can then trigger, build and test the entire set of services as a single unit.
When each micro-service is in its own repo, you tend to build and release each micro-service with its own unique version then release a set of compatible versions instead. Both patterns work and at Riot both patterns are in some form of use. It really depends on how the teams want to work and what works best for the release process.
There are lots of nuanced pro's and con's here. The "single version" number to rule them all (LOTR reference!) has a lot of benefits for moving a release through various test environments and deploying a unified set of services you know should work together. It has the downside that a single line code change to a single service has to produce a new bundle version number. The Runeterra pipeline compensates for this by not rebuilding services that didn't change (but thus has to have some more complexity in the build pipeline). Likewise Riot's deployment technology for containers is "smart" enough to not re-deploy services who haven't actually changed when you deploy an new bundle version. Again, adding complexity to what happens at deploy time.
On the flip side, it is very easy to see and find historical code changes and manage development branches of services that have a lot of dependencies and general makes it easier to work with on the developers as they work on major features, which often have to touch several services at once.
Probably could write an entire blog on that approach alone!
Context: I worked on this for a little while with "Some_other_guy" though I'm not on the project anymore.
Interesting. I respectfully disagree with this position however. Plenty of very very complex SaaS deploy new code hundreds of times per day. Of course games are different, but they're also still software. These companies manage to ship with extreme high confidence without any manual tests.
I hope you appreciate the challenge, I'm not discrediting your opinion and in the end you have more experience in Game devops than I do!
Come work with us and make it better! I love when people come tell me I'm wrong and demonstrate a better way :) And thanks for taking the time to chat about it and share your experiences.
Do you have remote opportunities? Sadly moving to the US is not an option 😔
Keep the tech blog posts coming! I'm very grateful you took the time to write this.
LoR does have software teams in Singapore and Hong Kong I believe. And in some instances Riot does work with remote engineers, though the majority of us are not remote, except during current pandemic circumstances. https://www.riotgames.com/en/work-with-us/helping-rioters-unplug-and-recharge has the latest on Riot's plans.
Interesting. I respectfully disagree with this position however. Plenty of very very complex SaaS deploy new code hundreds of times per day. Of course games are different, but they're also still software. These companies manage to ship with extreme high confidence without any manual tests.
I hope you appreciate the challenge, I'm not discrediting your opinion and in the end you have more experience in Game devops than I do!
I get into this conversation a lot. I spent a decade in Build/Pipeline/Continuous Delivery engineering in and outside of the games industry.
First, to be clear, for an individual small micro-service or collection of services, Riot absolutely has the ability to push dozens of changes a day if it wanted to, the tech to do so is pretty lock/stock standard these days.
Games though, games become an interesting piece of software to deal with. Without getting too into the weeds of argument, here's some things to consider. Unlike traditional pure service based development, a game "feature" typically includes art assets and game design. Assessing a "feature" release for a game can't be done with pure automation alone. You need to assess if it meets your visual/art standards and if the design is "fun" and balanced. These are parts of game software development that automated testing can't do and slow down traditional "continuous delivery processes". Or in your case, at "hundreds of deployments a day" more of a "continuous deployment" approach.
The last thing you need to consider about games is that they often have game "clients". These are not "services" and involve updating CDN's and "patching" out to players. In the case of mobile games, you can't really release any faster than mobile platforms will allow. Google that's typically 8-24 hours for a review and apple is usually minimum 24 hours. So even if you could have 100 builds of your client ready a day, you'd be impeded by the platform release.
Even if you remove "mobile" the PC client release has to factor in things like how big the patch size is etc. You don't want to force players to patch their client 100 times a day even if you could release it that fast.
In order to bring a game feature to players you often need to have things in the client, not just the services. In fact sometimes you don't even need to change the services, but you have to bring the client to players.
All of this is why Games typically don't change as fast in production as traditional pure service/web client only technologies.
Wow, thats a lot of machines for your build system. Do you have more machines running for your build system than your runeterra prod servers? (hopefully not too stupid of a question, not intending for it to be sarcastic or anything)
I'm not actually sure, but I suspect we have significantly more servers in prod than in our build infra. Though they may not really be comparable - our build infra is mostly persistent EC2 VMs, but the prod stuff is all containers.
Interesting article for sure!
Where do the manual testers fit in these pipelines?
So where do they give their feedback / how do their envs get maintained (are they different from the auto test envs) / is there feedback reflected in any pipelines (or is it something separate?)
Once builds have been deployed, manual testers can download them and start validating bug fixes, new features, etc, and put their feedback in the JIRA ticket. QA is a core part of our workflow.
Nice! I think I can make most of that work. I guess a first foray into game development probably won't include live service updates on a regular basis, but I still think this is important to rapid development. Thanks!
some__other__guy
Hi, I wrote this :) happy to answer questions!
some__other__guy
:thinking:
some__other__guy
I like https://santamonicapizzakitchen.com/
some__other__guy
If you look carefully, I've linked a lot of the technical terms in the article to definitions and explanations. The sort of tl;dr is that we've built a lot of automation to help the team work more efficiently. To use a car analogy, this article is about the factory full of robots that build cars, except instead of cars it's a video game.
https://en.m.wikipedia.org/wiki/Continuous_Integration could be a good starting point.
some__other__guy
Sorry :(
some__other__guy
I love you too! And I'm very grateful to have the opportunity to work on LoR and be here with wonderful people like you.
some__other__guy
For players, all I'd really want you to know is that we work really hard to provide yall with the best experience we possibly can. I don't directly work on the content in the game itself, but I do everything I can to help the people who do make all that amazing content.
This article's primary target audience is actually people in my role but on other games. There isn't that much literature out there about how games-as-a-service are built, tested, and shipped, and I'm hoping to encourage more sharing and collaboration in this field :)
A lot of this stuff is pretty esoteric tech that isn't especially relevant in day to day life. But it's critical in modern software development.
some__other__guy
This is great! One clarification - we don't update the live game servers every day unless we've got a lot of surprise bugs to fix. However we do deploy to our internal test environments many times per day.
some__other__guy
Np! Check out this great talk for more: https://youtu.be/cKfz2nEgaX8
some__other__guy
Thanks! I'm glad you liked it.
some__other__guy
A bit of both :) anything specific to a particular field can be considered esoteric. Git and other topics in the article are normal things for software engineers, but for anyone else it might be as meaningless as reading a paper about quantum mechanics. But even for software engineers, git can be tricky.
https://xkcd.com/1597/
some__other__guy
Great point! I consider a CI pipeline to be essential for software of this complexity. The fascinating part is in comparing the implementation details :)
some__other__guy
We do have openings! https://www.riotgames.com/en/work-with-us#product=Legends%20of%20Runeterra for the general LoR listings. For my team specifically: https://www.riotgames.com/en/work-with-us/job/3117338/software-engineer-ii-production-engineering-legends-of-runeterra-los-angeles-usa and https://www.riotgames.com/en/work-with-us/job/3117341/staff-software-engineer-production-engineering-legends-of-runeterra-los-angeles-usa
Windows machines: Unity can build on Linux, but that support is relatively recent. Given time we might well switch to primarily Linux build machines - I definitely prefer Linux as a CI/CD environment. Managing Windows with Ansible can be tricky. Note that at least currently, our game client builds and game asset builds are not containerized - they run on persistent workspaces on plain old EC2 VMs.
some__other__guy
I can tell you a little, though our services are mostly owned and operated by one of our sibling teams, so I'm definitely not a subject matter expert. There's a single services monorepo with all the services in it. I believe they each have their own subfolder. They share a common service framework they're built on. They have their own Jenkins multibranch pipeline that builds all of the services in parallel in one run (also optimized to only build services that have code changes in order to avoid wasted builds). And the services builds are all containerized (and are much, much faster than our game builds).
some__other__guy
Specifically in a game dev context, I do personally prefer monorepos - it makes all this complex custom tooling so much easier. We've actually talked about moving our services into the game monorepo, though it's not a priority to figure that out.
In my experience, choices like this are actually mostly irrelevant. What's important is to find out how your users like to work, and tailor your tools to their needs and preferences. For example - personally, I love test-DSL-based automation frameworks like Robot Framework or Cucumber. But the game engineers I work with asked to write tests in just plain Python, so that's what we built for them, and they're happy with it. When I joined LoR, the team was using a monorepo, and they liked it and it worked well, so we just kept using it. It's of course possible to implement a monorepo poorly or implement a many-repos system poorly.
some__other__guy
Depends on what changed in a dev's commit. Small changes to card scripts can be under ten minutes from build start to playable on your local work PC. Changes to assets could still take 45 minutes.
Yep, there's a local build system. The main limitation currently is that we don't have a good way to create sharable playtestable builds from a local build, because we've just focused on the CI/CD pipeline as a core workflow. Making it easier to share and playtest local builds is in our backlog.
some__other__guy
<3
some__other__guy
Space as in rockets and satellites and such? That sounds super cool.
ivey
Personally, I think of “devops” as a philosophy and not a team. Our services developers are also the ones who operate the live systems - we don’t have a separate ops team, although we do have many supporting infrastructure teams at various levels in the company.
Production Engineering is the initiative that our team and a few others are included under. PE or SRE (site reliability engineering) are the terms that seem to be getting industry traction for this kind of work. I think you can do devops the philosophy as a company even if you have an SRE team on the front lines, but it’s definitely different from a web startup where engineers handle everything from design to release.
ivey
If you’re talking about only card script changes, those can be tested locally basically instantly. If you want to playtest them with someone else, you’re looking at the 60 minute build and deploy.
ivey
Once you understand that git is a directed acyclic graph managed using simple text based references to commit sha hashes, doing things like dealing with detached heads becomes easy.
Yes, git is easy right up until it isn’t and you spend an hour thinking about how to get back to what you wanted.
some__other__guy
Ah, I think you may have clicked through to my older article (https://technology.riotgames.com/news/running-automated-test-pipeline-league-client-update) about the League Client update.
some__other__guy
Oh, sure. I can tell you what I remember, though keep in mind I haven't worked on the League Client in some four years now, so some of the tech may well have changed. Yes, there's access to Chrome developer tools just like in a regular browser. The test suite included tons of unit tests across the client as well as a few hundred functional tests. We did have game loop tests that could go from launching the client, to matchmaking, to running the game and doing some simple game functional tests, and then to validating the end-of-game screen. But yeah, you'd feel right at home writing automated tests for the League Client, it's very similar to what you do.
some__other__guy
We do in fact do this! And yes, we do have the ability to do data-only deploys that don't require a client redeploy, depending on the scope of any given change.
some__other__guy
One aspect of it is just the sheer amount of manual work involved in validating changes. Video games are so complicated that even with a bunch of test automation, we still need to do a ton of manual QA work, and that can take a lot of time and energy.
some__other__guy
I love BL2! Have you tried any of the mods? I've been playing with the Unofficial Community Patch and the Better Loot Mod (https://github.com/BLCM/BLCMods/tree/master/Borderlands%202%20mods/Apocalyptech/BL2%20Better%20Loot%20Mod)
some__other__guy
The hardest part for me personally has been balancing short term fixes vs long term sustainability improvements. There's constantly stuff breaking, because anytime you change software, you will break it. I could spend every waking moment just fixing stuff and still have more to do. There's also a lot we can do that has less immediate impact, but makes a bigger difference in our efficiency and success later. But if we always focus on the long term stuff, then we stop getting anything done and end up blocking ourselves. So there's some sort of in-between balance to find, but it's really hard to be sure we're making the right priority choices.
some__other__guy
I think it's sort of all of the above. And yeah, it's pretty much impossible to exhaustively test every possible interaction in the game. We do have some fuzz tests that run continuously just doing random stuff, but that's best for finding rare crashes, since it's sort of hard to automatically validate what behavior is correct in random situations like that.
some__other__guy
The way I like to think of it is that automation doesn't replace manual testing, it augments it. By automating the boring kinds of testing, we free up our manual testers to do more creative exploratory testing to reach a higher quality standard.
some__other__guy
Our Artifactory is backed by S3, so it basically (kind of) has infinite capacity, while our GitHub is running on a VM with fixed capacity.
some__other__guy
maxfields2000
One of the biggest things to consider when going with a "mono-repo" approach for something like a set of micro-services is the ability to have a unified commit history. I.E. When all your services are in a single repo, any single code commit to that repo creates a new "version", which you can then trigger, build and test the entire set of services as a single unit.
When each micro-service is in its own repo, you tend to build and release each micro-service with its own unique version then release a set of compatible versions instead. Both patterns work and at Riot both patterns are in some form of use. It really depends on how the teams want to work and what works best for the release process.
There are lots of nuanced pro's and con's here. The "single version" number to rule them all (LOTR reference!) has a lot of benefits for moving a release through various test environments and deploying a unified set of services you know should work together. It has the downside that a single line code change to a single service has to produce a new bundle version number. The Runeterra pipeline compensates for this by not rebuilding services that didn't change (but thus has to have some more complexity in the build pipeline). Likewise Riot's deployment technology for containers is "smart" enough to not re-deploy services who haven't actually changed when you deploy an new bundle version. Again, adding complexity to what happens at deploy time.
On the flip side, it is very easy to see and find historical code changes and manage development branches of services that have a lot of dependencies and general makes it easier to work with on the developers as they work on major features, which often have to touch several services at once.
Probably could write an entire blog on that approach alone!
Context: I worked on this for a little while with "Some_other_guy" though I'm not on the project anymore.
some__other__guy
Come work with us and make it better! I love when people come tell me I'm wrong and demonstrate a better way :) And thanks for taking the time to chat about it and share your experiences.
some__other__guy
LoR does have software teams in Singapore and Hong Kong I believe. And in some instances Riot does work with remote engineers, though the majority of us are not remote, except during current pandemic circumstances. https://www.riotgames.com/en/work-with-us/helping-rioters-unplug-and-recharge has the latest on Riot's plans.
maxfields2000
I get into this conversation a lot. I spent a decade in Build/Pipeline/Continuous Delivery engineering in and outside of the games industry.
First, to be clear, for an individual small micro-service or collection of services, Riot absolutely has the ability to push dozens of changes a day if it wanted to, the tech to do so is pretty lock/stock standard these days.
Games though, games become an interesting piece of software to deal with. Without getting too into the weeds of argument, here's some things to consider. Unlike traditional pure service based development, a game "feature" typically includes art assets and game design. Assessing a "feature" release for a game can't be done with pure automation alone. You need to assess if it meets your visual/art standards and if the design is "fun" and balanced. These are parts of game software development that automated testing can't do and slow down traditional "continuous delivery processes". Or in your case, at "hundreds of deployments a day" more of a "continuous deployment" approach.
The last thing you need to consider about games is that they often have game "clients". These are not "services" and involve updating CDN's and "patching" out to players. In the case of mobile games, you can't really release any faster than mobile platforms will allow. Google that's typically 8-24 hours for a review and apple is usually minimum 24 hours. So even if you could have 100 builds of your client ready a day, you'd be impeded by the platform release.
Even if you remove "mobile" the PC client release has to factor in things like how big the patch size is etc. You don't want to force players to patch their client 100 times a day even if you could release it that fast.
In order to bring a game feature to players you often need to have things in the client, not just the services. In fact sometimes you don't even need to change the services, but you have to bring the client to players.
All of this is why Games typically don't change as fast in production as traditional pure service/web client only technologies.
some__other__guy
;)
some__other__guy
I'm not actually sure, but I suspect we have significantly more servers in prod than in our build infra. Though they may not really be comparable - our build infra is mostly persistent EC2 VMs, but the prod stuff is all containers.
some__other__guy
Once builds have been deployed, manual testers can download them and start validating bug fixes, new features, etc, and put their feedback in the JIRA ticket. QA is a core part of our workflow.
some__other__guy
If you're using GitHub, Actions (https://github.com/features/actions) are a really easy way to get started with simple automation.
some__other__guy
I'd love that! Canada is great.
ivey
Me but with darcs