Aug 07, 2018

CI

Last month (maybe longer) I wrote about one of the top three production mistakes we made in Thimbleweed Park. Today like like to talk about the second one.

Just to be clear, these are production mistakes, not design or programming mistakes (although sometimes the line is blurry).

The first one was not integrating FMOD into my engine. As I wrote, it was a Penny Wise and Pound Foolish decision.

The one I'd like to talk about today is Continuous Integration, but first a little primer.

"What the hell is this witchcraft you call continuous integration!" I can hear you saying and don't feel bad. I wonder how many indie game devs use it. My guess is a lot fewer than should (a quick poll of friends is standing at 0%).

Continuous Integration (or, CI, as the pro's call it) is when a separate and often dedication machine is continuously building (compiling) your game whenever you check in code.

This is good for two reasons:

1) If you check in code that won't compile on one of the platforms (I don't care how good or careful you are, this will happen to you) the CI machine will let you (and the rest of the team) know. It helps ensure that the game can build at all times. If you can run a battery of unit tests, the CI machine will often do this as well.

2) Since the CI machine is a standalone machine, it's dev environment is a known quantity, so installing some goofy tool or new version of python on your personal dev machine isn't going to introduce oddities in the build.

3) Bonus point. You always have a build ready to put into test and then distributed to Steam, Xbox, etc.

I've used CI on previous projects and it can be a pain to set up, but it's a time saver from then on. It's indispensable and you should always use it!

Did we use CI on Thimbleweed Park? Of course not! That's why I'm writing this blog entry.

When the Thimbleweed Park engine got to the point where it could actually be used and David was brought on, I thought about CI. Previously I had used a dedicated machine in the office with Jenkins installed. I only ever needed to make a Windows build (or builds that used VS) so it was one machine with a few different flavors of builds.

For Thimbleweed Park I need to make Windows, Xbox, Mac, Android, IOS (and later Switch and PS4) and Linux builds. Without some fancy hoop jumping, this was going to require three machines.

My mind fuzzed over and I said "later".

Throughout the project I kept revisiting CI, and being overloaded with work, I kept saying "later". As the end of the project rolled around it seemed pointless since the project was almost over. Of course, I was wrong and we continued doing updates and ports for a year.

At the time I was looking at CI, cloud-based services like AppVeyor and TravisCI either didn't exist or never showed up on Google searches. Today you can just use these two cloud services to build Windows (or anything that needs VS), Mac (Mac and iOS) and Linux. Android can be built on any of them. If you don't mind waiting 10 or 15 minutes for the build to start, both services are very reasonable (and free for open source projects).

In the end, I built the Windows, Mac and Linux builds on my local machine. I had the entire process scripted, so it was running a single bash script and all the Mac versions were built. I ran a Windows VM on my Mac and would launch it was run a .bat file to built all the Windows flavors. Same with Linux, boot the VM and do the builds.

Thimbleweed Park was a tad complex because there were two parts that had to be rebuilt (the engine and the game), and while intertwined, they were separate and it was common for one to be rebuilt and the other not. It was also complicated by all the game build tools being on the Mac, so you couldn't actually build the game code on Windows. This didn't matter since the game code/files were 100% cross-platform. I could build them on the Mac and they would run on Windows, PS4, Switch, etc.

It makes CI a little more complex because your CI might be building the engine but something had to merge it all together (another CI process). The game code needs to be built first (and you don't need to rebuild it for each platform), then the engine process could grab it and merge it, or a third process could combine it all, but everyone would have to know when each piece was done (if a new build was needed). My point isn't that it's impossible, just that it's hard and a problem that I ultimately (and unfortunately) didn't need to solve.

Back when the project started, if I had known about AppVeyor and TravisCI, I would have set them up and ultimately saved me a lot of time and stress.

Next time for sure!!!

No, really, next time for sure.

Dom

6d ago
Agree, CI is really helpful and a pain to setup,

Even with Appveyor and TravisCI it can be a pain to setup and maintain in the long term, for us we also start up new repos/projects all the time and dont want to waste a day or two each time mucking wround with the initial CI setup.

We ended up writing up something that let us setup a bitbucket/github repo up at the same time as CI (see here: https://github.com/irreverent-pixel-feats/ecology).

Its been a real time saver for us, especially when we want to prototype something, but still want the CI, or if we a participating in a game jam, its nice to just have CI there, and we dont want to waste time during the jam setting it up (abut cant do it in advance when we may not know exactly whatwe are doing).

Its written in Haskell which isn't everyones cup of tea, so i'm not suggesting anyone uses what we have written, but its definitely worth writing something like this up

(we got a blog post on it here: http://irreverentpixelfeats.com/posts/development/2018-07-21-ecology.html)

Hanaindiana

6d ago
<3 Jenkins

Big Red Button

5d ago
Reading your articles on game development always has been very interesting - even for readers like me, who are not software developers.

Adam East

5d ago
Regardless of mistakes I loved the game and keeping my fingers crossed for more like it

Ron Gilbert

5d ago
If you're not making mistakes, you're not trying hard enough.

Stefan

5d ago
GitLab is very powerful, fun to use and you can host it for free on your own machines (you can also pay for it!). You can set it up relatively easily using Docker. https://about.gitlab.com/ (No, I am not working for them, but I'm a huge fan ;-))

Reagan W

4d ago
What's next in line for you Mr. G? Any more retro-inspired masterpieces? Maybe one involving a dashing young wannabe pirate...?

Howard

3d ago
I remember hearing on one of the podcasts that you used Git. Did you use Github or host your own server? Did you store both code and game assets in the same repo? Would this have factored into the choice of CI? (Jenkins locally vs cloud based service)

Ron Gilbert

3d ago
My Library is in one git repro, the Engine in another and the game (scripts, art, sfx) are in a third.  I didn't want game programers (David, Jenn) to have to pull down a bunch of c++ code they didn't need.

If I had done CI, I would have used a cloud CI service. I don't want a bunch of machines running 365/24/7 in my house.  The whole point is to NOT have to manage machines. I have better things to do.
Here are the rules for commenting.