Aug 07, 2018

CI

Last month (maybe longer) I wrote about one of the top three production mistakes we made in Thimbleweed Park. Today like like to talk about the second one.

Just to be clear, these are production mistakes, not design or programming mistakes (although sometimes the line is blurry).

The first one was not integrating FMOD into my engine. As I wrote, it was a Penny Wise and Pound Foolish decision.

The one I'd like to talk about today is Continuous Integration, but first a little primer.

"What the hell is this witchcraft you call continuous integration!" I can hear you saying and don't feel bad. I wonder how many indie game devs use it. My guess is a lot fewer than should (a quick poll of friends is standing at 0%).

Continuous Integration (or, CI, as the pro's call it) is when a separate and often dedication machine is continuously building (compiling) your game whenever you check in code.

This is good for two reasons:

1) If you check in code that won't compile on one of the platforms (I don't care how good or careful you are, this will happen to you) the CI machine will let you (and the rest of the team) know. It helps ensure that the game can build at all times. If you can run a battery of unit tests, the CI machine will often do this as well.

2) Since the CI machine is a standalone machine, it's dev environment is a known quantity, so installing some goofy tool or new version of python on your personal dev machine isn't going to introduce oddities in the build.

3) Bonus point. You always have a build ready to put into test and then distributed to Steam, Xbox, etc.

I've used CI on previous projects and it can be a pain to set up, but it's a time saver from then on. It's indispensable and you should always use it!

Did we use CI on Thimbleweed Park? Of course not! That's why I'm writing this blog entry.

When the Thimbleweed Park engine got to the point where it could actually be used and David was brought on, I thought about CI. Previously I had used a dedicated machine in the office with Jenkins installed. I only ever needed to make a Windows build (or builds that used VS) so it was one machine with a few different flavors of builds.

For Thimbleweed Park I need to make Windows, Xbox, Mac, Android, IOS (and later Switch and PS4) and Linux builds. Without some fancy hoop jumping, this was going to require three machines.

My mind fuzzed over and I said "later".

Throughout the project I kept revisiting CI, and being overloaded with work, I kept saying "later". As the end of the project rolled around it seemed pointless since the project was almost over. Of course, I was wrong and we continued doing updates and ports for a year.

At the time I was looking at CI, cloud-based services like AppVeyor and TravisCI either didn't exist or never showed up on Google searches. Today you can just use these two cloud services to build Windows (or anything that needs VS), Mac (Mac and iOS) and Linux. Android can be built on any of them. If you don't mind waiting 10 or 15 minutes for the build to start, both services are very reasonable (and free for open source projects).

In the end, I built the Windows, Mac and Linux builds on my local machine. I had the entire process scripted, so it was running a single bash script and all the Mac versions were built. I ran a Windows VM on my Mac and would launch it was run a .bat file to built all the Windows flavors. Same with Linux, boot the VM and do the builds.

Thimbleweed Park was a tad complex because there were two parts that had to be rebuilt (the engine and the game), and while intertwined, they were separate and it was common for one to be rebuilt and the other not. It was also complicated by all the game build tools being on the Mac, so you couldn't actually build the game code on Windows. This didn't matter since the game code/files were 100% cross-platform. I could build them on the Mac and they would run on Windows, PS4, Switch, etc.

It makes CI a little more complex because your CI might be building the engine but something had to merge it all together (another CI process). The game code needs to be built first (and you don't need to rebuild it for each platform), then the engine process could grab it and merge it, or a third process could combine it all, but everyone would have to know when each piece was done (if a new build was needed). My point isn't that it's impossible, just that it's hard and a problem that I ultimately (and unfortunately) didn't need to solve.

Back when the project started, if I had known about AppVeyor and TravisCI, I would have set them up and ultimately saved me a lot of time and stress.

Next time for sure!!!

No, really, next time for sure.

Dom

Aug 08, 2018
Agree, CI is really helpful and a pain to setup,

Even with Appveyor and TravisCI it can be a pain to setup and maintain in the long term, for us we also start up new repos/projects all the time and dont want to waste a day or two each time mucking wround with the initial CI setup.

We ended up writing up something that let us setup a bitbucket/github repo up at the same time as CI (see here: https://github.com/irreverent-pixel-feats/ecology).

Its been a real time saver for us, especially when we want to prototype something, but still want the CI, or if we a participating in a game jam, its nice to just have CI there, and we dont want to waste time during the jam setting it up (abut cant do it in advance when we may not know exactly whatwe are doing).

Its written in Haskell which isn't everyones cup of tea, so i'm not suggesting anyone uses what we have written, but its definitely worth writing something like this up

(we got a blog post on it here: http://irreverentpixelfeats.com/posts/development/2018-07-21-ecology.html)

Hanaindiana

Aug 08, 2018
<3 Jenkins

Big Red Button

Aug 08, 2018
Reading your articles on game development always has been very interesting - even for readers like me, who are not software developers.

Adam East

Aug 08, 2018
Regardless of mistakes I loved the game and keeping my fingers crossed for more like it

Ron Gilbert

Aug 08, 2018
If you're not making mistakes, you're not trying hard enough.

Stefan

Aug 09, 2018
GitLab is very powerful, fun to use and you can host it for free on your own machines (you can also pay for it!). You can set it up relatively easily using Docker. https://about.gitlab.com/ (No, I am not working for them, but I'm a huge fan ;-))

Reagan W

Aug 09, 2018
What's next in line for you Mr. G? Any more retro-inspired masterpieces? Maybe one involving a dashing young wannabe pirate...?

Howard

Aug 11, 2018
I remember hearing on one of the podcasts that you used Git. Did you use Github or host your own server? Did you store both code and game assets in the same repo? Would this have factored into the choice of CI? (Jenkins locally vs cloud based service)

Ron Gilbert

Aug 11, 2018
My Library is in one git repro, the Engine in another and the game (scripts, art, sfx) are in a third.  I didn't want game programers (David, Jenn) to have to pull down a bunch of c++ code they didn't need.

If I had done CI, I would have used a cloud CI service. I don't want a bunch of machines running 365/24/7 in my house.  The whole point is to NOT have to manage machines. I have better things to do.

m thomas

Aug 16, 2018
Finally finishing TWP play through and it's turning out to be one of my favorite games of all time. The music, x-files/twin peaks aesthetic, and jokes make it a perfect game to me.

Sunny Kalsi

Sep 02, 2018
Bitbucket also has pipelines, which is another alternative to Travis et al. If you don't like jumping around several tools, it's an option (Disclosure: I work for Atlassian).

Niklas Borglund

Sep 02, 2018
At work, when I set up a CI-machine for our Unity-jobs, I decided to write all the scripts in a well known scripting language that has support on many different platforms. In my case, I decided to go with Javascript+Node+Npm+Gulp.

I agree, as mentioned in your post, that setting up CI-systems can be quite time consuming and this was a great way to reduce the risk of being entirely reliant on a specific setup (which can, and probably will, break at some point).
The setup on the Jenkins machine was quite simple after that with just having to run a simple gulp job with npm and the added benefit of everyone being able to run the jobs from their machines as well - if needed.

It kind of changes the question of "how" a job should run to a mere "when" - which is a great deal easier to answer to me.

Aaron Ramshaw

Sep 02, 2018
Hi there Ron

I'm Aaron. I'm from Australia
I just wanted to say that your blog is truly an invaluable resource to people like myself who are aspiring to be adventure game developers. I was contemplating how best to design a point-and-click just before I played Thimbleweed Park. The game unexpectedly taught me about puzzle dependency graphs right when I was trying to figure out how on Earth you did it. I researched the graphs and they led me to your blog. Since then I've been re-visiting it for other nuggets of wisdom like the retrospective advice in this post of yours.

I've been designing puzzles with the graphs for months now. At first it involved some serious abstract thinking because I don't have any art yet (its hard to imagine puzzles in backgrounds you don't have). Anyway, I see a lot of value in designing the puzzles and story first and foremost and the charts really do that well. I really can't imagine how such a thing could be achieved without the use of the tools you've given me. After a lot of practice I'm starting to get more efficient and creative with my puzzles (It's also good to know that they prevent me from making leaps of logic that could result in serious game breaking consequences).

I'm a child of Monkey Island. Born in '89, my parents sat me down with TSOMI when I was only six or seven years old. They told me puzzle games would make me smarter and yours pretty much taught me how to read and write.

Now I'm designing my own game and you're still teaching me so much
I just really wanted to say thanks for everything :)

Helmut

Oct 09, 2018
Hello Ron,
One thing i always wondered was why Thimbleweed Park needed an engine done in C++ and special ports on different platforms. The game could have been done in Javascript an run in a webview which ultimately would have made porting much easier.  Would that have gone against your programmers honour or were there other reasons?  I hope i'm not touching any religious beliefs here.. just curious.
P.S.: Purchased TP, played it all to the end. Did make heavy use of hotline though.
Here are the rules for commenting.