Doom3 BFG Source Code Review: Introduction (Part 1 of 4) >>

On November 26, 2012 id Software released the source code of Doom 3 BFG edition (only one month after the game hit the stores). The 10 years old idTech 4 engine has been updated with some of the technology found in idTech 5 (the game engine running Rage) and it was an interesting reading session.

I call it "idTech4 prime" since it is idTech4 powered up with idTech5 :

• Threading system.
• Sound system.
• Resources system.

By far the most appealing aspect is the Threading System: Doom 3 was developed at the dawn of the multi-cores era when few PCs were SMP capable. Now the rules have changed, even cellphones feature multiple cores and a game engine must be multi-threaded in order to use the full potential of a machine: In this regard, idTech5 "Jobs System" is elegant.

I have cleaned up my notes into this article. I hope it will inspire people to read more source code and become better engineers.

Part 1: Introduction.
Part 2: Multi-threading.
Part 3: Renderer.
Part 4: DooM Classic integration.

Edit May 24th, 2013 : Thanks for the tweet from John Carmack :) !

First contact

    git clone https://github.com/id-Software/DOOM-3-BFG

          

2. Open Visual Studio 2010 Express and hit F8 for "Build".

Done !


Note : If Direct3D SDK is installed, the full project compiles with 5 minimal warnings in less than a minute.

Debug Mode

Only three steps to start tinkering with Visual Studio 2010 Express :

1. Update the Debug command parameters to Steam assets location.

    +set fs_basepath "C:\Program Files\Steam\SteamApps\common\DOOM 3 BFG Edition" +set r_fullscreen 0

          

2. Set the Startup Project to "Doom3BFG".

3. Hit F5.

Source code readability

C++ subset :

Doom 3 BFG is written in C++, a language so vast that it can be used to generate great code but also abominations that will make your eyes bleed. Fortunately id Software settled for a C++ subset close to "C with Classes" which flows down the brain with little resistance:

• No exceptions.
• No References (use pointers).
• Minimal usage of templates.
• Const everywhere.
• Classes.
• Polymorphism.
• Inheritance.

And despite being heavily multi-threaded, the source does not employ smart pointers or Boost: What a relief (those usually make the code unreadable).

Comments :

Comments are plenty and useful since they are usually one sentence describing what the next block does for each important parts of a function. Here is an example from ParallelJobList.cpp :

    
    int idJobThread::Run() {

	threadJobListState_t threadJobListState[MAX_JOBLISTS];
	int numJobLists = 0;
	int lastStalledJobList = -1;

	while ( !IsTerminating() ) {

    // fetch any new job lists and add them to the local list
		if ( numJobLists < MAX_JOBLISTS && firstJobList < lastJobList ) {
			threadJobListState[numJobLists].jobList = jobLists[firstJobList & ( MAX_JOBLISTS - 1 )].jobList;
			threadJobListState[numJobLists].version = jobLists[firstJobList & ( MAX_JOBLISTS - 1 )].version;
			threadJobListState[numJobLists].signalIndex = 0;
			threadJobListState[numJobLists].lastJobIndex = 0;
			threadJobListState[numJobLists].nextJobIndex = -1;
			numJobLists++;
			firstJobList++;
		}
    
    // if the priority is high then try to run through the whole list to reduce the overhead
    // otherwise run a single job and re-evaluate priorities for the next job
    bool singleJob = ( priority == JOBLIST_PRIORITY_HIGH ) ? false : jobs_prioritize.GetBool();

    // try running one or more jobs from the current job list
    int result = threadJobListState[currentJobList].jobList->RunJobs( threadNum, threadJobListState[currentJobList], singleJob );


    

Overall the reader gets an immediate understanding of each part of an algorithm and I hope it will inspire people to write better code: Modern software development is not about being an ace programmers anymore. It is about being a team player that empowers co-workers with:

• Elegant design.
• Easy to read and commented code implementing it.

Doom 3 BFG ranks high on both those metrics.

What has changed ?

The Visual Studio solution has been cleaned up:

Old Doom 3	New Doom 3 BFG

• Gone the two "Game" projects (one for Doom III classic and the other for "Ressurection") : Game is now an unified project.
• Gone cURL: Doom III does not download things anymore.
• Gone the silly DoomDLL name...that was actually generating DOOM3.EXE.
• Gone the outdated Maya tools to export md5 models, animations and camera path.
• Gone TypeInfo that provided in-house RTTI/Introspection.

Summary of the projects in Doom 3 BFG solution:

Projects	Builds	Observations
Amplitude	Amplitude.lib	Used for Doom Classic: Tool to adjust WAV amplitude.
Doom3BFG	Doom3BFG.exe	Doom 3 BFG Engine.
doomclassic	doomclassic.lib	Heavily refactored Doom1/2 game engine.
external	external.lib	jpeg-6 and zlib source code.
Game-d3xp	Game-d3xp.lib	Unified Game library, featuring original + extension + new levels. Note that it compiles to a static library now instead of a DLL.
idLib	idLib.lib	The id software Toolkit providing abstracted filesystem, threading, containers.
timidity	timidity.lib	Used for Doom Classic: Convert a MIDI input to WAV format.

New Architecture

The architecture differs significantly from the original Doom III: The solution now compiles into one monolithic executable (Doom III compiled to one executable and one DLL containing the game logic).

This was motivated by two reasons (according to the main Developer Brian Harris ):

• Consoles such as PS3/Xbox360 don't have very good support for dlls.
• Speed up development. With dlls, you have to be very careful about memory -- you can't allocate in one and free in the other. This makes things much more difficult and creates very hard to track down bugs.

Console driven changes

Targeting the Xbox 360 and the PS3 on top of PCs resulted in many important updates:

1. As previously mentioned the entire game is contained in one executable.
2. The game used to store assets in various PAK files (which were zip archives). The high latency of DVD readers pushed id Software to use a "Resource System": One file containing all the assets for one level is loaded in one batch.
3. Doom3 assets used to be all text based (ref). In order to lower loading time, some of the assets such as models and animations are now binary (.bmd5mesh and .bmd5anim).
4. Doom3 was designed to run at 640x480 with a 4:3 aspect ratio. Since TV and Monitors are now 16:9, all the menu have been redone. Probably in order to speed up development, they are implemented in Adobe Flash. Doom3 BFG runs its own Flash interpreter (/neo/swf/ folder). Again Flash was used in order to speed up development.
5. The renderer shaders have been rewritten using GLSL 1.5. HLSL shaders can be converted on the fly.
   
6. In order to have acceptable framerate, the "Flash light" could not be used simultaneously with weapons on PC. Now with Doom 3 BFG it can be duck taped on weapons but for performances reason it won't cast any shadows.
7. Since all menus are cross-platforms, the PC has lost much of the renderer settings: We get a simple version that satisfies both PC and Consoles.

Multi-threading

The new and the old engine materialize the shift of paradigm that occurred over the past 10 years: The "free lunch" is over and game engines must be designed with multi-threading in mind in order to wrap around multi-core machines.

Hence the most appealing thing to read from Doom III BFG is idTech5 threading architecture. A detailed review can be found on a dedicated page.

Renderer

The Renderer has been updated: The two main changes are:

• It runs on Open GL 3.2 Compatibility profile using GLSL 1.5 shaders.
• It is heavily multi-threaded (with up to four threads running concurrently).

It is detailed in a dedicated page.

Doom Classic

Doom III BFG allows to play Doom 1 and Doom 2.

At first it sounds like a trivial exercise to integrate the old Doom1 engine into the new Doom3 BFG: Just redirect all Inputs/Outputs ! But allowing multiple players in split-screen on PS3 and Xbox360 involved doing something idTech1 was never meant to perform.

More in the dedicated page.

Recommended Readings

Mathematics for Game Programming :

A few years ago I bought the famous Mathematics for 3D Game Programming and Computer Graphics. ..which is horrible and almost useless (more for mathematicians than game programmers). After that I stayed away from proclaimed "math for games books". Thanks to a co-worker I recently came across 3D Math Primer for Graphics and Game Development which is THE math book that any Game Developer MUST have on his bookshelf. If you don't have it already, order it now. As far as I know it is the only book that features all visualization tricks such as "see a dot product like a projection" and other gems.