The Event System Bug

Event System Overview: The event system is a back-end data structure which allows something to connect / disconnect to it. Anything connected to it when it is called, will be notified and can do things based on what happened. An example is an event for a client joining the area, or leaving, or a mine becoming armed etc. In the mine example, something would connect to it (for example the detonate mine skill) and make the mine detonate once it has armed. It is used a lot throughout the game in various places and is a great alternative to adding a listener to a whole class which doesn't work in many situations. It allows the triggering of a function from an external source without them having to care who is actually connected to the event. This implements the Observer design pattern and is found in most games. There are different connection types such as method (allows you to connect to a function of another class etc.), function (standard function connection) or event (can connect an event to an event in order to chain events etc.).

Event System Tech: The Event system stores a pointer to an array of unsigned chars (dynamic sink data) as well as an static unsigned char array of 120 bytes (128 bytes - 2 pointers for a null terminating pointer as well as a start pointer to store the amount of dynamic memory allocated. The sink_data is what is used to store pointers to the connected function, connected class or connected event etc.

// Dynamic sink data mutable unsigned char* sink_data; // Local sink data mutable unsigned char local_sink_data[ SinkDataSize ];

The system uses the static array until it runs out of space and then allocates dynamic memory and copies the data over. After that it works just like most common dynamic data structures where it will reallocate memory once it is required, each time allocated more than previously.

The event system handles event disconnects during event execution by implementing a linked list of "IterationState" structures which point to the current sink data. This linked list gets added to each time the event is called (a new node is added to the end) which allows for recursive event calls. If an event disconnects during the event call it will fix all of the iteration state pointers by adjusting them by the amount of data lost from the removal:

for( auto s = iteration_state; s; s = s->next ) { if( reinterpret_cast< unsigned char* >( sink ) < s->iter ) s->iter -= size; }

The check in there is because we only need to fix the iteration states that are "behind" us because they will be the ones to break once we go back up the call stack to where they were used. Linked list nodes after the one being removed are left intact. The iter here is the pointer to the sink_data.

The Bug: The game would crash deep inside the event system event call with the sink_data pointer being corrupt. This was the most common crash on the live realm after the deploy of 3.0.0 (Fall of Oriath) and all of the call stacks where extremely unhelpful. We did have one crash on the weapon_swap event when switching from 1 weapon with summon raging spirits (a skill) equipped to two weapons with 3 summon raging spirits in each weapon. All of the skill gems had microtransactions (skill effects) applied to them. We narrowed down the event connection to be related to the microtransaction system (each gem with a microtransaction disconnects and connects to the weapon_swap event when you switch weapons). The bug was not reproducable in any other situation and changing any of the items or gems would stop it from crashing. Even more annoying was even with the correct setup it only crashed sometimes, we could not figure out why. One of the QA members even went to the extent of copying his .GGPK file (the game data file) for us to use to reproduce the bug!

My co-worker implemented logging inside the event system to track the disconnects and connects. That in itself was a hassle because you have to understand, the event system is used A LOT in the game. He had to implement a global void pointer which would point to a specific event. It would be set to nullptr everywhere except for when we call our weapon_swap event connects / disconnects, and then after that we would switch it back to nullptr again. We immediately had issues with this due to the fact that events can be called recursively, so we had to ensure the global pointer would save its previous pointer so that it could switch back to the previous event after our logging.

After this we did the repro and got it to crash. The latest pointer that was trying to call the event wasn't anywhere else in the logs. What that meant is we somehow had an event executing that had never been connected to. How was that possible? My first thought was memory corruption. We investigated the whole implemention of the event system and how it worked. He assuredly told me the event system handles connections / disconnections while the event is being called. This is true, it does using the iteration state and it works. If it didn't the game would be crashing everywhere.

Finally we investigated the implementation for copying from the local_sink_data (static array) to dynamic sink_data. It does this when it tries to make a new connection and it doesn't have enough memory in the static array to fit it.

const size_t size_left = sink_data + GetDataSize() - ptr; const size_t new_size = sink.SizeOf(); if( new_size + sizeof( void* ) > size_left ) .. Expand() ..

Through this we realised the expand function doesn't handle updating the iterationstate linked list pointers. We quickly did some math to see if this was likely the culprit:

Method Event contains: - a pointer to the servant (the class / function owner) - 4 bytes - a pointer to the method / function - 4 bytes - a pointer to a virtual table (each sink overrides a base sink struct with 3 pure virtual functions) - 4 bytes This results in a total of 12 bytes per method event. Coincidentally in our logs, we were connecting to 10 of these events in a row (12 * 10 = 120 bytes; the maximum size of the static memory array) and we were crashing on the next one.. We added logging to the Expand function and, sure enough, the crash occurred after it expanded the sink_data (copying the data to the dynamic system) and tried to continue processing the event calls. The fix was relatively simple, we just had to fix the pointers in each of the iteration state nodes after we called Expand. This involved calculating the difference between the new pointer (dynamic array location) and the old pointer location and adding this difference to the saved iteration state pointer. Boom, problem solved!

Pretty strange to find a bug like this after the file / system was first implemented in 2009 and has only had a few commits to it since then. Even more surprising is we had never had a situation where the 120 byte limit for storing events had been surpassed inside an event execution. This also explained why it was so hard to reproduce; that 120 byte limit was getting passed very quickly from regular game-play, so to reproduce the bug, the weapon switch had to be one of your first actions when you first enter a new instance. None the less, it was an interesting and confusing bug to solve, especially when delving into the depths of the event system.

CubeRunner, Path of Exile

I have spent a lot less time working on my hobby game "CubeRunner" (yeah I don't have an actual name yet) than I would like but work and life keeps me busy! I recently implemented a level system for the game. I got inspired whilst I was playing Track Mania and realised a lot of people playing my game would enjoy a level system to unlock, work on and complete. The infinite game mode is fun and all but can get boring really quick. It is also at the downfall of randomisation and procedural generation rules which can sometimes be repetitive or simply uncreative compared to hand-made levels. I play to implement several difficulty stages with several levels each.

Once I have completed the levels system, I will need to add a save / load system so that players can properly progress. Luckily UE4 has great features available for saving / loading information which makes it easy! After that I plan to finish my upgrades (bonuses) system as well as some advanced features such as physics based obstacles, dynamic obstacles and the advanced control mode itself.

To aid my assistance as a developer at GGG I created a stat description generator tool. As a gameplay programmer I create quite a lot of stats for items and monsters etc. Most of these stats will need descriptions added so they can be displayed as mods on an item. Creating stats is reasonably fast but I spent almost just as much time creating the descriptions for them. This is mainly due to having to manually create the txt file, set the correct format and layout the file properly so it can be parsed by the client. My tool automates all of this and makes it much quicker to create the files.

Creating tools to help yourself with development (or anything) is very satisfying and shows how investing some time into something can feel like a tangent, but can end up saving you countless time and hours!

Forza Horizon 3

Since my new PC build, my brother and I have been playing the new Forza. I have an XBOX 360 controller running through a wireless receiver which makes racing smooth and fun! This is probably the only time or situation in which I can say that a controller is actually better than a keyboard or mouse..

Forza Horizon 3 is over $100 NZD to purchase which is at the highest end of the price range for a brand new range. None the less I felt that I would just do it, seems I had just spend thousands on my new PC and I haven't bought a brand new (expensive) game in a long time. After playing about 50 - 100 hours on it so far, I decided I would do a review of my thoughts and critiques surrounding Forza Horizon 3.

Merits

The first point I will make about the new Forza is regarding the graphical fidelity; It looks amazing! Although I don't play as many games (especially brand new AAA games) anymore, this game probably has the best visuals I have seen from a game. The weather effects are excellent with rain affecting the ground, creating small pools, puddles and slickness on the road. Sun rays beaming through trees and the eeriness of the night with the stars gleaming and low lying fog, you feel joyful simply driving through the landscape and environment in Forza Horizon 3. All vehicles have incredible detail as well which creates a strong realistic experience. I cannot fault any points for graphics, environment, effects or any visual aspects at all really!

I think Forza did a great job at replay-ability in regards to the races. I only realised quite far into the game that most races can be done in any car class / model. The AI (Drivatars) that you verse are simply chosen from cars matching your class and model with similar upgrades etc. This is great and means you can still progress far into the game with a not-so-fast car.

I think the gameplay in Forza Horizon 3 is excellent. I believe the bucket list challenges are what really create enough diversity to ensure the game-play is varied, fun and unique. The different environments, speeds and styles of races mean that races are different enough to be fun (particular with the off-road races).

The AI controlled "Drivatars" are incredible. They have excellent ability to simulate real drivers with various difficulty levels which do more than make them "faster". Added aggression, varying driving styles and techniques make the AI extremely fun to verse against. It is also great so not see (as far as I can tell) the infamous "catch-up" feature that is ever present in racing games. AI also crash from time to time which is entertaining and realistic.

Faults / Weaknesses

You cannot sell cars (except on the auction house to real players). I assume this is to stop people from making huge amounts of credits, but I still think this should be do-able, albeit at a fraction of the amount the car is actually worth. I hate how much this game encourages you to collect hundreds of cars.

Cars are far too easy to get; With the lottery / gambling system you seem to win a new car every few levels. Too me this is just getting handed money and unlock-ables which takes away from the progression. You also seem to get a free car when you create a new festival, which also seems a little too "spoon-feedy". My starting car was a Nissan Silvia with a wide body kit and full decals, paint and the works. I would call this a pretty customised race / sports car which I think is a ridiculous way to start the game. I want to start in a rusty old Corrolla and work my way to the top, but that is just my opinion I guess.

I must say that the game reuses particular roads in races a bit much, but the street races make up for that. I am somewhat disappointed there isn't an actual race track in the map, nor is there a drag strip, these additions would have been welcome.

The paid DLC car packs and monitisation is far too pushy and in your face. I just spent over $100 NZD on this game and already I am locked out of cars that are teased at me in every menu. It doesn't help the DLC cars are extremely cheap on the auction house and show no indicator they are part of the DLC until you try to bid on them. What';s more, the pushiness of micro transactions for the game is absurd. I cannot believe Microsoft is this desperate for money and it's frankly quite disgusting. I would be happy if Microsoft decided to create a large DLC expansion a few months down the line that added new cars, tracks, areas and events etc. The key difference being a better content for money ratio as well as the separation from DLC and main game. I should not have to be subjected to adverts to micro-transactions and DLC whilst playing a $100 NZD game.

I do feel the gameplay as a whole is a bit "arcadey" for me but I guess that is the style of the game. Racing against a plane, air-ship, train and boat are a little bit out there in my opinion but they do provide an enjoyable experience and help cater the game to a wider audience (not just hardcore / serious racing sim players).