Archive for December, 2006

SourceMod – What’s the Deal? (Part 2 of 2)

Sunday, December 31st, 2006

If you read the previous article, you now understand why SourceMod was a huge disaster. We learned a lot from those mistakes, and our goals have changed drastically over the past two years. As we announced a few months ago, we are no longer interested in providing a full AMX Mod X port to Source. Among other things, it is simply not possible given the nature of Valve and the Source engine.

In April 2006, three members of the original SourceMod team (Faluco, DS, and I) decided to restart the project. Our new goals were:

  • Optimize the Pawn language as much as possible
  • Create our own new language platform for maintaining and extending
  • Solve the “Unloadability” problem of dependencies
  • Create a highly extensible, flexible, and well documented API for embedded applications

As you can probably tell, the words “Half-Life 2″ don’t appear in this list. As such, we began rewriting SourceMod from the bottom up, using none of the original code, or even writing HL2 specific code. We completed various phases in roughly this order (dates are all 2006):

  • March to August: Complete rewrite of the Pawn JIT optimizer, which is almost 120KB of raw x86 assembly.
  • August to September: Complete C++ rewrite of the Pawn Virtual Machine and its associated helper functions, as well as a rewrite of the stored file format.
  • October: Added a myriad of features to the compiler, such as dynamic arrays, function pointers, and native strings.
  • November-December: Implemented the “Five Essentials” to the SourceMod Core — the HL2 independent pieces which make the language embeddable, functional, and extensible.

So, Where are We Going from Here?
Our original plan was to have SourceMod’s Core basically wrapped up and internally ready by December, so we could announce a very, very limited demonstration package before the new year. As a bunch of things came up in December which I’d rather not get into, that didn’t happen. But our progress on SourceMod has been very good over the past two months, enough such that for the first time in a while, I have the confidence to write this news post.

Our current goal is to deploy SourceMod “as soon as possible” as something with capabilities similar to Admin-Mod for Half-Life 1. As it gets more popular and more testers, we will build it up to something comparable to AMX Mod X. Minimally, the first initial public release of SourceMod will probably occur sometime in late January to mid February. Don’t hold me to this, it is simply a rough guess. It will feature, roughly:

  • Loading of C++ extensions
  • C++ API: Group based administrative permissions to clients
  • C++ API: Modular extensibility to scripting (adding functions/events)
  • C++ API: Access to most of SourceMod’s internals
  • Scripting: Basic string manipulation
  • Scripting: Basic client events and natives
  • Scripting: Basic floating point/vector manipulation
  • Scripting: Creating commands/cvars
  • Scripting: Creating timers

As we are not focusing on the actual “admin” plugins right now, I cannot say what they will contain. We’re trying to get SourceMod usable by third party developers before we start going too far into the feature list, to avoid the mistake we made last time.

In conclusion, I’m very pleased to say that we have made a lot of real progress on SourceMod lately. We’re very pleased with the solidity, extensibility, and organization of the code, and Faluco and I have been working hard over the past few months to churn out as much as possible.

If you’re interested in SourceMod, make sure to stay tuned to the devlog over the next few weeks. I will be posting and explaining API and scripts to the devlog directly from SourceMod. We’re at the point where documentation is matching what will be in the final product, so I will also be writing articles for the AlliedModders Wiki and discussing them here.

SourceMod – What’s the Deal? (Part 1 of 2)

Saturday, December 30th, 2006

This article is a two part series on, well, SourceMod! This time I’m laying off the code, and I’m going to discuss SourceMod’s past, present, and future. This first article is a quick timeline of events, and a look at why SourceMod died.

  • October 7th, 2004: SourceMod is officially announced as a multi-scripting and administration platform for the Half-Life Source engine.
  • December 1st, 2004: The Source SDK is first released.
  • December 25th, 2004: SourceMod releases its source code.
  • January 2005: PM decides to start a “SourceHook” project to embed inside SourceMod.
  • March 2005: Realizing that the Source SDK was highly inadequate for plugins, as plugins were already hacking memory without any cohesion, PM and I decide to split SourceMod into two separate projects: Metamod:Source (with SourceHook), and SourceMod.
  • May 26, 2005: After many revisions and much work on PM’s part, Metamod:Source is finished and released.
  • May 2005: The SourceMod Core is completely abandoned. Work briefly starts on a new version, “core2.” The idea of a new core is quickly abandoned, and SourceMod is internally considered “on permanent hiatus.”
  • August 2005: The SourceMod team goes back to AMX Mod X.
  • March 22, 2006: A public quorum in IRC is held to decide if the community still wanted SourceMod. The reaction is positive, and the remaining developers decide to continue the project at an unknown future date, using a single language: Pawn.

That is how we left things off. So, why did SourceMod fail if AMX Mod X was such a success?

Why SourceMod Died

1. Too Ambitious
SourceMod’s original goal was to have multi-way scripting. This means reimplementing something like COM or Mozilla’s XPCOM, a huge task that simply wasn’t feasible in a short amount of time. Logically, this goal doesn’t even make sense. Supporting multiple language platforms is an absolute nightmare, and does not make sense for an embedded system. It was an incredibly flawed goal in multiple ways, and is SourceMod’s single greatest point of failure.

Another ambitious goal was having an embedded administration server (sort of like WebMod). This was foolish because not everyone has access to MySQL or a GSP that would allow random ports to be opened. Furthermore, it would require an inordinate amount of work for an unnecessary, non-essential feature.

2. Too Complicated and Unstable
Do not read “unstable” as “crashy,” but as in API stability. The API was constantly changing as it grew more and more complex, and the fundamental design issues were never resolved (such as unloadable dependencies and plugin execution). On the lucky chance that the code actually compiled properly, it was unusable or hard to understand.

3. Top-down Structure
We built SourceMod from the top-down. It could hook things, add console commands/cvars, load modules, print messages, et cetera — the essentials of what AMX Mod X could do. But, it never had one important feature: scripting. Nothing was testable, demonstrable, or releasable, because the multi-way scripting platform was never completed. By the time it was usable, the design was flawed and had to be scrapped. The project largely stagnated because there was nothing to show – all of the working code was simply unusable without the scripting layer finished.

4. Unfamiliar Development Team
The AMX Mod X team was mostly a tightly knit group, where everyone was familiar with policies and coding styles. SourceMod introduced new members who proved to either be useless or unfamiliar with these practices, or eventually wandered away due to the problems above.

5. Valve
Valve had to be on the list. Half-Life 1 is far more flexible and usable for mod-independent plugin authors than Half-Life 2. As Valve continued (and continues) to break their software, their own API, add unpopular changes, ignore developers, and deny simple features which exist in many other games, motivation to complete SourceMod dwindled. Our internal list of grievances goes back to the beginning of 2005, and includes:

  • Denial of simple engine changes (such as being able to intercept log messages).
  • Denial of IGameEvent changes for property enumeration.
  • Denial of a fix to let plugins print text to the screen.
  • Inability to have a timely fix for UserMessage enumeration crashing.
  • Inability to have a beta release system for mods.
  • Inability to forewarn developers of major changes.
  • Denial of a better menu interface, which is universally regarded as garbage.
  • One-time removal of simple menus, which were supposedly re-enabled only “temporarily” due to huge public backlash.
  • Denial of trivial changes to make SourceMM easier to load, since GameInfo.txt is overwritten.
  • Inability to fix unloading issues with Valve Server Plugins.
  • Inability to fix the issue of VSP load failures not showing what the problem is.
  • Inability to fix VSPs not being able to intercommunicate.
  • Disabling client command execution (engine API!) with an extremely poorly thought-out update (whose breakage extended across a myriad of plugins and API calls), not admitting broken backwards compatibility, and providing no workaround or forewarning.
  • Inability to provide working, documented, mod-independent API for very simple functions, such as telling a bot from a player, telling whether a player is really dead, telling a bot from a spectator proxy, or even changing player/entity positions.

Supporting one mod alone is difficult, but supporting all of them is very hard, and with many mods altering the Source SDK significantly, cross-platform support becomes much more difficult. When you add Valve’s unannounced API breakage into the equation, Source looked like a doomed platform to us.

What Next, Then?
In tomorrow’s continuation of this article, I will write what our current goals are, and by when we hope to achieve them.

SourceMod Dependency System

Friday, December 29th, 2006

When we decided to restart the SourceMod project a few months ago, we broke the project down into five essential pieces. These pieces are tightly coupled with each other, and SourceMod was not be able to function even minimalistically until they were implemented:

  • Library System – Allow loading of dynamic libraries to provide extensibility
  • Plugin System – Load, unload, and manage SourcePawn plugins
  • Forward System – Abstract various methods of calling into plugins
  • Share System – Track and manage dependencies/shared data
  • Handle System – Abstract pointers into typed, secure integers

The first three were a breeze to implement compared to the second two (which we are just finishing up now). Why? Well, if you had the unfortunate chance of reading the original SourceMod articles on dependencies — dependency tracking isn’t easy, especially if you don’t know what you’re doing. In the old system, every object was “Unloadable,” and each Unloadable received callbacks about unloading objects. This system was messy, inefficient, and poorly designed.

I’d like to take a moment to explain the SourceMod Share System. First, consider the actual problem. Ingredients:

  • A Library, called ‘MySQL
  • A Plugin, called ‘BanManager‘, which requires MySQL

Let’s say that BanManager caches a global SQL connection from the MySQL library. SourceMod promises that you can unload any module or plugin. If we unload the plugin, the plugin could be poorly written and leak the SQL connection. If we unload the module, the plugin could crash trying to use the natives. We call this relationship a dependency. BanManager is dependent on MySQL. There is a reverse relationship as well, since if BanManager unloads, Core must be able to ‘free’ the SQL connection without knowing specifically about MySQL.

To solve this, SourceMod introduces two concepts AMX Mod X does not have: “Identities” and “Handles.” An Identity is a uniquely identifiable integer. Anything can create an Identity, although for simplicity, only a few special systems do:

  • Plugins each have their own Identity.
  • Libraries each have their own Identity.
  • Core has its own internal Identity.

A Handle is a unique integer that abstracts a pointer. Handles arose out of a conflict with 64bit systems. In AMX Mod X, raw pointers were given to plugins, and nobody was the wiser. This worked because AMX Mod X secretly built two copies of the plugin at once: a 32bit copy and a 64bit copy. Core loaded the correct build at run-time, so sizeof(int *) == sizeof(int) was guaranteed, and a pointer would always fit in a cell this way. In SourceMod, we didn’t want to use this ugly method of cross-platform support (thinking ahead here), so we decided to implement a system where a pointer can be assigned a unique, 32bit ID, called a Handle. These are very similar to the WinAPI data structure called a HANDLE.

The Handle System supports the following operations.

  • Type Creation: Creating a unique “Handle type,” which may derive from a parent handle type. For example, Core might have an SQL type, and MySQL will create a derived MySQL type.
    • Each type must register a “destructor,” which is called when any of its Handles are freed. For example, the MySQL type destructor might close the SQL connection.
    • You can also specify security rules, tied to an Identity, to prevent other Identities from creating Handles of your type. Each of the subsequent operations will require the same Identity token to be passed in, or else they will return an error. This is to prevent accidental damage to private implementations.
  • Handle Creation: Given a pointer and a handle type, you can create a new Handle, which can be safely passed to plugins as an integer. You can also specify an Identity which ‘owns’ the Handle, for bookkeepping.
  • Handle Duplication: The reference counter on a Handle is increased, and a new copy is returned. This is useful for sending Handles to other plugins, in case the original plugin is unloaded, the copied Handles will not be removed.
  • Handle Removal: A reference counter on the Handle is removed. If the counter hits zero, the Handle is freed and the type’s destructor is called to clean up the internal pointer being held.

Briefly, the benefits of using Handles instead of raw pointers:

  • Pointer Safety: The user cannot corrupt the pointer value itself.
  • Type Safety: The user cannot easily send a random incorrect handle, since the value itself is checked against both a type and a serial number embedded in the ID.
  • Double-free Safety: You can’t free the same Handle twice and corrupt memory.
  • Cross-Platform Compatibility: Handles can map any pointer size into a 32bit ID (as long as you don’t run out of IDs).
  • Dependency Tracking: As this article shows, Handles let you very nicely deal with unknown data being passed around, duplicated, destroyed, et cetera.

It should now be apparent how these two systems nicely tie into each other. There is a direct relationship between Identities and Handles, since each Handle is linked to two Identities: the owner of the Handle, and the owner of the Handle type. Going back to our original problem, let’s see what happens when the Plugin unloads:

  1. Core retrieves the Plugin’s Identity.
  2. The Handle System is told the Identity is about to be destroyed.
  3. The Handle System finds all Handles owned by the Identity (this is a very simple operation, the data structure is nicely compacted).
  4. Each Handle has its reference counter decremented.
  5. If a reference counter hits 0, the Handle is freed, and the MySQL library is notified of the object it must free.

Voila! The dependency problem is solved! What about the MySQL library being unloaded though? This one is a bit more complicated, but the idea is similar:

  1. Core retrieves the Library’s Identity.
  2. The Handle System is told the Identity is about to be destroyed.
  3. The Share System sees if any of the Library’s functions were in use by the Plugin, and unloads the Plugin in response. This triggers the plugin unloading case above.
  4. The Handle System finds all remaining Handles and Handle Types owned by the Library and removes them.

How might this look to the end user? On the C++ side:

/*Create our type */
HandleType_t g_hEntType = g_pHandleSys->CreateType("CBaseEntity");
/* Creating a type for a plugin */
Handle_t handle = g_pHandleSys->CreateScriptHandle(g_hEntType, pEntity, pScript);
/* Getting data from a plugin's handle */
Handle_t handle = static_cast<handle_t>(params[1]);
CBaseEntity *pEntity;
if ((g_pHandleSys->ReadHandle(handle, g_hEntType, (void **)&pEntity) != HandleError_None)
{
     /* The plugin passed an invalid handle! */
}

And from the SourcePawn side, creating/destroying entities might look like:

Handle:ent = CreateEntity();
/* ... */
CloseHandle(ent);

Next Time: The SourceMod Plan — Is there a release date or progress bar?