Archive for March, 2007

Small Hiatus

Wednesday, March 28th, 2007

The SourceMod Dev Log will be on a short haitus until Monday.

Half-Life Entity Properties

Monday, March 26th, 2007

Today’s Dev Log article is on the wiki: Entity Properties.

This article discusses about how the two major entity property types work in Source: network variables and save/restore fields. It also shows how they can be enumerated from server-side plugins.

User Functions in SourceMod, Part 2 of 2

Monday, March 19th, 2007

Last week, we saw that “callfuncs” and “forwards” from AMX Mod X were literally combined into one system. We also saw that eliminating the distinction between “single” and “multi” forwards allows for much more flexible coding. Today we’ll take a look at the other side to calling functions: dynamic (also called “fake”) natives.

Dynamic natives were added to make IPC easier for AMX Mod X scripters. They have a few hugely important benefits over forwards and callfuncs:

  • Plugins can act as modules. As a demonstration of this, the “amxmod_compat” plugin for AMX Mod X implements many AMX Mod natives merely as a script.
  • It’s faster and safer than callfuncs. The compiler handles all parameter checking and pushing for you.
  • It’s easier than callfuncs. You don’t have to implement a complex stock that finds the plugin, finds the function, pushes all its parameters, and then returns the result.

That said, how do they work? For AMX Mod X and SourceMod, respectively, any native must implement the following prototypes:

typedef cell (*AMX_NATIVE)(AMX *, const cell *);
typedef cell_t (*SPVM_NATIVE_FUNC)(IPluginContext *, const cell_t *params);

However, how do we make a native with this prototype, such that it knows which plugin and function to call? This is tricky. Technically, all we know about the native, while we’re inside the actual function, is the native index we’re using in the calling plugin. From there we can get its name, and with the name, we could look it up in a table of dynamic natives. Unfortunately, name lookup is slow. Another possible solution is making a secondary lookup table for each plugin, that says “native #X is dynamic native #Y in plugin #Z.” However, we opted for something more crafty, which uses less memory and less computation time.

Both AMX Mod X and SourceMod dynamically assemble the fake native function such that it knows exactly what to do. Basically, the native looks like this in assembly:

;cell_t FakeNativeRouter(IPluginContext *pContext, cell_t *params, void *pData);
extern FakeNativeRouter
 
;(IPluginContext *, cell_t *)
push DWORD_PTR    ;Push the fake native ID/pointer
push [esp+12]     ;Push the "params" array
push [esp+12]     ;Push the context pointer
call FakeNativeRouter
add esp, 12
ret

Once assembled, the new address is given to plugins as the native pointer. When called, it invokes FakeNativeRouter, which is the real meat of the solution. In steps, it:

  1. Decodes the index/pointer back to information about the plugin and function.
  2. Pushes the parameters and plugins onto a global stack.
  3. Calls the function in the plugin, where natives get parameter information from the global stack.
  4. The plugin returns to Core, and Core returns to the JIT/VM.

This is the optimal, albeit more complicated solution. The natives, as mentioned earlier, are nearly identical between AMX Mod X and SourceMod. The difference is in how the dynamic function is constructed.

AMX Mod X implements the dynamic native construction through an external assembly object.

  1. amxx_DynaInit is called. This caches the FakeNativeRouter address.
  2. amxx_DynaSize is called. This tells Core how many bytes are required for the memory buffer for building the function.
  3. amxx_DynaMake is called. This copies a dynamic native function “template” into the given executable buffer. It then modifies a “push” call such that it contains the dynamic native’s ID.
  4. The router function then receives an ID, which is an index into a vector of dynamic natives. From there, it knows how to execute into the plugin.

SourceMod’s system is much cleaner.

  1. CreateFakeNative is called in the JIT/VM. This handles executable memory allocation, casting, relocation, and function building. The JIT internally has macros for building functions in assembly, so it is both optimized and much more readable than the NASM version, which must copy and modify a function template.
  2. The router function receives a pointer, which it casts back into a structure that has all the information it needs. The reason it uses a pointer instead of an ID is subtle: since dynamic natives can be removed at runtime (plugin unloading), a simple vector does not suffice for efficiency or memory growth.

As you can see, dynamic natives are implemented with a rather hacky solution. On the other hand, they’re highly optimized and require very little additional memory to store.

Whoa?!

Friday, March 16th, 2007

Although a bit late in the day, the real news is over here.

User Functions in SourceMod, Part 1 of 2

Wednesday, March 14th, 2007

Inter-plugin communication was one of the most important features of AMX Mod X, and thus it was equally as important in SourceMod.

IPC started in AMX Mod X with the “callfunc” system, which we thought was enough to suffice for all IPC usage. It was implemented by PM OnoTo and appeared in the first AMX Mod X release. Simply put, they allowed you to call any public function in any plugin with three major steps:

  1. Start the call, given a plugin name and a public function.
  2. Push the parameters you want to send.
  3. End the call and get the return value.

The idea behind callfuncs was that plugins could share information between each other. It worked great when you wanted to notify other plugins of certain events. However, that only represented one direction of data flow, owner to listener. The opposite, listener to owner, was much harder with callfuncs. Say you wanted to provide external functionality to plugin B from another plugin A. You had to make an include file with something like:

stock my_function(params)
{
    callfunc_begin("my_real_function", "plugin_A.amxx")
    callfunc_push(params);
    return callfunc_end();
}

Then plugin B would have to call this stock, and hope that plugin A was loaded. To solve this problem, AMX Mod X introduced a three tier function calling system:

  1. “Callfuncs,” the lowest level of function calling. Version 0.1+
  2. “Dynamic Natives,” allowing plugins to create natives. Version 1.5+
  3. “Forwards,” allowing plugins to make event/callback systems more easily, especially with dynamic natives. Version 1.7+

In SourceMod, it’s a bit different. Internally, SourcePawn is object oriented. Callfuncs are equivalent to the IPluginFunction class, and forwards are equivalent to the IForward class. The trick is that they both derive from one type, ICallable, and share the same calling convention. Because of this, “callfuncs” and “forwards” are one and the same in SourceMod, and the methods for calling both are nearly identical, both internally and for plugins.

There is one subtle difference between a forward call and a function. A function is technically a “temporary” object. If the plugin unloads, the function unloads as well. Special care must be taken to remove references to functions that are being unloaded.

On the other hand, forwards are automated collections of functions. Calling the forward calls all functions in its collection. They also intrinsically remove the unloadability complication. Whenever a plugin is unloaded, all forwards are “cleansed” of the removed references. While forwards are more expensive to create and maintain, they are much easier to use and manage, especially since unlike AMX Mod X, they are true collections, and thus you don’t need to worry about storing individual function IDs in arrays. They allow completely arbitrary functions to be packed in.

There is one other subtle difference between SourceMod’s forward system and AMX Mod X’s, which has been touched upon before. In AMX Mod X, there were “multi forwards” and “single forwards.” A multi forward called a function of the same name in every plugin; for example, “plugin_init” was a multi forward. A single forward was simply one function, and a multi forward was a named global function.

In SourceMod, this design is split for usability purposes. IPluginFunction is the “single forward” equivalent, save for it being unsafe to cache. IForward is the “multi forward” equivalent. Derived from that is the new type, IChangeableForward, which lets you modify the internal collection. This important feature allows users to implement per-function hooks without worrying about local memory constraints. An IChangeableForward can act as either a multi forward or a single forward, or both.

As a quick example, AMX Mod X code for per-function events looked like:

#define MAX_FUNCS 80
new g_FuncsNum = 0
new g_FuncArray[MAX_FUNCS]   //Array of single functions
 
AddFunction(const plugin[], const name[])
{
    if (g_FuncsNum >= 80)
        return 0
    g_FuncArray[++g_FuncsNum] = CreateOneForward(plugin, name)
    return 1
}

A SourceMod version might look like:

new Handle:g_Funcs
 
AddFunction(Handle:plugin, Function:func)
{
    if (g_Funcs == INVALID_HANDLE)
        g_Funcs = CreateForward()
    AddToForward(g_Funcs, plugin, func)
}

That covers callfuncs and forwards, which have been combined into one system for SourceMod. On Monday we’ll see how dynamic natives are different from AMX Mod X, and how they are implemented. There’s something special planned for Friday, so stay tuned!

On Timer Design, Part 2 of 3

Monday, March 12th, 2007

Last time we saw four timer systems and their evolution. Today, let’s look at the mistakes in each, and how they affect SourceMod’s final design.

With AMX Mod X, as usual, the mistakes were plentiful. The most serious was that timers were non-uniquely identifiable, a direct fault of user-chosen ids. Unfortunately this was a convenience benefit so users could make up base ids and add an offset. For example, you could have:

#define PLAYER_TASK   0x3000 

Then if you passed PLAYER_TASK+id into set_task, you could subtract PLAYER_TASK from the task id in the callback, and retrieve the player index. Aside from this convenience, non-unique task ids were a hindrance and led to unreadable code, as users randomly created large numbers to avoid internal collision.

The second mistake was having so many confusing modes. In AMX Mod X, the entire timer list is checked every 0.1 seconds. This cannot really be optimized without splitting the timers into multiple lists, and that is a direct result of the extra modes based on map changes. Furthermore, these extra flags added unnecessary complexity for functions that are very rarely used

Lastly, AMX Mod X had a very, very horrible function, called task_exists(). It should be obvious why such a function is a bad idea, but let’s go over why in case you don’t see it: You should always know if a task exists or not. You either created it or you didn’t. If you did create it, it has either executed or it hasn’t. This function is a very expensive call and exposes two bad design practices:

  • Not writing deterministic code. If you have to make sure a timer is currently running or not, you should be using global variables that deterministically tell you whether the task in session.
  • Inventing task ids so you can check later if they’re still in session. Again, you can save this globally; why eat up another unique ID?

Task ids are a problem shared between both AMX Mod X and the proprietary system. In order to find the timer associated with a task id, you must iterate through all tasks until you get a match. This makes timer removal/lookup very expensive (and another reason task_exists is so bad in AMX Mod X). Luckily, properly written code usually doesn’t rely on task lookup; it only uses the TaskId for sanity checks, but the issue is still there when it is used.

However, let’s take a look at CSDM and CS:S DM’s timers, which use objects: in both systems, the object pointer you pass in is the same as the unique task identifier. The trouble here is: what if you want to create two timers on the same object? This fundamental flaw makes both systems very limited. While the proprietary system gets around this, it still exposes a ‘deleteThis’ function, meaning it treats each task instance as a temporary object. CS:S DM is the worst, and doesn’t even give you a choice of whether to delete the object or not. Conclusion here: user-owned timer identifiers are bad, and task ids are bad, which means all four systems are bad.

Lastly, AMX Mod X is the only system to actually tell the callback what timer is being run! In the other three systems, whether they support multiple timers on the same object or not, the unique timer identifier is never passed into the object. This is quite problematic.

How does SourceMod’s Timer System fix these issues?

  • Timers are identified by a pointer created by the timer system. They are not owned by the user.
  • Timer callbacks (there are two: one for executing, one for terminating) all receive the given timer identifier, even if it’s the same object being used.
  • There are only two modes: repeat or no repeat. There are no counters. Repeatable timers are stored in a flat list, single timers are sorted for optimal iteration.
  • Although repeatable timers can short-cut out like in CS:S DM, the timer removal functions are O(1) because of pointer usage.

As such, the SourceMod timer system is not only a lot simpler, but much more flexible and easier to work with. It’s also more optimized.

Bonus Question: Why were repeating counters left out?
Answer: You don’t need them. Since there are multiple ways to kill a repeating task, a counter is just syntax fluff; you can make your own. The system is easier to design if extra unnecessary options are left out. Furthermore, repeatable counters are not very common, and when they are needed, it’s usually for just one repeat where creating a new task suffices.

The third part of this series will be continued later.

Update March 14, 2007 4:46PM Fixed SVN link.

On Timer Design, Part 1 of 3

Friday, March 9th, 2007

In the past four large projects I’ve worked in, save for AMX Mod X, I’ve directly designed the timer system in each. In all four, the timer system was either cludgy or lacked some important feature or consideration. When I stepped up to prototype and document the SourceMod version (which faluco is implementing), I knew exactly what had gone wrong with previous iterations, and thus prototyping it was very easy.

In all systems, timers are encapsulated in “timer objects” — the difference is in how they are created, how their intervals are computed, and how they are destroyed.

AMX Mod X: The Worst of All
First, let’s take a look at what’s probably the worst of all, AMX Mod X: the nefarious set_task() function. Given an interval of time k, it has five modes of acting on this interval:

  • default: Execute the timed function once after k seconds.
  • repeat: Repeat the timed function n times over intervals of k seconds.
  • loop: Repeat the timed function every k seconds indefinitely.
  • after start: Execute the timed function k seconds after the map’s timelimit is up (I don’t really understand this one).
  • before timeleft: Execute the timed function k seconds before a map’s timelimit is up.

As if this wasn’t confusing enough, timer callbacks had two other parameters: the task_id and the data array. If a data array was specified, the task_id was the second parameter in the callback; otherwise it was the first. By default, task_ids were 0, but they could be any number you chose. When destroying tasks (usually repeatable ones) with remove_task, you had to specify the id and an “external” or “internal” flag. All tasks with a matching id were then removed; if you specified “external,” tasks outside of the given plugin were removed as well.

A Bit Better: CSDM 2
In CSDM 2, timer objects have no extra modes. They are always one-time executes with a given interval from the current time. A timer was represented by an object which contained the two callback functions:

  • void Run(void): Tells the object to run the task.
  • void deleteThis(): Tells the object to delete itself once the timer system is done with the object.

Rather than task ids, which were optional and non-unique, tasks are identified by the actual object pointer itself. Thus when you created a timer instance, the instance you passed to the timer system was the unique identifier.

CSDM Tasks got two callbacks, exposed through the task object (the system is object oriented):

Proprietary Tasks
For a proprietary project I worked on, I made a third task system. This one was designed to be more flexible than the CSDM one, but less complicated than the AMX Mod X version. The API was similar to CSDM 2′s, except with a few changes:

  • Tasks had a “repeatable” flag and a built-in counter to allow for both infinite and ending repeatable tasks.
  • Tasks were identified by task ids again, except they were unique integers decided by the timer system itself. Thus they were neither pointers nor user-given identifiers.
  • A new function was introduced: FireTask, which allowed you to short-cut and execute a task right away, by its TaskId.
  • The Run() function could return either Timer_Continue or Timer_End, which let you short-cut out of repeatable tasks without calling the expensive KillTask function.

CS:S DM Tasks
Lastly, CS:S DM had essentially the same design as the proprietary system, with minor convenience changes. The deleteThis was removed for virtual destructors instead. Also, task ids were again abandoned in favor of object pointers to the task event itself.

Conclusion
In Monday’s article, I will explain why the design kept changing, and the how and why of SourceMod’s evolved timer system.

Data Packs

Wednesday, March 7th, 2007

A new feature introduced in SourceMod is “data packs.” Before explaining what these are, let’s look at why they exist.

Observe the set_task function in AMX Mod X. It allows you to pass an arbitrary array of data onto the timer callback. This is ugly, as it requires bitpacking. Worse, it means the entire array has to be copied into each successive call.

As ugly as this is, we probably still would have done it in SourceMod, except for the fact that — we can’t. The String tag does not coalesce to any other tag properly, and thus it is impossible to pack strings through. That’s not acceptable.

Instead, commands such as CreateTimer (might not be public by the time this article is released) take in a Handle. This way, you can pass one Handle through asynchronous functions. It’s much faster to copy an integer than an array, and you can pass any Handle type.

However, users still need to be able to pack data into a structure for asynchronous calls. Thus, we created the DataPack type. A DataPack is simply a buffer of memory that stores whatever you write to it. You can write floats, cells, and strings. The DataPack itself is a Handle, so you can pass it through asynchronous functions. And as a special convenience, there is a CreateTimer function which explicitly handles allocation and deallocation of a DataPack for you.

Other places DataPacks will probably get used are the SortCustom1D/SortCustom2D functions, as well as SQL_ThreadQuery.