Archive for February, 2005

SourceHook Optimizations

Saturday, February 19th, 2005

OMGOMGOGM TEH PM R TALKIN ABOUT HIS SOURCEHOOK AGAIN STFU OMGOM BANANA BAN1111111

Yeah, it’s me again. I’ve decided to optimize SourceHook a bit. What I wanted to optimize is:
1) Execution time
2) Compilation time
3) Executable size


1) Execution time
One can see that SourceHook’s original function calling mechanism is a bit weird:

Change vtable entry to original function
Execute the function
Change vtable entry to hooked function

This has the advantage that it can execute the original function with the correct this pointer and without the use of inline assembly. But we can optimize it. If SH_SAFE is not defined, SourceHook now generates an other call class for each function: Its vtable points to call gates that alter the this pointer and call the original function.

mov ecx, [ecx+4]
mov eax, (original function address)
jmp eax

(msvc)

As you can see, the original function pointer is stored as a member of the call class (in [ecx=this] + 4). The original function address is assigned on runtime.
Because this generates code on runtime, it will not work with PaX. It is also not compatible with gcc and with amd64 at the moment. So you can disable it by defining SH_SAFE.

An other problem is the list lookup in the hook handlers:

# define SH_FIND_ENTRY() 
	SH_ACI(SH_HookedIfaceList)::iterator fetmpiter = std::find(SH_ACI(g_SH_HookedIfaceList_).begin(), 
			SH_ACI(g_SH_HookedIfaceList_).end(), this); 
	if (fetmpiter == SH_ACI(g_SH_HookedIfaceList_).end()) 
		SH_CF_NoEntryFound_FatalErr(); 
	SH_ACI(SH_HookedIface_) &entry = *fetmpiter;

Imagine this situation:
There are many plugins, and 30 instances of IGaben are hooked. The EatYams function is called on the one that was registered last. The hook handler has a pointer to the instance at this moment (the “this-pointer”). In order to get the original vtable, the pre and post functions list, etc., it needs to find the corresponding entry in g_SH_HookedIfaceList_IGaben. Because our instance is 30th in the list, everytime some hooked function is called, std::find has to iterate 30 times before it finds the entry.

But what else could we do? We can hide the entry pointer into the instance somehow. We are modifying the vtable all the time, so we can use it for this too. Here is what I did:

When an instance is added, its first vtable entry is replace with code generated on runtime. This code only jumps to the original vtable entry. But we can hide something behind the code! This code looks like this:

mov eax, (original vtable entry)
jmp eax
(ENTRY POINTER HERE!)

When the intance is removed from the hooked instances, and reset the vtable entry.

This way, we can replace the SH_FIND_ENTRY macro with this:

	inline void *SH_CF_GetPointer(const void *iface)
	{
		const unsigned char *jmpcode = *(*(const unsigned char***)iface);
		return *(void**)(jmpcode + SH_CF_POINTEROFFSET);
	}

# define SH_FIND_ENTRY() 
	SH_ACI(SH_HookedIface_) &entry = *(SH_ACI(SH_HookedIface_)*)SH_CF_GetPointer(this);

This is again generating x86 code on runtime, so the old method is used when SH_SAFE is defined.

Compilation time and executable size
I was trying to optimize these too, but I don’t have any impressive results so far.

Plugin Reloading

Monday, February 14th, 2005

Why did Admin-Mod and AMX Mod/X reload plugins on map change?

This issue was recently brought up on IRC by EKS. To be honest, I could not supply an answer. Why have plugins consistently been reloaded on map change?

I came up with two answers. One, it allows the plugins file to be easily edited for changes. You can update or remove plugins and simply call changelevel. This is good both for administration and for debugging. Secondly, you don’t have to worry about re-initializing data such as player info. On map change, your plugin is simply deleted from memory and reloaded.

However, it must be noted that this functionality is rarely needed, and as SourceMod is vastly more complex, it may not be a good idea to force a reload on every plugin. Furthermore, the issue of re-initialization can be solved through an extra forward. So in SourceMod I’ve decided that plugins will not reload on mapchange. They will, however, be notified of a mapchange so they can do basic checks.

This feature can be overrided for debugging or other purposes. An example plugins.ini can look like:

[plugins]
myplugin.js = debug,reload

This will put the plugin in debug mode as well as forcing it to reload on mapchange.

To make this change, plugins now exist in two categories: Persistent and Temporary. Persistent plugins are located in the plugins.ini file and are never resynced on mapchange. If the plugins.ini file or database changes, these changes are not merged on mapchange. This works because SourceMod allows the dynamic loading and unloading of plugins, unlike Admin-Mod/AMX Mod/AMX Mod X. Meanwhile temporary plugins are loaded either through the console or by special request in the configuration. These temporary plugins are always reloaded on mapchange. The only way to remove either is to use the plugin management interface, either through IVPluginManager, rcon/server console, or web interface.

Minor SourceHook changes

Thursday, February 10th, 2005

Yesterday we experienced a weird crash in SourceHook on server unload. After a lot of debugging, I have found that a part of SourceMod was calling GameEventManager->RemoveListener _after_ SourceHook has been shut down. Of course, parts of SourceMod should be calling interfaces through the CallClass, but it still has no reason to crash. The problem was that the vtable of the interface was not reset. So when the function got called, SourceHook was searching for the entry in the hooked interfaces list, it found nothing (because the interface had already been removed from CSourceHook::Shutdown), and tried to access it, so the server crashed. So I decided to add a check whether something has been found to SH_FIND_ENTRY. But what should it do if it hasn’t? It can’t call the original function, because the original vtable entry is not available. So I decided to shut down the server with a proper message. Still better than a crash.

Anyway, why is it possible to remove an interface from the list without reseting its vtable? When I was first testing SourceHook, it crashed when reseting the vtable on unload. I assumed that the pointers were already invalid, so I just made reseting the vtable optional. Later, DS found out that the pointers were available, but the vtables were reprotected (because VirtualProtect/mprotect affects the whole page, not only the range you pass). So now, we reset the vtables on unload, but do not reprotect them.

I have changed the SH_REMOVEINTERFACE macro a bit; it doesn’t take the “reset” parameter; instead, it always resets the vtable. If you are _sure_ that the pointer is invalid, you can still use the SH_REMOVEINTERFACE_NORESET macro. Note that if you use this on an interface that is still valid, and the interface is used later, the server will crash. This should prevent people from accidentally not reseting the vtable.

Faster String

Wednesday, February 9th, 2005

Let’s look at a basic example function first:

 public String:junk(id) 
{
     new String:stuff = string("Data:") 
     for(new x=0; x<4; x++) 
     {
           strnCat(stuff, x)
     }
     return stuff; 
}

All this crazy function does is return “Data:0123″. Now let’s look at what happens with the dynamic string while this function runs.

We declare the string like this:
String0

But what if your plugin has pre-allocation turned on?

- Each time a string expands in size it needs more space in the computer’s memory, it can’t simply request more space directly after its current space, it has to delete everything and start over. This process is time consuming. If pre-allocating is turned on then all strings start off with an invisible “buffer” zone to allow strings to expand a little without having to re-allocate that string.

String1

We now have 1/3rd more space allocated before the string has to be re-created in memory. But this isn’t good enough for our unusual function above (The empty characters at the end are not spaces, they are nothing/NULL). The string would still have to be allocated / destroyed a total of three times in this call.

In situations when the following is true
1. The programmer knows exactly how much space they will need
2. That space is more than 1/3rd the strings original size

Then the following natives are of use –
pString(num) – Pre-Allocate string
pCString(const word[], num) – Pre-Allocate + Data

Here is our code again with an improvement:

 public String:junk(id) 
{
     new String:stuff = pCString("Data:", 9) 
     for(new x=0; x<4; x++) 
     {
           strnCat(stuff, x)
     }
     return stuff; 
}

When the string is first created it is done so with 9 characters so it only needs to be allocated a single time and thus the function runs faster.

Try to keep in mind that the 9 character allocation is _NOT_ the same as a character array size, if you try and put a 10th character in this string it will work fine but behind the scenes it will have to completely re-allocate a new string to hold all ten characters.

MetaEng Revision 007

Tuesday, February 8th, 2005

After a suggestion from forum/IRC-goer lancevorgin (aka stosw), I have taken another step to making MetaEng more friendly.

If you read (I think Part 2), you knew that functions were called as dummy CFuncObjects. That has now changed. Functions are now separate objects, referred to as ICallables. CallFunction/PrepareArray/GetFunction are now removed from IScript, giving:

ICallable *IScript::GetCallable()
ICallable::Execute()
ICallable::PrepareArray()

This was a very logical move, and I thank lance for pushing me to it. As functions are now abstract objects, this leads the way for:

MetaEng without Connectors
Lambda functions
Easier memory management
Calling functions in functions/classes

While we’re a long way away from MetaEng without connectors, it’s an idea for the future that lance has shown is viable, and ICallable will become more powerful in order to adapt (with properties such as parent functions (for lambda functions) and parent classes (for member functions)), in order to abstract complicated design structures that are available in modern scripting languages.

SourceHook Reloaded

Monday, February 7th, 2005

No, I didn’t really like Matrix Realoded, and the original Matrix was much better, but in case of SourceHook, it is the other way round.

As you may know, my original SourceHook, an easy-to-use virtual-function hooking framework, was only able to hook interfaces that are available on startup, before anything is initialized, that is, exposed by the engine or the gamedll. That was a big limitation, so I decided to remove it. I have also added support for overloaded functions. Many things have changed, and so has the SourceHook API.

Interface declarations
Right. This is the shif_*.h stuff. Two changes in here:
1) You have to #define __SH_DYNAMIC to true or false before specifying SH_FUNCDECL_BEGIN.
2) The SH_FUNCDECL* macros have a new parameter.

Let’s look at these in detail:
1) If you #define __SH_DYNAMIC to false, SourceHook loads the interface on Initialization. For this, it uses the good old __SH_GETINTERFACE macro you also have to #define. Usually, for things like IVEngineServer, it looks like this:

#define __SH_DYNAMIC false
#define __SH_GETCURINTERFACE (IVEngineServer*)g_SM_HL2Connector.GetInterfaceFn()(INTERFACEVERSION_VENGINESERVER, NULL)

If you #define __SH_DYNAMIC to true, it means that the interface is not loaded on statup. You still have to #define __SH_GETCURINTERFACE to something although it’s unused; I use NULL.

2) The new parameter is the so called overload. Consider this interface:

class IGaben
{
public:
   virtual void EatYams(const char *location) = 0;  // Eats all yams from location
   virtual void EatYams(const char *location, int amount) = 0; // Only eats amount of yams from location

   virtual void Vomit() = 0; // Too many yams :X
};

When SourceHook builds the VTable struct and the delegate/function pointer typedefs, it needs an unique name for each function. That’s why the overload parameter was added. It is simply appended to everything that needs to have an unique name. You would do something like this in your interface declaration:

SH_FUNCDECL1_void(EatYams, 0, SH_NOATTRIB, const char*)
SH_FUNCDECL2_void(EatYams, 1, SH_NOATTRIB, const char*, int)

SH_FUNCDECL0_void(Vomit, 0, SH_NOATTRIB)

Note that this is the only place where you need to specify the overload parameter – when hooking functions, the compiler decides which one to hook depending on the type if the function you pass. (If you pass a type that does not correspond to any overloaded function, you get some weird error)

Usage
There are a few changes with the macros:
SH_HOOK_STATICFUNC and SH_HOOK_MEMBERFUNC remain the same; so do SH_REMOVE_STATICFUNC and SH_REMOVE_MEMBERFUNC.

The old SH_GET_CALLCLASS is now SH_GET_CALLCLASS_STATIC.

The SH_ADDINTERFACE and SH_REMOVEINTERFACE macros are new. They hook/unhook an interface pointer:

   SH_ADDINTERFACE(GetInterfaceFunc, Interface, Pointer);
   SH_REMOVEINTERFACE(GetInterfaceFunc, Interface, Pointer, reset);

GetInterfaceFunc is, as always, SourceMod’s GetInterface function; Interface is suprisingly the interface, and pointer is a pointer to an instance. SH_REMOVEINTERFACE also has a fourth bool parameter, reset. If this is set to true, the VTable is reset to its original values. Note that this is not necessary, but it should be done if you can ensure that the Pointer is not destroyed yet.

Also note that we do not reprotect vitual function tables anymore. Damaged Soul has found out that this can also reprotect other other vtables in the same page, which leads to crashes when we try to access it later. Thanks!

Ok, so what now, after you have called SH_ADDINTERFACE on some pointer? You can start hooking! It actually works the same way the “normal” hook functions work, but these have a new parameter:

#define SH_GET_CALLCLASS(GetInterfaceFunc, Interface, InterfacePointer)
#define SH_HOOKDYNAMIC_STATICFUNC(GetInterfaceFunc, Interface, InterfaceFunction,
   InterfacePointer, YourFunction, Post)
#define SH_HOOKDYNAMIC_MEMBERFUNC(GetInterfaceFunc, Interface, InterfaceFunction,
   InterfacePointer, YourInstance, YourMemberFunction, Post)
#define SH_REMOVEDYNAMIC_STATICFUNC(GetInterfaceFunc, Interface, InterfaceFunction,
   InterfacePointer, YourFunction)
#define SH_REMOVEDYNAMIC_MEMBERFUNC(GetInterfaceFunc, Interface, InterfaceFunction,
   InterfacePointer, YourInstance, YourMemberFunction)

I think these are pretty self-explaining. Don’t forget to SH_REMOVEINTERFACE after you are done.

I have also added a new macros that can be used from handlers: META_IFACEPTR. The value of this macros is the pointer to the interface that was meant to be called.

Here is a small illustration of dynamic hooking:

class OMGOMGOMG : public ConCommandBase
{
public:
	bool IsCommand()
	{
		SM_LOG("OMGOMGOMG::IsCommand called. Returning true. this=%p", this);
		return true;
	}
};
OMGOMGOMG g_Lol1;
OMGOMGOMG g_Lol2;

class OLOLHOOK
{
public:
	bool IsCommandHook()
	{
		if (META_IFACEPTR == &g_Lol1)
		{
			SM_LOG("IsCommandHook called; ptr=%p => superceding with true", META_IFACEPTR);
			RETURN_META_VALUE(MRES_SUPERCEDE, true);
		}
		else
		{
			SM_LOG("IsCommandHook called; ptr=%p => overriding with false", META_IFACEPTR);
			RETURN_META_VALUE(MRES_OVERRIDE, false);
		}
	}
};
OLOLHOOK g_omgomg;


// This is neccessary so the compiler is not too clever and does not bypass the vtable lookup
ConCommandBase *lol1 = &g_Lol1;
ConCommandBase *lol2 = &g_Lol2;

SH_ADDINTERFACE(GetInterface, ConCommandBase, lol1);
SH_ADDINTERFACE(GetInterface, ConCommandBase, lol2);

SH_HOOKDYNAMIC_MEMBERFUNC(GetInterface, ConCommandBase, IsCommand, lol1, &g_omgomg,
   &OLOLHOOK::IsCommandHook, false);
SH_HOOKDYNAMIC_MEMBERFUNC(GetInterface, ConCommandBase, IsCommand, lol2, &g_omgomg,
   &OLOLHOOK::IsCommandHook, false);

SM_LOG("g_Lol1.IsCommand() = %d", lol1->IsCommand() ? 1 : 0);
SM_LOG("g_Lol2.IsCommand() = %d", lol2->IsCommand() ? 1 : 0);

This example produces this log output:

L 02/07/2005 - 15:13:56: [SM] IsCommandHook called; ptr=06DF166C => superceding with true
L 02/07/2005 - 15:13:56: [SM] g_Lol1.IsCommand() = 1
L 02/07/2005 - 15:13:59: [SM] IsCommandHook called; ptr=06DF16E4 => overriding with false
L 02/07/2005 - 15:13:59: [SM] OMGOMGOMG::IsCommand called. Returning true. this=06DF16E4
L 02/07/2005 - 15:13:59: [SM] g_Lol2.IsCommand() = 0

Final Word on Scripting Support

Wednesday, February 2nd, 2005

SourceMod’s Scripting Framework can potentially adapt to many languages. How does this work out in terms of us supporting them?

C++ is the “official” native language of SourceMod. It’s also how we’re writing the base admin plugin.

The official supported scripting language is “JavaScript”, implemented by Mozilla’s SpiderMonkey. The MetaEngine for it, and the associated base connector and documentation, will be maintained by developer Blaubaer.

Finally, we’ve made a MetaEngine for “AMX/Small”. The core of this is a direct copy from AMX Mod X, so things like error tracing will remain the same. This is considered DEPRECATED and UNSUPPORTED. As Manip posted a few entries ago, he is the official maintainer for it, giving him a chance to play around with some new ideas.

On another note, you may remember the specialized plugin forums from www.amxmodx.org – I’m hoping to somehow acquire a vBulletin license (maybe some kind soul will donate one or money for one :) ), because there’s no way in hell I’m rewriting those hacks for phpBB again – the codebase is absolutely awful. It would also give me (or rather: Freecode) a chance to port our almost-ugly SourceMod theme. That’s up in the air though, but it would be nice to have good forum software before the first major release.

SourceHook

Wednesday, February 2nd, 2005

SourceMod will contain something like MetaMod. We have decided to call it “SourceHook” (and not SMMeta as initially planned by one of our devlopers that can’t think of good names < -- me).
SourceHook is able to hook virtual functions from interfaces. I wanted to make adding support for new interfaces easy, so each interface has its own header file, like this one for IVEngineServer. All supported interface header files are included in sh_supportedinterfaces.h.

So, as you can probably guess, SourceHook #defines things SH_FUNCDECL_BEGIN, SH_FUNCDECL_END, and SH_FUNCDECL4_void to some stuff, and then includes sh_supportedinterfaces.h. Actually, that file is included many times. For each interface, SourceHook generates a lot of stuff. Most shint_*.h header files define the SH_FUNCECL* macros to something, then include the interface headers. The things that are generated are things like a vtable struct, the hook functions, typedefs, the hook lists, a registration interface, etc. For each parameter count, there are 4 versions of the macros. Here are some examples:

SH_FUNCDECL2_void(function, attributes, parameter1_type, parameter2_type)
The only not self-explaining thing here is attributes, which can be const, or SH_NOATTRIB.

SH_FUNCDECL2(function, attributes, return_type, parameter1_type, parameter2_type)

SH_FUNCDECL_vafmt2_void(function, attributes, parameter1_type, parameter2_type)
This declares a function of the form: function(parameter1_type p1, parameter2_type p2, const char *format, …)
In other words, this is used for functions like ClientCommand, which use printf-style formatting.

SH_FUNCDECL_vafmt2(function, attributes, return_type, parameter1_type, parameter2_type)

Writing all these macros for all parameter counts would be a pain, so I have created a small utility called shworker, which generates all the macros from templates like this.

Note that SourceHook was only designed to hook interfaces that are publicly exposed by the engine or the gamedll; That is, you can not (yet?) hook things like the Dispatch() function from Jeff’s ConCommandBase pointer.

Ok, how to use this!?
You can hook functions to:

  1. Global functions / Static member functions:
    SH_HOOK_STATICFUNC(SourceMod's GetInterface function, Interface, Function, YourFunction, Post)
    
    
      
    • SourceMod’s GetInterface function:
      If you are in core, this is the global GetInterface function. If you are in a module, this is the GetInterface function pointer SourceMod passed to some init function.
    • Interface:
      The interface you want to hook
    • Function:
      The function you want to hook
    • YourFunction:
      Your hook function
    • Post: Set this to true if you want to make a post hook, to false otherwise

    For example, you can do something like this:

    void MyChangeLevelFunction(const char *s1, const char *s2)
    {
       LOG_SOMEHOW("ChangeLevel pre-hook called; Superceding! s1=%s, s2=%s", s1, s2);
       RETURN_META(MRES_SUPERCEDE);
    }
    
    ... (in some init function)
    SH_HOOK_STATICFUNC(GetInterface, IVEngineServer, ChangeLevel, MyChangeLevelFunction, false);
      

    Yeah, you can use the old meta macros in your hooks. Note that MRES_UNSET has been removed; not providing a meta result means the same as providing MRES_CONTINUE.

  2. Non-static member function:
    SH_HOOK_MEMBERFUNC(SourceMod's GetInterface function, Interface, Function, YourInstance, YourFunction, Post)
      
    • SourceMod’s GetInterface function:
      If you are in core, this is the global GetInterface function. If you are in a module, this is the GetInterface function pointer SourceMod passed to some init function.
    • Interface:
      The interface you want to hook
    • Function:
      The function you want to hook
    • YourInstance:
      A pointer to the instance of your class
    • YourFunction:
      Your hook function
    • Post: Set this to true if you want to make a post hook, to false otherwise

    Example:

    class MyTestClass
    {
    public:
       void MyChangeLevelFunction(const char *s1, const char *s2)
       {
          LOG_SOMEHOW("ChangeLevel pre-hook called; Superceding! s1=%s, s2=%s", s1, s2);
          RETURN_META(MRES_SUPERCEDE);
       }
    };
    
    MyTestClass g_TestClass;
    
    ... (in some init function)
    SH_HOOK_MEMBERFUNC(GetInterface, IVEngineServer, ChangeLevel, &g_Test, &MyTestClass::MyChangeLevelFunction, false);
      

Of course you can also remove hooks:

SH_REMOVE_STATICFUNC(SM's GetInterface function, interface, interface function, your function)
SH_REMOVE_MEMBERFUNC(SM's GetInterface function, interface, interface function, your instance, your function)

Note that these remove both pre- and post-hooks.

If a function is hooked, the hook always gets called, no matter what calls the function. That’s why you should not use the interface pointers provided by engine/gamedll. Instead, get your interface pointers using:

SH_GET_CALLCLASS(SM's GetInterface function, interface, is_gamedll_interface?, interface version)

the third and fourth parameters are used if SourceHook doesn’t support the interface – then, the engine’s / gamedll’s interface factory is used.
ie.

IVEngineServer *engine = SH_GET_CALLCLASS(GetInterface, IVEngineServer, false /*engine*/, INTERFACEVERSION_VENGINESERVER);
engine->ChangeLevel("xxx", "yyy");  // Calls engine's function directly, skips hooks

If you want to use SourceHook from modules, you will have to define the global g_SHPtr variable, and set it to GetInterface(SOURCEHOOKINTERFACE_NAME). For example:

#include "sourcehook/sourcehook.h"

IVSourceHook *g_SHPtr;
...
Init-Function:
g_SHPtr = GetInterface(SOURCEHOOKINTERFACE_NAME); // GetInterface is SM's GetInterface function

This pointer provides functions for hooks (=> you will be able to set up hooks without it, but you will not be able to use the metamod-style macros in them).

I’ve listed all macros you can use in hooks here:

	SET_META_RESULT(meta_result)
		Sets the meta result to meta_result, does not return
	RETURN_META(meta_result)
		Sets the meta result to meta_result, then returns
	RETURN_META_VALUE(meta_result, value)
		Sets the meta result to meta_result, then returns value.
	META_RESULT_STATUS
		Contains the highet meta result that was returned by now
	META_RESULT_PREVIOUS
		Contains the previous hook's meta result
	META_RESULT_ORIG_RET
		In post functions: Contains the original function's return value
				(or, if a hook specified MRES_SUPERCEDE, its return value)
		In pre functions: Undefined
	META_RESULT_OVERRIDE_RET
		Contains the override value (if any)

By the way, hooking of non-static member functions is accomplished using Don Clugston’s excellent FastDelegate class.

UPDATED Feb 03:
- Actually implemented the remove hook macros
- Changed get-call-class macros a bit