Archive for January, 2005

The Vault

Saturday, January 29th, 2005

How does SourceMod’s Vault work?

A Vault was an Admin-Mod concept that let you store key/value pairs in a text file. In SourceMod, we’ve taken this a step up. The vault is implemented as four data structures: The Manager, Vault Masters, a Vault itself, and a Vault Entry.

The Vault Manager lets you move between different storage mechanisms. The planned implementations are Text (standard text files, for old-timers), Binary (efficient storage that can’t be edited by hand), and SQL (for integration with the web interface).

An IVaultMaster is an implementation of a vault access structure – these will let you open a vault from the Text/Binary/SQL interfaces. In AMX Mod and Admin-Mod, the Vault was a global storage facility. In SourceMod, you can have individual vaults, for example:

var vault = Vault("my_vault");
var str = vault.GetKey("STEAM_0:%s", id);

This will greatly enhance a plugin’s ability to manage its own persistent data.

An IVault is a single structure with access to a chunk of data sets, called IVaultEntrys. When you open an IVault with a IVaultMaster, you get an IVault back. When you get a key from an IVault, you get an IVaultEntry. The entry has three parameters:
The Key (i.e., the variable)
The Value (the variable’s contents)
The Timestamp (the last edit of the variable. 0 for persistent).

The timestamp addition was suggested by Geesu for a new cool feature of the Vault API – pruning! Now you can prune and clear individual vaults based on their date, and even set dates to be persistent, or invulnerable to pruning.

Vaults are far more powerful in SourceMod, as they can be used for temporary tables, persistent configuration data, and abstract storage; and since you can choose more efficient storage mechanisms, I imagine they will become much more used as time goes on.

The Command Manager

Thursday, January 27th, 2005

So… I thought I could about write how the Command Manager of SourceMod works. You may ask, “What the hell? Who cares about some uninteresting command manager!?”. Well, I don’t know, and honestly, I don’t care. The devlog is so cool and I haven’t posted anything yet, and the first thing that came to my mind is the command manager, so that’s what I’m going to talk about now.

Ok, command manager…
First off, you can view its source here:
1) Public Interface
2) Class declaration
3) Implementation

Ok, so here is what you can do with it:
1) Register client, server, or mixed commands
2) Unregister registered commands
Hmm, that would be the most important things. You can also retreive information about registered commands, but I don’t want to cover that here.

1) Registering commands
The IVCommandManager interface has three RegisterCmd members: One of them is taking an ordinary function pointer, the second one takes an IScript pointer and a function name, and the third one takes a pointer to a class, which has to inherit from ICommandFunc.
1.1) Client commands
For client commands, the Server Plugin Interface provides the ClientCommand callback, so in theory, you should be able to catch all the client commands from there. However, if the client types a command that is already registered by the GameDLL, ClientCommand never gets called. That’s why the command manager registers a new command callback in such a case, which then calls the Command Manager’s generic command callback. But, as you might know, this completly kills the GameDLL’s command. So, to call the original gamedll function, such hooks first find it in the command list, and store a pointer to it. Then, when the command was not handled by SourceMod or its plugins / modules, the gamedll command gets executed.
1.2) Server commands
There is no generic ServerCommand function, so server commands use the second client command method.

But now, imagine that such a hook gets deleted. When the engine tries to access it, the game will crash. So, we need a method to remove any command from the command list. HL2 stores the commands in a linked list; it only stores the first one, and every command maintains its pointer to the next command. Note that there is no pointer to the previous command. New commands get added at the beginning of the list.

So, we have two cases:
1) The command we want to remove is NOT at the beginning of the list
In this case, we go through the list, until we find the command we want to remove. We set the previous command’s next pointer to our next pointer, and that’s all.
2) The command we want to remove IS at the beginning of the list
In this case, we have to modify the engine’s internal command list pointer. We don’t have direct access to it, so we need to use a workaround. What I have used is a command that is always registered (that’s why I gave it the name “eternal command”). It has a method, called BringToTop, that unregisters itself from the list and reregisters again. So, when you want to remove a command that is at the beginning, you call g_EternalCommand.BringToTop(), and then you do the same as in 1 (except more simple, as you only have to set g_EternalCommand’s next pointer to your next pointer).

Umm, yes, that’s all I wanted to post about; check out the source for more details.

PM

SourceMod web frontend: smWeb

Tuesday, January 25th, 2005

OK, time to write up what some of us have been up to…
Most of you heard about the fact that one of SourceMod features will be a web-interface. If not, go read bail’s interview here. I’ve been working on Amxbans for some time now, and it kinda took me by surprise on how popular web-interfaces are to a lot of admins out there.

When I found out about SourceMod I started to talk to bail, and mentioned that it would be great if SourceMod had a nice flexible, configurable and powerful web interface. We started exchanging ideas and quickly came to the conclusion that not only it would be a nice way to lay a lot of functionality bare to the server-admins but it was actually a good alternative to editing a bunch of ini files. As bail mentioned we will keep different set-ups in mind… just running one gameserver? You’ll get along fine, but the convenience of being able to manage multiple servers from one ‘console’ (while not being a new concept) is hard to find. Sure there are alternatives, but usually they’re limited in functionality.

So what’s it gonna be?!
Let’s start by summing up a few of the features that smWeb will have:

* Admin management.
Here you’ll find the usually functionality you would expect from something called ‘admin management’ :) By assigning admins to groups, and giving admins individual web-access levels you will quickly be able to assign admins to servers and create different ‘functions’ for different people.

* Server management
As you install SourceMod onto your gameserver(s), these servers will ‘register’ themselves with smWeb, enabling you to assign servers to groups, changing settings and more.

* Plugin management
The Plugin manager basically is a part of server management. Per server you will be able to enable/disable plugins. The Plugin manager allows for plugin authors to extend functionality across the web-interface. This means that plugin authors can write addons in order for their plugins to be configured from the web. Plugins usually contain commands, each available command will automatically register with the interface, where it can be assigned to groups, enabling you to control who will be able to use what command on which server. Commands can also individually be enabled/disabled; liked that plugin he wrote, just not that particular command? just disable it in the web-interface.

* Group management.
When you admin multiple servers with multiple mods, you will probably like the idea that groups are what tie admins, servers and commands together. In other words you can have people being admin on one server, but not another, and have various levels of admins across your servers. Admins might be able to perform a specific command on server A, but not server B. Is this all needed? Not for all of you, but we aim to please a wide variety of admins. You can as easily create 1 group, put all your admins and servers in there, assign all available commands to that group and your good to go, while others can create per-mod groups and assign specific servers with specific plugins installed to the various groups. Lots of talk which can be summed up in 1 word: flexible.

That’s it?
You might say that that’s not a lot of functionality if you take a step back from the details. After all you’ll ‘just’ be able to manage admins, plugins and servers… This is why we made smWeb extendable by addons. Addons basically can be a collection of php scripts (and server plugins) which extend the functionality of smWeb.

As we speak, some of the addons below are being worked on:

* smBans.
The successor of amxbans nicely integrated into smWeb by yours truly. Some of you may allready know and use amxbans. Pretty much all the functionality will be in smBans… and then some. I even managed to integrate with amxmodx here and there :). Share them bans!

* Server Config management
The ability to add/edit and store configurations for your server(s), manage mapcycles, add custom content like maps/sounds, edit motd files and more. No need to shell into your server and change those configs, or remote desktop to that server just to tinker with the mapcycle. The aim is that it can all be done from smWeb.

* Bot management
Enables you to easily add, remove and configure bots. simple as that.

Now to be completely clear about this; as with SourceMod itself, the development team will focus on the core. We all rely on you to come up with cool new features and enhancements. We will be providing some tools and examples to get you started. Hopefully plugin authors see the benefits from not only writing usefull and/or fun plugins, but making that easily configurable too. Did I mention that smWeb will be completely customizeable with themes, and localized to boot? Just making sure…

Wrap up.
We’re well underway with smWeb, actually we’re about to start working on interfacing with SourceMod itself, it’ll be nice to flick some switches and actually see the results on your server :)

Oh, should you come accross Geesu, make sure you hug him for putting so many hours into this, he likes being hugged :)

MetaEng – Part V

Tuesday, January 25th, 2005

How did MetaEng begin?

It all started back with AMX Mod X development in March. We had reached a roadblock with modules – AMX Mod’s original implementation was poorly documented and sketchy. Without going into specifics, the system did not work well, and there were possibilities of memory leaks and incompatibility with Listen Servers.

On April 12th, XAD (AMX Mod X developer) posted a thread about changing how modules connected to MetaMod, which led to the new advanced “function request/fakemeta” API found in AMX Mod X. It brought forth an entirely new level of power-scripting in Small for Half-Life 1.

However, XAD’s initial post had another note in it:

Another rearchitecture:
- If we also changed the Small engine to become a “module” instead (with forward functions and exported engine functions) and the core functions implemented as regular C/C++ functions (instead of Small natives directly) plugins could then be coded in C/C++ as modules. This would allow “huge” plugins written in Small beeing coded in C/C++ as modules instead. Some of those great but lag resulting plugins could be much slicker in code and performance…
- We could also create new modules allowing plugins to be written in other languages such as Perl, as the plugin engine is now not part of the AMXX core but as a module.

PM OnoTo’s initial reply to this was:

Other languages like perl and the abstract machine in a module: HL2AMX

(So I guess we can consider this the first post directly related to SourceMod)

A few weeks later I reread XAD’s post and replied:

I think this is an excellent idea [...]

This API would require a lot of thought though, so it is something to keep in mind for HL2 AMXx. Another thing to keep in mind is that making AMXx a plugin engine for random langauges could get very bad, as you then have to support multiple interfaces.

The First Version – ScriptMod

So, we had the idea in mind for what we wanted a long time before HL2 hit the shelves – what we didn’t know was how to do it. It was put on to the backburner until HL2 was glooming over the horizon. When I started, there was no differentiation between SourceMod and MetaEng – it was one tool called ScriptMod, but the code didn’t change when it split into MetaEng/SourceMod. It met a private and quick fate; it tried to do too much with a very complicated design.

In AMX Mod X’s Module API functions were stored as string/function-pointer pairs, so you could find any function pointer as long as you had the name and knew how to call it. ScriptMod attempted to take this a step further and made every single API call stored in a global repository of functions. Each function had to be registered with a calling convention. Then AMX’s amx_Callback() was reconfigured to automatically look up the function and call it dynamically. However, this was instantly realized to be impossible, as native to C calls do not map properly for various reasons (how do you tell a cell from a cell array? a cell with a heap address or a cell with a float address? it’s almost possible with very ugly hacking until you get to varargs – and this is just AMX).

So I introduced connectors, but the system was still grotesquely complicated. If a plugin wanted to use, say, 70 different calls, it would have to request each one, individually, on load. Talk about overhead! And connectors would have to request every function! The system was far too decentralized; everything was a “registered function” – instead of a ScriptMod returning a struct of information, it simply returned names of functions to call that when called, would return information. There was no IScript context – everything was handled through the silly global function map. It was looking really ugly, and it needed something to bring “order to chaos”.

What resulted – MetaEng
Eventually, I realized a C model just wasn’t working, and rewrote ScriptMod as an object oriented system called MetaEng. Instead of “function” registering, classes are now registered. If you have the header and name for a virtual global class in SourceMod, you can request it by name and get a pointer. This hugely reduced the overheard for requesting large groups of functions, greatly improved organization of mapping functions, and soon after, everything began falling into place.

My comment about “multiple languages being bad” still holds even today with SourceMod, and I stand by it. Supporting multiple languages is a terror. That was never the real goal of MetaEng, rather, the goal was to support plugins in C/C++ and a secondary language for lightweight plugins. Who wants to support 30 different and separate scripting languages? So even though SM can support 30 MetaEngines, it would be a bad idea to do this, and SM’s real usefulness is its simplification of managing plugins and modules internally.

In conclusion, there’s still a lot to consider. MetaEng is rapidly maturing, and there are likely to be more changes as time goes on. Right now, we’re taking a path and seeing where it goes. When it’s done, MetaEng will be stripped out and made its own package for embedding in other games/hosts.

This wraps up a rather lengthy exposť on the gory details, history, and theory behind MetaEng. I hope someone finds it interesting or useful.

MetaEng – Part IV

Monday, January 24th, 2005

How does MetaEng handle the ability for scripts of different languages to call a unified set of C/C++ functions, or “natives”?

Quite simply, it doesn’t. In Part Five I will post about the original MetaEng architecture which tried to do this and failed. It’s absurdly difficult. Yet, the problem still needs to be solved, just not by MetaEng. In the SrcMod Framework the job of unifying natives is left up to the host architecture – in our case, SourceMod. Unfortunately, this was one of the most difficult hurdles to jump, and the flaming wreckage that resulted is called “Connectors”. They are what makes SourceMod powerful, overly complicated, and in the end, both a blessing and a curse to development.

Connectors, simply put, are DLLs that hold very primitive wrappers and callback routines. Their sole purpose is to sit in between the MetaEngine’s loading of a plugin and SourceMod’s initial execution of a plugin.

Take this example. We have host application A (SourceMod) with an exported function called “PrintMessage”. The C/C++ MetaEngine obviously has no problem accessing that through the SourceMod Interface Manager… but how does an AMX or JavaScript plugin access A::PrintMessage?

Since we don’t want to hardcode either the MetaEngine or the core, we make a connector DLL that does this:
1. Provide a static wrapper function for accessing A->PrintMessage() natively in the scripting language.
2. Hook the script’s load process.
3. Inject the script with the wrapper information (depends on MetaEngine API).

This means that a connector is a many-to-many relationship, hopping through a MetaEngine. Each Connector connects N functions to M plugins, using a MetaEngine’s API as the gateway (this is trivial since the MetaEngine should be exporting its API through the Interface Manager).

They sound complicated, but more realistically, they are trivial to make and simply rather annoying to pump out. For each MetaEngine (other than C++) you must supply a connector DLL for every set of functions you wish to make available to that language.

This is pretty nice for SourceMod’s core, as then we simply have core_js and core_amx. Unfortunately, it gets fustrating for module writers, who are forced to make a wrapper for every module they write. If I make a module called Sockets, I must also write Sockets_js and Sockets_amx, otherwise, those people will not be able to use my provided functions in their plugins.

The last notable thing about connectors, to ease the pain on both the user and developer, is that they are totally transparent. You simply throw them in the MetaEngine’s directory and it will have one of three load states (Loaded, Not Needed, Load Failed). You don’t have to worry about adding them to configuration files and such.

For examples of Connectors, you can see /cvsroot/sourcemod/ core_js, core_amx, demos/sample_module/samplemodule_amx, samplemodule_js.

Interestingly, there is a project called “SWIG” which could (in theory) be used to easily build Connectors. It will take a set of C++ definitions and built a module for a specific language, automagically! It supports more heavy-duty languages like PHP/Perl, so you can’t use it for AMX/JavaScript. The site’s description is: ” SWIG is an interface compiler that connects programs written in C and C++ with scripting languages”. Hey, doesn’t that sound familiar? It could be modified to simplify the development of connectors, as you’d just input your module header and it would spit out the connector for a given MetaEngine. So, if MetaEng lets you call multiple scripting language functions from C, Swig is the opposite, providing an interface for letting you call C functions from multiple scripting languages. Cool stuff. However, it wasn’t intended for this exact usage, and since a MetaEngine has its own ways of exporting the scripting API, it would have to be changed. In Part V I’ll talk about how MetaEng originally tried to do what SWIG does, except dynamically, not at compile time.

Tune in next time for a conclusion and history of how MetaEng evolved into the current framework for SourceMod, tentatively called “SrcMod”.

Administration Revealed

Sunday, January 23rd, 2005

How does SourceMod handle administration and users?

In Admin-Mod, access checking, user authentication, and user privileges were entirely handled by the DLL. When OLO released AMX Mod, however, core administration/user authentication was done entirely by a plugin. Why? We may never know.

For the purposes of good design, it’s pretty obvious that this is not a good idea. If administration is meant to be a core part of your design, it shouldn’t be a plugin that can be switched off. But then again, what good is a tool that you can’t easily modify?

SourceMod attempts to combine these two areas. First, the actual configuration of SourceMod is stored in a separate DLL, which means it can be swapped for a different solution. For example, the WebGUI of SourceMod will use a different config DLL called webgui_config, and use SQL. Simple_config is the old AMX Mod/X style configuration.

Furthermore, access levels are managed by the config DLL and checked by the Core, not the plugin – so like Admin-Mod, access levels are built in. And unlike Admin-Mod, Plugins no longer hardcode access levels. They’re set in the configuration. That’s big for people who are always complaining about weird access levels – it’s now trivial to change them. It gets better – access levels are abstract. Simple_config uses the “abcde” flag system, but WebGUI_config uses a more advanced system of usergroups and permissions.

There are only three fields absolutely required to be implemented by a Config DLL: Unique usernames, unique steamids, and unique IPs. Don’t confuse usernames with aliases. In Admin-Mod and AMX Mod/X, user accounts were rather sparse, with only three real data types: “auth”, “password”, and “flags”. SourceMod implements true user accounts, where each user must have a unique username and then has sets of properties (such as multiple steamids, aliases, etc), and only two of those properties are requred (steamids, ips). By required, I mean that the Config DLL must support them as options.

Lastly, the actual authentication of a user is very loose. SourceMod doesn’t want to enforce too much, and the base admin plugin only serves to authenticate the server. It doesn’t do anything else, like reading access levels. It simply says “Hey, a user connected. Use the Config API to look them up by SteamID.”

This system is very powerful, allowing developers to implement their own configuration and authentication API for SourceMod — very easily. The idea behind this was letting gaming groups bind servers together and modify administration in virtually any way, without needing to change the plugins at all.

MetaEng – Part III

Sunday, January 23rd, 2005

What’s up with C++ plugins? Are they native code or what, and how does MetaEng work with them? Also, what happens when MetaEng Contexts (IScripts) are destroyed or unloaded? Wouldn’t all of the registered forwards cause a crash?

Well, C++ plugins are easy to explain: they’re just DLLs being wrapped around an IScript implementation. Their bytecode execution and API is handled directly by the operating system. In fact, the code for MetaEng-C is almost trivial, as IScripts map to system calls fairly well (Load==LoadLibrary, GetFunction==GetProcAddress). Unfortunately, it is inherently impossible to do variadic argument pushing in C/C++, so Param_Blk decoding must be done at the assembly level.

The stdarg/va_arg stuff in the standard library deals with retrieving already pushed arguments, but for reasons that probably have to do with weird syntax or non-usefulness, you can’t say “I have a variable number of parameters, and I’d like to just throw them into a function pointer.” I had numerous arguments with ex-developer malex about this, and he came up with several solutions involving pure C (such as passing va_list pointers or doing struct memory hacks), but the end result is that the safest way to do it is to use native machine code.

That is part of the reason that Param_Blk structure is so cryptic – it was designed around the first MetaEngine being able to decode a list of parameters, in memory, using assembly, and construct the required stack frame to call the function. The only difficulty here (which malex would not let me forget) is that each processor and each operating system has a different method of calling. x86 was a simple list of push operations, but the alpha and especially AMD64 ports required an annoying decoding routine.

Nonetheless, they work, and in theory these functions could be used to port languages like “Angel Script” to other CPUs, which (if I recall correcty) is x86 specific because it decodes strings as C calling conventions and uses assembly to bind them to bytecode. Oops, getting off track here…

Now, let’s take a breather from the low level stuff and move back to the overall architecture of MetaEng. Admin-Mod, AMX Mod, and AMX Mod X did not allow you to remove plugins during runtime. Why?

The reason is that it’s a pain. If you unload a plugin, every forward that contains callbacks into the script must be told the callback no longer exists. Otherwise, events will still be called in the deleted script, causing a crash. To get around this is a lot of coding for something that may have little to no use. Yet SourceMod supports it. How does it deal with this, and why bother?

There are two ways to ensure safety when creating MetaEng callbacks. The first is to always use IVPluginManager::RegisterForward(), which creates a global callback registered for every plugin that has it. This will cache your forward inside the plugin manager, so when the plugin manager unloads or loads a plugin, the forward will be automatically updated. It will never be freed by the PluginManager– you’re responsible for this — but you can be assured that if your IForward conforms, unloaded scripts will be filtered out automatically.

However, this is only used for registering global forwards, not specific script callbacks (a single function in a single IScript context). If you’re registering specific callbacks, there’s another process – based on Managers. If you are registering single forwards yourself, you will need to keep track of them. Our example class will be IVMessageManager, which (for the sake of argument) is meant to let plugins register callbacks for client messages. Our manager will be keeping a list of all the assigned forwards.

class IVMessageManager : public CInterface
//...
class CMessageManager : public IVMessageManager, public IUnloadable
{
public:
    void RemoveScript(IScript *); //go through and clean m_Forwards of IScripts
private:
    std::list<iforward *> m_Forwards;
};    
//...
CMessageManager g_MsgMngr;
g_PluginMngr->AddUnloadable(dynamic_cast<IUnloadable *>(&g_MsgMngr));

This will register your manager class as an IUnloadable, which must implement at least one function: RemoveScript(IScript *). RemoveScript() is called every time a script context is about to be destroyed and provides you with a nice callback for making sure you’ve removed all instances of it.

Why bother? Quite simply, so you can unload and reload a plugin for beta testing. Other than that, there aren’t too many reasons you would need this functionality, since a map change resyncs the plugin list in SourceMod.

That’s all for this MetaEng update – next time we’ll cover how Natives work (calling C/C++ host functions from scripts).

Small 2.7.2

Saturday, January 22nd, 2005

The Small Toolkit v2.7.2 has been released.

The new version doesn’t contain any new features and only a few bug-fixes.