Archive for February, 2007

Command API, Part 1 of 3

Wednesday, February 28th, 2007

Today I’d like to highlight the latest wiki article: Scripting Commands in SourceMod. SourceMod’s Command API is slightly different than AMX Mod X’s; this is partly for optimization, and partly because of how Half-Life 2 works.

The first difference we’ll discuss is that there is no command filtering. As noted in the article, this notation is no longer possible:

register_clcmd("say /ff", "Command_FF");

Why didn’t we accommodate this? Two reasons, and they’re both related to one fact: The only usage of this syntax seemed to be for “say” and “say_team” and no other commands. I have never seen a plugin use this syntax for anything else, although I’m sure it has been done.

So, the first reason is that we’d be adding a rather complicated functionality with very limited use. The second reason is that since it’s limited, every command that went through IVEngineServer::ClientCommand would essentially be compared against a list of say hooks, even though most commands aren’t actually “say.” Of course, we could find ways to optimize around this; for example, per-command filters or filters strictly for say — but it’s a lot of work for a feature that’s entirely a convenience.

There’s a third hidden reason which will be explained in part 3 – it’s not actually technically feasible to implement it, given how Source’s command system works.

Next article we’ll see how SourceMod handles admin commands.

Handle System API, Part 3 of 3

Monday, February 26th, 2007

Last article, we saw that Handles could be blocked from CloseHandle on a per-type basis. However, there are situations when you only want some Handles of a type to be blocked. This is addressed in today’s newest addition to the Handle System API article.

You may note that the article has been altered; there are now per-Handle permissions. Originally, there was no way to do this. In fact, this feature was added only a day ago. However, there are pieces of SourceMod already implemented which required the feature, so how was it done in the past?

We used the fact that child types do not inherit permissions from their parent. Imagine two types using BitBuffers as an example: a “read only BitBuffer” and a “writable BitBuffer.” First, you create a “writable BitBuffer” type. Then, you create a second, read-only type with the writable type as its parent. Because of the parent/child relationship, the same set of functions would work on both types; but CloseHandle() would only work on the writable version. This works because you can specify new security rules for child types.

Why did we opt for such a complex solution at first, rather than just implementing per-Handle permissions from the start? There’s no real excuse, but this was the mindset:

  • We did not actually foresee this being useful enough at the time. Each Handle uses about 40 bytes in memory, and the block of pre-allocated Handles is 16K entries long. That’s roughly a 660KB block of consecutive memory, and per-Handle permissions would increase that to 60 bytes per Handle (~990KB), so we opted to not do it.
  • The permissions system was already completed and in use by the time we realized the design flaw, and thus new functions would have to be added for backwards compatibility.

One thing to note about the prior workaround is that the order of inheritance does not matter. The parent OR the child type can be the locked type. The Handle System doesn’t care, as the types internally do not actually share anything. However, you must remember to use the parent type for generic accesses. If you use the child type, and a Handle of the parent type is passed in, the type check routine will fail. This is the same as in C++: You can’t necessarily cast a base type to a derived type safely.

Internally, the implementation of the new per-Handle security is optimized for backwards compatibility. The “quick” way to implement this, in pseudo code, would have been:

handle->access = access ? access : handle->type->access

This would make the Handle permissions equal to either user-supplied, or the type’s default, access rules. But the implementation is slightly different. CreateHandle() is now a wrapper around CreateHandleEx() (which itself wraps an internal function called CreateHandleInt(), which wraps MakePrimHandle!), and it passes NULL as the access structure — meaning that CreateHandle is general usage and will be called more often. So to prevent copying the entire permissions struct on every “normal” Handle create, it looks a bit different.

IF access
   handle->special_rules = true
   handle->access = access
   handle->special_rules = false

Internally, there is a simple CheckAccess() function which can decode this:

CheckAccess(handle, right, identity)
   access = handle->special_rules ? handle->access : handle->type->access
   if (access[right] & HANDLE_RESTRICT_IDENTITY
       && handle->identity != identity)
       return false
   return true

(Of course, this isn’t really an “optimization,” but it’s in case the HandleAccess struct grows more complex over time.)

That about sums it up for Handle Security. You, as the rest of the SourceMod developers are, should be thoroughly sick of it by now.

Handle System API, Part 2 of 3

Friday, February 23rd, 2007

The newest addition to the Handle API article shows an example of modifying the security permissions of a type.

This is in fact more useful than it might seem at first, despite the bad example. There are a few areas in SourceMod where CloseHandle is not supported from plugins on certain Handle types:

  • ConVars: Can’t be closed or cloned. They are global and not owned by any plugin, and thus already totally shareable, so there’s no need to allow for deleting or cloning. They are only freed when SourceMod is unloaded.
  • SMC Parsers: Can’t be cloned. They contain function pointers to a specific plugin. If that plugin unloads, the Handle’s internal contents are useless. Thus, they cannot be cloned into another plugin.
  • Plugins: These uniquely identify a plugin. If the plugin is unloaded, the Handle is destroyed, so naturally they cannot be cloned. They cannot be deleted either; it would be a disaster if a plugin accidentally called CloseHandle() on itself!
  • BitBuffers/Events: Closeable if manually created, but not if passed through a forward.

All of these, except the last, are implemented in a manner similar to today’s example. What’s special about the last? BitBuffers and Events require per-Handle permissions, which do not yet exist. Getting around this will be explained in Monday’s article.

Note: Unfortunately, this article was originally much longer, but it somehow got corrupted. Monday’s Wiki entry should be more detailed.

Handle System API, Part 1 of 3

Wednesday, February 21st, 2007

There’s a new wiki article up that goes over the Handle System API (from a C++ perspective) in somewhat scary detail. It also provides a complete introductory example. Visit it here:

The initial example is the typical usage of Handles: creating a temporary structure that can be safely shared in between plugins. When the Handle is destroyed (via CloseHandle once plus again for each time CloneHandle was called), the temporary structure is freed.

However, there are more advanced usages that take advantage of the security features. More examples will be posted about this in Friday’s article.

Choosing a Bug Tracker

Tuesday, February 20th, 2007

Unfortunately, I’ve been cramped for time lately, but over the weekend we finally chose a bug reporting system to use. This was a fairly lengthy task, and I reviewed about 8 systems before making a choice. We discounted anything commercial, as usually these don’t have good source-code editing flexibility, or they are licensed per-user, which is bad for community projects.

A quick run-down on what I encountered:

Bugzilla: Overall, a very comprehensive package. The source code is modular and very readable (especially for Perl). Although the default theme is horrible, it is easily editable since it is templated. Unfortunately, its permissions system is extremely basic (despite Mozilla claiming it’s “comprehensive”). Access rights to a task are either all or none, which means we couldn’t let users add comments to tasks without also letting them change priorities/assignees/et cetera. Although Bugzilla’s documentation had an article about altering the permissions source code, it was certainly not a clear-cut job.

eTraxis: A relatively new project with a slick interface. Unfortunately, the interface is also mind-bogglingly confusing, and the documentation is extremely light. We couldn’t figure out how to add a task within a few minutes of playing around and dropped it.

Eventum: By the MySQL team, this looked fairly promising. It has a huge feature set, a detailed interface, and integrates with SVN. However, it seemed to have no option for anonymous viewing, so we dropped it.

Mantis BT: Although 3 years ago it might have been popular to inline HTML into PHP scripts, these days it’s unacceptable, especially with the advent of tools like Smarty. We dropped this one because maintaining it looked like it would be an uphill battle. To its credit, it tries to condense HTML into separate PHP files, but the necessary abstraction isn’t there.

Scarab: This tool did not appear to have any concept of projects. Even stranger, it claimed that this was a “feature,” and used the word “Modules” instead. This silly design decision was bad enough, but we couldn’t see that Scarab had any sort of hierarchy or sub-categorization beyond “Modules.” Our projects tend to have many nested pieces.

Trac: Trac looks excellent for single-project setups. However, we were looking for an integrated environment. Trac doesn’t seem to allow for multiple projects under one hood. On top of that, it comes with a lot of extra baggage that we don’t need (though this can probably be disabled).

Eventually, we ended up choosing Flyspray. The PHP code is fairly easy to read. It is templated properly, thus abstracting design from code. It allows for projects, categories within projects, and categories within categories. It has comprehensive task permissions for both global groups and per-project groups. Users can be a member of a global group and one group in each project. All in all, this perfectly fit our setup, so after doing a quick vBulletin integration, we launched.

If we had more time and more developers, we probably would have chosen Bugzilla or Eventum and done modifications to implement our features. However, Flyspray had exactly what we needed at the right time.

My Beef with Valve Ads

Wednesday, February 14th, 2007

I’m sure this has been expounded on time and time again, and that other people have re-iterated some or all of these points. However, I’d like to get them out of my system regardless.

I speak of everyone on the AlliedModders team when I say that we do not like the advertisements Valve is adding to Counter-Strike 1.6.

1. They’re intrusive and not realistic. In-game ads in other games have been known to fit into scenery well, or be more subliminal than annoying. This isn’t the case with Valve’s ads. For example, the massive advertisement in the de_dust2 CT spawn, which you’re forced to glare at when your POV is locked during freezetime.

The spectator advertisements don’t really matter, but the scoreboard one is ridiculous. The majority of screenshots for Counter-Strike are of the scoreboard, and the scoreboard is a very frequently used VGUI panel. Every scoreboard screenshot will have ads in it unless otherwise altered.

2. Users did not ask for ads. Many people bought Counter-Strike retail as opposed to downloading it for free. Now, they’ve got ads in a game they paid for, and they’re not getting anything value-added back.

3. Server owners have no control or compensation. Valve has continually told server owners they can’t put ads in-game. Now Valve has turned around, not only forced server operators to have their clients see ads, but the ads are not targetable nor do server owners receive any compensation for providing the arena for displaying said ads.

4. The community feels “betrayed.” Valve is usually very slow with updates. Features are randomly denied, products are delayed over and over (for example, EP2 and TF2), and update (and release!) quality itself is usually extremely poor. The notion here is that Valve doesn’t have time to address critical issues or feature requests, and are in fact bogged down with all these cool projects. However, they do in fact have time to negotiate potentially massively profitable, intrusive, in-game advertising that offers nothing back to the user; for a game that they no longer support or update, and which has been unchanged for years. For a game that was originally very much a community game (not owned by Valve), this is awful.

5. Admission of Source’s slow adoption. By putting advertisements into Counter-Strike 1.6, Valve is, at the very least, admitting that they are not concerned with Counter-Strike 1.6 anymore. As a corollary, it is a subtle admission that Source is not as popular as 1.6, despite how much they’ve pushed it. You can attach all sorts of conspiracy theories to this. Valve might be trying to milk CS 1.6 to kill it off, or they could be planning to put advertisements into all free mods (maybe even 3rd-party mods, if they found a way). Regardless, the motivation is highly suspect, when there are other ways to benefit the community. And with CS 1.6 not receiving updates, putting ads in Counter-Strike:Source would be a much less hated tactic than modifying an old, stable game.

That’s about all I can think of. As a conclusion, I think that the majority of Counter-Strike players will simply accept the change as they have with the other unliked or really bad additions (DWP, Tactical Shield, et cetera). It’s Valve’s product, and they can do what they want. However, this is one of the most disappointing lows Valve has reached over their short history. Their interests can definitively be pointed as not being with the community anymore, and there’s no telling what they’ll do next.

As a Counter-Strike enthusiast and developer since the beta periods, it’s disappointing to see how the game has been modified under Valve, and how Valve’s treatment of the community and its concerns has changed.

Nightly Builds

Monday, February 12th, 2007

SourceMod has a nightly build system that will be used for both AMX Mod X and Metamod:Source nightly builds in the coming weeks. This build system took a decent bit of effort to set up, so I felt I’d explain it for people running into similar problems.

First, there’s the problem of hardware. To conserve hardware and processing power, all our builds are done on one machine. We don’t use cross compiling, as compilers usually don’t jump platforms well (or, in C++’s case, you lose ABI compatibility). We use VMware Server to host each platform.

The host operating system in our case is Linux; the guest is Windows XP. The “order” is irrelevant, but we ran into a few roadblocks that took a while to resolve. Next time I would probably run Windows as the host operating system instead — Linux doesn’t require a GUI to do builds, and VMware seems to work better on Windows.

Second, there’s two problems of automation. You need a good cross-platform way to build and package, and you need a good cross-platform way to automate the actual nightly build process itself. For our package building tools, we use C# (.NET 1.1) for both AMX Mod X and SourceMod. For nightly build scripts, we use Perl.

The package build tool has functions for copying files and folders, compressing folders, building libraries, and compiling plugins. For Linux, it runs ‘make‘ and ‘make clean‘ on each target library. For Windows, it runs ‘ /useenv /Rebuild Release project.vcproj‘. All of these commands work from a command line; no IDE involved.

The build tool also looks for any file called ‘svn_version.h‘. If found, it is edited to contain the current build number of that package. All subversion information is detected using svnversion. Once it’s done, the output folder is compressed into the final .tar.gz or .zip file.

The nightly build tool is written in Perl. Its job is to run the main build tool every night at 12AM CST. It does the following steps:

  • Checks if there have been any changes since the previous night (if none, the script dies).
  • Updates the local copy of the source tree. This is done using SSH Keys. On Windows, we use svn‘s --config-dir option and set the ssh config file option to point to TortoisePlink.exe, which has to be in the PATH environment variable.
  • Builds the C# packaging tool using (Windows) or mcs (Linux, Mono c sharp compiler)
  • For Windows, the INCLUDE, PATH, and LIB environment variables are preset to all of SourceMod’s local folders. This is so we don’t have to use its built-in path features, which are inherited from the IDE, rather clunky, and don’t play well in a multi-project environment. These also have to contain Visual Studio’s own include, binary, and library folders respectively.
  • The build tool is executed. On Linux, this requires mono for running .NET applications.
  • If the build tool fails, an e-mail is sent to all developers with a compilation log and revision #.
  • Each package is transferred to a final public location via FTP.

Finally, these scripts are slapped into crontab for Linux and Scheduled Tasks for Windows. Both the host and guest operating systems stay powered on 24/7.

As mentioned before, we will be using this system soon to have nightly builds for both AMX Mod X and Metamod:Source. However, since both of those projects are open source, it will be solely for user convenience.

SourcePawn File Format

Friday, February 9th, 2007

Since we just got finished discussing opcodes, it’s worth talking about the new SourcePawn File Format. The biggest question is: why bother changing it? The compiler already does a fine job outputting files.

The compiler’s native format has a few problems that we wanted to eliminate.

  • File sections are hardcoded. You cannot add or remove them easily.
  • The built-in compression format (“compact encoding”) is not as efficient as something like gz compression.
  • The debug format is unceremoniously tacked onto the end of the file. You get to it by jumping past the DAT section and seeing if there’s anything there. We figured this was kind of hacky, and exposed the lack of a good section layout.

Ultimately, it comes down to a more subtle problem. Pawn does not separate the concepts of a plugin and a context. A plugin is a file and all of its associated information. A context is an in-memory, “boxed” executable, with all of its runtime variables. Furthermore, since Pawn is tied to an architecture’s bit width, the file is basically a memory dump. Depending on the architecture, you can read the file into memory and relocate almost nothing at all; the structures can be used as-is. This is very convenient, but isn’t cross platform compatible.

SourcePawn has a few major changes to the file format. First, it is sectioned. The header contains a list of each section and its name. Each section contains private information and data. Second, the debug information is contained in individual sections, rather than an entirely new format tacked onto the end. Like the original format, strings reside in a name table. Third, the format allows for global compression of all sections. There are currently two compression methods: no compression, and gz.

The big difference is that you can’t just malloc() a chunk of memory, load the file into it, and begin executing plugin code. There are ultimately three separate data structures. The file format structures, the plugin structures, and the context structures.

The file format structures are used to write and read from the binary files. These are documented in sp_file_headers.h. The format is read back into an sp_plugin_t structure, documented in sp_vm_types.h. This holds all of the compile-time information about a plugin, such as its p-code, initial memory data and size, debug information, public, native, pubvar information, et cetera. Why bother having separate structures? Relocation. Since the file structures often have relative indexes (for example, indexes into the nametable), part of the job of loading a file is to convert all that local file information into nice pointers. This makes the API a lot easier for users, which wasn’t the case for the AMX format.

Once a file is loaded and converted to the sp_plugin_t structure, it is converted to an sp_context_t by the JIT. The JIT reads in an sp_plugin_t and basically dumps out a new version with run-time information attached. It also does absolute relocation; since all p-code addresses change when you go from p-code to native code, all of the offsets for the debug section and such must be updated. Thus, an sp_context_t contains all of the run-time data that an sp_plugin_t should not.

One of the easiest benefits of this method is that it is very easy to implement context swapping. In SourceMod, a plugin can be toggled in debug mode on or off without changing maps. Why? All it has to do is re-invoke the JIT, get a new context, and copy the old context’s memory usage. In theory, you could also set up contexts to share each other’s memory and whatnot. Contrast this to the AMX method, where the file has to be completely reloaded (either from file or a cache); the plugin and context are the same, and the p-code is destroyed when the JIT runs over it.

On a final note, it’s worth mentioning that AMX Mod X’s plugin format is not a true compiler-generated format; it’s a wrapper. It invokes two separate instances of the compiler (32bit and 64bit cell sizes), and gz compresses each one. Then it throws both into one file and writes a header about where to find each copy.