User:MascaraSnake/UDMF
As you might have heard already, we've added UDMF support to SRB2. This page gives an overview on what we're doing, why it's useful, and what you can do to participate.
What is UDMF and why is it good?
UDMF stands for Universal Doom Mapping Format. It's a text-based map format developed by the Doom community, the idea being that it supports all the different map features from all the various Doom source ports, and that it can be easily extended to support new features in the future. Before I go into detail about how it works and why we want it, let me motivate why it was created in the first place.
What is the problem with Doom's original map format?
Doom's original map format (which SRB2 uses to this day) is what you call a binary format – it takes the map data as it is stored in the game's memory during runtime and dumps it on the hard drive as raw, non-human-readable binary data. The advantage is that it's really fast and simple for the game to read and write because it doesn't need to do any conversion. But the big downside is that it's not extensible. Let's look at the way the Doom map format handles Things as an example. Each Thing is stored in the map data as a sequence of five 16-bit integers:
- X position
- Y position
- Angle
- Thing type
- Flags
The THINGS lump of a map is simply a bunch of these five-integer sequences all in a row, one for each Thing. It works the same way for sectors, linedefs, sidedefs and vertices. Each map element has a set of properties – which I will call fields from now on – and the map format just dumps these fields into a file as raw data, one after the other.
You will notice that Things have a field for X and Y position, but not for Z position (i.e., height). This is because the original Doom didn't have Thing heights! So what does SRB2 do to support Thing heights? It steals 12 bits from the "Flags" field and shoves the Z position information in there. This is why SRB2 only has four Thing flags – because there are only 4 bits left over for the actual flags. It's also why you can't have Thing heights beyond 4096, because that's the most you can fit in 12 bits. These kinds of hacks are why we need specialized map editors like Zone Builder instead of just using Doom Builder straight out of the box.
Why don't we just add a sixth "Z position" field? Because that would change the map format and therefore break map editor support in a much more drastic way. Telling Zone Builder to interpret the last 12 bits of the "Flags" field as the Thing's height is pretty easy to do. If we load an original Doom map that doesn't use that hack in Zone Builder, or if we load an SRB2 map in Doom Builder, it will show some of the Thing information incorrectly, but it will still basically work. However, if we added a sixth field to the Thing definition, cross-editor support would break completely. Zone Builder expects the THINGS lump to have exactly five 16-bit fields for each Thing. If we add a sixth field and don't tell Zone Builder about it, it will garble the Thing data completely and fail to load the map properly. We could change Zone Builder to recognize that sixth field, but then it wouldn't be able to read all those old maps with the original five-field format, so you'd need to keep an old version of Zone Builder around to load the old maps.
So this is the core of the problem: We could extend the map format to add new fields if we wanted to do, but every time we do so, we completely break support for older maps and lose any semblance of compatibility with the existing map editors for Doom. Some years ago, the creators for all the different Doom source ports were facing a similar problem: Sure, they could add new map functionality by adding new fields and changing the map format, but then you'd need a specialized map editor just for that one source port, which wouldn't be compatible with all the other source ports. And so they decided to come up with a new map format that they could all use and that didn't have this problem.
How does UDMF work?
UDMF is entirely text-based. All the map data (except for the stuff created by the nodebuilder) is stored in a single text lump called TEXTMAP. Here's what the entry for a Thing looks like:
thing // 0
{
x = 0.000;
y = -768.000;
angle = 90;
type = 1;
flags = 3;
}
As you can see, each entry is a simple list of the fields and their values. If a field is unused, you can simply leave it out and it will fall back on a default value. Take this for example:
thing // 0
{
x = 0.000;
y = -768.000;
angle = 90;
type = 1;
}
Here the flags value is missing, so the "Flags" field will default to 0.
Sectors, vertices, linedefs and sidedefs are also defined in text blocks just like this one. The TEXTMAP lump, then, is nothing more than a list of these blocks, one for each map element.
Why is this better?
The key difference to the binary format is that each field is labeled with its name. This allows us to leave out fields and have them fall back on a default value. This makes it possible to add new fields and still have backwards compatibility. Let's say we want to add a Z position field for Things. The Thing entry would then look something like this:
thing // 0
{
x = 0.000;
y = -768.000;
z = 32.000;
angle = 90;
type = 1;
flags = 3;
}
If the z field is missing, it simply defaults to 0. This future-proofs the map format: If we decide to add a new field in a future version, maps from earlier versions will still load fine both in the game and the map editor – the missing field will simply be set to its default value. What makes this even better is that GZDoom Builder (which Zone Builder is based on) has a "Custom" tab in its Thing/linedef/etc window that allows you to add and edit any field you want. This means that if we ever add a new field, ZB won't need to be updated to allow you to use it (although we will probably update it eventually to integrate it more nicely into the interface).
In the context of the Doom community, UDMF allows source ports to share a common map format even though they don't all support the same fields. The map loading code simply ignores any field that it doesn't recognize, so you can add a new field that other source ports don't support without breaking anything. In the SRB2 context, this means that source code mods and spinoffs like SRB2Kart will be able to extend SRB2's UDMF implementation as they see fit, without having to worry about downwards compatibility to SRB2 and map editor support. It may even be possible to allow Lua to define new map fields.
Why should I care?
If for some reason you're not yet convinced that UDMF is the best thing since sliced bread, here's a few examples of the things UDMF allows us to do. These come in two varieties: getting rid of old hacks that made life unnecessarily hard for mappers, and adding entirely new features that weren't possible before.
Linedef arguments
Because the binary map format has a limited amount of fields, we sometimes have to get creative in order to stuff extra information into the map. This is especially common with linedef specials. A lot of our linedef types are parametrized – their behavior depends on various settings that the mapper can control. Because we don't have any dedicated fields where we can store these parameters, we typically shove them into unrelated fields, like the texture offsets of the control linedef, or the floor height of the control sector. Sometimes even the length of the control linedef is used as a parameter, which leads to control sectors with bizarre contorted shapes. All of this can make it pretty tedious to set up special effects, because you have to look up which field controls which setting on the Wiki, and if it's something like linedef length, changing the setting requires more work than just swapping out one number for another.
In UDMF, none of this will be necessary because linedefs have a number of dedicated "argument" fields, arg0 through arg9. These fields exist solely for the purpose of supplying arguments to linedef specials so you can customize their behavior. As an example, let's look at linedef type 60, Activate Moving Platform (Adjustable Speed). Right now, you need the following to set it up:
- Tag: Tag of the FOF control sector
- Linedef length: Speed
- Front X texture offset: Time delay when the movement changes direction
- Front Y texture offset: Time delay before the platform starts moving
- Not Climbable flag: Platform begins by moving upward
In UDMF, it works like this instead:
- arg0: Tag of the FOF control sector
- arg1: Speed
- arg2: Time delay before the platform starts moving
- arg3: Time delay when the movement changes direction
- arg4: 1 if platform begins by moving upward, 0 if it begins by moving downward
In the map editor, the labels of the arg fields can be customized for each effect, so the arg1 field is labeled as "Speed:", for example. Fields where you need to choose from different options, like arg4 in this example, can be displayed as drop-down lists. Fields that are a collection of flags that are either toggled on or off can be displayed as checkboxes.
The result of this is that you can set up most linedef effects by simply typing the parameters you want into the arg fields. There is no more need for shoving parameters into unrelated fields or encoding them in the map geometry. And you won't have to look up which setting goes where because the map editor will tell you.
Native flat alignment
Another consequence of not being able to add new fields is that some things that really should be sector properties are instead hacked in via linedef specials. The most obvious example is flat alignment. In Doom's map format, sidedefs have fields for texture offsets, so you can align wall textures just by tweaking those numbers. But sectors don't have anything like that, so we hacked it in via linedef type 7, Sector Flat Alignment. This is pretty tedious to set up because you have to tweak settings on the control sector until it looks right in the target sector. In UDMF, there is no need for this mess: Sectors simply have fields that allow you to set the floor and ceiling offsets directly, and you can adjust them in Visual Mode just like wall textures.
Native vertex slopes
Vertex-based slopes are somewhat tedious to set up right now because you have to place three slope vertex Things and then link them together via a linedef special. In UDMF, vertices can have a Z position, just like Things. This makes it trivial to make sloped sectors – you just make a triangular sector and set the vertex heights to what you want them to be, and the surface will interpolate between them automatically.
Multitagging
UDMF has support for applying multiple tags to one sector/linedef/Thing. This is extremely useful when you have two special effects (such as FOFs) whose tagged areas overlap partially but not completely. In the binary map format, you have to give the overlapping area a new tag and then make duplicates of the two special effects that use that tag instead. With multitagging, you just give the overlapping area two tags – one for each effect.
What does this have to do with me?
We need you as testers! UDMF has given us an opportunity to redesign a lot of our mapping setup. All the existing linedef, sector and Thing types have been given a long, hard look to figure out the best way to set them up under the new system. Some have become obsolete (see the flat alignment example above). Many have become more comfortable to set up. A lot of similar linedef types have been merged now that we have more space for different options in the linedef arguments (this is the case for a lot of FOF types especially). The goal has been to make things as convenient as possible for mappers. This requires feedback from as many mappers as possible, so we invite you to participate.
Discussion of UDMF is happening in the #udmf channel on the Discord server. Everyone is welcome to participate!
Where can I get the newest SRB2 build with UDMF support?
2.2.11 has "full" UDMF support in the sense that the entire feature set of the binary format has been replicated in UDMF. This doesn't mean UDMF development is over now – since UDMF allows us to add new mapping features more easily, we will certainly do so in the future. But the UDMF implementation included in 2.2.11 is complete enough that you can start making "real" maps in it.
How do I make an UDMF map?
We are currently developing Ultimate Zone Builder (UZB), a map editor specifically for SRB2 UDMF mapping. It's based on the newest editor in the Doom Builder family, Ultimate Doom Builder (UDB), much like Zone Builder was based on GZDB. UZB isn't ready for an official release yet, but you can find beta releases here. If you encounter bugs, please report them here.
Unlike the binary format, UDMF doesn't require a lot of SRB2-specific map editor support. However, there are still a couple of reasons why we need our own editor instead of sticking with UDB:
- UDB's Visual Mode doesn't properly display a lot of SRB2's special features, such as our version of FOFs, skewed midtextures and so forth. We're currently adding support for these to UZB.
- The mapping interface is tailored to GZDoom's feature set. In the sector/linedef/Thing edit windows, a lot of space is taken up by fields that have no meaning in SRB2. All the fields that are exclusive to SRB2 are clumped together in the "Custom" tab, which isn't as convenient to use. UZB redesigns these windows to match SRB2's feature set.
- Over the years we've added some new features to ZB, such as custom Thing support, NiGHTS path rendering, PolyObject previews, etc. We will port these over to UZB.
Note that UZB will not support SRB2's binary map format. This means that the original Zone Builder will be the tool of choice for binary mapping, and UZB for UDMF mapping. We know that it would be nicer to have one program for both, but we decided against merging them in order to keep the UZB codebase as close to the UDB codebase as possible. The reason why we've had a hard time keeping the original ZB up to date with the development of GZDB/UDB over the years is that there are too many differences in the codebase. This time around, we want to keep the differences smaller and localized, so that we can merge in new UDB features and bugfixes as they are developed. This means that we have to leave binary map support behind, but we think it will be worth it. In the long term we want to make UDMF the default mapping format anyway, and the original ZB will still be around to edit those old binary maps.
Is there a mapping tutorial for UDMF?
Not yet! If you're familiar with mapping in the binary format, then hopefully UDMF mapping won't need a ton of explanation. For the more complicated special effects, you may have to consult the documentation. You can also check out SeventhSentinel's recent livestream.
Should I start mapping in UDMF now, or should I wait until UDMF is more stable?
If you only want to mess around and see what UDMF has to offer, then you can start right away! If you're planning to start a more serious project in UDMF, that's possible too, but you need to be aware of a couple of things:
- UDMF support is still experimental, and we might change how some things are set up based on the feedback we get. If we do that, you may need to fix up your existing UDMF maps. We hope that this won't happen a lot, but we can't make any guarantees yet. If you're not prepared to keep your map up to date with the current state of the UDMF implementation, you might want to stick to binary for now.
- UZB is still in the beta phase, so it's incomplete and might be buggy. If you stumble upon a bug or a missing feature that's stopping you from making progress with your map, don't hesitate to make an issue for it in the public issue tracker.
Can I convert existing binary maps to UDMF?
Yes! SRB2 comes with a built-in converter. It works like this:
- Launch SRB2 with the command line parameter
-writetextmap. - Warp to the map you want to convert. Once the map has been loaded, SRB2 will write a
TEXTMAPfile containing the converted map into your SRB2 folder. - Open SLADE, create a new .wad file and import the
TEXTMAPlump from your SRB2 folder. - Click on New Entry to create an empty lump and call it
MAPxx(where xx is the extended map number). - Create another empty lump and call it
ENDMAP. - Put the lumps in this order:
MAPxxfirst, thenTEXTMAP, andENDMAPlast. - Save your .wad file and open it with UDB/UZB.
- Build the nodes. If you don't do this, the map will crash on startup. In order to invoke the nodebuilder, you need to modify the map. You can do this by drawing a new sector and deleting it, for example. Then save the map to build the nodes.
Note that a few more of the more esoteric features in the binary format cannot be converted automatically. Most maps don't use these, but if yours does, you will have to fix it manually. If the game encounters an issue like this during the conversion, it will print a warning in the console. You can then open the log file from the session to aid you with the manual conversion.
(Thanks to community member klasky for writing up the original version of this guide!)
You can report bugs in our public GitLab repository. Please take note of the following when reporting a bug:
- Make sure that the SRB2 build you're using is as recent as possible. If your build is out of date and you have access to a newer one, please check if the bug still exists in the newer one.
- Describe the bug in as much detail as you can provide. The more detail you can provide, the easier it will be for us to figure out what's going on.
- If the bug happens in a custom map, please upload the map.
- Add the UDMF label to your issue.
I've tried out UDMF mapping and I wish XYZ was set up differently. Where can I go to yell at you?
In the #udmf channel. Don't hesitate to ping me. If you're worried your feedback might get lost in the shuffle, you can also open an issue in the GitLab repository. I can't guarantee that I will be able to fulfill your request – there are still some limitations that we need to design around. But I can certainly take it into account. The more feedback we get, the better the mapping experience will turn out for everyone.