NAME

nh4ct - NetHack 4 tileset formats

DESCRIPTION

The computer game NetHack (nethack(6)) has a screen-oriented view; each square of the map is drawn onscreen, as a formatted character or as an image. NetHack 4 (nethack4(6)) is a modernized version of NetHack, and comes with a modernized rendering system. This file describes the formats that NetHack 4 uses to specify the "tiles" used for this rendering.

HISTORY

Originally, tiles were introduced in the NetHack 3 series. This system stored the files in a text format; each tile was specified using an image, a name and a number. There were three files (one for objects, one for monsters, and one for other tiles); a particular tiles image was identified by its position in one of these files (the name was checked to ensure it was the expected name, and the number was ignored). The image format was very simple; each file had a palette of up to 62 colors, which were mapped to letters and digits, then those letters and digits were just formed into a rectangular image directly.

Slash'EM brought some improvements to this system. The image format was generalized to allow up to 4096 colors (via two-character keys), although Slash'EM refused to use more than 256 colors at once due to limitations of typical video cards from that era. It also supported transparency via a color key (71, 108, 108). Another major change was that it supported the merging together of multiple files to produce one tileset; the merge was by name, which was a little confusing/error-prone because multiple tiles could have the same name. The workaround for this was to match both the identified and unshuffled unidentified appearance of a tile, which makes no sense conceptually (tiles don't logically have an identified appearance; even if you know what an orange potion does, it should still look like an orange potion). Even then, there was an occasional fallback to file position, as in the case of the many tiles called "wall".

Early attempts at a tiles system in NetHack 4 were based on the Slash'EM system, but generalized still further. One innovation was to give every single tile a unique name, meaning that tile merging was a lot less error-prone, and meaning that tiles could be arbitrarily re-ordered in the source files without issue. (This involved some changes to the game engine; probably the only user-visible one was that tools representing disarmed traps gained unidentified appearances, to distinguish them from the traps they became.) These source files (that identified tiles by name) were merged into intermediate files that identified tiles by file position, and then into images the same way as in Slash'EM. Each tile name was also associated with a number; these numbers were in numerical sequence, so that they could be used as file positions directly.

This system eventually turned out to be unsatisfactory, however. There were two main issues. One is that even 4096 colors is too small a color depth for some tilesets, in 2014. (In fact, deeper color depths were common even in the NetHack 3 series, via the unofficially supported but common method of just generating an output tile image directly in an image editor. This is the tiles equivalent of writing a program in machine code, and unsurprisingly, it leads to nonportable tilesets; the resulting tilesets could typically only be used with the precompiled Windows version of NetHack, needing manual adjustment whenever even minor changes were made to the game.) The other was that the set of tiles that exist wanted to vary from tileset to tileset; in particular, aesthetic variations such as "substitution tiles" (tiles that look different in different branches) might or might not exist in any given tileset.

One other issue was that the code for rendering ASCII and Unicode had diverged dangerously from the code for rendering tiles, meaning two different sets of code needed to be supported. The method of rendering substitutions (such as dark rooms and lit corridors), in particular, was fragile and buggy. Thus, it seemed sensible to allow the tile selection code to also be used to calculate what to draw in ASCII, with the only difference between the two being rendering.

TYPES OF TILE FORMAT

At its heart, a NetHack 4.3 tileset consists of a set of images or cchars (the ASCII and Unicode equivalent of a tiles image), each of which is associated with a name. However, there are several different formats for expressing these, and converting between them is the main purpose of the tile utilities.

A tile image can be expressed in two different ways. One is to specify the image inline as text, in the same way as the NetHack 3 series and Slash'EM. Another is to specify the image by reference, giving the position of the tile within some external image (this is a 32-bit number, with 0 being the top-left tile, 1 being the tile to its right, and so on). (Nothing about this format conceptually limits the format the external image can be in; however, any tile utilities that need to be able to manipulate that image require it to be in PNG format.) A cchar can also be specified in two different ways: as a textual description such as underlined brightmagenta 'h', or as a 32-bit number which forms a packed representation of the cchar. The textual and binary representations of cchars are interchangeable.

Tile names can also be expressed in two different ways; as text (such as walls 0 or sub male sub gnome wizard), or as a 96-bit number that forms a packed representation of the text (32 bits for the base tile, such as 'wizard', and 64 bits for substitutions; substitutions take up more space because they form a bitfield rather than an enum). The textual and binary representations of tile names are also interchangeable.

A "compiled tileset", which is used by the program at runtime for actual tiles rendering, consists of a list of tile name / rendering pairs (totalling 128 bits each), with the tile names being expressed as numbers, and the images also being expressed as numbers. A cchar-based tileset will use cchar numbers; an image-based tileset will reference a sub-image of an image. (No tileset can contain both cchars and images, incidentally; the main reason is that it's unclear how the result could be rendered.) An image-based compiled tileset thus needs to be shipped along with an image; a cchar-based compiled tileset is usable standalone. They also have a header, giving metadata about the tileset (name, tilesize, image/cchar). Compiled tilesets are always stored sorted in numerical order; this makes it more efficient to find the relevant tile from them. A cchar-based tileset is identified via having the tile width and height set to 0.

The source formats for tiles is less fixed. All that is required is that it contains names (in some accepted format), and images or cchars (in some accepted format, including references); source formats are text-based, and thus if using numerical formats, spell the number out as digits rather than representing it as binary directly. The following possibilities are available:

For names:

For images:

The source format for an image-based tileset can also optionally contain a palette (the binary format has a palette if and only if the associated image has a palette). This can contain 1, 3, or 4 channels (separate channels, red/green/blue, or red/green/blue/alpha), and is necessary if any tile images are in text format and optional otherwise.

Also noteworthy is the phrase "map file". This is half of a source-format tileset: it specifies the tile names, but uses file position to determine images. This forms a full tileset when combined with a NetHack 3 series or Slash'EM tileset (although it will probably need to be merged with a few extra tile images to provide tiles that weren't present in the source image). Map files can also be used in reverse, to specify the order that tiles should appear in in a produced tileset image; this allows NetHack 4 tilesets to be "backported" to earlier versions.

Finally, because it is very common to use an external image together with a tileset that references it, the combination can be glued together into one file. This is a PNG file with up to two extra private chunks that contain the tileset: nhTb for a binary tileset and nhTs for a source-form tileset; any references to an image in the tilesets reference the file that it's embedded in. (If both are present, they should both have the same meaning; this means that nhTb can be used by the rendering code, and nhTs to automatically compensate for changes to the name to number mapping.) These chunks must appear before the image data.

TILE NAMES

As mentioned above, tile names can be represented either as text or numerically. The textual names are "stable"; in fact, many of them are chosen for compatibility with the NetHack 3 series or with Slash'EM. The numerical names are subject to change without warning, which means that every tileset should be distributed with the textual names available somehow (which is one of the big reasons why an image can have both an embedded binary tileset and an embedded source-form tileset, as the binary form is needed for actual display). One thing that is consistent about numerical names, however, is that the top 32 bits are used for the base tile, and the bottom 64 bits for substitutions (numerical tile names are 96 bits long). When multiple substitutions are available (e.g. sub orc and sub female for a female orc wizard), the substitution with a higher numerical value will be the one that's favoured.

For textual names, substitutions are specified with sub and another word as a prefix (again for consistency with the NetHack 3 series, which has tiles like sub sokoban walls 7). Any number of substitutions can be given on one tile, in any order. sub female sub orc wizard will attempt to render a female orc wizard, then an orc wizard (because sub orc has a higher precedence than sub female), then a female wizard, then just a wizard; the tile that's actually drawn will be the first of those four that exists in the tileset.

Textual names are mostly based on the unidentified appearance of an item, sometimes with an object class appended (e.g. orange potion), or just the common name of a monster or terrain feature. Sometimes, more than one item has the same unidentified appearance, or else multiple terrain features have the same common name; in both these cases, disambiguation is achieved via appending s and a number (such as orange gems 1 or walls 5). This happens even in cases which would otherwise be ungrammatical, such as the dubious Amulet of Yendors 0, for consistency when importing from old versions (which allowed duplicate tile names).

You can get a complete list of unsubstituted textual names by running tilecompile on an empty tileset, asking it to warn about missing base tiles.

There is various sugar allowed on textual names, that is desugared by the tile manipulation routines. Some of this is for the purpose of reducing the effort for importing old tiles. Some of it's to make tilesets easier to write by hand. Here's the list:

When writing a 96-bit numerical tile name into the binary tiles format, the least significant byte is stored first; this reduces byte-swapping operations on common hardware. The name is written before the tile itself in order to form a 128-bit name/image or name/cchar pair.

CCHAR REPRESENTATION

A cchar contains the following fields:

Any of these fields can be omitted, in which case the field is filled in with the appropriate "copy the layer below" behaviour. More than one field can be omitted, but at least one field must be given (to avoid the null string being a valid cchar). However, you can set all the fields to copy the layer below via giving one such field its default value explicitly (e.g. invisible); the binary representation of such a cchar would be 0xA2000000.

TILESET FORMAT

It's now possible to define the format of a tileset file.

A tileset file possibly contains a palette, and then contains zero or more tile definitions. It also permits comments, which are lines starting with !, that can appear anywhere except inside an image definition or transformation; these are ignored.

A palette consists of lines of the form:

 key = (channel, channel, channel)

where each key can provide 1, 3, or 4 channels (although either every key must have 1 channel, or every key must have 3+ channels). Keys themselves can contain letters, digits, underscores and dollar signs, and be up to two characters long (although all keys in any given file must have the same length). For single-letter keys, avoiding underscores and dollars gives the best possible backwards compatibility; for two-letter keys, using __ as the first key is recommended by the Slash'EM tiles developers to get old versions of the tiles utilities to bail out cleanly. Providing a palette with no tiles is useful when doing tiles merging, in order to adjust all the tiles onto a specific palette (especially when the -p option, to lock the palette to that in a particular file, is given).

A tile definition starts with the tile name. For backwards compatibility, this can be given in the form # tile 0 (name), where the 0 can be any number, and tile is written literally (with name replaced by the name). However, it's also possible just to write the tile name by itself (in text format, or numerical format in hexadecimal). The # tile version is recommended if the tile file is entirely made out of text-format images, for backwards compatibility; however, there's no point in using it otherwise, because old tiles utilities would not be able to read such tilesets anyway.

The format is then continued with the tile image or cchar:

Binary tilesets also have a header, consisting of the string "NH4TILESET" and two NUL characters, then 80 characters representing the name of the tileset (padded to the right with NUL characters), then the width and height of each tile (as little-endian 16-bit integers). In text-based tilesets, these header fields can be expressed as follows:

 # name My Example Tileset
 # width 32
 # height 48

The headers can be omitted in text-based tilesets, in which case they must be supplied to tilecompile using command-line arguments.

SEE ALSO

tilecompile(6), nethack4(6), http://nethack4.org.

BUGS

Substitution tiles and brandings are different concepts, and ones that are typically not interconvertible (although the sub * syntax allows you to use a substitution tile as a branding in a cchar-based tileset).