De Facto Data Somethings in Eulora's Client

There was a rather important question asked only recently in #eulora:

mircea_popescu: inasmuch as we do have a ~de facto~ data model, the gap must be bridged. "why aren't you using what you are already using again ?"

To answer the above, I went and extracted what and how can be found currently in the full client and its dependencies code, to have it all clearly in view for once. The report of that investigation is below in quite a lot of detail but that detail means also about 5k words in total, so here's directly the conclusion of it all namely the answer I can now give to that important question above:

  1. The de facto data model is not really one data model at all but overall as it stands more of a confused mix of several formats, approaches and accidental structures of all sorts, all wrapped up and intertwined together into one single large pile of code with fundamental assumptions that are the exact opposite1 of those of S.MG and that requires a lot of effort for the smallest adaptation to new/different requirements, working otherwise mainly for as long as it is used precisely as it stands.
  2. The thick layer of PS on top of the comparatively saner CS and Cal3D core makes access to and use of CS/Cal3D data models directly especially difficult/effort intensive because everything is built quite on purpose precisely to avoid such use. Moreover, the overall de facto data model such as it is has as fundamental assumption that *all* assets are known upfront with no new knowledge/assets ever added during runtime. Consequently, there is a significant tension between the de facto model and the desired model since the focus of ~everything in the current client is on handling local loading of potentially large but fully known upfront lists of assets while the desired model is instead meant for transmission and/or discovery and loading of new assets as an ongoing concern as the code runs.
  3. At the core of the current client (hence in CS and Cal3D essentially) there are some clearly defined data models and formats, most notably Cal3D's file formats and CS's internal data models (that pack however shaders and shader variables into materials as standard) as well as some basic file formats used otherwise such as .png, .dds, .ogg or .wav. However, none of those different sets cover *everything* that is needed, meaning that they can only provide perhaps a *part* of that desired definitive universal data model. The rest can be - it would seem to me as a result of the detailed investigation below - extracted/synthesised out of what CS allows but using it as a universal data model is problematic with the existing client because of the PS layer on top that would require significant effort to turn around from hiding/making direct access difficult to using it directly when and where desired.
  4. Making the best of it all, here's a proposed list with the parts for which there is at least some clear format already (if not necessarily easy to use in the desired manner with the existing client such as it stands otherwise) and the parts on which a format would need to be decided on/extracted out of various options, fallbacks and CS's own API even:
    1. Characters (animated mesh objects): Cal3D's formats, ideally the binary set (as they are more compact); this means: .csf (skeleton), .caf (skeleton-based animation), .cpf (morphanimation), .cmf (mesh), .crf (material) + raster images for the textures that are referenced by filename from .crf; the preference for the texture format would be .dds files with size as power of 2 as CS otherwise performs the calculations needed to bring anything to the equivalent of that anyway; alternatively, CS compiled with the usual set of plugins supports a variety of usual image formats including .png, .jpg, .gif.
    2. Items/Structures (non-animated mesh objects): the options here would be to either force the use of the same .cmf from Cal3D's set above (the trouble with this being that it assumes the presence of a skeleton/bones, which is not going to fit well with items/structures and moreover it will require some conversion for use with the current client even beyond the PS layer, namely to fit the internal CS non-Cal3D mesh) or otherwise extract and define a format based on CS's internal representation/specification of a mesh (this is essentially a plain and long enummeration of all vertices, triangles that make up the surfaces and submeshes if any).
    3. Terrain: there is no clear format for this; the de facto "model" relying on CS's Terrain2 plugin seems to require simply a bunch of image files for heightmap, material palette, base material and alpha as a minimum, using at all times some of CS's predefined shaders for terrain. PS reads/expects those packaged via an .xml file that matches image files to their role for the terrain so either such .xml specification is imported as part of the format or that part of the client (the loader essentially) has to be adapted/changed.
    4. Particle systems: there is no format or specific model currently available; in principle it could be extracted based on the concrete set of parameters required by CS's way of specifying existing particle systems (hence, a specification of emitter and effector parameters) but it's unclear just how powerful such a model would really be as there are only a few types of particle systems implemented in CS and each of them tends to have its own specific parameters too.
    5. Light sources: there is no format or specific model currently available; extracting it from CS's internal representation provides at least a relatively simple set of characteristics so this could be perhaps done as such.
    6. Ambiental light and fog/air effects: there is no format or specific model currently available; de facto, ambiental light values as well as fog and air effects (mostly weather-related) are currently scattered through several .xml files that are mainly PS's: some values for ambiental and fog are given in the world.xml file (art/world.zip) while hour by hour values for ambiental and rain are given in a different .xml file for a full day and the client expects them as such. The more reasonable approach here is to simply keep those as sets of rgb(a) values rather than force-define anything more complex than that.
    7. Effects: there is no format or specific model currently available; CS does not treat this as a separate category as such; PS on the other hand has a de facto .xml format for defining effects attached to items but there is no clear specification for it and existing samples focus mainly on defining essentially animations of sorts. Perhaps this can be skipped entirely as nothing different from particle systems and/or sound and/or some specific animation defined through the corresponding formats.
    8. Materials: .crf (Cal3D's format); this will require though some conversion to use with the current client even beyond the PS layer, since it is not a direct fit to CS's internal data model for materials. Moreover, using this format will fail to take advantage of some of CS's techniques for materials such as using animated textures (literally changing a set of textures in the render buffer so the result is an animated surface). Arguably such effects though will have to be obtained by other means if desired by any specific client.
    9. Textures: raster images with sizes as power of 2, preferably .dds format but possibly supporting a wider set including png, tga, bmp, jpg, dds, gif, mng, jng. NB: there is no provision or format available for any procedural texture generation in this case (CS provides procedural texture generation implementations for water, fire, plasma and dots).
    10. Icons/2D/GUI decorations: same as textures above. Basically any image can be used as either icon or texture at any time anyway. NB: there is no provision currently for using any vectorial representation directly.
    11. Sound: .ogg and .wav files.
    12. GUI/Skin: this is currently specific to PS and as such perhaps the "model" can simply be a .zip file that contains whatever files and everything else a specific client requires to skin its GUI entirely. This would be the only option I see to provide this too along the rest without making it either dependent on the current client implementation (hence PAWS in particular) as such or otherwise truly the protocol/server's concern since it's strictly a matter for the client to decide on, in my view.

Overall the parts and formats listed above focus on the assets themselves and entirely avoid the specification of function and structure that is currently de facto done via .xml files. The following section provides a more detailed look at how things happen with the currently deployed client.


Current Client Situation

While the de facto data formats in Eulora have been investigated previously and in quite some detail too, the focus was at the time on understanding the whole thing well enough to be able to extract out of it the useful and - hopefully - discard all the useless and the worse-than-useless currently bundled up with it. The attempt failed though, despite the progress made in that vein. So here I am taking stock of the various parts of the swamps I am stuck with, for lack of having anything better - meaning something that actually fully works - to take stock of, how lovely2. Anyways, them being the swamps is no excuse for not having otherwise a clear picture of what's in there, so I'll aim to bring the clarity even to an inventory of mud, what else is there to do anyway.

To start with and for an initial top-level, orienting view, note that there are effectively at least 3 (three!) layers of "formats" weaved and intertwined in the current client, from bottom up:

  1. Formats supported by the various external libs and plugins that CS3 uses. Those are usually standard formats such as .png and .jpg for images for instance, .ogg for sound or .caf (Cal3D's format) for animations. Note that the full list of those is not really a fixed thing since it *depends* on what components you have on your system and include/compile with the whole client! Moreover, *none* of the different plugins provides a single set of formats that would cover *fully* all assets or even all graphical assets so one can't just pick from there a single set and be done with it.
  2. Formats defined/imposed by CS. Those can be further split into internal and external (to the engine code as such). The internal formats are most relevant for textures, materials and meshes since they fix a certain data model and requirements for defining and working with those (see below for the more detailed descriptions). The external formats are based on xml but with specific tags, restrictions/rules and parsers for things such as "shaders" or scene/world/object definition in standalone - and at times or to some extent hardcoded even - files that CS knows how to work with.
  3. Formats defined/imposed by PS4. Those are yet another set of xml-based "formats"5, this time aiming - as far as anyone can tell by looking at the gnarly result - to provide the way to list and describe existing assets of the game, from 2D icons to 3D meshes and GUI widgets (windows in game).

Given that the original goal was to arrive at a clear definition of a definitive6 universal data model, I'll focus on the "clear" and therefore use the above three neat layers to structure the whole discussion of the de facto formats and emerging "model". The alternative would be to follow closely and describe exactly just "how the client currently works" but that results quickly7 in a quite predictable muddle (given the de facto muddle it reflects), so not much use for clarifying anything or for answering any question. To state it plainly though: I'm aiming to structure the text for clarity but not by glossing over any gross details - my aim is to mention such muddy details where they are relevant but to not let them drive my text even if they do get to drive the legacy code in quite a few places.

Before diving into the concrete details and exact formats currently in use on client side for all sorts, it's worth noting a few fundamental characteristics of CS that drive quite a lot of the whole de facto model such as it is:

  1. The focus is on supporting - somehow and to a best effort level essentially - a generic and not that well defined everything. The main implication of this is lack of clarity and added complexity of all sorts: there isn't a clear single data format and model as such but at most a generic structure (most notably at the way materials are defined and implemented to include specific reliance on predefined shader types and variables) that still ends up forcing some choices, only not fully supporting them either (since that would break the appearance of supporting everything); there are always rather intricate fallback and alternative mechanisms at all levels from type of rendering (software vs hardware and then depending on what operations are supported by different types of hardware) to high-level object or scene definition (e.g. from code or through xml).
  2. XML is used (and pushed) extensively for a double - if never explicitly stated as such - purpose:
    1. to describe the structure and organisation of sets of assets on disk (all sorts, including but not limited to graphics) for a CS application. Sets of graphics/sound/UI assets are called "library" in CS parlance, for all their xml-only definition.
    2. to script from outside the CS application all sorts, from rendering steps (in definitions of shaders) to scenes, objects and even actions in the world (in definitions of sectors, meshes, triggers, sequences and portals). Especially for shaders, this gets quite intricate, it's a sort of ad-hoc xml-based programming language, no 2 ways around it.

    Note that the scripting purpose of xml above goes well beyond mere description of structure and organisation or even statement of the purpose/use of things in there. To try and give an idea of this, the xml code at CS level is quite overgrown and does a lot of things, including but not limited to customising existing objects, describing how to create new objects out of existing assets, setting out what and where things are to be found in the world, calling and otherwise using (therefore fixing as requirements) specific plugins or parts of CS, defining new rendering sequences or even buffer contents and prescribing steps and parameters of all sorts for them.

    The saving grace of all the above is that most - though unfortunately not ALL - of it can still be done directly from code as well, without a lot of trouble. So in this sense, the xml reliance pushed by CS is more of an option than a mandatory part to deal with. Nevertheless, there is one significant part of this xml reliance that is deeply embedded and does not really have a clear code-only/no-xml alternative, making it more of a mandatory part than an option: the shaders which are fully described *only* through a spaghetti of xml files that rely on parts and "snippets" from one another, provide fallback or alternative to one another and otherwise end up hardcoded as part of the very definition of a material and at times of crucial plugins (the terrain generator for instance) in the CS engine.

  3. All identifiers (including those of assets loaded/created in the engine) are strings - combined with CPP's own specific brand of string-problems (most notably the null terminator that may or may not exist and the allocate/deallocate memory dance), this is a pain. The approach of CS/PS to avoid the pain is by having everything rely on CS's implementation of csString as a utility that takes care of the gritty details of using strings in CPP. Furthermore, there are several hashmaps used to map those cumbersome string ids to saner numerical values but those of the engine itself are not directly exposed. Adding to this, CS provides a way to "attach" data to any object so one could in principle add a numerical id but there is little gain from that since all searches for objects still rely on the string "id".
  4. There is an implicit assumption that all assets and the game's world are fully known and defined upfront as well as outside the code itself. The programmatic option - as much as it exists - ends up overall underdeveloped in comparison to the xml support for the fixed, outside of code definition of everything.

The above fundamental characteristics of CS get pushed even further at the PS level on top: the fallbacks and "support everything but don't commit to anything in specific" are pushed even on to graphics preferences and configuration files (of which there are several types and several layers); the xml gets added functionality as means to describe appearance of widgets (windows) in the client's interface while still relying on even more hardcoded paths/names both in the xml itself (which references resources as it expects them to exist) and in the code (which looks for various xml files by name and path (if at least the path is relative to some directory as defined in the VFS - virtual file system - configuration); the expected full upfront knowledge of everything that may exist in the world gets cemented by PS: there is virtually no programmatic creation of anything and instead, everything is both supposed to be described in external xml files (+ referenced graphics/sound/effects files) and otherwise known and loaded before the game can even start at all. Essentially the PS client chooses at most what to show to the human player at one time or another out of its otherwise full knowledge of what's possible and what not8. Things can get at most cloned or deleted, hence the player may encounter "in the game" something that seems "new" to them only if they really didn't look in the client's directory as otherwise everything's already there as long as it has some sort of asset-based representation at all.

Leaving aside for now the above characteristics of CS and their various implications, here's otherwise the structure such as it is of the de facto data model at the core of the engine itself, from top down:

  • the whole world is made of "sectors" that may be split into "regions" and contain any number of "sources of light", "effects" and "mesh objects";
  • sectors are connected to one another through "portals" (that are really just a special type of mesh objects);
  • sources of light do not have geometry and are defined instead through a set of functional characteristics, they are basically ideal lights that can be tweaked to some extent (see below for full list of characteristics);
  • effects include particle systems as well as sequences (a set of operations tagged with relative time so that they get executed in a sequence);
  • mesh objects can contain in turn other mesh objects (in a hierarchical structure) and are defined ultimately through their geometry (aka vertices and triangles) combined with assigned materials that describe the surface appearance and, where relevant, skeleton-based animation (CS supports animation through either its own sprite class or the external Cal3D lib);
  • materials are in turn made out of one or several textures combined with any number of shaders and parametrised through specific values assigned to variables from a predefined set;
  • shaders are defined via xml only and contain instructions for the rendering of the material data, introducing further their own variables, parameters and dependencies on other shaders;
  • textures are the most basic element and they are either directly a colour represented through the three components, red, green and blue or otherwise an image file that is meant to be of a power of 2 in size (with the exact list of supported formats depending on which plugins are linked with CS on any specific install). There is also a limited selection of procedural approaches that can be in principle applied to any texture, namely for fire, water, dots and plasma appearance. Confusingly though, CS still wraps all textures into a default "material" so despite loading textures, it's always materials one gets to work with from the code. Moreover, further tweaks such as animating textures are done through render buffers and as such essentially through shader code, whether it gets written as a separate xml-based shader specification or otherwise implemented directly in the code.

Adding more concrete details to the above overall description:

1. Sector - this is an abstract type (it does NOT define by itself any geometry and as such it does not have dimensions or position or anything of the sort) that works as a way of grouping together a set of assets for rendering purpose (overall rendering characteristics are defined at this top level, including ambiental light characteristics, some air effects such as fog, overall sources of light if any and specific choices of visibility culler and collision detection system). Everything that is shown/heard as part of the game has to be contained in a sector and at any given time there is only one sector that is fully visible. Characters may be in more than one sector at a time but only temporary and as part of the way in which crossings happen from one sector to another, namely through "portals". At CS level, everything in a sector and all properties of a sector can be changed in principle on the fly from the code although some changes may incur a significant computational cost (e.g. changes to dynamic lights that should trigger a recalculation of lightmaps).

While the above is given by CS, the current client PS implementation adds its own layer of complexity on top: the current code requires that all possible sectors are already defined in an xml file that it expects to find in /art/world.zip (this path may be changed but not on the fly, only from one of the many configuration files). Moreover, it also expects that it knows everything upfront about all sectors and that all sectors reference only assets/items/anythings that it already knows about (from other hardcoded xml files that serve basically as inventories and/or definitions of different types of assets). Finally, there's the added hardcoded dependency of some of the GUI on some sectors and graphical assets (most notably the requirement for the "podium" sector that is used to display the character's smaller version in the inventory window and/or at the initial character selection). Note also that PS's current client implementation makes it very hard to actually change anything on the fly from the code since it quite specifically assumes that things do not change - they may be copied or otherwise not shown on screen or even not fully loaded into memory at one time or another but otherwise not really changing much.

1.1 Region - this is yet another abstract type, basically providing a way to split a sector into smaller chunks of contained objects (of all sorts, so not only physical objects) if desired; the main reason for this to exist at all seems to be the fact that the assumption of knowing everything upfront can result otherwise in "too much to load at a time" and this got solved by allowing the split of sectors into regions9.

Here PS adds a whole layer that keeps track of various regions and "zones", what should be fully loaded for each and even what specific picture to show the user while the loading is crawling at its own pace. It also handles its own added complexity from the current interaction with the server as some entities may come before their sector/region is fully loaded and so the client has a special sector created as a temporary purgatory for those entities, checking continuously to see when they can finally be moved to some loaded sector/region.

1.2 Light - there is "ambiental light" as an overal property of each Sector, as well as "sources of light" as abstract objects (no geometry) that have nevertheless a position from which the light is considered to be emitted. Ambiental light is defined only through its colour (rgb). Sources of light can be of 3 types: static (do not move and do not change intensity or colour), pseudo-dynamic (do not move but can change intensity or colour), dynamic (can both move and chnage intensity or colour). Each source of light is defined at a minimum through its type (static, pseudo-dynamic or dynamic), position (3 float values), radius (float value) and "diffuse" colour (rgb) that influences as well the intensity of the light coming from this source. Optional characteristics include a second colour ("specular", rgb), an attenuation mode (none, linear, inverse, realistic, constant linear quadratic) and parameters (3 floats), a cutoff distance (float value) representing the maximum distance at which this light is considered to shine, a halo (flare, nova or cross, with corresponding parameters), falloff inner and outer angles for a spot light (2 float values).

1.3 Fog - this is an overall property of each Sector and is defined through colour (rgb), mode (none, linear, exponential, exponential2, crystalspace), density (float) and fading start/end distance (2 float values).

1.4 Effects - this is my own category really since CS doesn't bother as such to fully group everything. CS provides a set of particle systems that can be programmatically used. As I discussed those in more detail elsewhere, I won't repeat that part here. However, it's worth noting that the current implementation of those particle systems relies on the predefined set of shaders that CS comes with and so they are essentially customisable to some extent through the code rather than fully defined (unless one adds a redefinition of expected shaders too, I suppose).

The current client's take on effects is again - unsurprisingly - quite entirely reliant on various xml files. There is a hardcoded path art/effects that includes a bunch of .dds files mainly as well as a art/music/effects that includes a bunch of .ogg and .wav files. The paths to those are hardcoded throughout the client where the legacy code "knew" effects should be loaded from. The loaders generally look for an xml file that provides the inventory and tends to attach "resource names" to the physical files. At times, the xml file will also provide items to which effects are "linked" and the legacy PS code keeps effects in an array with references to the intended items. Even this de facto "model" is not all that respected really since there is at least one place where the "rain effect" is simply directly hardcoded as a texture to be loaded from the hardcoded path to the .dds file (see src/client/weather.cpp: const char* rainTex = psengine->GetConfig()->GetStr("PlaneShift.Effects.Rain.Texture", "/this/art/effects/raindrop.dds");).

In addition to the above, PS also adds its own plugin for loading effects defined in xml files as illustrated in data/effects. The full format for those is not clearly defined anywhere and currently I haven't fully investigated it either as it's not directly in use. At any rate, the .eff xml files defining the effects contain also the definition of needed textures and materials based on the .dds files included next to the xml. The rest of the .eff file seems to define an object type to which the effect is to be attached and otherwise keyframes so essentially yet another sort of animation format done ad-hoc in xml.

1.5 Mesh Objects - from CS's point of view, anything that has geometry at all is a mesh object. The main categories of mesh objects that I can discern are terrain, characters (moving) and items/structures (non-moving). For each of those, CS allows in principle any concrete implementation and inner data model to be used, as long as the corresponding SCF10 interface is provided. In practice though, the existing client uses the Terrain2 plugin for terrain, the Cal3D plugin for characters and CS's native basic mesh for items/structures. Note however that all of those are wrapped up in a few layers of PS obfuscation, most notably through the GEMObject hierarchy and psCelClient + psEntity classes. The GEMObject hierarchy is a core element of the current client too and as such it's not trivial to either change or remove. Getting directly and using CS's interface from the existing client is not straightforward at all because the PS layer works precisely and purposefully to insulate away from touching CS directly. Nothing is impossible, of course, but whatever clarity there is at CS level needs to be considered through the murky and intertwined layers of PS on top as things currently stand.

1.5.1 Terrain (from Terrain2 plugin) - at CS level, this requires as a minimum a heightmap file, a material map file and a material palette (aka several full material definitions including their texture(s), shader(s) and shader variable(s)). The CS Terrain2 plugin is directly linked to some specific shaders that handle the splatting and seem to further require - for better appearance - at least an alphamap of the whole terrain. The current PS implementation relies on XML to define the terrain too and expects as usual that everything - including all possible terrains and maps and even the pattern of sunlight throughout the various hours of the day! - is known and fully loadable upfront.

1.5.2 Characters (from Cal3d plugin) - when the Cal3d plugin is used, there is an XML description required (mainly by the PS layer since CS can otherwise simply do it from the code directly) that specifies the relevant Cal3D files for any given character. Those Cal3D files come in 5 main formats (the formats mentioned below are binary formats; they all have equivalent xml formats as well - those are directly readable but otherwise more bulky, of course):

  1. Skeleton file, .csf - this lists the hierarchy of bones that make up the character's skeleton. Each character is supposed to have at least one bone.
  2. Animation file, .caf - this defines the pose of the character at each keyframe (the animation will interpolate the pose between those keyframes). Keyframes are further grouped by tracks with one track per animated bone and have to include the time, the relative position and the relative rotation with respect to the parent bone (if .caf file).
  3. Meshanimation file, .cpf - this is meant for changes in expression for instance or more generally animation of meshes/submeshes without relying on /requiring bone movement as such. Note that this format is mentioned in the Cal3D docs, but it is not currently in use at all and it's unclear how well it is actually even supported otherwise (to establish this, I'd need to investigate in more detail both Cal3D itself and the CS plugin for Cal3D).
  4. Mesh file, .cmf - this describes in detail the mesh as a hierarchy of submeshes and with references to materials in use as well as the weighted influences of the bones on each vertex. This would also include morphtargets if the .cpf format is used for additional animation(s).
  5. Material file, .crf - this describes a material through its ambient, diffuse and specular colours (each given through 4 values for red, green, blue and alpha respectively), its shininess (a float) and a number of textures that have String identifiers (those can be filenames but can equally well be anything else really).

1.5.3 Items/structures (from native CS's mesh) - the de facto format for those follows closely CS's internal data model of mesh objects: each mesh object can contain submeshes in a hierarchical organisation; each mesh/submesh is defined through its geometry (as an enumeration of vertices and triangles made with those vertices) and attached materials that are in turn defined through textures, shaders and specific values for any shader variables used. Textures themselves are raster images with .dds as preferred format of CS (as other formats such as .png or .jpg effectively incur a penalty although at this point it's unclear what this penalty adds up to in real terms). CS allows the creation and definition of meshes either as tedious enumerations of everything or through a few limited generators that tend to rely on regular shapes (e.g. spheres and cones). Both of those can be directly called from code or otherwise wrapped up in xml and "loaded" from code.

As everywhere else, the current client relies on xml for the definition of all possible types of meshes. It loads mesh definitions from the xml files in art/meshes.zip and expects that all entities ever required /encountered for the whole duration of the game reference one or several of the meshes in that archive. In turn, the xml in meshes.zip references of course materials that are -all of them!- expected to be found in art/materials.zip and are defined as such in art/materials.zip/materials.cslib xml file. The materials.cslib xml file first lists an inventory of textures (assigning a "name" id for each image file in the archive basically) and then defines various materials that make use of those textures and/or specify various shaders and corresponding values for the relevant shader variables. As usual, the shaders are simply referenced by name and assumed to exist as expected under that name, basically making art/materials.zip dependent implicitly on the whole contents of data/shader (and through that, further on to data/shader-snippets and data/renderlayers at the very least).

1.6 Sound
For sound, CS simply relies on external plugins for handling a few standard formats, most notably .ogg and .wav (apparently there are some .spx files too in data/voice but I haven't seen them in use from anywhere within the current client's code). PS adds on top of this a soundmanager that is intertwined with a whole bunch of code otherwise so getting to the heart of the simple matter of playing one sound file from the current client can be surprisingly messy. Still, from the point of view of data formats here, things are quite simple since not even PS managed to do much about sound files being simply .ogg, there is that.

1.7 GUI and icons

With respect to GUI and more generally any icons shown in game (so any 2D representations really, such as items in the inventory for instance), CS relies on external plugins. In principle there is CEGUI supported but the currently deployed client does not use that one, relying instead on PS's own PAWS. Both CEGUI and PAWS share a design of GUI as a set of widgets but other than that I didn't investigate CEGUI any further since the current client is rather deeply intertwined with PAWS. Consequently, the de facto data model for GUI is more of a reflection of what PAWS expects rather than anything else: art/skins contains .zip files that provide any number of .png images in whatever hierarchy of directory one sees fit, together with an imagelist.xml file that provides effectively the inventory, assigning a "resource name" to which file and listing the relative path to it. Note that this imagelist.xml as well as the art/skins location are both hardcoded in the current client. Moreover, the imagelist.xml is meant to be the full list of all possible icons and GUI elements in the game and known when the client starts since it's all a pre-requisite of even creating the GUI widgets at all.

Besides the above, there's another part to GUI, since the PS client further expects xml to be used (in data/gui) to define each and every window that can ever be shown by the client. Those make use of resources by name and as such are inevitably -and implicitly- dependent on the contents of art/skin. Moreover, some of the current code goes as far as referencing directly and hardcodedly resources from that same art/skin (most notably the mouse pointer icon). Not sure where exactly should this go into a "definition of the de facto data model" but there it is.

A rather funny part of the xml reference by name of various resources is that font resources (data/ttf currently) get the same treatment and the xml widget definitions that dully specify - thus fixing too! - their favourite fonts by name again. Then the legacy PS code also hardcodes a font as default in quite a few places and rather central to the GUI places at that (src/common/effects/pseffectobjtext.cpp and pseffectobjtext2d.cpp in the same place; src/common/paws/pawswidget.cpp and src/client/gui/psmainwidget.cpp), specifying the exact "data/ttf/reteprelieum.ttf" path as well. Except the actual file in data/ttf is called "reteprelleum" rather than the hardcoded "reteprelieum" and so errors are thrown and whatever else gets used instead for all the usefulness of that careful hardcoding of one font in that many places. Are you laughing yet?


  1. In particular fixed, finite and fully known upfront world and assets vs flexible, infinite and continuously discovered world and assets. 

  2. Isn't this what you always thought "choice" meant in practice, namely choosing between the inexistent and the impalatable? No? 

  3. CrystalSpace 

  4. Planeshift, the ancestor of the current client's code, such as it is. 

  5. More like accidents than formats. 

  6. Let me add here the very clear and useful definition of this definitive: mircea_popescu: it's definitive in the sense death&damnation is definitive, not in the sense mathematics is definitive. 

  7. Yeah, I tried it out, I have the draft to show for it. And it was even a good rant really but not exactly serving the purpose it was meant to serve, so I discarded it anyway. 

  8. It does sound like an interesting choice when set out this way, doesn't it? The code you run on your own machine is meant to know more than you, to choose what to show you at one time or another and even to specifically thwart any attempts - cheater! - to get to know more than it decides to share with you. 

  9. There is also "Collection" as yet another way of grouping resources based on whatever reason the programmer may come up with but this is not used by the current client at all so I won't go into detail on it. 

  10. Shared Class Facility 

2 Responses to “De Facto Data Somethings in Eulora's Client”

  1. The thick layer of PS on top of the comparatively saner CS and Cal3D core makes access to and use of CS/Cal3D data models directly especially difficult/effort intensive because everything is built quite on purpose precisely to avoid such use.

    There's an unexamined (and imo counteproductive) jointure here that needs disjunction.

    It may very well be that using the data model such as it exists is inconvenient, in practice, because of accidental complexity. This however does not speak at all to its quality, which is what we're trying to discern currently.

    To take an analogy : if (1) a married woman makes sandwiches, and then (2) gives one to her preteen son, thereby (3) sending him out of the family hut to play (4) whereby the child goes to play in the stable (5) where he promptly drops his bread on the floor (6) thus returning to his mother early, (7) upon her inquiry as to whether the sandwich is any good, his response that (8) no because it is covered in shit is not a good answer. 7 and 8 speak to different things : she can make another sandwich of the same kind, and he doesn't have to drop that one into the shit also. She's trying to review 1, not 5, even though the son may well be fixated on 5 by virtue of age-appropriate self-absorbtion.

    So : in a purely theoretical sense, can we just use the CS/Cal3D part for ~pretty much everything we need ? For actually everything we need ? For everything we need except X specific part that's just not there ?

    Consequently, there is a significant tension between the de facto model and the desired model

    Try as I might I don't comprehend this tension.

    Suppose it comes to pass that the latest Windows machine comes with a pre-set palette of nine colors ; there's a syscall to add more colors, but it requires a system reboot. (Honestly, it wouldn't surprise me if this is what they come up with next, god knows it's exactly how "innovation" at Apple went.)

    Now, what's the tension ? You a) pick a color off that palette to be the default color displayed if the color you want displayed isn't loaded ; b) feed the user whatever the fuck colors you feel like using, which get saved upon first encounter and c) whenever the fuck he's had enough black on black for a session, he reboots therefore increasing his palette.

    Where was the tension ?

    I'd say, if you were to ask me, that the tension was inside the engineer's head, who made some assumptions about instantiation and acceptable functioning that just aren't valid. Nobody says everyone playing Eulora must see the same things ; we don't care what they see, and therefore it's perfectly acceptable for the game to download new assets as it meets them, but only start displaying them once it's restarted. It's shockingly inelegant verging on stupid, but elegance is not nearly a requisite -- certainly not nearly like functioning at fucking all is a requisite.

    One of those things that yes should be fixed eventually, but that eventually must come afterwards, not aforewards, and in any case it can't possibly be blocking.

    The rest can be - it would seem to me as a result of the detailed investigation below - extracted/synthesised out of what CS allows but using it as a universal data model is problematic with the existing client because of the PS layer on top that would require significant effort to turn around from hiding/making direct access difficult to using it directly when and where desired.

    In which foregoing context, this now needs a rehash.

    raster images for the textures that are referenced by filename from .crf

    This sounds like a potentially fucking nasty piece of stupidity.

    CS compiled with the usual set of plugins supports a variety of usual image formats including .png, .jpg, .gif.

    It would indeed be preferable to do the precomputation once and save whatever as dds rather than just keep reiterating it every server reset or w/e. Is there code for this gif->dds conversion for instance anywhere ?

    the options here would be to either

    The houses currently standing, how are they made ?

    but it's unclear just how powerful such a model would really be

    Possibly the hole through which the night comes in. Then again, how much of a fuck do we even give is entirely unclear to me as of yet.

    while hour by hour values for ambiental and rain are given in a different .xml file for a full day and the client expects them as such.

    What the fuck, I don't even remember seeing rain or anything like this. I mean, I guess the terrain turns black at night, but...

    procedural texture generation

    I don't even know what the fuck that is, to be honest. Vectorial rather than rasterized now it's suddenly "procedural" bla bla ? Is that it or is there more that I'm not seeing ?

    In general, I'd much prefer disposing with bitmapped textures altogether, because tiling simply fucking sucks as a visual, and the insane ammount of sheer labour involved in trying to plough by hand is staggering. I'd honestly much rather go plough on my feet while giving a 200 HP agricultural tractor a piggyback ride than tile hand-cut textures.

    perhaps the "model" can simply be a .zip file that contains whatever files and everything else a specific client requires to skin its GUI entirely. This would be the only option I see

    Fine by me, honestly.

    the current code requires that all possible sectors are already defined in an xml file that it expects to find in /art/world.zip

    Evidently this'll have to go away before any kind of meaningful activity can continue.

    the main reason for this to exist at all seems to be the fact that the assumption of knowing everything upfront can result otherwise in "too much to load at a time" and this got solved by allowing the split of sectors into regions

    Motherfucker.

    These idiots should never have been permitted out of the rural-appropriate camin de nefamilisti.

    Are you laughing yet?

    At least in the limited sense that all the pantsuit/moronworld idiocy is in the end deeply irrelevant to anything.

    The story goes, one day a young man visited the land of the pantsuit. He sat down in a cafe, pulled his .5 DE out of its holster and set it squarely on the table. Everyone panicked, running around like headless chickens, because guns weren't allowed in the cafe, or in the world, and besides utterly fucking forbidden and so on. There were news crews reporting, the police cordoned the building off, etcetera. Then eventually a trembling guy sorta-awkwardly approached, pointed out that he's the Whatever-Pantsuitist-Pompous-Title and therefore has to take the man's gun away "because it's illegal" and "not done" and bla bla bla. The man asked him calmly if shooting people in the face is just as illegal, and upon hearing the answer in the stuttering positive, promptly shot the pompously titled vacuole in the face. Everyone panicked, running around like headless chickens, because guns weren't allowed in the cafe, or in the world, and besides utterly fucking forbidden and so on. There were news crews reporting, the police cordoned the building off, etcetera.

    Shoot them in the fucking breathing hole, motherfucker, break everything.

    The last thing we need to give the first inkling of a fuck about is just how the girly thinks she's to be used, and so on.

  2. Diana Coman says:

    It may very well be that using the data model such as it exists is inconvenient, in practice, because of accidental complexity. This however does not speak at all to its quality, which is what we're trying to discern currently.

    Well, for starters I actually thought we were trying to assess - given the focus on pragmatic and the question itself, which inquires as to why not use it, not as to why isn't it good quality - a very practical fitness for purpose that includes therefore both quality and usability as it stands. And from where I see it, the analogy would be more along the lines of access to mom's sandwiches going only through the inept child and as long as we don't want to rewrite the client it means we don't kill the inept child, hence the very reason why that inconvenience of access gets into this discussion at all in the first place. Because otherwise yes, I agree that there are those 2 different aspects but I don't think I mixed them up as such anywhere, no. I did have both of them in mind and the text does cover both of them but doesn't really either mix or mistake one for another at any point (does it somewhere?). I think that leaving one out of the discussion entirely is not that compatible with the pragmatic approach of focusing on what works as it risks we end up with "great, it should work" because theoretically yes, it should and then come practice, it takes 2 years to make that should materialise and not even step by step.

    So : in a purely theoretical sense, can we just use the CS/Cal3D part for ~pretty much everything we need ? For actually everything we need ? For everything we need except X specific part that's just not there ?

    Hm, I thought point 4, Making the best of it all is precisely answering this. (What's missing there?) To re-phrase/summarise it though, since you asked the question: cs/cal3d covers most but not all; what is covered does not fully fit (because still raster images and direct references to files by name inside the material format and enumerations of everything for instance); what is not covered can be in parts extracted/made to cover, perhaps (e.g. particle systems) and in part sidestepped (e.g. gui).

    I'd say though that this "purely theoretical sense" here sits very uneasy with me especially as it gets already mixed in the very first sentence with the suddenly practical "just use".

    Try as I might I don't comprehend this tension.

    I do see now at least the sort of practical approach you have in mind, basically more of a sidestepping/working around the limitations of the client rather than solving them (unless absolutely having to). I'll keep this focus in mind as it's not my usual approach, indeed, though I see the need for it to move this forward, yes. And thank you for the example, it helps.

    Otherwise, specifically to "what's the tension" - it rears up in more concrete detail down the line but perhaps you either don't call it tension or don't assign it to this root: all the formats/models, such as they are, focus on organizing something fixed (a set of whatever, from a set of bones if we talk of cal3d's format for meshes for instance to assets as such if we talk of CS's format for "library", hence all the raster formats and references by filename and enumerations of whatnots) while our focus is on modeling (in its proper sense of extracting the underlying model, not the mess CG made of this poor word; hence vectorial for instance). Sure, we don't "care" about the original focus in the abstract but its concrete results can't be ignored and do clash with what we want. We can sidestep it perhaps as above at the user end but sidestepping it similarly at the format level seems to me to be pretty much along the lines of don't use their format, make your own and then write some converter for the client basically (what I even mention anyway for where there's no direct fit, since I don't see anything else really).

    Is there code for this gif->dds conversion for instance anywhere ?

    I had a look and there seems to be a CS plugin (cs/plugins/video/loader/dds and its ImageLib dir especially) that supposedly provides conversion to dds. So far I haven't used it/tried it out/read it - and it's about 22k LOC by itself. There is also otherwise a GIMP plugin for dds still in C, possibly smaller but still an unknown/unexplored.

    The houses currently standing, how are they made ?

    Not sure why you consider those any special really (I get it that they were added afterwards and not PS-made but they play along otherwise, more of a test of can we do as they do than anything else). They are made replicating exactly the client approach to everything really so manual change to all the xmls so that they get preloaded etc. Specifically (there are 3 house objects but they are similar so I'll use only 1 to illustrate): the "House03_001_object" is added to the Island xml file (in art/world.zip) requesting the use of CS's genmesh plugin to create a mesh object using the House3 factory and placing it at specified position -30, 45, 340 in the sector Island; the House3 factory is defined - as a CS "library" - in its own xml called HOUSE3 that is placed as expected in art/meshes.zip and enumerates the vertices + triangles, as well as specifying by name the material to be used, namely "House03"; this material is defined as expected in the materials.cslib inventory xml (in art/materials.zip) using directly numerical values for 2 shader variables (specular and diffuse) as well as a texture name for the "tex diffuse"; this texture is defined in the same materials.cslib through more xml that gives a name as identifier to the .png file in art/materials.zip (this is a baked texture so the whole outside appearance of the house made into a 2d image and saved as such, to be painted on the geometry basically).

    What the fuck, I don't even remember seeing rain or anything like this. I mean, I guess the terrain turns black at night, but...

    The light (ambiental so sector's property) gradually changes until it "turns black at night" (and then back again). The rain is not seen because the part of the code that would handle that was chopped/not fully made to work in the client - the original PS seems to have had a whole "weather manager" mushroom on the side of which possibly only some tendrils (still a mess) remain.

    I don't even know what the fuck that is, to be honest. Vectorial rather than rasterized now it's suddenly "procedural" bla bla ? Is that it or is there more that I'm not seeing ?

    From what I understand it so far, the procedural in CS's approach is not just about "generate this image" (this part would indeed be vectorial representation simply) but about combining images (whether rasterized or vectorial) to achieve whatever effect. In principle procedural textures are not static - meaning they are animated/contain some way of changing appearance as they are displayed. Whether it's something we want or not, I can't even tell, since it might very well be that one can achieve the same result by specifying somewhere else (where exactly is unclear atm) changes of texture every x frames or something, no idea.

    In general, I'd much prefer disposing with bitmapped textures altogether, because tiling simply fucking sucks as a visual,[...]

    Uhm, tiling is "avoided" from what I can see via this approach of "bake the whole skin and then stretch it on the object, attaching the corners where specified". So a sort of in between the 2 things you seem to mention there: it's not tiled as it doesn't repeat the same thing but it's still rasterised as the image is already generated and only placed where the model indicates. Maybe I don't fully get what you want/don't want here.

    Fine by me, honestly.

    Ack re .zip with everything for GUI.

    Shoot them in the fucking breathing hole, motherfucker, break everything.

    Breaking is fine by me but my understanding is that we broke too much too soon with not enough build-as-you-break, hence the backtrack.

Leave a Reply