Those Zip Files of Eulora's Client Graphics



July 24th, 2020 by Diana Coman

Wandering all alone and growing quite bored of seeing the same familiar empty space, the twisted hopeful glanced all of a sudden an unexpected shiny item sticking out of the ground and in plain view:

csmesh_1_640.png

What could that be and where did it come from? It wasn't there before and there's no guarantee it will remain there either but there is now finally the possibility for any number of such items to be anywhere at all in Eulora. And now that this part is done as well, the client has finally all it needs to render -though not yet to control or interact with- all three main types of 3D things: the world itself (terrain, sky, water), moving characters be they NPCs or players (Cal3D) and non-moving items (CS meshes). In all cases, the "graphical representation" is simply a .zip file. The contents of this .zip file can change and there's plenty of flexibility in there with only a few "must-have" things nailed down for now - even those can change later on without much trouble, since a .zip file remains a .zip file otherwise and knowing how to use what is inside is entirely a client concern.

The current client 2.0 aims to look for and use what it knows, while not getting stuck on anything additional that might be in there. So it uses currently some very simply conventions regarding filenames inside those .zip archives and simply picks up the first matching files it finds inside: anything starting with "tex" is assumed to be a texture and its extension is assumed to match the format (.dds, .png, .jpg all work fine); anything starting with "mesh" is assumed to be a mesh description and interpreted depending on the context as either CS mesh format or Cal3D format (binary or xml depending on extension); anything starting with "skel" is a skeleton file; animation files are always exactly walk.caf, idle.caf (so far those 2 are the only ones the client knows about). As examples, here's what the current test set of .zip files contain:

  • For Outdoors (the filenames here are fixed indeed as they all have very specific roles):
    • basemap.png
    • heightmap.png
    • lightmap.png
    • materialmap.png
    • norm_1.dds
    • norm_2.dds
    • norm_3.dds
    • normalmap.png
    • sky.png
    • tex_1.dds
    • tex_2.dds
    • tex_3.dds
  • For Cal3D:
    • skel.xsf - the file containing the skeleton in Cal3D format. This can be either xml (.xsf) or binary format (.csf) and the client will simply use whatever is provided.
    • walk.caf - the animation to be used when this character "walks". This has to be currently in binary Cal3D format because reasons.
    • idle.caf - the animation to be used when this character is standing/sitting still. As for walk, this has to be in binary Cal3D format.
    • mesh_*.xmf - this hopeful happens to be made out of 26 meshes so there are 26 such files, simply numbered mesh_1.xmf to mesh_26.xmf. The client will just load as many mesh files as it finds, that's all.
    • tex.png - the texture to use on all meshes. In principle there could be as many texture files as meshes so that each mesh has its own texture. It's not a big trouble to adapt the client to use them but at the moment it just picks the first tex* file it finds and it uses it for all meshes in that .zip file.
  • For CS mesh:
    • meshfact_1.xml - the description of the mesh in CS "meshfact" xml format (NOT library format, see below the more detailed discussion why not library).
    • tex_1.png - the texture to use for this mesh. This can be in principle *any* graphical format. The current client can and will happily use .dds, .png or .jpg just as well.

Compared to the previous iteration, the "outdoors" type bundled together water, terrain and sky because in practical terms they always go together indeed. There is here one part that I'm still hesitating about: while the size of "sectors" is fixed and doesn't need to be therefore sent (or at least not at this level), it's unlikely that this will also match any indoor areas that are to be separate areas as such and the trouble is that the loader needs to specifically set those. One way to go about this would be perhaps to make a different type specifically for "indoors" and give all this additional information required there in the euloran object itself (arguably even the outdoors type might be better off with some specified there, if or when they may change). The main question I have here is: what is fixed about outdoors and indoors (as the cs part has a lot of parameters and very few turn out to be of any use in the end)? More specifically, given the protocol specification, the size of an outdoors sector is fixed and as a result, the client can just create a fixed-size sphere for "sky" for instance and a fixed "cell" size for the terrain and so on. Is this size fixed for indoor places too or could those be smaller/larger?

As to the latest addition from the list above, namely the CS mesh items, getting them to work included a few ugly surprises that slowed me down a bit more than I initially expected. Nevertheless, as there's otherwise less resistance left in that paste than at the initial stages, it didn't take all that much to whip it up into something useful. First, the format I had used previously based on existing client's "art" files turned out to be less than ideal for a rather unexpected reason: the CS loader would happily load it but it wouldn't provide any means to obtain directly a way to reference the loaded mesh - it assumed the caller would know that from somewhere else or otherwise find it somehow1. To fully get this: CS insists on referencing everything by "name" hence effectively by string-ids and the names of loaded stuff from a library file are ...specified in the xml but otherwise not returned by the loader itself! So to find a factory or a mesh, the code would need to either read it specifically from the xml (ugh!) or otherwise assume it matches something else through some convention forced thus by... lack of options (because no, I don't want to have the server now be concerned with fixing the names of meshes clientside, wtf). The solution I found to this is to not use "libraries" at all since anyway each of them contains exactly one meshfactory and then load that factory with whatever name there might be in the xml but *change* it from the code to the name that is useful to the client (and that's ...the ID of the darned object, ofc, what else, only made into a string to fit CS's expectations, yes). It took a while to figure this out, mainly because finding the way to simply rename things is made... interesting by means of Object Oriented (OO) obfuscation layers so that the code can end up at times like this: meshWrapper->GetFactory()->GetMeshObjectFactory()->GetMaterialWrapper()->QueryObject()->GetName(). Count the layers and laugh at it but know that it's not even the worst that OO can do!

Once the above sorted, there was some more work to cut that GEM crap on items too and then to follow up and look into all the resulting, inevitable crashes given the spaghetti pointers running all around. At least I used the opportunity to look also into the way that PS used to do "effects" - it's a useless pile of crap on top of some tidbits of usefulness from the underlying CS, not that it comes as any surprise. The surprising part was that "labels" (e.g. names of characters) used to be... "effects" and thus bringing in the whole crashing monster of stuff. Well, labels presumably will be useful so they are on my list to add as part of sorting out the 2D part of the interface but I refuse to bring in for this reason a whole set of useless "managers" and corresponding plugins and 1001 classes that do the same thing only now it's called labels and then it's called shadows and then again it gets its name changed so that a different "contributor" can sign at the top of the file. Such contributions that one is way, way better off without any of them.

At any rate, having now sorted non-moving items as well, the next step really has to be looking into the 2D GUI stuff more closely, hence... PAWS. There's no estimate I can really make (only heavy guesstimates) on how long it will take to make *that* into something useful but I'll keep to my heuristic that served me very well so far and I'll write up whatever stage I got to, whenever it needs unloading and certainly not later than in 2 weeks' time.


  1. How? well, that's I suppose exactly how one ends up with all those .xml files just to tell what is in those other .xml files and then load some upfront and expect they know it all, too. Like will all madness, it's not that there's no logic involved, there is in fact plenty and at each step - it's only that the root is entirely misplaced. And yes, I am fully aware that those perceiving the roots of my own work as misplaced will therefore assign the madness label to my own work and my own self. A label though never did anything though, nor will it ever do by itself. 

Comments feed: RSS 2.0

5 Responses to “Those Zip Files of Eulora's Client Graphics”

  1. You know it's wandering.

    > it's unlikely that this will also match any indoor areas

    Nah, fuck it, all indoor areas are the same +-1000 x +-1000, and if the level designer has issues with it, can just isolate the intended 3x3 square in impassable objects. There's nothing saved in terms of "space" (which, being imaginary, ie worthless anyways, has neither cost nor value) through the application of logic (which, being rational rather than imaginary, is the premiere good and most expensive thing in Creation). Fuck that, let space be kneeling at our fingertips for no work at all, it's where it belongs.

    > perhaps to make a different type specifically for "indoors" and give all this additional information required

    Fuck that!

    > otherwise assume it matches something else through some convention forced thus by... lack of options

    Remarkably instructive showcasing of the drawbacks of involving idiots in human activity : they produce incorrect abstractions, these limp, half-way, broken testaments of their own internals. No doubt here's how that design meeting didn't go but should have gone :

    "Obviously the server will know and care what the client calls its dolly..."
    "And why ?"
    "Because... because... because mommy loves me!"
    "Maybe she told you so, but nevertheless, nobody knows anybody. Not that well. Mommy principally wants you to come to her funeral. In any case neither does she know nor does she care to know your innards, not even to the moderate degree you presumably were acquainted with hers."
    *spazz*

    In any case, it's entirely possible the one reason they even had the xml wrapper was this "only way to label things, which we need working quite in this way because we're narrow-minded idiots".

    > the ID of the darned object

    Does this mean that if there's 500 rocks in view, the client will have 500 copies of the rock mesh separately allocated in RAM ?

    As a subsidiary question : since files are sent with keccak hashes ; and since indeed there's no need or point to libraries -- wouldn't it make more sense to call the object by that hash ? Perhaps truncated, if need be (though I am not so sure there is), but in any case, a string for free.

    So in this model, the client builds a local list of objectids-keccakhashes equivalencies, and just uses that as a lookup table. (And, since we're discussing this : something very similar should probably occur re disk caching, keep the client from filling the user-allocated space with n copies of the same thing). Thusly the server needn't bother with what the client thinks or knows, and the client can discover experimentally what the server thinks and knows : "hm, when I asked for X it gave me M and now asking for Y I get the same M, well, time for an indirection layer here, we'll call it knowledge".

    but I refuse to bring in for this reason a whole set of useless "managers" and corresponding plugins and 1001 classes that do the same thing only now it's called labels and then it's called shadows and then again it gets its name changed so that a different "contributor" can sign at the top of the file.

    Heh. Since I spent some time in the simpswamps recently, here, I have a gift for you : https://crookedtimber.org/2020/07/20/economists-versus-epidemiologists

    "Free and open source" "collaborative models", or how to turn the civilised world into an african village on the cheap.

  2. Diana Coman says:

    You know it's wandering.

    Fixed, ty. Though I think it might have been a wondering sort of wandering!

    Nah, fuck it, all indoor areas are the same +-1000 x +-1000,

    Cool if they are all the same, only it's -500 to +500, lol. I should add also that at least the current client further "fixes" a water-level in that the "water" is currently a... sheet set at some coordinate. This however is entirely client-side concern anyway and otherwise at most designer concern, entirely nothing to do with the server.

    Does this mean that if there's 500 rocks in view, the client will have 500 copies of the rock mesh separately allocated in RAM ?

    For the sadness of it, as far as I currently understand it, that IS the way CS works really - if you have 500 objects, then each gets separately allocated - possibly because at any rate they are different objects from engine's point of view since they get rendered in different positions or something. The *only* part where CS focused on reuse is the *factory* ie having at least just one blueprint loaded for those 500 copies of the rock mesh.

    I hesitated to use the hash of the file for the factory name as it's then an additional value to pass around (and it can get to a rather long "around") each and every darned time an actual mesh is required. I'll have to think of this again as there are in fact a few options here: arguably the better in terms of performance would be indeed to use for factory name the hash of the zip file (not truncated, there's absolutely no gain to that); the easiest is to simply discard the factory as soon as one object is created - with the risk of course to have to reload that factory.

    Detailed client-design was not at all initially on my list though well, I guess it has got to be now. But in any case, no fucking lookup table for this, I'm sick and tired of how many lookup tables the gui-part had/still has and then it's yet another thing to keep in sync for no really good reason, so no, no need for an additional "the client builds a local list of objectids-keccakhashes equivalencies" At most, the gui will fucking request from EuCache this hash for a given object id every time it needs it since it has got to be in the tree-structure of the world anyway and moreover it is in fact EuCache that ultimately decides which graphics representation is going to be used for which object (recall, in principle EuCache can just plonk some defaults based on type if for instance it hasn't yet got the actual file from the server). So to the extent that the world-tree acts as "lookup table" sure, it can be but I don't see at all any reason for a separate thing, the information is there anyway.

    It is true that currently EuCache doesn't *also* index by hash as such ie there's another layer in there, namely the full path of the file - on one hand I even wondered if the name of the file shouldn't be its hash to start with but I didn't want to force this or rely on it.

  3. > Cool if they are all the same, only it's -500 to +500, lol.

    So they are in current Eulora. I am thinking about quadrupling the sector map, which I suppose got exposed here accidentally though I was reserving discussion for later.

    > "fixes" a water-level in that the "water" is currently a... sheet set at some coordinate

    Yes, but per-sector, which really should be fine. Any waters appearing at other levels should be part of objects (like a fountain or w/e).

    > At most, the gui will fucking request from EuCache

    So you keep your lookup tables in EuCache, that's fine :)

    > I even wondered if the name of the file shouldn't be its hash to start with

    This would make sense, really. I think it was even specified in some version of the protocol though possibly not a published/publicly discussed one.

  4. Diana Coman says:

    Heh, at least I find out new things if I make the time to do the write-ups!

  5. [...] Eulora's communication protocol can handle easily the transfer of any data files that the client might need, the size of such files still matters, mainly from a user perspective1 - large files take time to [...]

Leave a Reply