Ossa Sepia

Against my own expectations, it seems there’s little resistance left even in that mess of paws: it took less than a week to map it all out quite fully this time, discard yet another set of dead parts of it, cut its remaining tendrils and pretense to “independent plugin, part of a common trunk, not just some code in the client”, bring it therefore quite in line (though not to internal sanity, it still is what it is) and ready to be made use of, without crashing and without complaining that it doesn’t have graphics or skins or resources or whatever else it previously imagined that the user must provide. For instance, here’s the gui just using the system’s mouse pointer if nothing else is available and the help window gladly showing whenever it’s asked to show, looking as best it can with whatever is available at that time:

Having thus made sure that indeed, I can make paws do something useful after all, I’m literally staring at the pile of neatly sorted entrails of the whole thing and I can’t make up my mind for the life of me – *what* is even the desired use of this paws-paste? So I’ll first write down the brief description of what’s there and then I’ll spell out this generic question into several more specific questions that can hopefully be answered.

The current client’s GUI interface is essentially a lame and broken attempt at implementing the world hierarchy from a… visual perspective ¹. There’s a top “controlling” ² window that keeps track of all the others and does in fact the generic figuring out as to what window is the user interacting with, which ones should show and which ones should hide, all that sort of thing. Being of course a visually controlled world, the top window ends up being… that menu bar at the top (see? it’s at the top, right?) of the screen, with the inventory and the quit button and whatnot. It’s true that there is otherwise *also* a “mainwidget” thing but… that’s just the overall container and doesn’t do much at all. Don’t ask me why can’t that mainwidget be actually main and therefore control the rest, it just isn’t. And because those are just windows and the whole codebase at the time simply couldn’t do without managers, there’s on top of this top a pawsManager with its very own textureManager.

The pawsManager is meant as far as I can tell as the entry point to the paws itself – at least when entertaining the delusion that paws is or can somehow be separate of the rest. The textureManager is one of those endless glorified list-keepers: it keeps its own list of filename-resourcename equivalencies for all images that the GUI uses and otherwise interfaces load/find from the engine’s own textureManager, of course. The meagre claim to purpose it has beyond that of list-keeping would be that it packs in one place some format-related processing of 2D images that are supposedly helpful in some cases (though it’s unclear if there are all that many such cases in current use). Anyways, aiming to keep changes to a minimum, it can live as such and keep its list too, since I have no intention of saddling EuCore with any internals of the GUI/skin now. There’s of course a whole dance around in both pawsManager and its textureManager with translating strings (literally, the “localization” because apparently it makes somehow sense to shove that concern in there), defining “styles” (there’s skins and then there’s… styles; because one S is not enough and nevermind there’s no game worth even one capital letter, yet, we’ll add styles to the interface and that’ll be grand!) and making oh-so-sure that one is never, ever, under any possible circumstances somehow left without – gasp! – a loaded “skin”. Additional, even smaller concerns abound too but by now we get to the microscopic so I’ll let them be already, suffice to say that if I’ll ever need an example of pointless polishing of irrelevancy, I can certainly use that code.

Past the 2 managers and 2 top-widgets, there is the indistinct mass of …more widgets. Basically *everything* in the GUI is a widget, from one single slot in the inventory window to the whole inventory window itself. Here I can happily report that indeed, the “off with its dolly” task was most satisfyingly carried out: there were 2 (two!) classes on top of all the usual widget+xml defined just to *support* that and several tendrils sneaking out into non-paws parts too and oh, what joy to cut them out already!

Dolly cast aside and readily forgotten, the reduced pile of widgets is still a considerable pile and otherwise waiting for either full chop or partial cut, depending on what exactly turns out in the least useful in there. The “design” of the whole thing is that each widget has a code class (in client/gui mostly) that is meant to implement what the widget does (and even if it doesn’t – there still has to be this code present as a means to register and create the thing, useless as it might be) and an .xml file (in data/gui currently) that defines in principle the layout of the corresponding window to be shown in game, including size, position, contained widgets and used “resources” aka icons and other images. In practical terms this means that new or different widgets still require touching the client, there’s no “download this” and run with it really. At a stretch, the .xml layout files could be made perhaps part of the “GUI” .zip and so something that can be downloaded from the server and used but there’s a very tight coupling between the code and the .xml so that it’s unclear to me if there’s much point to it anyway. So Q1 of those more specific questions would be here: is the layout of various GUI windows something we even consider part of a “skin” definition and hence up to GUI designers to produce as part and parcel of offering a client skin for use?

Even after the above trim, there still are about 100 widgets with their corresponding 100 .xml files, so I would rather not list them in here – the list is anyway quite easy to get with a simple ls data/gui in the deployed client’s dir. The more specific question here is Q2: what windows do we actually need in the whole interface anyway? I can’t quite see the point of 100 windows and it really all seems to me more a matter of ill-thought “design” in the first place, fitting slots in places by hand and adjusting them to be just-so, the usual sort of “artistic” approach to it all. I rather think there has to be a small subset of elements that will do it all and better anyway but at least currently I don’t even quite know where to start from with it, as it’s not all that clear to me what sort of things we need in the first place. I can even see the point in saying it’s too early to bother with that and move on to implementing more of the functionality so that what is needed emerges from there but on the other hand the GUI has to start with something/from somewhere too.

Going through the list of main things as seen from a player perspective so far in Eulora, the Q2 above can get further refined: the most complex window in there is the chat window – does this even still make any sense or should this be only and at most a cli window in fact? Windows like the storage for instance were relatively ok in the beginning and then an utter pain when one has a full storage, reminding me quite precisely of the more generic windows-vs-cli computer use really – do we still want a storage window at all? And where does this stop anyway, as on one hand I can keep going and wonder if the inventory window is useful as such (perhaps it’s at least pretty to see them all!) or the container or …which of those 100 total, in the end? (Then again, I’m a cli radical apparently (hence my asking here), regardless of all the fun I had making textures, heh.)

Windows and widgets aside for a moment, the GUI graphical elements as such would be a set of tiny icons like arrow-this-way and that-way, backgrounds, buttons, scrollbars and the like. Currently even this is a bit of a mess in that there’s no standard set at all, but a proliferation of arrow-up and arrowUp that are two different arrows pointing “up” (and this is probably what they tried to solve with the whole “styles” layer of grime on top, myeah). Nevertheless and as boring as it sounds to do it, a basic set can certainly be extracted and then forced even upon the existing widgets, certainly. The potentially troublesome part though comes from a current ugly mix of actual GUI elements (e.g. buttons) and 2D representations of game objects (e.g. a key in inventory). The currently deployed client makes no real difference and effectively lists those 2D representations as GUI “resources” like all the rest (so for all the fun, one could in principle very well set their buttons to look like Omlette Du Disgorge, why not). While I have nothing against buttons looking like omlette-du-disgorge, game objects are a very different category from GUI elements and I don’t think they should be mixed at all. Even focusing solely on the very practical perspective, GUI elements get their representations (if any) from a skin.zip file (if it exists at all), while game objects are first of all a potentially infinite set, not known upfront and even potentially changeable on an item by item basis during play. So question Q3 here would be: does this separation make sense or is there some better way to go about sorting this out? (As I’ll need to sort out this part at some point anyway, it can’t really remain as it is.)

All the above out of the way and in the open for discussion, I can happily report that the client’s code continues to shrink for now as it increases in usefulness and otherwise there are still parts that will end up most likely discarded in the end, as they get increasingly left behind (the original net part comes first to mind). While there is still quite a lot to implement on both client and serverside, I think the next steps will have to be alternating client and server development, introducing the new things (finally!) and sorting out otherwise what troubles remain, as they pop up. In other words, there’s still a lot of hard work ahead but finally getting out of the swamps and it’s such joy to be out of them too!

1. The attempt is so visible and the whole thing is so broken from so many points of view that I don’t even have the heart to laugh at it anymore. It’s one of those things that are not even wrong yet.[↩]
2. The corresponding parent classes are even called exactly that, for bonus points of similarity of names being a great thing in code: controlwindow and controlledwindow.[↩]

Wandering all alone and growing quite bored of seeing the same familiar empty space, the twisted hopeful glanced all of a sudden an unexpected shiny item sticking out of the ground and in plain view:

What could that be and where did it come from? It wasn’t there before and there’s no guarantee it will remain there either but there is now finally the possibility for any number of such items to be anywhere at all in Eulora. And now that this part is done as well, the client has finally all it needs to render -though not yet to control or interact with- all three main types of 3D things: the world itself (terrain, sky, water), moving characters be they NPCs or players (Cal3D) and non-moving items (CS meshes). In all cases, the “graphical representation” is simply a .zip file. The contents of this .zip file can change and there’s plenty of flexibility in there with only a few “must-have” things nailed down for now – even those can change later on without much trouble, since a .zip file remains a .zip file otherwise and knowing how to use what is inside is entirely a client concern.

The current client 2.0 aims to look for and use what it knows, while not getting stuck on anything additional that might be in there. So it uses currently some very simply conventions regarding filenames inside those .zip archives and simply picks up the first matching files it finds inside: anything starting with “tex” is assumed to be a texture and its extension is assumed to match the format (.dds, .png, .jpg all work fine); anything starting with “mesh” is assumed to be a mesh description and interpreted depending on the context as either CS mesh format or Cal3D format (binary or xml depending on extension); anything starting with “skel” is a skeleton file; animation files are always exactly walk.caf, idle.caf (so far those 2 are the only ones the client knows about). As examples, here’s what the current test set of .zip files contain:

• For Outdoors (the filenames here are fixed indeed as they all have very specific roles):
  • basemap.png
  • heightmap.png
  • lightmap.png
  • materialmap.png
  • norm_1.dds
  • norm_2.dds
  • norm_3.dds
  • normalmap.png
  • sky.png
  • tex_1.dds
  • tex_2.dds
  • tex_3.dds
• For Cal3D:
  • skel.xsf – the file containing the skeleton in Cal3D format. This can be either xml (.xsf) or binary format (.csf) and the client will simply use whatever is provided.
  • walk.caf – the animation to be used when this character “walks”. This has to be currently in binary Cal3D format because reasons.
  • idle.caf – the animation to be used when this character is standing/sitting still. As for walk, this has to be in binary Cal3D format.
  • mesh_*.xmf – this hopeful happens to be made out of 26 meshes so there are 26 such files, simply numbered mesh_1.xmf to mesh_26.xmf. The client will just load as many mesh files as it finds, that’s all.
  • tex.png – the texture to use on all meshes. In principle there could be as many texture files as meshes so that each mesh has its own texture. It’s not a big trouble to adapt the client to use them but at the moment it just picks the first tex* file it finds and it uses it for all meshes in that .zip file.
• For CS mesh:
  • meshfact_1.xml – the description of the mesh in CS “meshfact” xml format (NOT library format, see below the more detailed discussion why not library).
  • tex_1.png – the texture to use for this mesh. This can be in principle *any* graphical format. The current client can and will happily use .dds, .png or .jpg just as well.

Compared to the previous iteration, the “outdoors” type bundled together water, terrain and sky because in practical terms they always go together indeed. There is here one part that I’m still hesitating about: while the size of “sectors” is fixed and doesn’t need to be therefore sent (or at least not at this level), it’s unlikely that this will also match any indoor areas that are to be separate areas as such and the trouble is that the loader needs to specifically set those. One way to go about this would be perhaps to make a different type specifically for “indoors” and give all this additional information required there in the euloran object itself (arguably even the outdoors type might be better off with some specified there, if or when they may change). The main question I have here is: what is fixed about outdoors and indoors (as the cs part has a lot of parameters and very few turn out to be of any use in the end)? More specifically, given the protocol specification, the size of an outdoors sector is fixed and as a result, the client can just create a fixed-size sphere for “sky” for instance and a fixed “cell” size for the terrain and so on. Is this size fixed for indoor places too or could those be smaller/larger?

As to the latest addition from the list above, namely the CS mesh items, getting them to work included a few ugly surprises that slowed me down a bit more than I initially expected. Nevertheless, as there’s otherwise less resistance left in that paste than at the initial stages, it didn’t take all that much to whip it up into something useful. First, the format I had used previously based on existing client’s “art” files turned out to be less than ideal for a rather unexpected reason: the CS loader would happily load it but it wouldn’t provide any means to obtain directly a way to reference the loaded mesh – it assumed the caller would know that from somewhere else or otherwise find it somehow ¹. To fully get this: CS insists on referencing everything by “name” hence effectively by string-ids and the names of loaded stuff from a library file are …specified in the xml but otherwise not returned by the loader itself! So to find a factory or a mesh, the code would need to either read it specifically from the xml (ugh!) or otherwise assume it matches something else through some convention forced thus by… lack of options (because no, I don’t want to have the server now be concerned with fixing the names of meshes clientside, wtf). The solution I found to this is to not use “libraries” at all since anyway each of them contains exactly one meshfactory and then load that factory with whatever name there might be in the xml but *change* it from the code to the name that is useful to the client (and that’s …the ID of the darned object, ofc, what else, only made into a string to fit CS’s expectations, yes). It took a while to figure this out, mainly because finding the way to simply rename things is made… interesting by means of Object Oriented (OO) obfuscation layers so that the code can end up at times like this: meshWrapper->GetFactory()->GetMeshObjectFactory()->GetMaterialWrapper()->QueryObject()->GetName(). Count the layers and laugh at it but know that it’s not even the worst that OO can do!

Once the above sorted, there was some more work to cut that GEM crap on items too and then to follow up and look into all the resulting, inevitable crashes given the spaghetti pointers running all around. At least I used the opportunity to look also into the way that PS used to do “effects” – it’s a useless pile of crap on top of some tidbits of usefulness from the underlying CS, not that it comes as any surprise. The surprising part was that “labels” (e.g. names of characters) used to be… “effects” and thus bringing in the whole crashing monster of stuff. Well, labels presumably will be useful so they are on my list to add as part of sorting out the 2D part of the interface but I refuse to bring in for this reason a whole set of useless “managers” and corresponding plugins and 1001 classes that do the same thing only now it’s called labels and then it’s called shadows and then again it gets its name changed so that a different “contributor” can sign at the top of the file. Such contributions that one is way, way better off without any of them.

At any rate, having now sorted non-moving items as well, the next step really has to be looking into the 2D GUI stuff more closely, hence… PAWS. There’s no estimate I can really make (only heavy guesstimates) on how long it will take to make *that* into something useful but I’ll keep to my heuristic that served me very well so far and I’ll write up whatever stage I got to, whenever it needs unloading and certainly not later than in 2 weeks’ time.

1. How? well, that’s I suppose exactly how one ends up with all those .xml files just to tell what is in those other .xml files and then load some upfront and expect they know it all, too. Like will all madness, it’s not that there’s no logic involved, there is in fact plenty and at each step – it’s only that the root is entirely misplaced. And yes, I am fully aware that those perceiving the roots of my own work as misplaced will therefore assign the madness label to my own work and my own self. A label though never did anything though, nor will it ever do by itself.[↩]

There’s still quite some work to do before having a fully working client as well, but I can happily report that the most important part for that auto-updating version of the client is indeed in place and working: an initial prototype client 2.0 can now start happily with nothing at all (no keys, no graphics, no knowledge of the world at large) and then generate keys, obtain an account, retrieve (and check) sector and model files, load them when it got them, even let the resulting euloran move about and admire its reflection in the wonderful water by the rocks:

For future reference and current unloading of a fresh set of troubles sorted, here’s the list of main updates done on the clientside:

1. EuCore (data acquisition logic, Ada):
   1. Setting RNG source via config file: whatever string the user provides in there (e.g. “/dev/urandom” or “/dev/ttyUSB0” or “/silly/ass”) will be used as source of “random bits”; NB: the indicated source, whatever it might be, is considered already initialised in whatever manner is required, since it makes no sense at all to try and initialize from EuCore’s code a source that can be set to anything at all. For all the apparent “easy” label on this additional knob, there was a significant headache to make it right, since the RNG source is used from both C/CPP code and Ada code so that it has to be set across language-borders, not to mention in the precise direction previously-avoided-to-keep-things-neat-and-less-confusing. Not that I can recall *any* time when I got to keep over long term any of those parts that were easier, it’s just not my thing or something. What can I say – it’s done now and it’s working (the default is urandom).
   2. Setting a “sync” interval – meaning how often the user wants EuCore to attempt to resync its local data with the server. This is important mainly as a means for the user to find the compromise they want between generating traffic (which has a cost) and otherwise keeping their local data cache up to date (since out of date information can presumably make play rather less than ideal). Worth noting here that this “sync” interval is important for movement update and for requesting World Bulletins regardless of other triggers that might apply.
   3. Creating account and obtaining BOTH sets of Serpent keys from the server – this got a refactoring and update to reflect the clear separation between EuCore that does *all* data logic, acquisition, storage and so on (hence it’s the only part that can request creation of RSA keys for a new account for instance) and what is simply a dumb GUI – the former “client” that will focus exclusively on using and presenting data that is present, nothing more. There was also still pending – due to previous lack of clarity at least in my understanding of this part – the issue regarding the initial set of Serpent keys for communication with the client. While the protocol allows for the client to generate those and send them to the server, at the moment the client will simply keep sending request for an account until it receives those too. Perhaps at a later stage I’ll get back to this and implement also the option + knob to generate them locally and send them to the server but this can certainly still wait.
   4. Updated EuCache (the actual data storage structure that provides thread-safe access) to reflect the latest, streamlined data model.
   5. Updated the data acquisition logic to actually send requests to the server for each and any bit of missing data (be it object or file) as soon as it becomes aware it’s missing. This includes proper packing of file requests in as few messages as needed (instead of firing up the wasteful one message for each request).
   6. Updated handling of objects to fully update local data cache correctly (in short: adding those in server’s view and not in client’s view, deleting those in client’s view but not in server’s view, asking for anything missing and fully walking the structure at regular intervals, to ensure re-sync).
   7. Updated handling of file received messages so that chunks are saved as received and the Keccak hash of the file is calculated + checked against expected value at the end, deciding on whether the resulting file is kept and made available to the GUI or otherwise deleted (+ new request sent to the server, to resend it). NB: the approach here is very basic for now in that chunks are literally saved in the order received and therefore it’s enough to have 1 message lost or out of order to result in an incorrect file and therefore repeat the request. I’m sure that this can be made better – not so sure though that it’s worth the trouble to sink time to make it better *right now* and so there it is, low hanging fruit for anyone who wants to improve their client.
   8. Updated interface towards the GUI to reflect the new understanding that the actual needs of the GUI are way, way simpler than the full complexity that the protocol and more generally the world structure allows (and here’s an example of where the doing-it-all kind of made me stumble because even when it’s the “same thing”, it’s from different perspectives, of course and so I need to keep them all in mind at the same time and change swiftly between them, oh joy). At any rate, now the interface provides the sort of packing that the GUI finds useful, while taking otherwise advantage of EuCache’s data structures that reflect the actual hierarchy of the world. That’s pretty much the part that wasn’t quite that obvious for me until now – the dumb GUI is in fact way, way flatter than the actual world as it were and as such, there’s really no point to make the GUI more intelligent than it can handle. Let the GUI be an idiot that is forever concerned only with “what can I see *right now*” and that’s that, everything all of a sudden starts to fit, too.
2. GUI (the former “client”, cpp) side:
   1. Refactored the SMGData interface to work with the latest EuCore interface and provide a single point of access to all data, from objects to files. Sorted out here some unexpected issues with importing constants from Ada into C as well.
   2. Nailed down a working model for this whole GUI stuff: there’s a logic (such as it is) part that will then make use of all the paste that used to be pompously “the client”, from pscelclient to psmovementmanager and whatever. Sorting this out was a pain but the result is a very welcome insulation from the rapidly-spiralling-into-madness mess. In other words: while the mess is still there for it will take ages to fully eradicate, one doesn’t have to work neck-high into it *at all times* anymore! At times, a descent into that is still needed but I’m finally at the stage where it’s “at times” as opposed to “all the time”.
   3. Updated and refactored loader actually capable of making full use of .zip files that were acquired *after* client startup! This was such a pain to get working, as the virtual file system thing is rather poorly documented and quite clunky at times. Nevertheless, it … works!
   4. Prototype implementation of “soft reset” – essentially the PS code insisted that one needs to go through 1001 things in 10001 places to just clean up if/when the sector changes for instance but then it turned out that CS has this nice and shiny red button to …DeleteAll! And once that is done, PS can go and get stuffed since there’s not much it can do anyway. So the prototype so far aims to simply do such a soft reset if something crucial like the sector or the player themselves suddenly change – it would seem possible that we don’t actually have to end up with the sort of Windows-style-restart-the-client, after all.

Aside from the above, there are many other smaller details and issues sorted as they appeared – some of them might even further change as everything takes further shape. So far though I’m satisfied that the former client-code is finally just about paste-enough to be of some actual use instead of unrelenting pain and misery.

The next steps are for now to iron out some remaining weirds in the current minimal prototype and to further develop the actual use of data from the GUI since for now there’s really only the sector+player check, retrieval and loading of corresponding graphics assets but nothing further connected. Local movement works, of course, but it’s all local and moreover, the GUI doesn’t yet request from EuCore anything *other* than the player itself and the sector where they happen to be.

On the write-up front, there’s the structure of the .zip files for sector + cal3d models to write and publish. All is quite clear and neat there but I need to find some time for it and I might possibly do first the loading of CS meshes too.

On the iffiest client-side part remaining, the paws widgeting mess looms largest. At the moment I can’t tell if or to what extent I can sidestep it/any of it but I’m certainly not keen at all to get into that swamp even an inch more than is absolutely and unavoidably required.