Ossa Sepia

August 6, 2019

The Widgeting Paws of Pointerism (Notes on Graphics in Eulora, V)

Filed under: Coding, Eulora — Diana Coman @ 5:02 p.m.

It took indeed more than a week and I'm rather surprised it didn't actually take even longer than two weeks to get *anything visible* out of it. It's all one strand like Stan's spittoon after all, so there's rarely just "one sip" that doesn't bring in the whole rot with it and I have to fight over and over the urge to just burn it all down and start a sane thing from scratch1. Nevertheless, a lot of deep breaths and assorted cussings later, here's a brief summary of the main achievements, in the order they were made:

  • some useless crap was removed and the whole thing made to work without (e.g. the doll with its dependence on a special sector in the map file);
  • the initial loading and "pre-caching" of meshes, textures, materials, maps and 2D stuff was excised (there are still the shaders remaining though as well as 1001 "preferences" in as many files in yet as many different places, sigh);
  • the actual handling of 2D icons and GUI images in general (e.g. buttons, borders, arrows, window decorations) aka the PAWS was dissected and exposed and roughly mapped as ugly as it is;
  • as recorded proof of breaking PAWS and because of its shameful lack of proper way to load images on actual demand, the whole thing has had a run with the stupidest "mouse pointer" available in there and otherwise with absolutely no 2D images at all for the very lulzy effect that it even looks better in some ways than with all of the various pieces of "art", here it is:
    That snot on the screen is the mouse pointer on the black background starting screen.

    That snot on the screen is the mouse pointer on the black background starting screen.

    PAWS's shame: the better-looking interface with no images whatsoever (clicking the invisible widgets still works!)

    PAWS's shame: the better-looking interface with no images whatsoever (clicking the invisible widgets still works!)

The path that led in the end to the above started with me looking - as initially planned - at how the character (as moving sprite) is handled or rather mishandled, consider: there is the CEL layer that supposedly provides on top of CS the sort of structures needed by games to store and manage the various entities and objects; PS however "uses" it in principle but in practice it still mainly does its own thing anyway (and by now I strongly suspect that this pattern is simply because nobody ever spends the time to understand what they use); specifically, the PS way is to have a pscelclient class that supposedly manages game entities on client side (shouldn't CEL have been enough for that, you ask? most probably yes but that'd have taken someone to figure out first exactly how it works and who has time for that!!); but then there is *also* a pscharapp class that handles the appearance of the character (apparently that's too low stuff for "management" to handle) and there is also a psclientchar that moves about equipped items because this is yet neither management nor appearance but absolutely something separate - though you'll be hard pressed to notice much separation in the code for all this theoretical separation; and to top it all, the inventory and doll views that precisely reflect character's appearance and equipped items are actually the domain of something else entirely - the PAWS; and the PAWS (which is supposedly so separate as to bother to be a plugin by itself) in fact messes about with pretty much anything and everything from the engine to shaders and textures and 2D images (which are still loaded as... textures, but those are textures-this not textures-that) and xml-defined maps and hardcoded paths to all sorts of files it expects - nay, demands - to be present and just-so and at-start but oh-so-configurable. At which point it became clear that the PAWS mess is indeed such a large part of the whole mess that I'll simply have to get to the bottom of it just in order to move further at all.

Looking at PAWS more closely revealed that it's meant to be a GUI system. This initially rather puzzled me because there IS in fact already another GUI system in there, one that PS uses/relies on as a dependency, namely CEGUI. Then again, the PS style is to "use" something by simply sitting on top of it and mushrooming a lot of managers with hardcoded paths and expectations that they know the whole possible world upfront so there should be no surprise at PAWS, which is precisely this sort of thing - it mushrooms widgets and pointers rather than something-else and pointers, that's about all the difference from all the other PS masterpieces.

Masterpiece aside, what IS this GUI system PAWS? As far as I can tell, it's a lot of widgets so well organised that there are no less than 3 methods "FindWidget" because the 1st one (defined in pawswidget) checks the current widget and its children only, the 2nd one (in pawsscript) checks also the parent of the current widget while yet the 3rd one (in pawsmanager because yes, of course there are all sorts of managers) simply delegates it all to "mainwidget" aka a sort of calling the 1st one from the top of the tree (but no, they couldn't possibly notice that it's meant to be a tree and there are well-known algorithms for searches in a tree). Anyways, the fun part with widgets is the way in which they are defined really as each widget has an .xml file (in data/gui) that supposedly defines appearance only (though more concretely it's about which elements it contains and on what position) and a .cpp file (in client/gui usually) that supposedly defines behaviour. The funny bit however is that both of those, for all their pretense of clear separation (they even are in totally different dirs, yes?), quite directly reference both one another's bits and pieces AND global, all-of-a-sudden-made-hardcoded parts: the appearance references by name other widgets (that it uses for specific roles and in specific places so it's not just a name really) and "resources" aka pre-loaded images; the behaviour code in turn has to reference various components of course. And in all this, apparently nobody noticed that before you can talk of behaviour and of appearance, there has to be somewhere a *structure and nature* of the thing that actually is the core and therefore the driver of everything else... Then again, what structure and nature and such silly notions, when the whole thing is bent on defining anything and everything by its current, temporary position or at best some role it might fulfill for a while2.

The widgeting PAWS further has all sorts of notions regarding appearances: there are "skins" and "styles" and "resources" and they are all both hardcoded and expected to be fully known upfront. The Great Paws knows not only all the world that is but also all the world that could possibly be (the world as it defines it is all nothing but labels anyway so it makes sense in a way). And it enforces this by whining and even wagging its finger at the user if it ever finds that a file with what it *knows* is the "right" name is missing from its expected place, quite literally giving orders to the user, as the message spit is this:

"ART ERROR: PawsTextureManager loaded the image %s which was missing from the imagelist.xml and was loaded on demand. Add the image there!"

I'll add to the above: and blow your nose and stand up straight and cut your nails shorter! Imagine that, being such a shitty user that the poor, poor PAWS had to actually load an image on demand (how degrading!!), you nasty user, you! And note that this "on demand" is anyway the sort of "you demand what I'm willing to do or no dice" not any sort of actual demand: it only means it deigned to look (because it had to use that image) in the *same place* it looked initially (when it tried to load everything that there should ever be) just because you are the sort of shit that failed to put the image there when it looked the first time, you know? What sort of user does such a thing to a poor piece of code, you tormenter, you, shame on you!!!

Before you get your hopes up that there is at least some sort of "loading on demand", bash those hopes one over the head: for one thing this "on demand" still relies on the predefined list of what there *can* ever be so that if you request a "resource" it doesn't know about than tough luck, it won't try to look for it, it's not that stupid to do something useful; for another thing, there is no way to actually call this on demand: it's just a way of doing rigidly the same thing but expecting a different result because meanwhile the world should have better come to its senses and fit the expectations already, what.

If the above sadness is not enough, add to it this significant bit: PAWS considers itself a plugin and as such entirely separated from the actual client and therefore unable to use whatever the rest of the client knows. Sure, in practice this separation "works" by means of the main client class simply passing pointers to PAWS on creation - why use CPP if not for pointerism, after all. Moreover, the added benefit of this separation is that PAWS also gets therefore to have its own set of repeated XML parsing of all sorts since it can't use the already-repeated XML parsing that the client anyway suffers from. And finally, it makes somehow perfect sense to "separate" the main client from the GUI system seeing how they both use the same graphics engine anyway and moreover the main client loads some images as textures while PAWS loads...well, some images as textures, yes. But they are not the same "class" of textures, so that's significant separation, see?

Besides deep red, I tell you what I further see: since I have no intention to pass even more pointers on or otherwise to force this "plugin" nonsense on everything just for the very specialness of PAWS, it's way more likely that the paws will be clipped more to size and therefore will take their place as yet another of the many classes inside the main client, possibly together with all the rest of very-special-plugins if need be (of course the dependencies are not that simple). I'm still pondering this, mainly because the idea is to touch as little as possible the code rather than as much as it takes to clean it (mainly because clean it won't be anyway). But one way or another, I will have to find some reasonable way to bring PAWS in line with the rest and have it go and load resources from wherever (and whenever) it is told to. This might even require more discussion in S.MG's boardroom but at the moment and as a result of all the adventure above, I have at least a much clearer idea as to what the 2D part is all about so there is therefore a point to even start the discussion from.

  1. While it might *seem* faster that way, it's not, nor does it really make sense if one calmly considers everything involved. Urges like this are simply focused on getting rid faster of the current set of problems and do not care nor stop one second to consider the alternative set of problems that are bought by such "solution". So yeah, urges make very poor guides, how surprising. 

  2. And no matter how much I try to *not* see it this way, it always is the case that the closer one gets to the "how things look" and therefore in this case to graphics, the more confusion and more monkeying there is. 

July 29, 2019

Overview of FFA Ch1 - Ch19

Filed under: Coding — Diana Coman @ 9:51 a.m.

Over the past few days I've taken the time to finally go through all the currently published chapters of FFA (1-19 with some bis and a,b,c on the way too) and got as a result a clearer idea of the whole thing as it builds up from basic definitions of large integers to the Peh interpreter machine with its own registers, two stacks (data and control, respectively) and the Cutouts mechanism for controlling access to sensitive data. While initially I was planning to add to my previous notes on FFA Ch1-Ch4, my experience of getting back to FFA after a significant break pointed to the need for a higher-level overview of what-where-why even before looking for such detailed illustrations of the code involved. So here's my current overview of FFA - open to any criticism you are able to throw at it, as always - consisting first of a brief list of main approaches in FFA and then the 4 "layers" that I see to the whole thing so far.

Main FFA Approaches and Algorithms

  1. Strictly non-branching on input values and therefore practically the death of "if". No ifs, no buts, so what's left is MUXes1: essentially the "if" goes down one level, from a logical switch (which operation to perform) to a mechanical one (which buffer to copy the result from); the same operation is always performed but its result is not always used.
  2. Strictly constant-time execution for any given inputs size. This means in practice strictly worst-time, of course, as it's always the case when you require all to be the same. The only way for all to be equal is by making them all equal to the worst, no way around it. Then again, some people's worst turn out better than other people's best so take "worst" with a pinch of context, as always.
  3. Comba multiplication algorithm: calculating sequentially the *columns* of a product; additional optimisations for the squaring case (mainly avoiding calculating the same products twice).
  4. Karatsuba multiplication algorithm: calculating the product XY as a combination of cheaper operations (additions and shifts preferred by far to multiplications) on the half-words of X and Y; additional optimisations for the squaring case.
  5. Barrett's modular reduction: calculating X mod M by means of an approximation using the "Barrettoid", B, that can be precomputed from any given module; X mod M is approximated first as X - M*B and then adjusted by at most 2 subtractions of M (i.e. the error of X - M*B as approximation of X mod M has only three possible values: 0, M or 2M).
  6. Stein's binary GCD: rather quasi-Stein as it's adapted to FFA requirements, most notably the constant-time execution requirement (by iterating over the full number of bits rather than stopping when the smallest number is 0).
  7. Miller-Rabin compositeness test: an adapted version of MR, forcing any given witness into the acceptable range and providing a "one-shot" test for any given value N and witness W.
  8. Peh interpreter machine: basically the "cryptographic machine" that the user gets to push the buttons of; Peh has a data stack and a control stack, a set of registers and an instruction set including the ability to define and call subroutines and loops as well as to define Cutouts aka a protected area for sensitive data.

FFA Layers (from bottom up)

  1. Representation of arbitrarily large integers as fixed-size (i.e. specified for every run) arrays of machine-words (with iron-specific size of machine-words changeable via knobs in the code). In FFA terminology, such integers are called "FZ".
  2. Basic arithmetical, modular arithmetical and logical operations on FZ entities in constant time and handling any overflows/underflows. This part builds up from the most "naive" approach (aka most straightforward if rather costly) dubbed "egyptological" through successive refinements (optimisations of the original algorithm followed by radical changes of approach) until the final result uses Barrett's reduction for modular operations, Karatsuba with Comba core-case (i.e. below empirically-established thresholds) for multiplications and a variation of those (Karatsuba + Comba + specific threshold) specifically optimised for squarings.
  3. Primality-related algorithms working in constant time on FZ entities: Stein's binary GCD2 and an adapted MR3 that can adequately handle any value of a given witness (by forcing it effectively within the valid range).
  4. Peh, the finite-tape, dual-stack and cutouts-enabled interpreter. While I still need to turn this inside out a few times more, I can tell already that I want a Peh-iron!

And while there are still a lot of experiments and timings and playings to go through with all the FFA, I'm happy at this point to sign the remaining chapters as I went through them with pen in hand and as far as I can tell they do what it says on the tin (not to mention that I certainly would use FFA and in fact I'm itching to drop already the shitty gpg, ffs!):

  1. Multiplexer aka a way to select only one of several inputs. 

  2. Greatest Common Divisor 

  3. Miller-Rabin 

July 24, 2019

Naked Models (Notes on Graphics in Eulora, IV)

Filed under: Coding, Eulora — Diana Coman @ 11:30 a.m.

What's the first change you'd do to an empty-headed character such as Testy? Why, chop some bits off, change his sex and get him naked at least, even transparent at that, since he's pretty much just as tedious and boring as all children dolls 1. Of course the first attempts failed to produce anything visible because it turns out that using Cal3D sprites in Eulora's client goes through several layers of inmisdirection: Crystal Space (CS) wraps the Cal3D code in its own SprCal3D plugins and factories and whatnots; on top of that, there is also the rather elusive Cel (Crystal Entity Layer) that is meant to be a game-specific part of CS keeping track of game entities; and messing within and across all is Planeshift's (PS) own GEM concoction interlaced with delayed loaders and Dead-Reckonings2 and cloning of materials (why? no idea yet) and surprising, undocumented transformations. Still, after a while of hacking through all the above, I could at least get some parts of Testy showing... if lying down rather than upright:

Spot the bottom in the grass!

Spot the bottom in the grass!

The prostrate position makes at least sense in that "movement" also looks a bit like some swimming look-alike: parting the legs and advancing sort of thing. Then again, it turns out that using a different model (e.g. Cally) makes it even more interesting - the model is upright when waiting but it dives straight back to the horizontal as part of moving and it furthermore cares not one bit for whatever movement animation one loads. So there's a clue for another day - current movement needs a good bashing over the head and a deep dive to figure out what's all this nonsense. But until then, here's naked Cally who borrowed Testy's skin for a spell:

What bits is she missing?

What bits is she missing?

Anyway, since position of the model is always a matter of applying one transformation or another, I went ahead and applied the obvious transformation, 90 degrees and upsy daisy on her feet:

Upsy-Daisy with muddy skin!

Upsy-Daisy with muddy skin!

And funnily enough, once the 90 degrees rotation is applied to the factory (i.e. presumably to ALL entities built out of that factory) the Testy model remains upright even during movement without any perceived change. On the other hand, the Cally model is upright without the 90 degrees rotation but nevertheless dives down for movement and pops back up as soon as movement is done. So it would seem that movement has its own assumptions that it doesn't bother to make public, how lovely. Seeing how it's most probably to do with the model's initial position or similar, I might even need to end up again with the full environment, Blender and all installed to get to the bottom of it and probably see exactly what or how the exporter is also broken and specific. Such joys to come but for the moment they are not yet here since the more pressing need is first of all to disentangle some more the PS+CEL+CS+CAL3D mess regarding animated entities so as to actually be able to *use* the walk/fight/move animations rather than just "add" them and moreover to be able to add and remove on the fly as many and as diverse entities as wanted, beyond the "main character" guy. And all this is getting deeper into PS swamps so it's likely to take more than a week indeed - ideally by then I'll be able to cut off at least some parts of the existing code (precaching and preloading especially) and lighten the client start a bit if nothing more.

  1. Changing graphics is this exercise in tediousness and pointlessness, it eerily reminds me of playing with dolls, just one step lower than that since those are just images of dolls, fancy that. And playing with dolls was utterly boring even when I was 5! It's like an endurance test along the lines of watching paint dry. Tinker with this and tinker with that, all for "the looks" of it and can't even satisfyingly throw it at anything to hear it break. 

  2. Not kidding, although it's way less exciting than the name they had the chance to land on: it's mainly looking to see if your character falls down and breaks their neck. 

July 17, 2019

The Mirror Land (Notes on Graphics for Eulora, III)

Filed under: Coding, Eulora — Diana Coman @ 10:26 a.m.

Getting some actual landscape as in getting the height right turns out to be only half (the reasonably well-working half) of the terrain task: once I had in place at least the generic mechanism for creating factories and meshes, the terrain factory required only a few iterations, false starts and mandatory dives into CS's entrails to get working and save Testy from his usual fall into the sky-sphere:
Terrain generated from heightmaps only (no mix of materials, only base material).

After setting up the lighting a bit better too, it looks even more reasonable (next to the crater that is currently empty, yes):
Base terrain from heightmap.

The truly frustrating bit turns out to be the painting of the landscape in the desired materials. For the relatively easy part, the Terrain2 construct of CrystalSpace provides a "terraformer" that can be fed a "material palette" (i.e. a set of materials to use) and an indexed image - as a result, the terrain will have a reasonable mixture of the materials depending on heights and an ok fading of the materials into one another where they meet so it doesn't look that terrible, especially once all normals are in place too so that lights stand a chance to work:
Terrain from height map *and* material palette with 3 materials: grass, sand and rock.

The trouble however is that so far I haven't quite managed to get to the bottom of turning down the reflective mirror-like surface of this landscape! In theory, materials have those 2 light-reflecting properties, namely the extent to which they are "diffuse" or "specular" (i.e. "shiny"). Still, for all my futzing around with those for all three materials (sand, grass, rock), the observed effects are pretty much non-existent. So much for the theory and welcome to this graphics thing of tinkering. To see clearly what I mean, here's a rather surreal rendering with the ambient light turned on to higher levels so it's more obvious (note that the effect is precisely the same even with low ambient light - it's just harder to realise what it is exactly; similarly and as expected, one can mask the effect to the point that "it's fine" by turning on the fog button - it makes sense since it dulls the ambient light, I'd say):

On the positive side, the futzing with lights and whatnots is really something of less concern for me at this moment since I'm not that interested in obtaining an image *just so* - my goal is in figuring out *what* sort of things one can set, what do they do and how to set them while the game is running. In this sense, at least the basics of the terrain seem to be in place for now: heights are read from a heightmap file, the materials are indeed used according to heights, the rest is for another day. I suspect though that in usual tradition, the only real solution here is to dig in the end in that shaders pit as well since it's most probably a shader thing: the terrain object uses a different, terrain-specific shader and for some reason the defaults for that end up producing more of a mirror than a soil, at least given *everything else* (such is graphics, everything and anything can be the culprit). Note that I specifically used and set precisely the same parameters for materials as they are in the client currently in use - so it's most probably something else/somewhere else that I'm not setting just right for now.

The next step now is to figure out animated meshes too, hence Cal3d objects. So in the next episode, it's Testy himself that will change and get created directly from code. Once I have at least one way to do this too, I can go back a bit and look again into integrating them into (or extracting them out of...) the rest of the PS infrastructure. Then again, before doing that, I might still need to dive in and figure out the shaders in more detail to extract some way of specifying them without xml and directly from code too, perhaps. That promises though to be a lot of figuring out and additional mess so depending on whether it can or not wait a bit longer, it might not be exactly next in line - basically I have lots of work competing for my attention, a sort of inverted looking-for-work, it's the... (lots and lots of) work looking for me!

July 14, 2019

A Working Cuntoo Install on AMD FX-8350 (with script)

Filed under: Coding — Diana Coman @ 10:20 p.m.

Since the relatively recent discovery that Cuntoo work is apparently not really going anywhere for lack of more public testing and reporting, I'm doing my bit and publishing what I've got so far on installing Cuntoo on a local box and prodding it for a while. First, to introduce the box in question, so you can easily replicate it:

  • AMD FX-8350, 4GHz, Octa Core
  • Gigabyte Ultra-Durable GA-78LMT-USB3
  • 2*8GB DDR3 RAM
  • 1TB Seagate Barracuda

Since I wanted to do this from scratch to make sure it's easy to replicate, I ignored my previous Cuntoo-compilations and I made myself first a bootable USB-stick with Gentoo's livecd. Initially I took the minimal install_amd64_minimal_20190703T214502Z.iso but that proved iffy to use because it was apparently a bit too spartan for my requirements (esp. no make on it). So I traded space for time as it were and used instead the much larger livedvd-amd64-multilib-20160704.iso (2.2G vs 292M for the minimal iso). Perhaps this was quite overkill and there is something in between that does the job just fine so if you find/have that, just use it - the bootable USB-stick really is just a way to "get in," nothing more. Note that on some .iso images you might need to run isohybrid first if you want to use them to boot from USB (rather than burning them on a CD/DVD). As for making the bootable USB stick, that's as easy as using dd to transfer the .iso to your USB drive - do be patient if your .iso is big as dd is silent.

Armed with the bootable USB stick, I popped it into one of my USB 2.0 slots and turned the machine on. If you have the same setup as mine, the F12 key will give you the boot menu (or Delete for the full BIOS menu) and you'll need to select the USB stick *from the hard-drives* (you'll need the PgUp/PgDown keys and not the +/- as some docs mislead me). Once the live-dvd Gentoo image boots, you can proceed with the Cuntoo-related tasks, in order (note that you'll be doing all this inside the live-dvd image so if you reset, *the environment will reset too and your changes will be lost* so you'll need to start from the beginning again. With many thanks to Hannah for her mini-series of posts detailing her ordeal with Cuntoo and therefore inspiring my script-making right from the start, here are the steps and the relevant script parts:

  1. Add to your /etc/hosts file the IPs of any websites you'll need - the exact list depends on what you choose to do at each of the following steps (e.g. where do you take your V from) but at the very least you'll need the IP1 for trinque.org to get the Cuntoo tarball!
  2. Obtain and import any gpg public keys you need - similar to the step above, the exact list here depends on whose stuff you use but as a minimum you'll need Trinque's key, of course. As I've used only Trinque's and my own stuff, my gpg-importing bit of the script looks like this:
    # gpg keys
    curl -v "http://ossasepia.com/vpatches/diana_coman.asc" -O
    gpg --import diana_coman.asc
    curl -v "http://wot.deedbot.org/FC66C0C5D98C42A1D4A98B6B42F9985AFAB953C4.asc" > trinque.asc
    gpg --import trinque.asc
  3. Get and install a working GNAT. My choice here for expedience is to use the frozen Adacore version that I'm hosting on this site. Correspondingly, the relevant part of the script grabs it from my website, checks the provided sha512sum on it and installs it (if the GNAT dir doesn't already exist from a previous run perhaps):
    # GNAT
    if [ ! -e $GNATZIP ]
      curl -v "http://ossasepia.com/available_resources/gnat-gpl-2016-x86_64-linux-bin.tar.gz" > $GNATZIP
    if [ ! -e $GNATZIP.sha512 ]
      curl -v "http://ossasepia.com/available_resources/sha512sum_gnat-gpl-2016-x86_64-linux-bin.tar.gz.txt" > $GNATZIP.sha512
    if [ "$(sha512sum $GNATZIP)" = "$(cat $GNATZIP.sha512)" ]
      echo "Matching sha512sum for Adacore-GNAT."
      echo "FAILED sha512sum test for Adacore-GNAT, stopping."; exit
    if [ ! -d $GNAT ]
      tar -xvf $GNATZIP
      cd $GNAT
      sudo ./doinstall
      cd ../
      echo "GNAT dir already exists so GNAT is assumed installed! Skipping GNAT install."
    PATH="/usr/gnat/bin:$PATH"; export PATH
    echo "Updating PATH variable to $PATH in order to use GNAT."
  4. Get and install your favourite V. Make *sure* it can be found (e.g. update your PATH variable with V's location). Once again, since I can rely on my previous work on packing V (namely mod6's implementation of V), my life is that bit easier now and my script relies on my website to get and check everything:
    # V
    if [ ! -e $VZIP ]
      curl -v "http://ossasepia.com/vpatches/starter_v.zip" > $VZIP
    if [ ! -e $VZIP.diana_coman.sig ]
      curl -v "http://ossasepia.com/vpatches/starter_v.zip.diana_coman.sig" > $VZIP.diana_coman.sig
    echo "gpg --verify $VZIP.diana_coman.sig $VZIP"
    gpg --verify $VZIP.diana_coman.sig $VZIP
    if [ ! $? -eq 0 ]
      echo "FAILED sig check on $VZIP, stopping."; exit
      echo "Sig check fine for VZIP"
    if [ ! -d $VDIR ]
      unzip $VZIP
    cd $VDIR
    chmod +x build.sh
    cd ../
    PATH="$PWD/$VDIR:$PATH"; export PATH
    echo "The PATH var for using gprbuild, vk.pl, ksum, vdiff and vpatch is: "$PATH
  5. Get Cuntoo itself (note that it's ~800M so depending on your connection, this step may take a while):
    # Cuntoo itself
    if [ ! -e $CUNTOOZIP ]
      curl -v "http://trinque.org/$CUNTOOZIP" > $CUNTOOZIP
      curl -v "http://trinque.org/$CUNTOOZIP.sig" > $CUNTOOZIP.sig
    echo "gpg --verify $CUNTOOZIP.sig $CUNTOOZIP"
    gpg --verify $CUNTOOZIP.sig $CUNTOOZIP
    if [ ! $? -eq 0 ]
      echo "FAILED sig check on $CUNTOOZIP, stopping."; exit
      echo "Sig check fine for $CUNTOOZIP"
    tar -xvf $CUNTOOZIP
    cd $CUNTOO

Once all the above finished without any error or problem, all you need to do is to run the bootstrap.sh script according to Trinque's instructions. If you are at a loss as to the kernel config - by far the ugliest part in this - and your hardware is similar to mine, I was able to use Stan's "Dulap-II" config with minimal (possibly it works even without any) changes so go grab that one and feed it to the script - note that you'll get a quite spartan but absolutely functional environment and you can customize it from there as you want it. Here's the full bash script too if you want to use it: cuntooprep.sh.

NB: at the moment I'm writing this, the known path-issue within Cuntoo's scripts is still present in the Cuntoo.tar file so you'll need to do the change manually for the signature to verify on the genesis.vpatch that you obtain at the end. The change you need to make is explained in the logs and you can do it after running the bootstrap.sh script if you already started that.

Once you get the genesis.vpatch, verify it against Trinque's signature - if it doesn't verify, chances are that you still need to fix the paths issue so see the paragraph above and remember to actually re-run the scripts/make_portage_tree.sh script after you made the change! Then simply reboot and choose to boot from the disk on which you installed Cuntoo. You can then recompile the kernel to tweak any configs you want, for instance with those steps:

  1. cd /usr/src/linux
  2. make oldconfig
  3. make menuconfig (and/or change whatever you want directly in the .config file before running this - I've ended up finding the *text* file BETTER than the bloody gui thing simply because nobody can tell where something is in there, ugh.)
  4. make && make modules_install
  5. cp arch/x86/boot/bzImage /boot/bzImageNew
  6. change /etc/lilo.conf to add the bsImageNew as another boot option (DO keep the old one too, at least until you are sure the new one is fine!)
  7. lilo
  8. shutdown -r now

Once you had enough of recompiling the kernel, the next step would be to have a look at what you'd like to add to your system - chances are there'd be quite a lot of things that you need before you can actually do a lot in there. Cuntoo has its own preferred repository in /cuntoo/portage and a distant second in /usr/portage. So when you run emerge, you'll see either ::cuntoo or ::gentoo appended to the packages required, depending on which of the repositories is used. To enrich the cuntoo repository and effectively get rid of the madness where gentoo just doesn't "have" anymore the packages you want, you'll need to copy the tarball of your package into /cuntoo/distfiles and the ebuild+manifest+patches+shit into /cuntoo/portage/package-path. My initial attempt with MySQL was a step too far at this stage as it served only to point out more clearly what a mess porting the dependencies to Cuntoo will be - MySQL itself requires all sorts! So I looked among those "all sorts" and picked out one with better chances of being standalone, namely curl. With this adjusted aim, I was actually able to "port" the curl ebuild to Cuntoo as it were, simply like this:

  1. Download the tarball without installing it: ebuild /usr/portage/net-misc/curl/curl-7.60.0.ebuild fetch
  2. Copy then the tarball to /cuntoo/distfiles
  3. Copy /usr/portage/net-misc/curl to /cuntoo/portage/net-misc/curl
  4. Go into /cuntoo/portage/net-misc/curl and snip away the ebuilds that are other versions than 7.60.0 as well as the references to them in the Manifest file (NB: you'll need to keep all the patches though regardless of what their names say as they are still applied to the 7.60.0 version anyway).
  5. Now run emerge --ask -v net-misc/curl and look closely: it should say ::cuntoo and it should report that it has to download precisely 0Kb. If all that is true, let it do it and you'll have gotten Curl in your new and sparkly Cuntoo!

Now I must say that I'm a bit at a loss as to whether the above curl-experience should get the grand name of "making an ebuild" or not or what - to me it looks more like hacking an ebuild at best and moreover a quick look at gentoo's docs re making ebuilds reveals enough *other* stuff to eat that I don't see it happening all that soon. So I don't even know - is *this* sort of hacking ebuilds that Trinque was asking for? Or writing them from scratch or something else entirely or what?

  1. Obviously, unless you use DNS but why would you still use that anyway? 

July 8, 2019

The Rocky Moon (Notes on Graphics for Eulora, II)

Filed under: Coding, Eulora — Diana Coman @ 9:35 p.m.

My client-side Eulora task set last week is completed and as a result, the Testy character gets now to see something more than just fog in his Sphere-of-all-beginnings, look:

Testy under a Wavy-Blue Moon, in the Sky-Sphere of all beginnings.

Testy under a Wavy-Blue Moon, in the Sky-Sphere of all beginnings.

It turns out that CS can indeed eat - though not very willingly1 - "geometry" for fixed/static entities (aka "generic meshes") even without any xml in sight2. For static aka non-moving entities, there are 5 sets of numbers required:

  1. Vertex coordinates
  2. Normals at vertices
  3. Colours at vertices
  4. Texture mappings at vertices
  5. Triangles

The main trouble with the above was the exact "how" and "where" to actually set them so CS understands what they are. It turns out that the most obvious "those are the vertices and those are the normals etc" approach is "obsolete" and replaced apparently by having to specify a render buffer for each set of numbers. Of course it's still the same arrays of numbers either way, but with render buffers you need to also specifiy which *type* of render buffer you have in mind (i.e. what are those values for) and in pure CS style this is specified by giving the... name. Name that is afterwards internally translated into a numerical id of course but nevermind since it's supposedly as easy as it can get to have to remember that the name is exactly "position" and not "positions" nor "vertices" nor anything else3). So easy in fact that I ended up having the list of "buffer names" at hand all the time, great help indeed.

Before moving on to some basic descriptions of those 5 sets of numbers, it's worth noting that they are just the most usual 5 sets really: there are plenty others that one can specify though it's unclear exactly when and precisely why would one do. The easiest example concerns texture coordinates: one can specify as many as 4 different texture mappings. And why stop at 4 if one went all the way there, why not 5 or 6 or 7 or 1000? No idea. Anyway, for the basic descriptions:

Vertex coordinates are exactly what the name says: the 3D coordinates of vertices that define the entity's shape in space. Those are all relative to the position at which the entity is drawn at any given time, of course, since they are internal to the entity and unaware of anything else in the larger world.

Normals at vertices are perpendiculars on the surface of the entity at each of the vertices defined earlier. Those are quite important for lighting (together with the orientation of each triangle surface defined further on).

Colours at vertices specify the colour of the object at each vertex and allow therefore colouring of the full object through interpolation for instance. While in principle this can be skipped when a texture is given, the "skip" simply means default colour (black) so it's a "skip" only in the sense that it's less obvious it exists.

Texture mappings at vertices - those are tuples (u,v) that specify a texture point (2D because textures are 2D images) that is to be mapped precisely to each vertex. Just like vertex coordinates provide fixed points for the shape of the engine that is otherwise interpolated at non-specified points, the texture mappings provide fixed points for the "painting" of the shape with a given texture.

Triangles define the surface of the entity as approximated through triangular shapes. All surfaces of static shapes in CS are approximated through triangles and the only difference between curved and flat surfaces is essentially in the number of triangles required so that your eye is fooled enough so you like rather than dislike the lie. This is after all the whole computer graphics in a shell: a lot of effort spent to lie to you the way you like it.

Triangles in CS are given as triplets (A,B,C) where the values are simply indices of the vertices previously defined. So the simplest cube will have for instance 8 vertices for its 8 corners as well as 2*6=12 triangles for its 6 faces since each face, being a square, needs 2 triangles to approximate. And those 12 triangles mean 36 values in total since each triangle needs the indices of all its 3 vertices. Moreover, the order in which the vertices are specified matters as it gives the orientation of the approximated surface4 and that in turn is crucial for calculating lighting (and ultimately for figuring out whether you get to even see what is on that surface at all). Since it's not intuitive perhaps, it's worth noting the basic fact that a surface has 2 sides and the "painting" of a texture is strictly on one side only. So if you mess up the surface's orientation, you can easily do the equivalent of dyeing a garment on the inside without any colour whatsoever showing when you wear it unless you turn it first inside-out.

As you might have guessed from the above example with the most basic cube if from nothing else, even very simple entities require a lot of values tediously and boringly specified - hence quite the bread and butter of automation5, yes. For now and for testing purposes, I simply extracted the numbers of interest from the xml description of the moon object in game and then I used them as default values for an object created directly from code. And lo' and behold, Testy got a moon to stare at and the moon even got to paint itself stony when I'm bored of its usual wavy blue:

Stony Moon and Testy in the Sky-Sphere.

Stony Moon and Testy in the Sky-Sphere.

The next biggish step further in this line is to figure out how to generate terrain from just a heightmap and then how to create an animated (hence, Cal3D) entity from code rather than through the sterile xml-packaging. Looking even further from there, monstrous xml-shaders await in the same direction but in other directions there is still a lot of work to do on teaching the PS code to use the EuCore library and to discover as a result that the world is way larger than what fits in its inherited, well-worn and time-proven, traditional xml-bible.

  1. A bit like those finding out that they can actually eat food that was never packaged in plastic or washed in chlorine or even re-heated. 

  2. At least not in geometry-sight. In a bit wider perspective, all shaders so far are still balls of xml that will probably have to be untangled sooner rather than later. 

  3. If you give a non-existent name, the buffer is simply ignored since supposedly you can define your own buffers too so it's not an error. Combined with the wonderful way in which everything influences everything else in graphics, you get - if you are lucky - to puzzle over some unexpected visual effect that might even seem at first to have nothing at all to do with the names of the render buffers at all (maybe you messed up a few numbers in that long list, you know? 

  4. By convention, vertices are given clockwise as read when facing the surface 

  5. Automation can be here exporters from supported tools such as Blender and generators of known shapes/patterns. 

June 30, 2019

Notes on Graphics for Eulora

Filed under: Coding, Eulora — Diana Coman @ 9:31 p.m.

Eulora's client uses the CrystalSpace (CS) engine for all its graphics needs and CS in turn uses the Cal3D library for animated objects in the world1. Some two years ago2 I wrote this stop-gap glue-and-string script to supposedly help artists and all those eager to contribute but held back by a lack of direct way to experiment and see their "art" coming alive in CS. Well, I wrote it and until the beginning of this month, those scripts plus some basic notions from introductory computer graphics courses taken way back during my uni years were about as much as I could say I actually knew regarding computer graphics.

As it seems quite conceivable by now that I'll end up having to get all the way to the bottom of the graphics pit too, I spent most of this month reading the graphics reference book3 and in parallel trying to make sense in a very practical way of the whole way in which Eulora's client handles all the graphics part from the Planeshift (PS) top layer down to CS and CAL3D. And while I don't yet have a fully working new pipeline for actually loading on the fly all and any sorts of graphics into Eulora's client, I have at least a hard-won better understanding of what goes on in there (despite the layers upon layers of obfuscation) as well as some bits and pieces that might conceivably even make it into such a pipeline in the end. So I'll jot down here my findings so far and those yet-moving ideas about it all, as it helps for sure to structure and unload a bit in preparation for the next stage of work. And it might even help - says ever optimistic me - to start perhaps a discussion with anyone who knows this better than I do so if that's you, kindly comment below and let me know your thoughts.

One difficulty with CS (and it seems to me with Computer Graphics in general really) is this persistent idea of accomodating all sorts of "alternative" ways of doing things that turn out in fact to be precisely the same thing only packaged differently so that nobody is left behind or something. So instead of having a clear structure of knowledge to acquire, one is supposedly left to be creative and find what "feels natural" or "comfortable" to them. Well, so much for actually learning anything there, welcome to lots of code that does the same thing, only it's harder to tell that upfront without sinking hours in it, to follow it all from one end to the other. Nevertheless, after all the reading and poking it and changing it and having my way with it, my current basic structure of CS4 notions of interest for client graphics is this (ordered from top down, as much as I could):

  1. Sectors - those are the highest-level containers of "graphics" of any sort. A sector simply stands for a "location" that is effectively defined by nothing more than a name5 and two mixed characteristics: on one hand the visibility culler in use (so a computational matter) and on the other hand the ambient light (i.e. the light that affects everything in that sector in the same way at any position and as such a matter of aspect pure and simple).

    CS supports two visibility cullers, namely Frustum and Dynamic (dynavis) with the default for the game's needs being the dynavis culler. So a first building block for the desired client pipeline is simply a "GetSector" function. Like all the rest of the "GetX" functions that I have in mind, this will retrieve any information it needs about the sector from EuCore / EuCache (via a CPP-to-Ada layer, as needed) and then either retrieves the required sector from the engine itself if it exists already or otherwise proceeds to creating it and loading it into the engine so that at the end of it, one way or another, the sector is known to exist. When information is missing/incomplete/unavailable, the GetSector method fails and lets the caller decide what to do (try again at later time, most likely). At this moment I have a prototype of this method as well as some similar prototypes for GetLight, GetTexture and GetMaterial - in other words, I can create from code without any xml and parametrised four main things: sectors, lights, textures and materials.

  2. Lights - those are sources of light in a sector beyond and in addition to any ambient light. They can be customised in all sorts of ways (e.g. intensity, colour, position, direction). Whether they are elements inside a sector or an environmental characteristic of a sector is unclear to me at this stage since they could easily be seen either way. From an implementation point of view however they are currently clearly elements inside a sector, NOT characteristics. This means among other things that one could in principle add/estinguish/modify lights as they please at any given time and I'd say this is how it should be, given that it's not inconceivable to be able to turn on a candle or any other light in one's own house for instance.
  3. Weather - the more general name for this would be environmental factors I suppose, but so far it's really at most "weather" and in practice mainly... fog. Supposedly one could add here all sorts of *effects* really so from an implementation point of view I'd even call this part "environmental effects" i.e. whatever effects manifest themselves in the sector as a whole and are specific to it. The fog itself so far appears to be a rather simple matter as it's essentially a limited modifier of ambiental light. I don't really know if this is just a limitation of CS or a more general approach and for that matter I can't for the life of me quite grasp yet exactly how do graphics-people decide whether some effect they want is rolled into "fog" or into "ambiental light" or into anything in there. This is an issue at all levels and so far it really looks like an individual preference than a choice, quite everywhere. In other words a matter of "am I more skilled in making it look just so by tinkering with this or by tinkering with that"?
  4. Meshes - those are any and all elements in a sector that have what CS calls "geometry". This geometry concept is a bit fuzzy at times but as far as I managed to pin it down, it would seem to mean essentially that the object has a shape defined mainly as a set of "vertices" aka points in the 3D space that the object occupies. There are 3 main types of Meshes:

    1. Movement-capable entities
    2. Fixed entities
    3. Terrain

    Each of the above types of meshes has its own specific requirements and even ways of defining itself, including its geometry (hence part of the fuzziness...). At least in the current setup and as a very brief overview, movement-capable entities are effectively CAL3D models defined as hierarchical structures of meshes with attached action representations (e.g. run/move/jump/fight animations) and skeleton-based geometry; fixed entities are CS's "generic mesh" with vertex and triangle-based geometry; terrain entities are CS's "terrain2" mesh type with attached terraformer for generating the geometry based on a heightmap. The details of each are much gnarlier than this list might suggest and I'll be teasing them out one by one as there is no way around that but at any rate, CS's approach seems to have been to make different xmls for each of them and all sorts of "loaders" on top of loaders and on top of plugins, services, parsers and whatnots. Then PS added its own loader on top of CS's pile of loaders. As you might imagine, this tends to rather obscure the precise way in which one can create those different meshes *without* xml.

  5. Factories - those are abstract concepts containing effectively the blueprint for creating identical meshes. The idea behind them - as far as I can tell - is that it's worth loading in memory what is needed for a mesh only once and then reusing it every time a new specific instance of that precise mesh is required. In practice I'm not sure that this is really worth much for anything other than snowflakes or raindrops really, since otherwise pretty much each instance of a mesh seems to have its own factory anyway. There is however the argument that the factory should contain only truly common structure (i.e. all players of one race have the same structure) and then leave the customisations to each instance. So keeping this in mind, here they are in the list, factories for meshes (of all sorts, though indeed, each of the three different mesh types has its own specific factory type on top of a generic type).
  6. Materials - as far as I can tell so far, a "material" is essentially any specific combination of one shader + a whole set of values for the shader's parameters (aka "shader variables" in CS terms). Note that the texture itself (or themselves as apparenlty one can use several although it's not clear to me WHY is that any better), if there is one, is effectively an argument passed on to the shader.
  7. Textures - those are the pretty picture that is lovingly painted on some/any surface that an object in the world may show to the player at some time or another. Essentially each object is but a set of surfaces linked together and each of those surfaces may be covered with one or another texture fixed with some spit so that it goes exactly over that bump or starting from that point to end up just-so. In principle, one could provide instead the way to calculate the texture at any point but that seems to be less frequently used in practice (CS claims to support it though, although I can't tell to what extent and with what results really).
  8. Shaders - this is where the fuziness increases to madness levels. On one hand, "shader" seems to really mean all sorts of things in different places, from yet another description of a surface to any code running on the GPU. There is obviously more reading that I need to do on this aspect but so far what I've got is that CS's "shaders" are really just another layer of encapsulation meant to be for types of surfaces. So if one has types of meshes encapsulated in factories, one also has types of surfaces encapsulated in shaders and is able therefore to apply some "surface type" to any given texture so that the same picture will look "cloth-like" or "stone-like" depending on the nature of the surface on which it is applied. This is about as clear as I can get it now and I can't say I like it much. On top of this, shaders in CS are yet another pile of layers and layers of xml with intricate dependencies on one another and on who-knows-what-else in the name of reusability6.
  9. Render Loops - those are (in usual graphics style as far as I can tell) simultaneously in parallel and on top of the "shaders" above. They are yet another encapsulation, this time of types of rendering + lighting objects. To make the difference between shaders and render loop abundantly clear, default render loops are of course provided by CS, as files in the /data/shaders directory. Admitedly I totally lack the expertise to say at this stage just why are BOTH shaders and renderloops needed as separate entities and/or why not either do with only one of them or otherwise merge them together in a single thing at least but hopefully one of my more knowledgeable readers clarifies this for me. Failing such clarification, I'll need to keep on reading and experimenting with it, of course, for as long as it takes, what else.

In addition to the list above, there would be also the easier part of interface "skin" aka the variious icons and decorations for windows, buttons, labels and whatnots. This part is also on the list to be made fully reliable on data received from EuCache but so far I haven't touched it yet (other than keeping an eye out for what I could do about it, whenever other bits and pieces make me go close to that part of the code that has anything to do with it, of course).

For a fresh image, the TestChar at least moved out of its cubic box and into a rather faceted sphere, in his quest for the promised infinite landscape. Lights got a bit better, shader variables were sorted so as to neither block nor directly crash CS anymore. Next on the list is fully figuring out just how to properly feed "geometry" to CS as what it is, namely several sets of numbers (vertices, texels, normals, triangles) rather than all those + a ton of xml on top. This is a required step in order to have the GetFactory prototype too for static meshes at least. Once that is done, the next step is to move on to factories for terrain and then for animated meshes. Until then though, here's Test-in-his-sphere:

  1. For completeness, I should mention that CS has also its own native support for animated objects via 2 different sort of objects even but neither of those is as flexible and overall useful as CAL3D is. Therefore, I simply consider that animated object = CAL3D from now on. 

  2. Already two years passed! And to think that I'm picking it up precisely from where I left it all this time ago, kind of boggles the mind. 

  3. Foley, J.D., van Dam, A., Feiner, S.K. and Hughes, J.F., "Computer Graphics: Principles and Practice", 2nd ed., Addison-Wesley. Yes, I'm aware that there is a newer edition of this book, supposedly improved, better, shinier, more fashionable whatever have-you. I'm also aware that it's full of what can only be called advertising for Microsoft while being way lighter on what I'm interested in, namely precisely principles of computer graphics. So I'll stick with this edition, thank you. 

  4. Note that there is no desire to make all Eulora somehow CS-dependent as such but at this stage, I have to at the very least support CS as it were so I'm aiming for using CS as a practical example of handling client graphics, not as a driver of standards or content. 

  5. CS uses names aka text/strings as identifiers to the extent that it *requires* those for ~all the Find functions at all levels. It *does* have its own internal IDs and it even exposes them if one tries really, really hard to get to them but they are basically not very useful otherwise and moreover they can't even be set to match an external set of IDs (i.e. one can't create an item with a specified ID.)  

  6. "Reusability" seems to mean that the original author can skip writing a few lines of code at the cost of nobody else being able to understand afterwards at all how the thing works unless they spend at least double the time that it would take them to write it from scratch anyway. 

June 18, 2019

Euloran Blu's and Boos

Filed under: Coding, Open Sores — Diana Coman @ 9:24 a.m.

The legacy code base on client side can best be described as a pile of tangled pointerisms and assorted CPP mess under the name of Planeshift (PS) on top of a smaller ball of ifdefism1 and XMLade called CrystalSpace (CS), on top of a comparatively saner and at any rate smaller ball of CPP called Cal3d, the whole construction bursting with a lot of good intentions and precious little useful structuring or maintenance consideration. While sanity seems to quickly decrease from bottom (cal3d) up (to CS and then various parts of PS), what strikes one even more is that the whole thing if afflicted with a grave case of XML addiction that starts from CS and then simply comes to its ugly fruition in PS. It has been obviously written at the very top of the XML hype - everything and I do mean *everything* and everywhere is best described in XML! It could be argued that the authors simply wanted to have a lot of XML of their own but since they couldn't do just that2, they had to also do a graphics engine and a sort of game on the side, almost as a second consideration really. Come to think of it, why not call it then at least XMLSpace and XMLShift?

While CS itself has at least some clear initial design and therefore some structure (if a very clunky one at best of times mainly because the author really had too many good intentions), its main underlying assumptions are precisely the opposites of mine: besides the total focus on xml, there is also this obsession with event-driven (which makes it very hard to actually follow anything from one end to the other), the generic plugins (which again means that doing the simplest thing involves a huge scaffolding upfront) and the fact that the whole thing tries rather desperately to make itself "easy to use" by those who don't understand (nor even look into) the code at the expense of course of those who do since there is no possible way to have one without the other. Leaving aside for a bit how one can never really do the work of others for themselves, the sad result is that tracking down anything in full becomes extremely difficult. The documentation helps here but only up to a point since it is not exactly up to date and more importantly, it doesn't really focus on helping the reader understand the code - it focuses instead with all its good intentions on helping the reader use the library without really understanding it, as a sort of black or at best graying box. And moreover, pervasive everywhere, there is the wrong sort of "simplification": while the weight of it all becomes obvious to the author too (it's quite usual for various constructors to have at least 10 parameters and they are already several layers deep so they anyway call parent constructors with yet another set of parameters), he doesn't see or choose a better abstraction that would help rather than hinder the effort. Instead, he arbitrarily fixes some bits and pieces: for instance, the terrain2 plugin simply expects the heightmap to be in a file called sectorName_heightmap in exactly the current path of the VFS3. In other, fewer words, weaning the legacy code base off its unsavoury assumptions and xml-addiction so that it simply loads art as it becomes available is extremely frustrating and time-consuming, making the overall situation quite foggy blue with cornered cases and hardly anything useful in sight:

The art4 that you just admired above might not look like much but it's a big achievement on Euloran soil: for the first time in Euloran history, a Testy character (still sadly xml-based himself) made it outside the pre-defined sphere of the XML-Island and into the great unknown of non-predefined, non-xml, fully code-loaded and discovered as you go world! It's true that the world is for now only a cube with rather sharp and unyielding corners and no landscape at all, it's true that the very three suns that gave reasonable light on the Island look foggishly blue in here and it's sadly true that the character itself is still trapped in the XML bubble. But it's a working start and moreover it's a seed that has just caught roots within the hostile soil of the legacy code so that it can grow out of it, hopefully discarding as it grows more and more of the unhelpful mess!

  1. This is a term of art, the logs will help you. 

  2. Why? Why can't you do *just that*, whatever it is you actually want, did you ever ask yourself? 

  3. Virtual File System inside CS.... 

  4. In computer graphics EVERYTHING is art, ok? There is no such thing as mere fitted drawings and clumsy, clunky approximations, no, no. All the games have nothing but art in them! Hell, they *are* nothing but art. Well, of snorts, yes. 

June 11, 2019

Eulora Client Hierarchy of Data Draft

Filed under: Eulora — Diana Coman @ 10:29 p.m.

This is a very first draft to help with the current work on enabling Eulora's GUI client to make use of new information as it becomes available from the server rather than expecting everything to be fully known upfront (as it currently does, laugh you not!).

Eulora's data is one big hierarchy with 3 main types of information: structure, description, representation.

  • 1. Structure (i.e. what entities are inside what other entities)
    Structure is always a tree of entities where each entity is given through ID (unsigned 32 bits) + Position (x,y,z,rx,ry,rz):

    The Root of the world will ALWAYS contain:

    • an *abstract object* with meta-info:
      • ID of "self" (SelfID)
      • ID of current location (aka "sector" or room or world or whatever else it is) (LocID)
      • List of Skills: list of (skillID, skillName, skillCategory, skillRank, skillXP, skillKnowledge) (?)
      • Game Date and Time (?) --- is this a meta-matter or is it a location-property?
    • the *top-most ID* (aka topmost entity); NB: this is not always the same as LocID since character may move into a location inside another location, presumably.
    • from top-most ID down, EACH node will have zero or more contained entities (hence, child nodes given through their unique ID and concrete position within their parent).
    • NB: each node can also have in addition to its structure-children (hence entities), any relevant number of description-children (see 2 below) or representation-children (see 3 below).

    • 2. Description of what entities themselves "are" through their relevant properties

      ANY structure-node may further have, alongside its children, one or several of the following properties (given as a tuple property,value):

      • name
      • description (??)
      • (- equipment slots (list of (slotID, entityID) ?) -- are those any different from "contained"? I can't see a truly significant difference.)
      • stack count
      • quality
      • HP (hitpoints)
      • BP (bloodpoints)
      • SP (stamina points)
      • MP (mana points)
      • SpP (spirit points)
      • MSP (mana spirit points)
      • weight
      • bulk
      • weight limit (max weight)
      • bulk limit (max bulk)
    • 3. Available representation of entities (graphics, sounds, effects, maps etc)

      NB: in principle there can easily be a structure + properties split at this level too since a graphical representation may have its own structure (e.g. map defined as a hierarchy of contained terrain + geometry meshes that have in turn different submeshes with corresponding textures/materials). This seems rather unseemly though and it's unclear if there really is a need for a hierarchy of graphical detail that is anything OTHER than hierarchy of world entities to start with. So perhaps this can be flattened so that representation of entities means just that: a list of relevant graphical characteristics of ONE entity.

      ANY structure node may further have, alongside its entity-children and its properties, one or several of the following representations:

      • Mod2D (aka 2D model; at its most basic, the filename of icon to show for this; tbd)
      • Mod3D (aka 3D model; at its most basic, spec for mesh including material/texture and sockets if any; tbd) ; NB: sockets allow graphical representation of contained (specifically "equipped" or "component") entities and therefore they should probably be identified by "position" so that the entity "child" on that position will then give the actual representation shown to the user (?).
      • ModAnimations tbd
      • ModMap (aka map for this entity; at its most basic heightmap + size (or is this fixed?); possibly also a list of materials to use + corresponding density)
      • ModWeather (unclear yet if this is any different from sounds+effects really)
      • ModSounds tbd
      • ModEffects tbd (unclear yet if this is substantially different from animations+sounds)
      • ModSkin tbd (stuff like window borders and button colours)

      ANYWAY: GUI may request whatever it wants from local cache, regardless of cache's own internal representation (just as cache may store data in any form, regardless of smg's own description of hierarchy).

    TO DO: flesh out in more detail each possible representation, for GUI to ask & work with + get it to actually work outside of/inside the intricate mess of ps paws, widgets, managers and whatnots maggots.

May 27, 2019

Eulora's Client Core: The Dedicated Requester

Filed under: Coding, Eulora — Diana Coman @ 9:38 p.m.

A crucial requirement of Eulora's new client is to actively request from the server ANY data that it may find itself missing at any point in time. At first glance, this seemed to me simply a matter of providing request services1 from Eulora's new Ada-based core and then adjusting the existing C/CPP code of the legacy client to make use of those services. This rather optimistic idea is of course plain wrong: "adjusting the existing C/CPP code" in this context is similar to saying that one "adjusts" a sheep to use the library - while it can certainly be done for various definitions of "done" and the sheep may indeed use the library one way or another, it's at best a huge waste (of time, of resources, even possibly of steak) for everyone involved and no matter how one looks at it.

Even leaving aside for a moment the trouble with "adjusting" the legacy tangle in any direction, the more important issue here is that this requirement is not as much a functional requirement as a non-functional, quality of service requirement: whichever part of the client provides data services, it should better be dedicated to the task and do whatever it takes to get it done instead of "providing" it only fair-weather style - if it's easy, there you are and if it's not easy then it's your problem really. In other words and in marked contrast to the very democratic "best"-effort-you-can-do-anything-anytime existing C/CPP code2, the new code should have a clearly defined task and then either complete it or die trying over and over again, taking full responsibility for the process involved, not just for some specific detail conveniently chosen nor - as an excuse for not delivering - for the outcomes that are not fully under its control3.

Considering therefore "active and dedicated request" as a quality of data service on Eulora's client side, it follows that its place is rather close to the data cache mentioned previously and at any rate inside the new Client Core since it's certainly not some additional part of client logic nor some bit of user interface. However, I'm reluctant to make it the responsibility of the cache itself since the cache is a passive structure that focuses on *storing* data and *providing access* to it. Mixing passive data storage with active data acquisition doesn't make much sense to me and even seems ill-advised for Eulora's client given the competing requirements: on one hand passive, immediate-response local data storage and on the other hand active, possibly-delayed and world-facing (i.e. communicating with the server) data acquisition. So I'd rather avoid this passive-active construction and have instead the two as separate entities: a EuCache dedicated to storing and retrieving on demand *any* data; a Requester dedicated to acquiring *any* data that is demanded of it. Note that the definition of "acquiring" here has nothing to do with the means through which the Requester actually gets this data (specifically nothing to do with the exact messages sent/received/exchanged with the server). Acquiring some data means simply that the required piece of data becomes available in the local cache aka EuCache. So the Requester will keep requesting this data from the server through whatever means it knows until either the data arrives and becomes available from EuCache or otherwise the whole client kills itself for lack of server4 and therefore of any possibility of playing the game.

Specifically, the Requester will be implemented in EuCore (and therefore in Ada) as a protected object exposing only a few procedures that are essentially notifications: some are notifications of demands for some specific piece of data (either an Object or a File really since those are the only 2 overall types of game-data that one can request from the server); the others are notifications of data being received or of timeout interval having elapsed (in other words a notification of failure to receive data). Note that the demand for an "Object" effectively means a demand of its properties and those might be indeed anything at all but each and every one of them will be either directly a value or otherwise an ID of another Object or of a File. All notifications (including those demanding data) are always accepted by the Requester but not necessarily immediately acted upon. Clients of the Requester do NOT control the actual requests to the server or messages exchanged and are not even concerned with those at all - the production of actual requests, their content and their timing are entirely the job of the Requester and under its sole control. Implementation-wise, the Requester will simply keep queues of requested Objects/Files and will then proceed as soon as it can to pack a request for as many of them as possible; this request will then be posted to the server and the Requester will set a timer to the timeout value so that in the worst case it is at least notified that nothing happened; when/if any data is received or when this timer expires, the Requester will check in EuCache to see what items (if any) of those requested are now available; any items that have become available will be discarded from the watchlist of the Requester (i.e. the demands for them are considered completed) and a new request may be created and posted to the server for any items that are still in demand but not yet available. Note that even in the event of a timeout, a "repeated" request to the server may not be identical to the previous request since the list of demanded data possibly changed in the interval.

One potentially iffy point for the Requester is its need to be notified of any incoming data. At the moment I don't see any real way around this, short of making the Requester poll at set times the EuCache and checking if any data of interest has meanwhile arrived. I don't really like this polling approach here because it's rather wasteful without good reason: any incoming data is indeed received and processed by another unit that is also on the same level with the Requester, namely the Consumer (the part that processes messages from the inbound queue). So the Consumer will have to notify the Requester when new data is received. While several Consumers may be active at the same time (at least one for Serpent and one for RSA messages) this is not a problem at all since the Requester is anyway a protected object i.e. thread-safe. Note also that even if (some of) the consumers fail to notify the Requester of some incoming data, the whole thing will still work if only slower than it could: the timeout timer will wake up the Requester and the check of data will happen there at any rate. In other words, the Requester is capable of reacting to events if notified of them but not dependent on those notifications to do its job correctly.

Given its rather complex task, the Requester is currently on the top conceptual layer of EuCore, making use of quite a lot of other units from lower levels. Currently, the main relevant units on this top level are the following:

  • Data Cache aka EuCache - this is a passive, thread-safe entity responsible for storing all and any data given to it and retrieve or delete it on demand. As such, it *owns* the specific format in which data is stored5 and it simply exposes functions and procedures for storing, retrieving, deleting and checking for data.
  • Communication Link aka Comm_Link - this is a passive, thread-safe entity responsible for persistent storage and updating of communication details, most notably RSA and Serpent keys for inbound and outbound communications as well as a message counter. This is effectively a specialized cache - where EuCache is for game data, Comm_Link is for communication protocol data. The requirements (including use contexts) and specifics of the two seem to me sufficiently different to keep them separate at least for now.
  • Consumers of incoming messages (RSA and Serpent) - those are separate, active tasks, responsible for processing incoming messages. Consumers own and get to define what "processing" means exactly but their role is to extract the data contained in the messages received and make it available for use by Eulora's client. In practice this means currently that Consumers will pass any data received on to EuCache for storage and will notify the Requester of any data receipt.
  • Requester - this is an active, thread-safe entity responsible for acquiring data that is in demand. It owns the process of data acquisition from the server and it accepts any demands of data specified by some identifier. While it guarantees that all demands will be served at some point in time as long as the whole program is running, it does not (and can not) guarantee when or if the corresponding data becomes available. It can't even guarantee that there IS any corresponding data and so ALL it can do is to guarantee that it will try with the same dedication for each and every bit of data to acquire it. Anyone demanding data can of course pester the Requester with more demands or give up or decide for themselves for how long they can or will wait.

And now that the main idea and overall design of this whole thing is at least quite clear to me, there remains of course the small bit of actually implementing it in full (rather than the current skeleton I already made) and sorting out the various implementation-level troubles that will appear as they always do, in all sorts of details. Relatively easy work compared to what follows it inevitably, namely teaching that C/CPP sheep to use the EuCore library...

  1. Mainly picking, packing and encrypting everything in the right format + sending it through to the correct place. 

  2. Seriously, think of it: existing client code is this event-driven thing where *anyone* can subscribe to any event and then "do" anything at any time provided of course that the any and anything are in fact very narrowly defined and set in stone even to the level of names of various art files (it has to be a zoneinfo.xml inside this and that and put exactly there, sort of thing). If this narrowing of "do" was not a price high enough to pay for such code "liberty", there is also the added reality of a huge tangle that does precious little indeed since everyone ends up calling anything from anywhere and no piece of code is really and truly responsible for anything bigger than a few variables here and there. And at the end of the day how could any code even be responsible for anything since it can't *own* any process by itself (shared! event-driven!) and it has to be passive, mainly reacting to some events or at most... signalling through events of its own but never able to rely on anyone giving a fig about its signalling! So there it is, code - like anything you do - is more of a mirror than you might think. And "teaching people to code" has way more layers than "teach them Java" and certainly more issues than current "courses" even seem to be able to imagine. 

  3. And now that it's clearly stated like this, tell me first just how many people you know who actually handle their own work like that? And just what bit of "programming language" you think one needs to teach so that you get programmers to actually design their code this way? 

  4. This "lack of server" is defined in practice as a number of successive timeouts on requests sent to the server, where the specific threshold value is chosen by the user via a config file. 

  5. Currently game objects are stored in memory in Hashmaps aka associative arrays while art/files are stored directly on disk. Note however that any of this can be changed without touching anything outside EuCache as it's nobody else's business. 

Older Posts »

Theme and content by Diana Coman