Archive for the ‘Coding’ Category

Pretty Petty Particles (Notes on Graphics in Eulora, VI)

Friday, September 27th, 2019

As the latest client data hierarchy draft revealed a gaping hole regarding the exact use and requirements of "effects", I've spent those past days diving into the tediousness of "particle systems" - those are the core of CS effects since they are used to simulate rain, snow, explosions, fires, fountains etc. On the bright side, particle systems are actually explained in a bit more detail than other things in the CS manual, mainly because they have been apparently changed quite significantly so the manual's author felt the need to provide examples for using the new code. On the darker side, they are stuffed into the clunky system of various plugins + xml + factories + meshes + shaders + 1001 attributes to set for the smallest thing, so that 3 drops of rain get you sweating and swearing through the usual multiple layers of all-possible-things in slightly-different-ways.

To start with, trying to find a proper place where those particle systems belong in a sane hierarchy of CS items is rather dubious because particle systems are a combination of almost everything in there plus "emitters" and "effectors". And ~everything has to do the dance of factory+instance although each time pretty much everything gets re-set anyway1 so it all feels like an exercise in bureaucratic-coding, to make one's stomach heave anew. Anyway, the core components of particle systems are those:

  1. A non-moving mesh that is the "particle". Like all other non-moving meshes, this can naturally have one or several materials/textures and shaders (with the load of shader variables) but unlike all other non-moving meshes, the geometry is not something one sets explicitly via vertices and triangles and all that - instead, the geometry is fixed and set by the provided "plugin"2 for particle systems! Setting various parameters allows one to tweak this geometry to some extent but that's about it. According to the manual, the existing plugin simply makes all particles rectangles - you can set however their size and their orientation with respect to the camera.
  2. Any number of emitters. As the name suggests, the emitters create and launch about new particles according to another set of parameters. For the same particle system, one can in principle add any number of emitters but there has to be at least one emitter for anything to happen, of course. There are a few types of emitters that differ mainly in the shape that is covered by the launched particles: sphere, box, cylinder, cone.
  3. Any number of effectors. From my tests, those are not even mandatory. They provide some ways to influence the existing particles (usually their movement but supposedly any aspect of them; the manual says that effectors "put different kinds of effects" on particles, "be it physical or non-physical effects"). There are only three available types of effectors: force, lincolor (interpolates the colour of each particle between two values based on the remaining time to live) and forcefield.

Once a particle system is created, the way it works is relatively straightforward: the engine calls on every frame the update loop of each particle system that is visible to the user; the update loop deletes old particles, polls all emitters to emit new particles, polls all effectors to apply their effects and then performs the final transformations if/when such are required. Basically a particle system is designed as a sort of predefined show happening at some place in the world for some specified interval of time with rectangle-shaped actors only (but possibly various costumes via materials).

For practical tests, I've created from code several types of particle systems including "rain", "fountain" and "fire". While the general mechanics of it all seem to work fine and the parameters seem to do indeed what is expected, there is still some weird issue with the materials - all the stuff looks brown-ish, no matter what texture or material is loaded. Since otherwise a material is just a material and it's clearly loading fine for anything else, I quite suspect that the trouble here is the shader: although in principle any shader could be used, there is a specific particle_shader that is used but not really documented anywhere (ie there is the xml of course but that has some variables and possibly references and it is *one* area that I rather avoided to fully get into, so far). Added on top of all previous experience, I'd rather say that sooner or later I will still have to dig shaders too, what can I do. Anyways, at the moment I am mostly interested in having the mechanics in place and figuring out the parameters, so in this sense it's a success, here's some snow and rain falling down on naked girl, at times with erupting particles too:

Euloran Snow - it's more brown than white!

Euloran Snow - it's more brown than white!

Undisturbed euloran woman among erupting stuff and pouring rain.

Undisturbed euloran woman among erupting stuff and pouring rain.

Let it rain with brownish nothings.

Let it rain with brownish nothings.

Given the above, I'll mark the "particle systems (Air)" as explored for now and move on to procedural textures (Water) to see what further catralliard of parameters I find there too. The test of movement for effects (ie following a char or something) is still on the list too but I see it as more general really - since the whole particle system is simply a mesh, having it follow another mesh should be just a matter of giving its position as relative to that rather than absolute. At the moment I'm not sure where exactly would be the best place to do this but I think it requires a bit more consideration because it's possibly a more generally useful thing to have - some items/characters might simply follow others at some point/for some time, anyway.


  1. It would even be hard to keep track of what is same and what is different given how many darned setThis and setThat one calls. 

  2. The plugin's code can be found in CS/plugins/mesh/particles/object/. 

Eulora Client Data Hierarchy v2.0

Saturday, September 14th, 2019

This replaces the previous draft and is based on the new knowledge of client entrails of all sorts as acquired during those past 2 months. It is however still a draft in that it uncovers some new questions (at least new to me) and it requires anyway some feedback and discussion given its scope and crucial position in the SMG Communication Protocol. It's published as it is at this stage and by the promised deadline not because it's "complete and final" but mainly because I need to have this version exposed for further discussion in order to be able to advance it further.

While I was already trying in the previous draft to figure out more clearly the scope and nature of the desired hierarchy, this became clearer only after getting a better understanding of the various data the client uses and especially after further discussion in the SMG boardroom of the protocol specification. The previous observation that there are 3 types of information in there (structure, description, representation) is as correct now as it was then. But meanwhile, as I got to understand better all the different domains involved, the high-level view evolved to this:

  1. Game World
    1. Concrete objects
      1. Terrain
      2. Water
      3. Air
      4. Characters
      5. Structures/Items1.
    2. Abstract objects
      1. Time
      2. Self
      3. Sector
  2. Representation
    1. Graphical
      1. 3D
      2. 2D
      3. Skin/GUI
    2. Audio

Before going further and into the hairier details, it's worth stating the important distinction that the protocol specification makes between the two types of "resources" that the client can ask for:

  1. Files - relevant protocol messages 4.4 File Request and 4.3 File Transfer.
  2. Objects - relevant protocol messages 4.7 Object Request and 4.8 Object Info (ie "Transfer")

The Files are - from the protocol's point of view - "big" resources that never contain other resources. Files are referenced by their full name that works therefore exactly like an id (although a text-id rather than a numerical id).

The "objects" are - from the protocol's point of view - "small" resources of a potentially composite sort. This means specifically two things: any object will always fit into one single message; an object may contain references to any number of other resources (ie references to other objects and/or to any number of files). An object is given (via a 4.8 message) as a list of tuples (id:type). Each such tuple is the reference (the "id" part) to one "property" of the object, where the term property is quite generic: any object can be at some time or another a "property" of another object. Note also that "id" is not necessarily a numeric id - a filename (with full path included, where relevant) is effectively the "id" of the actual file.

The hierarchy starts with the single object that has no id (otherwise put it has the "null" value as id). Since all things have to start from somewhere, so does Eulora's language of knowledge - in the beginning, it is always "null". If you think of the hierarchy correctly as a tree, this object with "null" as id is the root from which there can be ultimately any number of levels - the hierarchy is *not* restricted on either direction (ie any node can have in principle any number of children and any path may be arbitrarely long). The leaf nodes will *always* have "Final" as given type in the tuple (id:type). The id in the tuple (id:type) of a leaf node is either a filename (ie the id of a file, one can think of this as value given by reference) or otherwise a value that is used directly as it is, as data (e.g. helmet.png:Final or 5:Final). All values given at a leaf node will be one of the following basic types:

The root of all the world (the object with "null" as id) is the object that the server sends (via a 4.6 World Bulletin message) in response to an Object Request (4.7) that contains an unknown/non-existent object id. The hierarchy as I managed to flesh it out so far is given as id:type with comments after > :

  1. Root > there is only ONE object of this type and that object has id "null"

    1. Top_Location_ID : Sector > current sector; player's character is somewhere in the subtree of this object (eg may be in a room that is in a house in this sector, perhaps).
    2. Self_Info_ID : Self > id of the abstract object, not the id of the player's concrete character object.
    3. Current_Date_ID : Date > euloran date
    4. Current_Time_ID : Time > euloran time
    5. Client_Skin_ID : ModSkin > graphical user interface resources
  2. Sector

    1. CurrentNameID : Name

      • Value (text): Final
    2. Terrain_ID : Terrain > the heightmap (+ material palette)
    3. Air_ID : Air > ambiental light and weather (?)
    4. (Light_ID : Light)* > NB: zero or more; those are purely sources of light (NOT ambiental light which is a property of Air), without physical representation other than the light itself.
    5. (Water_ID : Water)* > bodies of water
    6. (Character_ID : Character)* > anything that is active/can move;
    7. (Structure_ID : Structure/Item)* > anything that is passive/can't move and is not terrain/air/water/weather
  3. Self > this includes all the stuff that is specific to self (ie known only about own Character) as opposed to the generic info given via Character type (ie info available for all characters encountered, whether self or not)

    1. CurrentPlayer_ID : Character > player's actual character with data that everyone can access/see
    2. Current_Loc_ID : LocID > should the location ID be here? (otherwise in principle client can fully search the tree until it finds CurrentPlayerID as contained in some location)

      • Value (uint32): Final
    3. Date_Of_Birth_ID : DOB

      • Value (date) : Final
    4. Vitals_Profile_ID : VitalsProfile
    5. Skills_Profile_ID : SkillsProfile
    6. Inventory_ID : Inventory
    7. Equipment_ID : Equipment
  4. Inventory

    1. (Item_ID : Structure/Item)*
  5. Equipment > How is equipment different really from Inventory? I can't see any real reason to make it a separate type. "Equip" is an action and as such it will fail if the item doesn't "fit" the destination, be it because it's not equippable or whatever. Similarly, the "how it looks" is a matter of checking what is in some slots if one so wants. The only thing: the meaning of "positions" in inventory should be defined, at least for those that have a specific meaning beyond "in the bag".

    1. (ItemID : Structure/Item)*
  6. Date > this could be in principle simply unixtime number but that wouldn't match well/easily with custom year/month sizes (?) + presumably custom month names rather than numbers (?)

    1. Year_ID : Year
      • Value (text) : Final > should those be numbers?
    2. Month_ID : Month
      • Value (text) : Final > should those be numbers?
    3. Day_Of_Week_ID : DoW
      • Value (text) : Final > should those be numbers?
  7. Time

    1. Hour_ID : Hour
      • Value (uint8) : Final
    2. Minute_ID : Minute
      • Value (uint8) : Final
    3. Second_ID : Second
      • Value (uint8) : Final
  8. ModSkin > should this cover *also* layouts & windows & UI components or only strictly "art" resources? If it's the later, in principle it's simply a list of images (with at most width, height, alpha, transparency)??

    1. (UI_Art_ID : UI_Image | UI_Spritesheet) *
  9. UI_Image

    1. (Name_ID : Name)? > CS uses everywhere names for IDs and sprite sheets won't be fine with filename as name for each sprite; onth perhaps client should just generate its own "names" out of object id and/or filename if it needs them?

      • Value (text) : Final
    2. Size_2D_ID : Size

      • Value (size2d) : Final
    3. Transp_ID : Transparency

      • Value (rgb) : Final
    4. Alpha_ID : Alpha_Lvl

      • Value (float) : Final
    5. File_ID : Image > should all be .dds or .png or fixed format?

      • Ref (text) : Final
  10. UI_Spritesheet

    1. UI_Image_ID : UI_Image > for the sheet as a whole, it still has all the details
    2. (UI_Sprite)+
  11. UI_Sprite

    1. (Pos_ID : Position)? > for sprite sheets - position in parent image

      • Value (position) : Final
    2. (Name_ID : Name)? > CS uses everywhere names for IDs and sprite sheets won't be fine with filename as name for each sprite; onth perhaps client should just generate its own "names" out of object id and/or filename/whatever if it needs them?

      • Value (text) : Final
  12. VitalsProfile

    1. (VitalID : Vital)*
  13. Vital

    1. Name_ID : Name
      • Value (text) : Final
    2. Current_Value_ID : CurrentValue
      • Value (uint8) : Final > assuming that those are still given as % of max
  14. Skills_Profile

    1. (Skill_Group_ID : SkillGroup)*
  15. Skill_Group

    1. NameID : Name
      • Value (text): Final
    2. (Skill_ID : Skill)*
  16. Skill

    1. Name_ID : Name
      • Value (text): Final
    2. Rank_ID : Rank
      • Value (uint64): Final
    3. Practice_ID : Practice
      • Value (uint8) : Final

      > assuming given as %

    4. Knowledge_ID : Knowledge
      • Value (uint8) : Final

      > assuming given as %

  17. Character

    1. Name_ID : Name
      • Value (text): Final
    2. Sex_ID : Sex > is this given and in what form? is it known for anyone or just for self?
      • Value (uint8): Final
    3. Position_ID : Pos
      • Value (position): Final
    4. Rotation_ID : Rot
      • Value (rotation): Final
    5. Model_3D_ID : ModCal3D
    6. (Model_2D_ID : Mod2D)? > 0 or 1
    7. (Model_Audio_ID : ModAudio)? > 0 or 1
    8. (Model_Effects_ID : ModEffects)? > 0 or 1
  18. Structure/Item > the difference between structures and items is minimal to the point of inexistent (eg "structures" can't be picked up perhaps, but that's not a property - simply a response to an attempted action and as such of no interest/relevance here!; or structures will have stack count = 1 but that doesn't mean that they don't have stack count...)

    1. Name_ID : Name
      • Value (text): Final
    2. Stack_Count_ID : StackCount > this will just be 1 for houses / some cases.
      • Value (uint64): Final
    3. Quality_ID : Quality > this may be called "durability" for eg tools or houses but it's same thing anyway.
      • Value (uint64): Final
    4. Position_ID : Pos
      • Value (position): Final
    5. (Model_3D_ID : ModCal3D)? > ONE of the models 3d/2d should be present in general
    6. (Model_2D_ID : Mod2D)? > 0 or 1
    7. (Model_Audio_ID : ModAudio)? > 0 or 1
    8. (Model_Effects_ID : ModEffects)? > 0 or 1
  19. Terrain

    1. NameID : Name
      • Value (text): Final
    2. Position_ID : Pos
      • Value (position): Final
    3. Height_Map_ID : Heightmap > NB: while both materialmap and heightmap are images (and even of same type + same size, listing them as "IMAGE" type means that the role (i.e. which one is Heightmap and which one is MaterialMap should be defined somewhere else - would that be better/how?)
      • Ref (text): Final

      > filename, hence "Ref", currently "Island.png"

    4. Material_Map_ID : Materialmap
      • Ref (text): Final

      > filename, hence "Ref", currently "Island_alpha.png"

    5. (Material_ID : Material)* > material palette to use (eg sand, grass, stone)
  20. Air

    1. AmbientLightID : AmbientLight > all lights should be "dynamic" or at any rate the static/dynamic distinction is entirely client-side concern ; currently those values are given at different "times of day" + interpolated by client in between, but I don't see the *need* for hours in here; client can decide on its own if /when it wants to sample this and what/whether to interpolate or do anything else with the values it gets.
      • Value (rgb) : Final
    2. Weather_ID : Weather
  21. Water > is this needed as separate type? CS has 4 types of procedural textures: fire, water, dots, plasma
    > therefore "water" could simply be in principle an item/structure with the "water" texture applied - TO CHECK.

    1. Item_ID : Structure/Item
  22. Weather > CS uses "particle systems" for this sort of thing (i.e. actual snow/rain)
    > as such, those could be just part of the "sector" at some point + corresponding changes to light & sky, if any. TO CHECK
    > however, there can be perhaps the point of having this separate to set specific params such as emission rate, velocity etc. ?

    1. Particle_Syst_ID: Particle_System > if going this route, this is TBF
  23. ModCal3D > aka moving/character stuff

    1. BB_Size_ID : BoundingBox > this is used for collision detection

      • Value (size3d) : Final
    2. Skeleton_ID : Skeleton > this is MANDATORY; all moving stuff have to have a skeleton

      • Ref (text) : Final

      > this is a .csf file for now; should the format be a further level/property?

    3. (Animation_ID : ModAnim)* > a character may have any number of animations
    4. (Submesh_ID : ModMesh)* > parts/subparts of the model; NB: atm sockets is still not used/unclear!
    5. Scale_ID : Scale > intended scale of the model, this is a transformation to apply before use

      • Value (float) : Final
    6. Translation_ID : Trans > pre-transform to apply, translation; same as for scale - could be standard and therefore spec not needed?

      • Value (3xfloat): Final
    7. Rotation_ID : Trans > pre-transform, rotation; same std issue as above.

      • Value (4xfloat): Final > (x,y,z,angle)
    8. RotX_ID : RotX > IF the model should be pre-rotated along the x-axis; again, should be std and be done with. Default is true

      • Value (boolean): Final
    9. FlipTex_ID : FlipTex > whether v coordinates of texture should be flipped (i.e. whether 0,0 in texture space should be top left or bottom left); default is false

      • Value (boolean): Final
  24. ModAnim

    1. Name_ID : Name > name is used as an id by CS... eg "walk", "run", "jump"

      • Value (text) : Final
    2. File_ID : File > this is a .caf file for now; should format be a deeper level?

      • Ref (text) : Final
    3. Type_ID : TypeAnim

      • Value (uint8) : Final > idle, travel, cycle, action, none, style_cycle currently from CS&PS - WTF?? do we even want such a thing?
    4. Vel_ID : Velocity
    5. Time_ID : Interval > this is "interval for idle or override actions", in seconds

      • Value (2xfloat): Final
    6. Idle%_ID : Idle% > this is "probability (as %) of this action being the overriding action"

      • Value (float) : Final
  25. Velocity > this is for animations

    1. Base_ID : Base_Vel > speed of translation when animation is used on its own

      • Value (float) : Final
    2. MinVel_ID : Min_Vel

      • Value (float) : Final
    3. MaxVel_ID : Base_Vel > min and max are used by CS's internal "blender" to get what precise speed is required at one point or another

      • Value (float) : Final
  26. ModMesh

    1. MeshFile_ID : MeshFile > format is .cmf ; should format be a lower level?

      • Ref (text) : Final
    2. Size_ID : Size > is this actually mandatory/needed here? Can't quite yet tell.

      • Value (size3d) : Final
    3. Name_ID : Name > name of this part of the model, eg "Head", "Neck", "Left_Arm"

      • Value (text) : Final
    4. Material_ID : Material > CS allows several materials mixed on a mesh (though on occasion with this sort of thing it "allows" while at the same time complaininig if too many textures or too many this and too many that) but I'd stick to ONE material only.
  27. Mod3DGeneric > aka structure/item (non-moving)

    1. Material_ID : Material > while several materials can in principle be used, I'd rather stick to ONE material since *that* can combine several textures anyway.
    2. (Vertice_ID : Vertex)* > one vertex as a minimum

      • Value (position) : Final > this is given in model's own coordinate space
    3. (Triangle_ID : Triangle)* > each triangle is given as (i1,i2,i3) where i1-3 are indices of the vertices given previously

      • Value (3xuint64): Final
    4. (Normal_ID : Normal)* > there should be one normal vector given for *each* vertex

      • Value (3xfloat) : Final
    5. (Colour_ID : RGBA)* > there should be the colour given for *each* vertex

      • Value (4xfloat) : Final > this means r,g,b and alpha channel
    6. (Texture_Map_ID : TexMap)* > there should be mapping from each vertex to a texture point (ie u,w coords)

      • Value (2xuint) : Final > these are coordinates in the 2d, discrete space of the texture
  28. Mod2D

    1. Texture_ID : Texture > NB: this currently is "texture2d" but it's unclear if the distinction is worth much (ALL textures are anyway 2d images, ffs)
    2. Transparent_ID : rgb

      • Value (3xfloat) : Final
    3. Alpha_ID : Alpha > this is int in the current code, hm.

      • Value (int8) : Final
    4. Pos_ID : position

      • Value (3xfloat) : Final
    5. Size_ID : Size > height and width

      • Value (2xint) : Final
  29. Material

    1. Name_ID : Name > CS uses "names" in fact as IDs; so there can be several materials with same texture/file/data but different names...

      • Value (text) : Final
    2. Texture_ID : Texture > CS works with materials (it wraps any plain texture into a material anyway)
    3. (Shader_ID : Shader)* > a material CAN have several shaders applied.
    4. (ShaderVar_ID : ShaderVar)* > those are used *across* shaders to customise them, hence put here as separate, ugh; I think though that they belong more logically as a property of each Shader.
  30. Texture > CS supports: png, tga, bmp, jpg, dds, gif, mng, jng; common use given: png/tga for textures with alpha; jpg for those without; "best" = dds because it stores mipmaps directly + no need for additional steps (for a png there is uncompressing + mipmaps calc + recompressing to upload; a dds is directly uploaded to graphics hardware)
    > There are ALSO, "texture classes" but it's totally unclear if they bring anything useful or if they are anyway a server concern at all; from the manual (p. 331/644): texture class = collection of settings controlling how a texture is uploaded to the Graphics hardware (e.g. lookup, normalmap, nocompress) + "adding semantics".

    1. Name_ID : Name

      • Value (text) : Final
    2. Image_ID : TexImage > at this stage it's unclear if stuff like size should be given explicitly since it should rather be either extracted from the file itself (?) or default/predefined (?)

      • Ref (text) : Final
  31. Shader

    1. Name_ID : Name

      • Value (text) : Final
    2. ShaderTypeID : Name > ugh; those are again predefined CS stuff/available shader types...
    3. File_ID : File

      • Ref (text) : Final

      > filename containing the shader description; atm those are a bunch of xml files, full of dependencies & referencing one another + cs-specific.

  32. ShaderVar

    1. Name_ID : Name > NB: those names need to match what shaders (those xml files) know/expect, for anything to work...

      • Value (text) : Final
    2. Value_ID : Texture|Float|2xFloat|3xFloat > the value of the shader variable has one of those types

      • Value : Final
  33. Light > ALL lights are "dynamic" since they can always change/be added/removed etc.

    1. Name_ID : Name

      • Value (text) : Final
    2. CentreID : Centre

      • Value (3xfloat): Final
    3. ColourID : rgb

      • Value (3xfloat): Final
    4. AttenuationID : Attenuation > "type" of attenuation (?)

      • Value (uint8) : Final
    5. RadiusID : Radius

      • Value (uint16) : Final

As full of holes and possibly over-wordy and all that as the above draft finds itself, it was still quite the toil to extract. Nevertheless, extracted and written down it had to be as I couldn't quite move any further without this step. Now that it's pinned down, I can go through it and discuss the questions/issues that are still unclear throughout. There are also a few aspects that are currently still entirely missing (not that the rest is guaranteed to be complete/final), namely:

  1. Some meta resources - most notably authorship of art files - are still missing entirely. I'd rather discuss/iterate first those concrete (relatively) parts extracted from client-entrails before moving on and fixing the details of that, since it's more likely to be actually easier anyway as it's way more clearly and sanely defined to start with.
  2. ModAudio - the definition of this is still missing as I never even bothered yet to get the SoundPlugin to work at all. Perhaps it can be added/fleshed out later on and then clients that know it / want it will use it, while the rest will simply ignore it as an unknown/un-usable part.
  3. ModEffects - this is also still missing but unlike the audio part, this could be more readily investigated and fleshed out more, by figuring out how to load/create effects from code directly, similar to the previous work on characters for instance.
  4. Water - this requires an investigation into creating water bodies from the code directly; specifically: some practical experience using CS's procedural textures and some tests to see if/to what extent this could/couldn't be folded into structure/item
  5. Weather - this requires an investigation into CS's particle systems at the very least (for snow/rain/etc) and some more reflection + testing to check if it makes perhaps more sense to fold this into Air anyway.

  1. This is just one type, the name is given like this to make it clear that both "items" such as an axe or a flower and "structures" such as a house or a bridge are in here. Basically anything that is not terrain/water/air and can't move by itself is in here (if it can move by itself then it's in Characters)  

  2. NB: CS uses floats, with 1,1,1 for white and 0,0,0 for black. 

A Summer's Summary and Next Steps in Eulora

Thursday, September 5th, 2019

Coming back from any holidays, no matter how short, means that I spend initially a few days simply with the task of picking up again each and every bit of work that I put on hold while I was away. Note that I didn't say "dropped for the holidays" but rather specifically put on hold in a prepared manner so that picking it up again is not a horrendous chore but rather an opportunity to re-load the context with a fresher mind. And this makes it also a good moment to review the work done over the past months and state the next steps just as I usually do anyway for my own benefit before setting to do any work. The only difference this time is that I'm publishing here this whole summary of work done + work in line to be done next, so that anyone interested can follow along. Why wouldn't I publish it anyway?

May was spent on two main pieces of work:

  1. A working initial1 implementation (both on client and on server side) of the SMG protocol parts that are already clearly defined, namely network communication, account request/creation, serpent keys request & management, file request & transfer. This part meant Ada implementation of the protocol but also wrestling the legacy Eulora client into using the network communication structures of SMG Comms as encapsulated in the standalone Ada lib called EuCore. It also included some server-side work so that the end result is proper communication: the client can now ask for a new Eulora account and the server will reply according to SMG Comms; the client can also use its already established Serpent keys to communicate with the server and the server will reply accordingly. As required part of this work, a broad mapping of the client and its dependencies had to be done as well and basic docs for the client were published.
  2. The initial design of client logic for communicating with the server i.e. moving up on to how to use the messages that the protocol (and the implementation at point 1 above) makes available. This involved more reading, thinking, discussing and structuring than coding. The concrete result is a clear client strategy for knowledge acquisition and a very basic initial skeleton of its implementation. However, as the *structure* of the knowledge to be acquired was not yet defined or even fully clear, the next steps (see June below) had to focus on clarifying that, before the implementation can be pushed any further.

June started with fleshing out the hierarchy of data in Eulora as an initial draft capturing the relevant knowledge so far. This helped clearly identify the big areas that required further detailed investigation, most notably the graphics part where neither the required building blocks nor the specific client approach were yet known in enough detail to allow any meaningful attempt to extract a relevant data hierarchy. So the rest of June was spent on figuring out the graphics part: acquiring a better theoretical understanding through study of the reference book for computer graphics2 and at the same time acquiring also a very practical knowledge of how things are done by first mapping broadly (and then ripping apart) the existing approach as it found itself spread and scattered through the legacy client, several plugins of the same and the several plugins more of CrystalSpace (CS) + the code of external libraries (cal3d especially). By the end of the month, the better understanding of the whole graphics part and the practical experience with it resulted in a working initial example of loading some specific type of graphics object and a clear structure of the notions involved. This structure doubles as the map to follow for the next steps towards making the client able to load everything on demand.

July focused entirely on graphics in Eulora, figuring out one by one and implementing a prototype loader for each of the sections in the map written at the end of June:

  1. Loading a static (incapable of movement) object given through its list of vertices, normals at vertices, colours at vertices, texture mappings at vertices and triangles that make its faces. This includes already loading factories and textures.
  2. Loading the landscape aka loading the terrain given through its heightmap and material palette. This includes also loading lights, materials and shaders. The whole was pursued only to the extent that the terrain and materials were indeed loaded as specified by the heightmap + material palette but without fully aiming to solve other potential issues such as the "mirror" effect that is more related to lighting and - possibly - some material or shader properties.
  3. Loading a character made of several parts that can in principle change on the fly. This part was pursued for now only to the extent that the character is loaded and shown + answers the player's control but without fully going into the details of equip/dequip, actions and the full extent of changing bits and parts.

August was a bit split, with the holidays in the middle:

  1. At the beginning of the month, I continued the work on graphics but moved from the 3D issues to 2D, namely the user interface with all the different icons and images and various bits that are sometimes known as the "skin" of a client. This required a deep dive into yet another plugin of the legacy client, the widgeting PAWS.
  2. At the end of the month, I started working on writing down the 2nd draft version of the client data hierarchy integrating the results of all the work on graphics + all the pondering and figuring out done throughout the month + the further discussions and clarifications in the SMG boardroom on the desired working of Eulora's communication protocol. This is still in the works currently but I plan to publish it for discussion as soon as I've got down all that I have so far and certainly no later than the end of next week (ie no later than 15th of September).

The current plan for September and beyond is this:

  1. Write and publish for discussion/revision the deeply revised (re-written really) draft of client data hierarchy. Deadline is 15th of September.
  2. Pending feedback and discussions, possibly dive again for more details if needed and/or revise further the data hierarchy, if/as required. This is a crucial part for moving forwards but it's also one that once fixed will be very costly to change.
  3. Work on the implementation of the client and server-side pieces of the SMG protocol that are still missing, as soon as they are set/clarified. At the moment, they are listed broadly and in this order: client-side data storage (aka cache, of which currently there is only a skeleton, pending better understanding of what the data is going to be exactly), object request/reply, client actions, full client logic for knowledge acquisition and storage, full client loading/use of local data cache, client and server management of own keys.

  1. to be refined/reviewed as other parts such as key storage and bootstrapping become clear 

  2. Foley, J.D., van Dam, A., Feiner, S.K. and Hughes, J.F., "Computer Graphics: Principles and Practice", 2nd ed., Addison-Wesley 

Ossabot and Its Flask of Python 2.7 on CentOS 6

Monday, September 2nd, 2019

Ossabot is a clone of Stanislav's logger, consisting of 2 separate parts:

  1. An IRC bot that connects to specified channels and records everything it sees to a local Postgres database.
  2. A "reader" that takes everything from the database and formats it as handy html pages of the sort you can admire at http://logs.ossasepia.com.

The main reason to run a logger in the first place is simply that I quite need it in order to keep track of and reference conversations in my #ossasepia channel. But on top of this and as mentioned previously, providing a public logger for tmsr channels is a basic contribution and one where redundancy is key in the long term. Not to mention that running a bot as part of a synchronised fleet is the closest concrete step I can see at the moment towards Gossipd - as soon as you stop to consider the issue of how to synchronize with others, you stumble towards/upon Gossipd quite inevitably, even if you had no previous idea as to what that was. In one word, running a logger is needed, useful and a stepping stone in the desired direction so it jumped to the front of the (long) queue of work waiting to be done.

For lack of a better option, my logger clone is currently running on CentOS 6. For once, it turned out that my choice of environment was too old even for republican code - the logger and especially its dependencies1 require Python 2.7 and Postgres 9 or 10 while CentOS 6 runs with Python 2.6 and Postgres 8 as default version. Given that the logger had been previously deployed and tested only on Gentoo and combined with the fact that software versions are rather meaningless across different environments, the initial attempt was an exercise in frustration that nearly ended in total abort of the attempt to cross the Gentoo-CentOS divide. Nevertheless, a second attempt was successful, taking advantage of the experience and of a different set of version numbers graciously made public by lobbes. Here's what it took to have Ossabot running on CentOS 6:

  1. Postgres 9.6 and related packages (-devel, -contrib2 ) installed, configured, initialised and started as a service:
    sudo rpm -Uvh http://yum.postgresql.org/9.6/redhat/rhel-6-x86_64/pgdg-redhat96-9.6-3.noarch.rpm
    
    sudo yum install postgresql96-server postgresql96
    Installed:
      postgresql96.x86_64 0:9.6.15-1PGDG.rhel6
      postgresql96-server.x86_64 0:9.6.15-1PGDG.rhel6                                         
    
    Dependency Installed:
      postgresql96-libs.x86_64 0:9.6.15-1PGDG.rhel6        
    
    sudo yum install postgresql96-devel postgresql96-contrib
    
    Installed:
      postgresql96-contrib.x86_64 0:9.6.15-1PGDG.rhel6
      postgresql96-devel.x86_64 0:9.6.15-1PGDG.rhel6
    
    sudo service postgresql-9.6 initdb
    sudo service postgresql-9.6 start
    sudo chkconfig postgresql-9.6 on
    

    The path for pg_config was for some reason not correctly set/updated when installing the 9.6 as opposed to default 8 Postgres, hence:

    sudo ln -s /usr/pgsql-9.6/bin/pg_config /usr/sbin/pg_config
    
  2. Creating the db and user for the bot according to its handy README:
    # su - postgres
    -bash-4.1$ psql
    psql (9.6.15)
    Type "help" for help.
    
    postgres=# create user ossabot createdb;
    CREATE ROLE
    postgres=# alter role ossabot superuser;
    ALTER ROLE
    postgres=# create database ossalog;
    CREATE DATABASE
    postgres=# grant all privileges on database ossalog to ossabot;
    GRANT
    
  3. To set local connections to Postgres as trusted so that the various scripts may run without requiring passwords/authentication:

    sudo nano /var/lib/pgsql/data/pg_hba.conf
    changed in pg_hba.conf (trust for local connections)
    sudo service postgresql-9.6 restart
    
  4. And creating the database for the logger, according to its README, simply run from logger's dir:

    sh init_db.sh
    
  5. The first dependency of the bot is psycopg2 that works fine when installing the default system version, i.e. simply:

    sudo yum install python-psycopg2
    
  6. For Python 2.7 on CentOS 6, one needs the software collections (scl) repository:
    sudo yum install centos-release-scl
    sudo yum install python27
    
  7. Once scl and python27 are installed, the best approach is to enable it in a separate session of its own, to avoid mixing stuff and going nuts. For example:
    $ scl enable python27 bash
    $ python --version
    Python 2.7.16
    

    The scl will install a whole bunch of executables specifically for the Python2.7 environment. It's unclear to me how the exact place of those is chosen and I didn't bother to dig that up but at any rate, on my machine at least it ended up in /opt/rh/python27 so that the bin dir in there contained for instance:

    $ ls /opt/rh/python27/root/usr/bin/
    easy_install      pydoc             python-config       rst2s5          sphinx-quickstart
    easy_install-2.7  pygmentize        rst2html            rst2xetex       virtualenv
    nosetests         python            rst2latex           rst2xml         virtualenv-2.7
    nosetests-2.7     python2           rst2man             rstpep2html     wheel
    pip               python2.7         rst2odt             sphinx-apidoc
    pip2              python2.7-config  rst2odt_prepstyles  sphinx-autogen
    pip2.7            python2-config    rst2pseudoxml       sphinx-build
    
  8. Within the Python2.7 enabled session, you can start installing the various dependencies required by the bot, specifying the precise version required (note the export of LD_LIBRARY_PATH when sudo-ing):
    sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install Flask==0.12.2
    Successfully installed Flask-0.12.2 Jinja2-2.10.1 MarkupSafe-1.1.1 Werkzeug-0.15.5 itsdangerous-1.1.0
    
    sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install requests
    Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
    
  9. With all the above installed (and the unused cache stuff removed from the bot's code), the eat_dump.py, reader.py and bot.py finally run, hooray! Note that you might want an up-to-date dump from one of the older bots to get historical data. If you use such a dump, you'll need to change manually in it the database user name to match what you have (or the restore/import will fail). Still, you need now to take care of installing and configuring Apache (or similar) too:

    sudo yum install httpd
    sudo nano /etc/httpd/conf/httpd.conf
    

    To configure virtual hosts on apache, you need to change the httpd.conf file above, adding for instance:

    < VirtualHost *:80 >
      ServerName logs.ossasepia.com
      KeepAlive On
      ProxyPreserveHost On
      ProxyPass "/" http://127.0.0.1:5002/
      ProxyPassReverse / http://127.0.0.1:5002/
    < / VirtualHost >
     

    And then start Apache and set it so it starts automatically on reboot:

    sudo service httpd start
    chkconfig httpd on
    
  10. The last step is to write scripts to backup the database and to start the bot + reader parts of the logger as well as set up cron jobs to call those scripts. Note that the scl use means that scripts for starting the bot/reader on CentOS will have to first enable python27, for instance:

    scl enable python27 "python /home/youruser/bot.sh /home/youruser/ossabot.conf >> /home/youruser/bot_log.out"
    

As it turns out, standing up the logger on CentOS 6 is an overnight success, quite literally. The above steps should hopefully make it more of an 1-hour task but if they don't, kindly let me know in the comments below. Meanwhile the bot has been running quite nicely and even turned out some interesting issues regarding loglines numbering and sync between several bots. Interesting means of course that they aren't yet solved either so just in case you were looking for something to do, here's your chance.

The code of the logotron is mirrored together with Stan's and my own signatures, on my Reference Code page.


  1. My full list as gathered from the installation notes on CentOS is this: postgresql96-server, postgresql96, postgresql96-libs, postgresql96-devel, postgresql96-contrib, python-psycopg2, python27 (centos-release-scl), Flask (Jinja2, MarkupSafe, Werkzeug, itsdangerous), requests (idna, chardet, certifi, urllib3). Infuriatingly, the python uninstaller does *not* care about the very dependencies that the python installer brought in so if you want to uninstall stuff, you need to do it manually for *all* the above packages. Such progress, such automation, such (w)ow. 

  2. NB: those are needed e.g. -contrib contains the pg_trgm extension that provides fuzzy string matching and is used by the logger 

The Widgeting Paws of Pointerism (Notes on Graphics in Eulora, V)

Tuesday, August 6th, 2019

It took indeed more than a week and I'm rather surprised it didn't actually take even longer than two weeks to get *anything visible* out of it. It's all one strand like Stan's spittoon after all, so there's rarely just "one sip" that doesn't bring in the whole rot with it and I have to fight over and over the urge to just burn it all down and start a sane thing from scratch1. Nevertheless, a lot of deep breaths and assorted cussings later, here's a brief summary of the main achievements, in the order they were made:

  • some useless crap was removed and the whole thing made to work without (e.g. the doll with its dependence on a special sector in the map file);
  • the initial loading and "pre-caching" of meshes, textures, materials, maps and 2D stuff was excised (there are still the shaders remaining though as well as 1001 "preferences" in as many files in yet as many different places, sigh);
  • the actual handling of 2D icons and GUI images in general (e.g. buttons, borders, arrows, window decorations) aka the PAWS was dissected and exposed and roughly mapped as ugly as it is;
  • as recorded proof of breaking PAWS and because of its shameful lack of proper way to load images on actual demand, the whole thing has had a run with the stupidest "mouse pointer" available in there and otherwise with absolutely no 2D images at all for the very lulzy effect that it even looks better in some ways than with all of the various pieces of "art", here it is:
    That snot on the screen is the mouse pointer on the black background starting screen.

    That snot on the screen is the mouse pointer on the black background starting screen.

    PAWS's shame: the better-looking interface with no images whatsoever (clicking the invisible widgets still works!)

    PAWS's shame: the better-looking interface with no images whatsoever (clicking the invisible widgets still works!)

The path that led in the end to the above started with me looking - as initially planned - at how the character (as moving sprite) is handled or rather mishandled, consider: there is the CEL layer that supposedly provides on top of CS the sort of structures needed by games to store and manage the various entities and objects; PS however "uses" it in principle but in practice it still mainly does its own thing anyway (and by now I strongly suspect that this pattern is simply because nobody ever spends the time to understand what they use); specifically, the PS way is to have a pscelclient class that supposedly manages game entities on client side (shouldn't CEL have been enough for that, you ask? most probably yes but that'd have taken someone to figure out first exactly how it works and who has time for that!!); but then there is *also* a pscharapp class that handles the appearance of the character (apparently that's too low stuff for "management" to handle) and there is also a psclientchar that moves about equipped items because this is yet neither management nor appearance but absolutely something separate - though you'll be hard pressed to notice much separation in the code for all this theoretical separation; and to top it all, the inventory and doll views that precisely reflect character's appearance and equipped items are actually the domain of something else entirely - the PAWS; and the PAWS (which is supposedly so separate as to bother to be a plugin by itself) in fact messes about with pretty much anything and everything from the engine to shaders and textures and 2D images (which are still loaded as... textures, but those are textures-this not textures-that) and xml-defined maps and hardcoded paths to all sorts of files it expects - nay, demands - to be present and just-so and at-start but oh-so-configurable. At which point it became clear that the PAWS mess is indeed such a large part of the whole mess that I'll simply have to get to the bottom of it just in order to move further at all.

Looking at PAWS more closely revealed that it's meant to be a GUI system. This initially rather puzzled me because there IS in fact already another GUI system in there, one that PS uses/relies on as a dependency, namely CEGUI. Then again, the PS style is to "use" something by simply sitting on top of it and mushrooming a lot of managers with hardcoded paths and expectations that they know the whole possible world upfront so there should be no surprise at PAWS, which is precisely this sort of thing - it mushrooms widgets and pointers rather than something-else and pointers, that's about all the difference from all the other PS masterpieces.

Masterpiece aside, what IS this GUI system PAWS? As far as I can tell, it's a lot of widgets so well organised that there are no less than 3 methods "FindWidget" because the 1st one (defined in pawswidget) checks the current widget and its children only, the 2nd one (in pawsscript) checks also the parent of the current widget while yet the 3rd one (in pawsmanager because yes, of course there are all sorts of managers) simply delegates it all to "mainwidget" aka a sort of calling the 1st one from the top of the tree (but no, they couldn't possibly notice that it's meant to be a tree and there are well-known algorithms for searches in a tree). Anyways, the fun part with widgets is the way in which they are defined really as each widget has an .xml file (in data/gui) that supposedly defines appearance only (though more concretely it's about which elements it contains and on what position) and a .cpp file (in client/gui usually) that supposedly defines behaviour. The funny bit however is that both of those, for all their pretense of clear separation (they even are in totally different dirs, yes?), quite directly reference both one another's bits and pieces AND global, all-of-a-sudden-made-hardcoded parts: the appearance references by name other widgets (that it uses for specific roles and in specific places so it's not just a name really) and "resources" aka pre-loaded images; the behaviour code in turn has to reference various components of course. And in all this, apparently nobody noticed that before you can talk of behaviour and of appearance, there has to be somewhere a *structure and nature* of the thing that actually is the core and therefore the driver of everything else... Then again, what structure and nature and such silly notions, when the whole thing is bent on defining anything and everything by its current, temporary position or at best some role it might fulfill for a while2.

The widgeting PAWS further has all sorts of notions regarding appearances: there are "skins" and "styles" and "resources" and they are all both hardcoded and expected to be fully known upfront. The Great Paws knows not only all the world that is but also all the world that could possibly be (the world as it defines it is all nothing but labels anyway so it makes sense in a way). And it enforces this by whining and even wagging its finger at the user if it ever finds that a file with what it *knows* is the "right" name is missing from its expected place, quite literally giving orders to the user, as the message spit is this:

"ART ERROR: PawsTextureManager loaded the image %s which was missing from the imagelist.xml and was loaded on demand. Add the image there!"

I'll add to the above: and blow your nose and stand up straight and cut your nails shorter! Imagine that, being such a shitty user that the poor, poor PAWS had to actually load an image on demand (how degrading!!), you nasty user, you! And note that this "on demand" is anyway the sort of "you demand what I'm willing to do or no dice" not any sort of actual demand: it only means it deigned to look (because it had to use that image) in the *same place* it looked initially (when it tried to load everything that there should ever be) just because you are the sort of shit that failed to put the image there when it looked the first time, you know? What sort of user does such a thing to a poor piece of code, you tormenter, you, shame on you!!!

Before you get your hopes up that there is at least some sort of "loading on demand", bash those hopes one over the head: for one thing this "on demand" still relies on the predefined list of what there *can* ever be so that if you request a "resource" it doesn't know about than tough luck, it won't try to look for it, it's not that stupid to do something useful; for another thing, there is no way to actually call this on demand: it's just a way of doing rigidly the same thing but expecting a different result because meanwhile the world should have better come to its senses and fit the expectations already, what.

If the above sadness is not enough, add to it this significant bit: PAWS considers itself a plugin and as such entirely separated from the actual client and therefore unable to use whatever the rest of the client knows. Sure, in practice this separation "works" by means of the main client class simply passing pointers to PAWS on creation - why use CPP if not for pointerism, after all. Moreover, the added benefit of this separation is that PAWS also gets therefore to have its own set of repeated XML parsing of all sorts since it can't use the already-repeated XML parsing that the client anyway suffers from. And finally, it makes somehow perfect sense to "separate" the main client from the GUI system seeing how they both use the same graphics engine anyway and moreover the main client loads some images as textures while PAWS loads...well, some images as textures, yes. But they are not the same "class" of textures, so that's significant separation, see?

Besides deep red, I tell you what I further see: since I have no intention to pass even more pointers on or otherwise to force this "plugin" nonsense on everything just for the very specialness of PAWS, it's way more likely that the paws will be clipped more to size and therefore will take their place as yet another of the many classes inside the main client, possibly together with all the rest of very-special-plugins if need be (of course the dependencies are not that simple). I'm still pondering this, mainly because the idea is to touch as little as possible the code rather than as much as it takes to clean it (mainly because clean it won't be anyway). But one way or another, I will have to find some reasonable way to bring PAWS in line with the rest and have it go and load resources from wherever (and whenever) it is told to. This might even require more discussion in S.MG's boardroom but at the moment and as a result of all the adventure above, I have at least a much clearer idea as to what the 2D part is all about so there is therefore a point to even start the discussion from.


  1. While it might *seem* faster that way, it's not, nor does it really make sense if one calmly considers everything involved. Urges like this are simply focused on getting rid faster of the current set of problems and do not care nor stop one second to consider the alternative set of problems that are bought by such "solution". So yeah, urges make very poor guides, how surprising. 

  2. And no matter how much I try to *not* see it this way, it always is the case that the closer one gets to the "how things look" and therefore in this case to graphics, the more confusion and more monkeying there is. 

Overview of FFA Ch1 - Ch19

Monday, July 29th, 2019

Over the past few days I've taken the time to finally go through all the currently published chapters of FFA (1-19 with some bis and a,b,c on the way too) and got as a result a clearer idea of the whole thing as it builds up from basic definitions of large integers to the Peh interpreter machine with its own registers, two stacks (data and control, respectively) and the Cutouts mechanism for controlling access to sensitive data. While initially I was planning to add to my previous notes on FFA Ch1-Ch4, my experience of getting back to FFA after a significant break pointed to the need for a higher-level overview of what-where-why even before looking for such detailed illustrations of the code involved. So here's my current overview of FFA - open to any criticism you are able to throw at it, as always - consisting first of a brief list of main approaches in FFA and then the 4 "layers" that I see to the whole thing so far.

Main FFA Approaches and Algorithms

  1. Strictly non-branching on input values and therefore practically the death of "if". No ifs, no buts, so what's left is MUXes1: essentially the "if" goes down one level, from a logical switch (which operation to perform) to a mechanical one (which buffer to copy the result from); the same operation is always performed but its result is not always used.
  2. Strictly constant-time execution for any given inputs size. This means in practice strictly worst-time, of course, as it's always the case when you require all to be the same. The only way for all to be equal is by making them all equal to the worst, no way around it. Then again, some people's worst turn out better than other people's best so take "worst" with a pinch of context, as always.
  3. Comba multiplication algorithm: calculating sequentially the *columns* of a product; additional optimisations for the squaring case (mainly avoiding calculating the same products twice).
  4. Karatsuba multiplication algorithm: calculating the product XY as a combination of cheaper operations (additions and shifts preferred by far to multiplications) on the half-words of X and Y; additional optimisations for the squaring case.
  5. Barrett's modular reduction: calculating X mod M by means of an approximation using the "Barrettoid", B, that can be precomputed from any given module; X mod M is approximated first as X - M*B and then adjusted by at most 2 subtractions of M (i.e. the error of X - M*B as approximation of X mod M has only three possible values: 0, M or 2M).
  6. Stein's binary GCD: rather quasi-Stein as it's adapted to FFA requirements, most notably the constant-time execution requirement (by iterating over the full number of bits rather than stopping when the smallest number is 0).
  7. Miller-Rabin compositeness test: an adapted version of MR, forcing any given witness into the acceptable range and providing a "one-shot" test for any given value N and witness W.
  8. Peh interpreter machine: basically the "cryptographic machine" that the user gets to push the buttons of; Peh has a data stack and a control stack, a set of registers and an instruction set including the ability to define and call subroutines and loops as well as to define Cutouts aka a protected area for sensitive data.

FFA Layers (from bottom up)

  1. Representation of arbitrarily large integers as fixed-size (i.e. specified for every run) arrays of machine-words (with iron-specific size of machine-words changeable via knobs in the code). In FFA terminology, such integers are called "FZ".
  2. Basic arithmetical, modular arithmetical and logical operations on FZ entities in constant time and handling any overflows/underflows. This part builds up from the most "naive" approach (aka most straightforward if rather costly) dubbed "egyptological" through successive refinements (optimisations of the original algorithm followed by radical changes of approach) until the final result uses Barrett's reduction for modular operations, Karatsuba with Comba core-case (i.e. below empirically-established thresholds) for multiplications and a variation of those (Karatsuba + Comba + specific threshold) specifically optimised for squarings.
  3. Primality-related algorithms working in constant time on FZ entities: Stein's binary GCD2 and an adapted MR3 that can adequately handle any value of a given witness (by forcing it effectively within the valid range).
  4. Peh, the finite-tape, dual-stack and cutouts-enabled interpreter. While I still need to turn this inside out a few times more, I can tell already that I want a Peh-iron!

And while there are still a lot of experiments and timings and playings to go through with all the FFA, I'm happy at this point to sign the remaining chapters as I went through them with pen in hand and as far as I can tell they do what it says on the tin (not to mention that I certainly would use FFA and in fact I'm itching to drop already the shitty gpg, ffs!):


  1. Multiplexer aka a way to select only one of several inputs. 

  2. Greatest Common Divisor 

  3. Miller-Rabin 

Naked Models (Notes on Graphics in Eulora, IV)

Wednesday, July 24th, 2019

What's the first change you'd do to an empty-headed character such as Testy? Why, chop some bits off, change his sex and get him naked at least, even transparent at that, since he's pretty much just as tedious and boring as all children dolls 1. Of course the first attempts failed to produce anything visible because it turns out that using Cal3D sprites in Eulora's client goes through several layers of inmisdirection: Crystal Space (CS) wraps the Cal3D code in its own SprCal3D plugins and factories and whatnots; on top of that, there is also the rather elusive Cel (Crystal Entity Layer) that is meant to be a game-specific part of CS keeping track of game entities; and messing within and across all is Planeshift's (PS) own GEM concoction interlaced with delayed loaders and Dead-Reckonings2 and cloning of materials (why? no idea yet) and surprising, undocumented transformations. Still, after a while of hacking through all the above, I could at least get some parts of Testy showing... if lying down rather than upright:

Spot the bottom in the grass!

Spot the bottom in the grass!

The prostrate position makes at least sense in that "movement" also looks a bit like some swimming look-alike: parting the legs and advancing sort of thing. Then again, it turns out that using a different model (e.g. Cally) makes it even more interesting - the model is upright when waiting but it dives straight back to the horizontal as part of moving and it furthermore cares not one bit for whatever movement animation one loads. So there's a clue for another day - current movement needs a good bashing over the head and a deep dive to figure out what's all this nonsense. But until then, here's naked Cally who borrowed Testy's skin for a spell:

What bits is she missing?

What bits is she missing?

Anyway, since position of the model is always a matter of applying one transformation or another, I went ahead and applied the obvious transformation, 90 degrees and upsy daisy on her feet:

Upsy-Daisy with muddy skin!

Upsy-Daisy with muddy skin!

And funnily enough, once the 90 degrees rotation is applied to the factory (i.e. presumably to ALL entities built out of that factory) the Testy model remains upright even during movement without any perceived change. On the other hand, the Cally model is upright without the 90 degrees rotation but nevertheless dives down for movement and pops back up as soon as movement is done. So it would seem that movement has its own assumptions that it doesn't bother to make public, how lovely. Seeing how it's most probably to do with the model's initial position or similar, I might even need to end up again with the full environment, Blender and all installed to get to the bottom of it and probably see exactly what or how the exporter is also broken and specific. Such joys to come but for the moment they are not yet here since the more pressing need is first of all to disentangle some more the PS+CEL+CS+CAL3D mess regarding animated entities so as to actually be able to *use* the walk/fight/move animations rather than just "add" them and moreover to be able to add and remove on the fly as many and as diverse entities as wanted, beyond the "main character" guy. And all this is getting deeper into PS swamps so it's likely to take more than a week indeed - ideally by then I'll be able to cut off at least some parts of the existing code (precaching and preloading especially) and lighten the client start a bit if nothing more.


  1. Changing graphics is this exercise in tediousness and pointlessness, it eerily reminds me of playing with dolls, just one step lower than that since those are just images of dolls, fancy that. And playing with dolls was utterly boring even when I was 5! It's like an endurance test along the lines of watching paint dry. Tinker with this and tinker with that, all for "the looks" of it and can't even satisfyingly throw it at anything to hear it break. 

  2. Not kidding, although it's way less exciting than the name they had the chance to land on: it's mainly looking to see if your character falls down and breaks their neck. 

The Mirror Land (Notes on Graphics for Eulora, III)

Wednesday, July 17th, 2019

Getting some actual landscape as in getting the height right turns out to be only half (the reasonably well-working half) of the terrain task: once I had in place at least the generic mechanism for creating factories and meshes, the terrain factory required only a few iterations, false starts and mandatory dives into CS's entrails to get working and save Testy from his usual fall into the sky-sphere:
Terrain generated from heightmaps only (no mix of materials, only base material).

After setting up the lighting a bit better too, it looks even more reasonable (next to the crater that is currently empty, yes):
Base terrain from heightmap.

The truly frustrating bit turns out to be the painting of the landscape in the desired materials. For the relatively easy part, the Terrain2 construct of CrystalSpace provides a "terraformer" that can be fed a "material palette" (i.e. a set of materials to use) and an indexed image - as a result, the terrain will have a reasonable mixture of the materials depending on heights and an ok fading of the materials into one another where they meet so it doesn't look that terrible, especially once all normals are in place too so that lights stand a chance to work:
Terrain from height map *and* material palette with 3 materials: grass, sand and rock.

The trouble however is that so far I haven't quite managed to get to the bottom of turning down the reflective mirror-like surface of this landscape! In theory, materials have those 2 light-reflecting properties, namely the extent to which they are "diffuse" or "specular" (i.e. "shiny"). Still, for all my futzing around with those for all three materials (sand, grass, rock), the observed effects are pretty much non-existent. So much for the theory and welcome to this graphics thing of tinkering. To see clearly what I mean, here's a rather surreal rendering with the ambient light turned on to higher levels so it's more obvious (note that the effect is precisely the same even with low ambient light - it's just harder to realise what it is exactly; similarly and as expected, one can mask the effect to the point that "it's fine" by turning on the fog button - it makes sense since it dulls the ambient light, I'd say):
mirror_land

On the positive side, the futzing with lights and whatnots is really something of less concern for me at this moment since I'm not that interested in obtaining an image *just so* - my goal is in figuring out *what* sort of things one can set, what do they do and how to set them while the game is running. In this sense, at least the basics of the terrain seem to be in place for now: heights are read from a heightmap file, the materials are indeed used according to heights, the rest is for another day. I suspect though that in usual tradition, the only real solution here is to dig in the end in that shaders pit as well since it's most probably a shader thing: the terrain object uses a different, terrain-specific shader and for some reason the defaults for that end up producing more of a mirror than a soil, at least given *everything else* (such is graphics, everything and anything can be the culprit). Note that I specifically used and set precisely the same parameters for materials as they are in the client currently in use - so it's most probably something else/somewhere else that I'm not setting just right for now.

The next step now is to figure out animated meshes too, hence Cal3d objects. So in the next episode, it's Testy himself that will change and get created directly from code. Once I have at least one way to do this too, I can go back a bit and look again into integrating them into (or extracting them out of...) the rest of the PS infrastructure. Then again, before doing that, I might still need to dive in and figure out the shaders in more detail to extract some way of specifying them without xml and directly from code too, perhaps. That promises though to be a lot of figuring out and additional mess so depending on whether it can or not wait a bit longer, it might not be exactly next in line - basically I have lots of work competing for my attention, a sort of inverted looking-for-work, it's the... (lots and lots of) work looking for me!

A Working Cuntoo Install on AMD FX-8350 (with script)

Sunday, July 14th, 2019

Since the relatively recent discovery that Cuntoo work is apparently not really going anywhere for lack of more public testing and reporting, I'm doing my bit and publishing what I've got so far on installing Cuntoo on a local box and prodding it for a while. First, to introduce the box in question, so you can easily replicate it:

  • AMD FX-8350, 4GHz, Octa Core
  • Gigabyte Ultra-Durable GA-78LMT-USB3
  • 2*8GB DDR3 RAM
  • 1TB Seagate Barracuda

Since I wanted to do this from scratch to make sure it's easy to replicate, I ignored my previous Cuntoo-compilations and I made myself first a bootable USB-stick with Gentoo's livecd. Initially I took the minimal install_amd64_minimal_20190703T214502Z.iso but that proved iffy to use because it was apparently a bit too spartan for my requirements (esp. no make on it). So I traded space for time as it were and used instead the much larger livedvd-amd64-multilib-20160704.iso (2.2G vs 292M for the minimal iso). Perhaps this was quite overkill and there is something in between that does the job just fine so if you find/have that, just use it - the bootable USB-stick really is just a way to "get in," nothing more. Note that on some .iso images you might need to run isohybrid first if you want to use them to boot from USB (rather than burning them on a CD/DVD). As for making the bootable USB stick, that's as easy as using dd to transfer the .iso to your USB drive - do be patient if your .iso is big as dd is silent.

Armed with the bootable USB stick, I popped it into one of my USB 2.0 slots and turned the machine on. If you have the same setup as mine, the F12 key will give you the boot menu (or Delete for the full BIOS menu) and you'll need to select the USB stick *from the hard-drives* (you'll need the PgUp/PgDown keys and not the +/- as some docs mislead me). Once the live-dvd Gentoo image boots, you can proceed with the Cuntoo-related tasks, in order (note that you'll be doing all this inside the live-dvd image so if you reset, *the environment will reset too and your changes will be lost* so you'll need to start from the beginning again. With many thanks to Hannah for her mini-series of posts detailing her ordeal with Cuntoo and therefore inspiring my script-making right from the start, here are the steps and the relevant script parts:

  1. Add to your /etc/hosts file the IPs of any websites you'll need - the exact list depends on what you choose to do at each of the following steps (e.g. where do you take your V from) but at the very least you'll need the IP1 for trinque.org to get the Cuntoo tarball!
  2. Obtain and import any gpg public keys you need - similar to the step above, the exact list here depends on whose stuff you use but as a minimum you'll need Trinque's key, of course. As I've used only Trinque's and my own stuff, my gpg-importing bit of the script looks like this:
    # gpg keys
    curl -v "http://ossasepia.com/vpatches/diana_coman.asc" -O
    gpg --import diana_coman.asc
    curl -v "http://wot.deedbot.org/FC66C0C5D98C42A1D4A98B6B42F9985AFAB953C4.asc" > trinque.asc
    gpg --import trinque.asc
    
  3. Get and install a working GNAT. My choice here for expedience is to use the frozen Adacore version that I'm hosting on this site. Correspondingly, the relevant part of the script grabs it from my website, checks the provided sha512sum on it and installs it (if the GNAT dir doesn't already exist from a previous run perhaps):
    # GNAT
    GNAT="gnat-gpl-2016-x86_64-linux-bin";
    GNATZIP=$GNAT.tar.gz
    
    if [ ! -e $GNATZIP ]
    then
      curl -v "http://ossasepia.com/available_resources/gnat-gpl-2016-x86_64-linux-bin.tar.gz" > $GNATZIP
    fi;
    
    if [ ! -e $GNATZIP.sha512 ]
    then
      curl -v "http://ossasepia.com/available_resources/sha512sum_gnat-gpl-2016-x86_64-linux-bin.tar.gz.txt" > $GNATZIP.sha512
    fi;
    
    if [ "$(sha512sum $GNATZIP)" = "$(cat $GNATZIP.sha512)" ]
    then
      echo "Matching sha512sum for Adacore-GNAT."
    else
      echo "FAILED sha512sum test for Adacore-GNAT, stopping."; exit
    fi;
    
    if [ ! -d $GNAT ]
    then
      tar -xvf $GNATZIP
      cd $GNAT
      sudo ./doinstall
      cd ../
    else
      echo "GNAT dir already exists so GNAT is assumed installed! Skipping GNAT install."
    fi;
    
    PATH="/usr/gnat/bin:$PATH"; export PATH
    echo "Updating PATH variable to $PATH in order to use GNAT."
    
  4. Get and install your favourite V. Make *sure* it can be found (e.g. update your PATH variable with V's location). Once again, since I can rely on my previous work on packing V (namely mod6's implementation of V), my life is that bit easier now and my script relies on my website to get and check everything:
    # V
    VDIR="starter_v"
    VZIP=$VDIR.zip
    if [ ! -e $VZIP ]
    then
      curl -v "http://ossasepia.com/vpatches/starter_v.zip" > $VZIP
    fi;
    if [ ! -e $VZIP.diana_coman.sig ]
    then
      curl -v "http://ossasepia.com/vpatches/starter_v.zip.diana_coman.sig" > $VZIP.diana_coman.sig
    fi;
    
    echo "gpg --verify $VZIP.diana_coman.sig $VZIP"
    gpg --verify $VZIP.diana_coman.sig $VZIP
    
    if [ ! $? -eq 0 ]
    then
      echo "FAILED sig check on $VZIP, stopping."; exit
    else
      echo "Sig check fine for VZIP"
    fi;
    
    if [ ! -d $VDIR ]
    then
      unzip $VZIP
    fi;
    
    cd $VDIR
    chmod +x build.sh
    ./build.sh
    cd ../
    PATH="$PWD/$VDIR:$PATH"; export PATH
    echo "The PATH var for using gprbuild, vk.pl, ksum, vdiff and vpatch is: "$PATH
    
  5. Get Cuntoo itself (note that it's ~800M so depending on your connection, this step may take a while):
    # Cuntoo itself
    CUNTOO="cuntoo"
    CUNTOOZIP="$CUNTOO.tar"
    
    if [ ! -e $CUNTOOZIP ]
    then
      curl -v "http://trinque.org/$CUNTOOZIP" > $CUNTOOZIP
      curl -v "http://trinque.org/$CUNTOOZIP.sig" > $CUNTOOZIP.sig
    fi;
    
    echo "gpg --verify $CUNTOOZIP.sig $CUNTOOZIP"
    gpg --verify $CUNTOOZIP.sig $CUNTOOZIP
    
    if [ ! $? -eq 0 ]
    then
      echo "FAILED sig check on $CUNTOOZIP, stopping."; exit
    else
      echo "Sig check fine for $CUNTOOZIP"
    fi;
    
    tar -xvf $CUNTOOZIP
    cd $CUNTOO
    

Once all the above finished without any error or problem, all you need to do is to run the bootstrap.sh script according to Trinque's instructions. If you are at a loss as to the kernel config - by far the ugliest part in this - and your hardware is similar to mine, I was able to use Stan's "Dulap-II" config with minimal (possibly it works even without any) changes so go grab that one and feed it to the script - note that you'll get a quite spartan but absolutely functional environment and you can customize it from there as you want it. Here's the full bash script too if you want to use it: cuntooprep.sh.

NB: at the moment I'm writing this, the known path-issue within Cuntoo's scripts is still present in the Cuntoo.tar file so you'll need to do the change manually for the signature to verify on the genesis.vpatch that you obtain at the end. The change you need to make is explained in the logs and you can do it after running the bootstrap.sh script if you already started that.

Once you get the genesis.vpatch, verify it against Trinque's signature - if it doesn't verify, chances are that you still need to fix the paths issue so see the paragraph above and remember to actually re-run the scripts/make_portage_tree.sh script after you made the change! Then simply reboot and choose to boot from the disk on which you installed Cuntoo. You can then recompile the kernel to tweak any configs you want, for instance with those steps:

  1. cd /usr/src/linux
  2. make oldconfig
  3. make menuconfig (and/or change whatever you want directly in the .config file before running this - I've ended up finding the *text* file BETTER than the bloody gui thing simply because nobody can tell where something is in there, ugh.)
  4. make && make modules_install
  5. cp arch/x86/boot/bzImage /boot/bzImageNew
  6. change /etc/lilo.conf to add the bsImageNew as another boot option (DO keep the old one too, at least until you are sure the new one is fine!)
  7. lilo
  8. shutdown -r now

Once you had enough of recompiling the kernel, the next step would be to have a look at what you'd like to add to your system - chances are there'd be quite a lot of things that you need before you can actually do a lot in there. Cuntoo has its own preferred repository in /cuntoo/portage and a distant second in /usr/portage. So when you run emerge, you'll see either ::cuntoo or ::gentoo appended to the packages required, depending on which of the repositories is used. To enrich the cuntoo repository and effectively get rid of the madness where gentoo just doesn't "have" anymore the packages you want, you'll need to copy the tarball of your package into /cuntoo/distfiles and the ebuild+manifest+patches+shit into /cuntoo/portage/package-path. My initial attempt with MySQL was a step too far at this stage as it served only to point out more clearly what a mess porting the dependencies to Cuntoo will be - MySQL itself requires all sorts! So I looked among those "all sorts" and picked out one with better chances of being standalone, namely curl. With this adjusted aim, I was actually able to "port" the curl ebuild to Cuntoo as it were, simply like this:

  1. Download the tarball without installing it: ebuild /usr/portage/net-misc/curl/curl-7.60.0.ebuild fetch
  2. Copy then the tarball to /cuntoo/distfiles
  3. Copy /usr/portage/net-misc/curl to /cuntoo/portage/net-misc/curl
  4. Go into /cuntoo/portage/net-misc/curl and snip away the ebuilds that are other versions than 7.60.0 as well as the references to them in the Manifest file (NB: you'll need to keep all the patches though regardless of what their names say as they are still applied to the 7.60.0 version anyway).
  5. Now run emerge --ask -v net-misc/curl and look closely: it should say ::cuntoo and it should report that it has to download precisely 0Kb. If all that is true, let it do it and you'll have gotten Curl in your new and sparkly Cuntoo!

Now I must say that I'm a bit at a loss as to whether the above curl-experience should get the grand name of "making an ebuild" or not or what - to me it looks more like hacking an ebuild at best and moreover a quick look at gentoo's docs re making ebuilds reveals enough *other* stuff to eat that I don't see it happening all that soon. So I don't even know - is *this* sort of hacking ebuilds that Trinque was asking for? Or writing them from scratch or something else entirely or what?


  1. Obviously, unless you use DNS but why would you still use that anyway? 

The Rocky Moon (Notes on Graphics for Eulora, II)

Monday, July 8th, 2019

My client-side Eulora task set last week is completed and as a result, the Testy character gets now to see something more than just fog in his Sphere-of-all-beginnings, look:

Testy under a Wavy-Blue Moon, in the Sky-Sphere of all beginnings.

Testy under a Wavy-Blue Moon, in the Sky-Sphere of all beginnings.

It turns out that CS can indeed eat - though not very willingly1 - "geometry" for fixed/static entities (aka "generic meshes") even without any xml in sight2. For static aka non-moving entities, there are 5 sets of numbers required:

  1. Vertex coordinates
  2. Normals at vertices
  3. Colours at vertices
  4. Texture mappings at vertices
  5. Triangles

The main trouble with the above was the exact "how" and "where" to actually set them so CS understands what they are. It turns out that the most obvious "those are the vertices and those are the normals etc" approach is "obsolete" and replaced apparently by having to specify a render buffer for each set of numbers. Of course it's still the same arrays of numbers either way, but with render buffers you need to also specifiy which *type* of render buffer you have in mind (i.e. what are those values for) and in pure CS style this is specified by giving the... name. Name that is afterwards internally translated into a numerical id of course but nevermind since it's supposedly as easy as it can get to have to remember that the name is exactly "position" and not "positions" nor "vertices" nor anything else3). So easy in fact that I ended up having the list of "buffer names" at hand all the time, great help indeed.

Before moving on to some basic descriptions of those 5 sets of numbers, it's worth noting that they are just the most usual 5 sets really: there are plenty others that one can specify though it's unclear exactly when and precisely why would one do. The easiest example concerns texture coordinates: one can specify as many as 4 different texture mappings. And why stop at 4 if one went all the way there, why not 5 or 6 or 7 or 1000? No idea. Anyway, for the basic descriptions:

Vertex coordinates are exactly what the name says: the 3D coordinates of vertices that define the entity's shape in space. Those are all relative to the position at which the entity is drawn at any given time, of course, since they are internal to the entity and unaware of anything else in the larger world.

Normals at vertices are perpendiculars on the surface of the entity at each of the vertices defined earlier. Those are quite important for lighting (together with the orientation of each triangle surface defined further on).

Colours at vertices specify the colour of the object at each vertex and allow therefore colouring of the full object through interpolation for instance. While in principle this can be skipped when a texture is given, the "skip" simply means default colour (black) so it's a "skip" only in the sense that it's less obvious it exists.

Texture mappings at vertices - those are tuples (u,v) that specify a texture point (2D because textures are 2D images) that is to be mapped precisely to each vertex. Just like vertex coordinates provide fixed points for the shape of the engine that is otherwise interpolated at non-specified points, the texture mappings provide fixed points for the "painting" of the shape with a given texture.

Triangles define the surface of the entity as approximated through triangular shapes. All surfaces of static shapes in CS are approximated through triangles and the only difference between curved and flat surfaces is essentially in the number of triangles required so that your eye is fooled enough so you like rather than dislike the lie. This is after all the whole computer graphics in a shell: a lot of effort spent to lie to you the way you like it.

Triangles in CS are given as triplets (A,B,C) where the values are simply indices of the vertices previously defined. So the simplest cube will have for instance 8 vertices for its 8 corners as well as 2*6=12 triangles for its 6 faces since each face, being a square, needs 2 triangles to approximate. And those 12 triangles mean 36 values in total since each triangle needs the indices of all its 3 vertices. Moreover, the order in which the vertices are specified matters as it gives the orientation of the approximated surface4 and that in turn is crucial for calculating lighting (and ultimately for figuring out whether you get to even see what is on that surface at all). Since it's not intuitive perhaps, it's worth noting the basic fact that a surface has 2 sides and the "painting" of a texture is strictly on one side only. So if you mess up the surface's orientation, you can easily do the equivalent of dyeing a garment on the inside without any colour whatsoever showing when you wear it unless you turn it first inside-out.

As you might have guessed from the above example with the most basic cube if from nothing else, even very simple entities require a lot of values tediously and boringly specified - hence quite the bread and butter of automation5, yes. For now and for testing purposes, I simply extracted the numbers of interest from the xml description of the moon object in game and then I used them as default values for an object created directly from code. And lo' and behold, Testy got a moon to stare at and the moon even got to paint itself stony when I'm bored of its usual wavy blue:

Stony Moon and Testy in the Sky-Sphere.

Stony Moon and Testy in the Sky-Sphere.

The next biggish step further in this line is to figure out how to generate terrain from just a heightmap and then how to create an animated (hence, Cal3D) entity from code rather than through the sterile xml-packaging. Looking even further from there, monstrous xml-shaders await in the same direction but in other directions there is still a lot of work to do on teaching the PS code to use the EuCore library and to discover as a result that the world is way larger than what fits in its inherited, well-worn and time-proven, traditional xml-bible.


  1. A bit like those finding out that they can actually eat food that was never packaged in plastic or washed in chlorine or even re-heated. 

  2. At least not in geometry-sight. In a bit wider perspective, all shaders so far are still balls of xml that will probably have to be untangled sooner rather than later. 

  3. If you give a non-existent name, the buffer is simply ignored since supposedly you can define your own buffers too so it's not an error. Combined with the wonderful way in which everything influences everything else in graphics, you get - if you are lucky - to puzzle over some unexpected visual effect that might even seem at first to have nothing at all to do with the names of the render buffers at all (maybe you messed up a few numbers in that long list, you know? 

  4. By convention, vertices are given clockwise as read when facing the surface 

  5. Automation can be here exporters from supported tools such as Blender and generators of known shapes/patterns.