Eulora Client Data Hierarchy v2.0

September 14th, 2019 by Diana Coman

This replaces the previous draft and is based on the new knowledge of client entrails of all sorts as acquired during those past 2 months. It is however still a draft in that it uncovers some new questions (at least new to me) and it requires anyway some feedback and discussion given its scope and crucial position in the SMG Communication Protocol. It's published as it is at this stage and by the promised deadline not because it's "complete and final" but mainly because I need to have this version exposed for further discussion in order to be able to advance it further.

While I was already trying in the previous draft to figure out more clearly the scope and nature of the desired hierarchy, this became clearer only after getting a better understanding of the various data the client uses and especially after further discussion in the SMG boardroom of the protocol specification. The previous observation that there are 3 types of information in there (structure, description, representation) is as correct now as it was then. But meanwhile, as I got to understand better all the different domains involved, the high-level view evolved to this:

  1. Game World
    1. Concrete objects
      1. Terrain
      2. Water
      3. Air
      4. Characters
      5. Structures/Items 1.
    2. Abstract objects
      1. Time
      2. Self
      3. Sector
  2. Representation
    1. Graphical
      1. 3D
      2. 2D
      3. Skin/GUI
    2. Audio

Before going further and into the hairier details, it's worth stating the important distinction that the protocol specification makes between the two types of "resources" that the client can ask for:

  1. Files - relevant protocol messages 4.4 File Request and 4.3 File Transfer.
  2. Objects - relevant protocol messages 4.7 Object Request and 4.8 Object Info (ie "Transfer")

The Files are - from the protocol's point of view - "big" resources that never contain other resources. Files are referenced by their full name that works therefore exactly like an id (although a text-id rather than a numerical id).

The "objects" are - from the protocol's point of view - "small" resources of a potentially composite sort. This means specifically two things: any object will always fit into one single message; an object may contain references to any number of other resources (ie references to other objects and/or to any number of files). An object is given (via a 4.8 message) as a list of tuples (id:type). Each such tuple is the reference (the "id" part) to one "property" of the object, where the term property is quite generic: any object can be at some time or another a "property" of another object. Note also that "id" is not necessarily a numeric id - a filename (with full path included, where relevant) is effectively the "id" of the actual file.

The hierarchy starts with the single object that has no id (otherwise put it has the "null" value as id). Since all things have to start from somewhere, so does Eulora's language of knowledge - in the beginning, it is always "null". If you think of the hierarchy correctly as a tree, this object with "null" as id is the root from which there can be ultimately any number of levels - the hierarchy is *not* restricted on either direction (ie any node can have in principle any number of children and any path may be arbitrarely long). The leaf nodes will *always* have "Final" as given type in the tuple (id:type). The id in the tuple (id:type) of a leaf node is either a filename (ie the id of a file, one can think of this as value given by reference) or otherwise a value that is used directly as it is, as data (e.g. helmet.png:Final or 5:Final). All values given at a leaf node will be one of the following basic types:

The root of all the world (the object with "null" as id) is the object that the server sends (via a 4.6 World Bulletin message) in response to an Object Request (4.7) that contains an unknown/non-existent object id. The hierarchy as I managed to flesh it out so far is given as id:type with comments after > :

  1. Root > there is only ONE object of this type and that object has id "null"

    1. Top_Location_ID : Sector > current sector; player's character is somewhere in the subtree of this object (eg may be in a room that is in a house in this sector, perhaps).
    2. Self_Info_ID : Self > id of the abstract object, not the id of the player's concrete character object.
    3. Current_Date_ID : Date > euloran date
    4. Current_Time_ID : Time > euloran time
    5. Client_Skin_ID : ModSkin > graphical user interface resources
  2. Sector

    1. CurrentNameID : Name

      • Value (text): Final
    2. Terrain_ID : Terrain > the heightmap (+ material palette)
    3. Air_ID : Air > ambiental light and weather (?)
    4. (Light_ID : Light)* > NB: zero or more; those are purely sources of light (NOT ambiental light which is a property of Air), without physical representation other than the light itself.
    5. (Water_ID : Water)* > bodies of water
    6. (Character_ID : Character)* > anything that is active/can move;
    7. (Structure_ID : Structure/Item)* > anything that is passive/can't move and is not terrain/air/water/weather
  3. Self > this includes all the stuff that is specific to self (ie known only about own Character) as opposed to the generic info given via Character type (ie info available for all characters encountered, whether self or not)

    1. CurrentPlayer_ID : Character > player's actual character with data that everyone can access/see
    2. Current_Loc_ID : LocID > should the location ID be here? (otherwise in principle client can fully search the tree until it finds CurrentPlayerID as contained in some location)

      • Value (uint32): Final
    3. Date_Of_Birth_ID : DOB

      • Value (date) : Final
    4. Vitals_Profile_ID : VitalsProfile
    5. Skills_Profile_ID : SkillsProfile
    6. Inventory_ID : Inventory
    7. Equipment_ID : Equipment
  4. Inventory

    1. (Item_ID : Structure/Item)*
  5. Equipment > How is equipment different really from Inventory? I can't see any real reason to make it a separate type. "Equip" is an action and as such it will fail if the item doesn't "fit" the destination, be it because it's not equippable or whatever. Similarly, the "how it looks" is a matter of checking what is in some slots if one so wants. The only thing: the meaning of "positions" in inventory should be defined, at least for those that have a specific meaning beyond "in the bag".

    1. (ItemID : Structure/Item)*
  6. Date > this could be in principle simply unixtime number but that wouldn't match well/easily with custom year/month sizes (?) + presumably custom month names rather than numbers (?)

    1. Year_ID : Year
      • Value (text) : Final > should those be numbers?
    2. Month_ID : Month
      • Value (text) : Final > should those be numbers?
    3. Day_Of_Week_ID : DoW
      • Value (text) : Final > should those be numbers?
  7. Time

    1. Hour_ID : Hour
      • Value (uint8) : Final
    2. Minute_ID : Minute
      • Value (uint8) : Final
    3. Second_ID : Second
      • Value (uint8) : Final
  8. ModSkin > should this cover *also* layouts & windows & UI components or only strictly "art" resources? If it's the later, in principle it's simply a list of images (with at most width, height, alpha, transparency)??

    1. (UI_Art_ID : UI_Image | UI_Spritesheet) *
  9. UI_Image

    1. (Name_ID : Name)? > CS uses everywhere names for IDs and sprite sheets won't be fine with filename as name for each sprite; onth perhaps client should just generate its own "names" out of object id and/or filename if it needs them?

      • Value (text) : Final
    2. Size_2D_ID : Size

      • Value (size2d) : Final
    3. Transp_ID : Transparency

      • Value (rgb) : Final
    4. Alpha_ID : Alpha_Lvl

      • Value (float) : Final
    5. File_ID : Image > should all be .dds or .png or fixed format?

      • Ref (text) : Final
  10. UI_Spritesheet

    1. UI_Image_ID : UI_Image > for the sheet as a whole, it still has all the details
    2. (UI_Sprite)+
  11. UI_Sprite

    1. (Pos_ID : Position)? > for sprite sheets - position in parent image

      • Value (position) : Final
    2. (Name_ID : Name)? > CS uses everywhere names for IDs and sprite sheets won't be fine with filename as name for each sprite; onth perhaps client should just generate its own "names" out of object id and/or filename/whatever if it needs them?

      • Value (text) : Final
  12. VitalsProfile

    1. (VitalID : Vital)*
  13. Vital

    1. Name_ID : Name
      • Value (text) : Final
    2. Current_Value_ID : CurrentValue
      • Value (uint8) : Final > assuming that those are still given as % of max
  14. Skills_Profile

    1. (Skill_Group_ID : SkillGroup)*
  15. Skill_Group

    1. NameID : Name
      • Value (text): Final
    2. (Skill_ID : Skill)*
  16. Skill

    1. Name_ID : Name
      • Value (text): Final
    2. Rank_ID : Rank
      • Value (uint64): Final
    3. Practice_ID : Practice
      • Value (uint8) : Final

      > assuming given as %

    4. Knowledge_ID : Knowledge
      • Value (uint8) : Final

      > assuming given as %

  17. Character

    1. Name_ID : Name
      • Value (text): Final
    2. Sex_ID : Sex > is this given and in what form? is it known for anyone or just for self?
      • Value (uint8): Final
    3. Position_ID : Pos
      • Value (position): Final
    4. Rotation_ID : Rot
      • Value (rotation): Final
    5. Model_3D_ID : ModCal3D
    6. (Model_2D_ID : Mod2D)? > 0 or 1
    7. (Model_Audio_ID : ModAudio)? > 0 or 1
    8. (Model_Effects_ID : ModEffects)? > 0 or 1
  18. Structure/Item > the difference between structures and items is minimal to the point of inexistent (eg "structures" can't be picked up perhaps, but that's not a property - simply a response to an attempted action and as such of no interest/relevance here!; or structures will have stack count = 1 but that doesn't mean that they don't have stack count...)

    1. Name_ID : Name
      • Value (text): Final
    2. Stack_Count_ID : StackCount > this will just be 1 for houses / some cases.
      • Value (uint64): Final
    3. Quality_ID : Quality > this may be called "durability" for eg tools or houses but it's same thing anyway.
      • Value (uint64): Final
    4. Position_ID : Pos
      • Value (position): Final
    5. (Model_3D_ID : ModCal3D)? > ONE of the models 3d/2d should be present in general
    6. (Model_2D_ID : Mod2D)? > 0 or 1
    7. (Model_Audio_ID : ModAudio)? > 0 or 1
    8. (Model_Effects_ID : ModEffects)? > 0 or 1
  19. Terrain

    1. NameID : Name
      • Value (text): Final
    2. Position_ID : Pos
      • Value (position): Final
    3. Height_Map_ID : Heightmap > NB: while both materialmap and heightmap are images (and even of same type + same size, listing them as "IMAGE" type means that the role (i.e. which one is Heightmap and which one is MaterialMap should be defined somewhere else - would that be better/how?)
      • Ref (text): Final

      > filename, hence "Ref", currently "Island.png"

    4. Material_Map_ID : Materialmap
      • Ref (text): Final

      > filename, hence "Ref", currently "Island_alpha.png"

    5. (Material_ID : Material)* > material palette to use (eg sand, grass, stone)
  20. Air

    1. AmbientLightID : AmbientLight > all lights should be "dynamic" or at any rate the static/dynamic distinction is entirely client-side concern ; currently those values are given at different "times of day" + interpolated by client in between, but I don't see the *need* for hours in here; client can decide on its own if /when it wants to sample this and what/whether to interpolate or do anything else with the values it gets.
      • Value (rgb) : Final
    2. Weather_ID : Weather
  21. Water > is this needed as separate type? CS has 4 types of procedural textures: fire, water, dots, plasma
    > therefore "water" could simply be in principle an item/structure with the "water" texture applied - TO CHECK.

    1. Item_ID : Structure/Item
  22. Weather > CS uses "particle systems" for this sort of thing (i.e. actual snow/rain)
    > as such, those could be just part of the "sector" at some point + corresponding changes to light & sky, if any. TO CHECK
    > however, there can be perhaps the point of having this separate to set specific params such as emission rate, velocity etc. ?

    1. Particle_Syst_ID: Particle_System > if going this route, this is TBF
  23. ModCal3D > aka moving/character stuff

    1. BB_Size_ID : BoundingBox > this is used for collision detection

      • Value (size3d) : Final
    2. Skeleton_ID : Skeleton > this is MANDATORY; all moving stuff have to have a skeleton

      • Ref (text) : Final

      > this is a .csf file for now; should the format be a further level/property?

    3. (Animation_ID : ModAnim)* > a character may have any number of animations
    4. (Submesh_ID : ModMesh)* > parts/subparts of the model; NB: atm sockets is still not used/unclear!
    5. Scale_ID : Scale > intended scale of the model, this is a transformation to apply before use

      • Value (float) : Final
    6. Translation_ID : Trans > pre-transform to apply, translation; same as for scale - could be standard and therefore spec not needed?

      • Value (3xfloat): Final
    7. Rotation_ID : Trans > pre-transform, rotation; same std issue as above.

      • Value (4xfloat): Final > (x,y,z,angle)
    8. RotX_ID : RotX > IF the model should be pre-rotated along the x-axis; again, should be std and be done with. Default is true

      • Value (boolean): Final
    9. FlipTex_ID : FlipTex > whether v coordinates of texture should be flipped (i.e. whether 0,0 in texture space should be top left or bottom left); default is false

      • Value (boolean): Final
  24. ModAnim

    1. Name_ID : Name > name is used as an id by CS... eg "walk", "run", "jump"

      • Value (text) : Final
    2. File_ID : File > this is a .caf file for now; should format be a deeper level?

      • Ref (text) : Final
    3. Type_ID : TypeAnim

      • Value (uint8) : Final > idle, travel, cycle, action, none, style_cycle currently from CS&PS - WTF?? do we even want such a thing?
    4. Vel_ID : Velocity
    5. Time_ID : Interval > this is "interval for idle or override actions", in seconds

      • Value (2xfloat): Final
    6. Idle%_ID : Idle% > this is "probability (as %) of this action being the overriding action"

      • Value (float) : Final
  25. Velocity > this is for animations

    1. Base_ID : Base_Vel > speed of translation when animation is used on its own

      • Value (float) : Final
    2. MinVel_ID : Min_Vel

      • Value (float) : Final
    3. MaxVel_ID : Base_Vel > min and max are used by CS's internal "blender" to get what precise speed is required at one point or another

      • Value (float) : Final
  26. ModMesh

    1. MeshFile_ID : MeshFile > format is .cmf ; should format be a lower level?

      • Ref (text) : Final
    2. Size_ID : Size > is this actually mandatory/needed here? Can't quite yet tell.

      • Value (size3d) : Final
    3. Name_ID : Name > name of this part of the model, eg "Head", "Neck", "Left_Arm"

      • Value (text) : Final
    4. Material_ID : Material > CS allows several materials mixed on a mesh (though on occasion with this sort of thing it "allows" while at the same time complaininig if too many textures or too many this and too many that) but I'd stick to ONE material only.
  27. Mod3DGeneric > aka structure/item (non-moving)

    1. Material_ID : Material > while several materials can in principle be used, I'd rather stick to ONE material since *that* can combine several textures anyway.
    2. (Vertice_ID : Vertex)* > one vertex as a minimum

      • Value (position) : Final > this is given in model's own coordinate space
    3. (Triangle_ID : Triangle)* > each triangle is given as (i1,i2,i3) where i1-3 are indices of the vertices given previously

      • Value (3xuint64): Final
    4. (Normal_ID : Normal)* > there should be one normal vector given for *each* vertex

      • Value (3xfloat) : Final
    5. (Colour_ID : RGBA)* > there should be the colour given for *each* vertex

      • Value (4xfloat) : Final > this means r,g,b and alpha channel
    6. (Texture_Map_ID : TexMap)* > there should be mapping from each vertex to a texture point (ie u,w coords)

      • Value (2xuint) : Final > these are coordinates in the 2d, discrete space of the texture
  28. Mod2D

    1. Texture_ID : Texture > NB: this currently is "texture2d" but it's unclear if the distinction is worth much (ALL textures are anyway 2d images, ffs)
    2. Transparent_ID : rgb

      • Value (3xfloat) : Final
    3. Alpha_ID : Alpha > this is int in the current code, hm.

      • Value (int8) : Final
    4. Pos_ID : position

      • Value (3xfloat) : Final
    5. Size_ID : Size > height and width

      • Value (2xint) : Final
  29. Material

    1. Name_ID : Name > CS uses "names" in fact as IDs; so there can be several materials with same texture/file/data but different names...

      • Value (text) : Final
    2. Texture_ID : Texture > CS works with materials (it wraps any plain texture into a material anyway)
    3. (Shader_ID : Shader)* > a material CAN have several shaders applied.
    4. (ShaderVar_ID : ShaderVar)* > those are used *across* shaders to customise them, hence put here as separate, ugh; I think though that they belong more logically as a property of each Shader.
  30. Texture > CS supports: png, tga, bmp, jpg, dds, gif, mng, jng; common use given: png/tga for textures with alpha; jpg for those without; "best" = dds because it stores mipmaps directly + no need for additional steps (for a png there is uncompressing + mipmaps calc + recompressing to upload; a dds is directly uploaded to graphics hardware)
    > There are ALSO, "texture classes" but it's totally unclear if they bring anything useful or if they are anyway a server concern at all; from the manual (p. 331/644): texture class = collection of settings controlling how a texture is uploaded to the Graphics hardware (e.g. lookup, normalmap, nocompress) + "adding semantics".

    1. Name_ID : Name

      • Value (text) : Final
    2. Image_ID : TexImage > at this stage it's unclear if stuff like size should be given explicitly since it should rather be either extracted from the file itself (?) or default/predefined (?)

      • Ref (text) : Final
  31. Shader

    1. Name_ID : Name

      • Value (text) : Final
    2. ShaderTypeID : Name > ugh; those are again predefined CS stuff/available shader types...
    3. File_ID : File

      • Ref (text) : Final

      > filename containing the shader description; atm those are a bunch of xml files, full of dependencies & referencing one another + cs-specific.

  32. ShaderVar

    1. Name_ID : Name > NB: those names need to match what shaders (those xml files) know/expect, for anything to work...

      • Value (text) : Final
    2. Value_ID : Texture|Float|2xFloat|3xFloat > the value of the shader variable has one of those types

      • Value : Final
  33. Light > ALL lights are "dynamic" since they can always change/be added/removed etc.

    1. Name_ID : Name

      • Value (text) : Final
    2. CentreID : Centre

      • Value (3xfloat): Final
    3. ColourID : rgb

      • Value (3xfloat): Final
    4. AttenuationID : Attenuation > "type" of attenuation (?)

      • Value (uint8) : Final
    5. RadiusID : Radius

      • Value (uint16) : Final

As full of holes and possibly over-wordy and all that as the above draft finds itself, it was still quite the toil to extract. Nevertheless, extracted and written down it had to be as I couldn't quite move any further without this step. Now that it's pinned down, I can go through it and discuss the questions/issues that are still unclear throughout. There are also a few aspects that are currently still entirely missing (not that the rest is guaranteed to be complete/final), namely:

  1. Some meta resources - most notably authorship of art files - are still missing entirely. I'd rather discuss/iterate first those concrete (relatively) parts extracted from client-entrails before moving on and fixing the details of that, since it's more likely to be actually easier anyway as it's way more clearly and sanely defined to start with.
  2. ModAudio - the definition of this is still missing as I never even bothered yet to get the SoundPlugin to work at all. Perhaps it can be added/fleshed out later on and then clients that know it / want it will use it, while the rest will simply ignore it as an unknown/un-usable part.
  3. ModEffects - this is also still missing but unlike the audio part, this could be more readily investigated and fleshed out more, by figuring out how to load/create effects from code directly, similar to the previous work on characters for instance.
  4. Water - this requires an investigation into creating water bodies from the code directly; specifically: some practical experience using CS's procedural textures and some tests to see if/to what extent this could/couldn't be folded into structure/item
  5. Weather - this requires an investigation into CS's particle systems at the very least (for snow/rain/etc) and some more reflection + testing to check if it makes perhaps more sense to fold this into Air anyway.
  1. This is just one type, the name is given like this to make it clear that both "items" such as an axe or a flower and "structures" such as a house or a bridge are in here. Basically anything that is not terrain/water/air and can't move by itself is in here (if it can move by itself then it's in Characters) [↩]
  2. NB: CS uses floats, with 1,1,1 for white and 0,0,0 for black.[↩]

Category: Eulora | 9 Comments »

« Older Entries
Newer Entries »