Eulora's communication protocol aims to be indeed as short and simple as possible while providing at the same time the full means for sending over to the client *anything and everything* at all that might be wanted now or later on, whether known currently or not. Given this broad scope and combined with the terrible state of the original client, it's perhaps no wonder that I had a rather frustrating time with my previous attempts at pinning down a practical implementation of it: here's for instance one attempt of defining the client-side data hierarchy and there's its further refinement too, but those are only two cases that I fully documented in an attempt to clarify matters better - the effort was certainly not wasted but the results in both cases are still lacking and I found myself repeatedly getting tangled in all sorts in there, as it's indeed very easy to run into circular definitions of things or get confused by the required combination of structure, meaning and representation. Nevertheless and much to my recent relief, it seems that I finally got my head around it - it took only yet another re-read of *everything* around it, previous discussions and attempts included, as well as all the previous hand-banging and failures, of course, what else.

As it happens though in such cases, once the last bits and pieces finally fell into place, it all started to make sense and fit together (although it *also* meant that half of my initial test/exploration implementation had to be thrown away; sigh). Since the actual implementation is still in the works and it's likely to still take a while as it's not exactly a walk in the park, I am taking this time now to try and set down my current clearer understanding of how it's all going to work - so that on one hand I have a reference point for later on and on the other hand it's out in the open for any further discussion and fault-finding.

For clarity and keeping it all contained here, note that I'll focus strictly on the data transfer part of the protocol, meaning messages for File Request, File Transfer, World Bulletin, Object Request and Object Info¹. Of those, the last three are not even fully defined in the current protocol precisely as further details needed clarification - in particular the "object list" in the World Bulletin and the "object properties" in the Object Info are essentially "to be defined". Skipping through all the intermediate attempts that either didn't quite fully work or turned out to be in practice so clunky that I couldn't live with them at all, the breakthrough basically came when I finally managed to set down a few basic things (that even seem very simple in retrospect, of course). To get it all in one place:

1. There's only *one* primitive to this whole thing and that is "object", no exceptions. For a long time I kept tripping over "properties" and "values" and whatnots, as I didn't quite grasp the boundaries (admiteddly blurry of times) of the different perspectives involved and kept trying to sort out or pin down too fine details before having a solid higher-level structure to rely on.
2. Objects may be concrete or abstract. The concrete objects are essentially those with a direct graphical representation in the game (such as your character or even the terrain as a whole). The abstract objects provide all and any sorts of other information required (such as what time it is in game perhaps or which object from all those in the world is actually your character!).
3. The whole game world (and this does include *everything* that may be requested or received through the set of messages mentioned above) is a hierarchy of objects with a single "root" node that is quite special in that it's essentially the "null" object itself. So to "request" the World Bulletin, the client basically needs to request the "null" object. This hierarchy of objects that is a full representation of the game world can be arbitrarely deep and wide at any given time - the only guarantee is that it is indeed finite at all times but the number of levels or the number of siblings on any level can and will change, there's no restriction on either of them (nor any need for such restriction).
4. The hierarchy of objects given as above provides basically the *structure* of the world, effectively describing where *is* each concrete object (or where it belongs, if it's an abstract object). As such, concrete objects can of course move about from one parent object to another (and take all their corresponding subtree with them). Abstract objects are, logically speaking, less likely to suddenly move to a different parent but there is no guarantee that this can't or shouldn't happen either.
5. The full set of object types can be neatly split into TWO main categories: "leaf" types and "non-leaf" types. Realising this was big for me, as it suddenly got rid of a lot of mess in there: non-leaf types of objects can only ever contain other objects (of any types though and in any number) but *nothing else*; by contrast, leaf types of objects do NOT contain other objects (obviously!) but have instead a predefined and fixed structure (basically, values). A lot of my previous trouble seems to have come from mixing those two groups ie considering that an object type may legitimately contain both other objects AND values². There's no real need for this mix that I can see and it only causes such a mess that it's not even worth fully describing. Once those 2 types of objects are neatly separated, everything falls in place and there is both the desired full extensibility of the whole thing (one can add any number of non-leaf or leaf types, as well as any number of intermediate levels in the hierarchy anywhere at any time without breaking anything) and the needed clarity for a working implementation: as long as a non-leaf type of object is encountered, the "properties" of it will simply be again an "object list" just like in the World Bulletin message; as soon as a leaf type of object is encountered, its structure is clearly known and *fully fixed upfront* so it can be actually worked with! Certainly, it may well be that one implementation or another is not aware of some newer types, be they leaves or not - in such cases, those types will be ignored since what else can the code do anyway, not like it can use what it doesn't know about. Note though that this doesn't break any of the rest - it will work for and with what it knows, while dropping what it doesn't and that's that.
6. All objects have a unique ID and a type, meaning that an object is effectively a tuple (Object_ID, Type_ID). In practical implementation terms, this means that the "object list" in the World Bulletin message is *always*³ simply a set of pairs of numbers so if World Bulletin gives 5 as "count of objects", then there will be 5 pairs of IDs following, interpreted exactly like that: first ID in a pair is object's ID, second ID in a pair is type ID and there's no need for separators or any further information provided.
7. For a non-leaf object type, the Object Info message will *always* contain as "object properties" simply an object list (the text field still works perfectly fine since it's known that there are 4 octets per ID), just like the World Bulletin. (Note though that the Object Info message may refer to more than one object and so some of those may be leaf types, too, so the properties for those will be different, see next item.)
8. For a leaf object type, the Object Info message will *always* contain as "object properties" the exact structure known for that specific type. Obviously, a full list of those structures is in the works and will be published but the crucial thing here is that the contents of those structures are always *values*, never objects. Sure, it can very well be that a value in there stands for an id of some object but the *meaning* is very different since it is NOT about structure anymore and so that id in there will then reference an *existing* object from somewhere else in the tree instead of introducing a new object (and therefore it will NOT need to come with the type id either)!
9. Values in the structures defined for leaf node types may be in principle of any known type, whether basic or not. From a practical point of view, the only truly special case is that of a value that represents a filename, since in this case there's effectively a further level to it all - the implementation will have to send a File Request message to get the actual content of the file, if it doesn't yet have it. Nevertheless, this is not any significant trouble as such.

For a practical example, here's my current must-have basic set of object types that I think are absolutely needed in order to get the client to the stage where it finds and requests graphics assets that it doesn't have:

• RootObj (the null object is the only one having this type and therefore the type itself is essentially an abstraction, no need for a type id as such)
  1. SelfObj containing for now⁴:
     1. PlayerIDObj - this is LEAF type with a single value, an Unsigned_32 that references the object in the world that represents the character controlled by the player.
  2. SectorObj (the id of this object will be given as top_id in a World Bulletin, too) containing for now:
     1. NameObj - this is LEAF type with a single *text* value.
     2. OutdoorsObj, containing:
        1. ZipObj - this is LEAF type with a single *text* value representing the full path and name of a zip file that contains *all* the graphics assets required for a sector. Note that the exact contents may change at later times but I don't see this as any problem - for one thing the plan is that the client will be able to select one overall schema at any given time and that schema will determine what exact file is made available here and for another thing, those schemas will be known upfront so a client will select what it knows and therefore won't have surprises regarding the content of this zip file either. For a long time I tripped over the trouble of defining the graphics in too much detail, quite possibly precisely because I lacked that knowledge myself. As meanwhile I sorted out this issue and I know all too well what exactly is needed, I finally realised that the only way to cut this knot neatly is to just specify a .zip file as a general "graphics assets for a logical object", no need for more at this level.
        2. HashObj - this is LEAF type with a single Unsigned64 value that represents the Keccak hash⁵ of the .zip file above.
     3. Any number of PlayerObj, containing:
        1. NameObj - this is LEAF type with a single *text* value.
        2. PosRotObj - this is LEAF type, with its value ("properties" text field in the Object Info message type) sent over the wire as 3 int16s for position followed by 3 int8s for rotation, as per the protocol specification
        3. Cal3DObj, containing:
           1. ZipObj - LEAF type, as above (see at OutdoorsObj - the big gain here is that although a Cal3D model will have an entirely DIFFERENT set of files in that zip than an OutdoorsObj, from the point of view of protocol spec and data acquisition, there is NO difference! They are simply all .zip file and that's that, no matter what they currently contain or may contain in the future or whatever else, hooray.)
           2. HashObj - LEAF type, as above (ie the hash of the zip file, for checking that it is indeed the intended file).

The above is "all" for now - if still just a plan to be implemented. Sure, I have otherwise the concrete details of what those .zip files for instance will contain and all that but they don't really matter all that much from the point of view of the overall data model and data acquisition - the contents are in the end of interest strictly for the GUI/graphical part itself so they'll be handled there and only there.

Regarding the recent discussion of the potential need to be able to choose sizes for textures and so on - those I think are also best chosen separately, at a higher level, as an overall thing, so not going about specifying them for each and every texture or image file or something. At any rate, I don't think it's something that absolutely must be included in the spec at this time or here but do let me know if you see any trouble with any of the above, as usual!

1. The encryption and lower level packaging part of the protocol has been already cleared up, implemented and working fine, see the SMG_Comms category for detailed documentation on it and the basic docs for a high-level overview of the client's structure overall. ↩

2. In turn, this came from my still-not-wide-enough definition of objects, since I didn't quite grasp that there will be objects strictly for this purpose basically e.g. there's an object that would be fully described as graphical-model-of-Sue's-helmet-at-night-on-the-moon or some such. Once I got my head around this, I even find it difficult to fully point the trouble I previously had with it, go figure. Though in hindsight, I suspect at least part of the trouble was the unhelpful overlap with "object-oriented" idiocy, ffs. ↩

3. There's a reason for NOT ever having directly a "leaf" type of object in the root: leaf types of objects provide concrete values in predefined structures but they do NOT contain, nor can they contain at the same time the meaning and role that those values are to play at any given time. Hence, the root node will always have to contain at least one level of non-leaf objects and I kick myself for not realising this clearly any earlier! ↩

4. More will be added to this and to most other types described here but right now this is on purpose as small as possible, to have it first implemented and working. ↩

5. I chose this as it's been used elsewhere, though so far not for full files - just let me know if something else would work better, since I need to implement this part anyway so there's nothing set in stone yet. ↩