The Syntax of World Knowledge in Eulora2

August 6th, 2023 by Diana Coman

~This is a work in progress, to be discussed and expanded as the game progresses further. This article brings together and builds on the work already done and the preliminary concepts that were already described in previous articles, including a first draft of a data hierarchy, its subsequent refinement and an initial brief discussion of how it works in practice.~

To support players in their quest to explore, make their name and write their own history in the infinite and infinitely expanding world of Eulora2, the game's client acquires and maintains at all times a dynamic view of the game's world, as it becomes accessible and known to each player and as it responds to their actions or as it changes otherwise, with time or due to other players' actions and to the game's own internal rules.

The above means specifically that the game's client successfully retrieves and makes use of any new data in its relevant context on the fly, as it becomes available and without knowing it upfront - because it simply can *not* be known upfront, since it's the players' actions that decide the very evolution of the game's world! Playing Eulora2 truly means exploration and discovery in full, not just a pretense kept alive by some sort of agreement between server and client to merely not show or not use some data until the server says it's "available" and the client is "updated" to have it and be able to handle it. In Eulora2, the player doesn't need to avoid for whatever reason to even look at the files on their own disk. Quite on the contrary and unlike in any other game to date, in Eulora2 the player is warmly invited to look at the files on their own disk, to make full use of all the data and knowledge that they can access at any time, to write or use their own bots and anything and everything else that they can gather or even think of.

The game's own client as I made it to date supports the above by design, as it is entirely and fully dedicated to serve the player with all it has and all it can actively and continuously acquire. These are not empty words but fundamental principles of mine and thus I made the client according to them, quite on purpose: it continuously checks, discovers and acquires new data and new knowledge of the world as it becomes available and it makes full use of it and of any new assets on the fly and as unobtrusively and quickly as it can.

How is such continuous discovery of knowledge and on the fly use of new assets even possible? In short, it's made possible by knowing the very syntax of the game's world representation - thus, instead of trying to enumerate upfront the world entire and getting stuck as soon as a new element is added, my client for Eulora2 simply starts off knowing how to discover and make sense of new elements as they appear.

My client starts without any of the game data but with the full ability to acquire and make sense of any and all such data, as soon as it has a working connection to the server. The way it does this is by knowing the rules of how the Euloran world describes itself, the very syntax of world knowledge in Eulora2. Unlike the usual game clients that start by having upfront a ton of data and graphics assets, my client starts without any of those but with a full working knowledge of the solid structure that contains and gives meaning to such data within the game. In other words, my client starts knowing how to obtain and make sense of data and then it proceeds to do exactly that, for as long as it runs and handling without any trouble all sorts of new situations as they appear, supporting thus the expansion of the player's own increasing knowledge of the game's world as it evolves with active play.

Knowledge of the game's world as seen by any given player in Eulora2 is at all times a tree - a finite tree for sure but one that can have any number of branches and any number of levels, changing continuously as well, in response to the player's actions and to the evolution of the game's world as the time passes. To support the player, my client repeatedly explores this tree, pruning dead branches, growing new ones as relevant, updating and further interrogating parts of current interest and showing to the user the corresponding world view and its changes as they are obtained from the server in response to specific requests1. Such showing of the world view is done through the graphical user interface (GUI) by default and through command line on demand. Playing the game relying entirely on the GUI alone is certainly possible but like all and any exclusive reliance on GUIs alone, this is a very limiting proposition really. Up to each player though how they play the game at any given time, for sure, but if you choose to limit yourself in some way, don't expect that everyone else will necessarily limit themselves the same way, that's all.

The above approach works to make sense of new content as it appears for a very simple reason: while the exact tree of knowledge for each player will be different and will keep changing throughout the game, its structure will always be syntactically correct, obeying thus at all times the known rules of describing the Euloran world. And since the game is by design and with intent fully open and encouraging players and developers to make or tweak their own clients as it serves them best, this syntax is public and will remain so at all stages. Moreover, any new additions to it will expand its capabilities as needed but without breaking at any point the compatibility with what came before - meaning that there is no requirement to "update" anything at any time. There is the option to do so or not, as and if it suits, nothing more. Like all Euloran evolution, even the world's own language and corresponding syntax may certainly expand but in an organic manner, opening up new options on top of and building on existing ones, not contradicting nor "obsoleting" anything. In Eulora2 and as a fundamental principle, experience is an asset that keeps on giving - if one builds it up and uses it for all its worth, of course.

As for this knowledge tree itself, let's illustrate it with an actual small example first and see how it describes the Euloran world or, more accurately, how it describes what one player happens to know about the game world at one given time:

(null, null)
- (Self, 23523903281)
-- (PlayerID, 123503920)
--- [10950238522]
-- (Salt, 8908319)
--- [2a891d0f]2
- (Sector, 513250013)
-- (NameVal, 120945693)
--- [Waypoint]
-- (Ground, 935410)
--- (CSModel, 10853092)
---- [ea3420d8982]
-- (Sky, 1930582)
--- (CSModel, 8273509)
---- [8ff24351ab92c]
-- (PC, 10950238522)
--- (NameVal, 82082501)
---- [Edward Wealthgore]
--- (Cal3DModel, 98357102)
---- [183cca82bf82e]
--- (Image, 125935601)
---- [1f98eb246caa28]
--- (PosRot, 2795927491)
---- [101, 252, 114, 124, 42, 90]
--- (Item, 9873592011)
---- (NameVal, 873798211)
----- [Coarse Cordage]
---- (CSModel, 17349272)
----- [ff32ea1d89cb0]
---- (Image, 72985711)
----- [91a0b3f28ea8d]

Reading the above would yield the following description of the world: starting from the root of the tree (that null,null first line), there is first of a all a Self (with ID 23523903281) and a Sector (ID 513250013). Further asking the server as to what exactly that Self with id 23523903281 contains yields as children nodes a PlayerID (ID 123503920) and a Salt (8908319). Both PlayerID and Salt are leaf types in our tree and this means that upon asking the server to expand further on these two IDs (123503920 and 8908319), the response will provide directly values rather than further tuples (type,ID): the actual ID of the player's own character is thus revealed to be currently 10950238522, while the value of the Salt turns out to be 2a891d0f, standing for a file with this exact keccak sum3.

Going back up in the tree and asking for further information about the Sector (ID 513250013) reveals that it contains a NameVal (ID 120945693), a Ground (ID 935410), a Sky (ID 1930582) and one PC with ID 10950238522. At which point, the client will have found therefore the player's own character in game, since this ID is exactly the one indicated by that earlier PlayerID node. Given that there is currently no other PC in this sector, it seems that for the time being, this player is either alone in the world or at least not yet seeing anyone else - possibly others are somewhere in the world but not yet within interacting/seeing range.

Drilling deeper into this PC node will reveal that the player's character goes by the name of Edward Wealthgore, is located at some specific point as given by the value of that PosRot node, carries a Coarse Cordage item and uses as 3D representation as obtained from the graphics assets packed in the file with keccak hash 183cca82bf82e (the value of the Cal3DModel node) and as 2D representation what can be obtained using the file with keccak hash 1f98eb246caa28.

Repeating the same sort of exploration as above for the remaining nodes, in turn, there will be detailed information about each, as relevant, including any contained nodes as well as graphical representations and additional characteristics. I'll leave the detailed description of the remaining nodes in the above example as an exercise for any reader wishing to test their understanding with a practical application.

While the above may seem tedious when written down in full detail, note that it's done at all stages internally, automatically and extremely quickly by the client code - it's exactly the perfect task for computers anyway, using a well and clearly defined structure to hold and process data repeatedly. As the player moves about, they are likely to encounter others and perhaps items too and thus more PC and Item nodes, each with their own IDs, will appear as children of that same Sector node above. If the user drops the Coarse Cordage that they are currently carrying, the corresponding Item node will simply move from its current place as child of the player's own PC node to become a direct child of the Sector node and perhaps other characteristics of the PC node (not shown in this minimal example) will change to reflect the lighter load carried. Similarly, if the user picks up an item, that item's node will move to become a direct child of the player's own PC node and other values may change as well. When the user starts some work, a corresponding node will appear, holding the relevant information and allowing thus the client to provide feedback to the user as to the effect of their action in the world etc.

As to obtaining new data and assets as they become available, the mechanism is the same throughout: on-demand discovery followed by explicit request. For instance, when the client encounters in the world tree a filename that it doesn't currently have, it will simply request it from the server according to the lower level communication protocol that fully supports such requests and then proceed to use the file as soon as it's fully downloaded and checked to match the expected hash. Similarly, when it encounters a new node where previously there was none, it will request further information drilling down repeatedly until it gets to the leaves and it knows thus that there is no further information to obtain.

Following exactly that earlier mentioned principle of the client aiming to fully serve the user and empower them to make use of any and all information whenever available, the player can directly interrogate and explore themselves the current tree at any time, simply via an ls command in the game's own console, quite similarly to how one explores the tree of files on disk from the command line on any computer. Note that this is made available without being either mandatory to use nor in the way of play otherwise - the whole point is to enable and support users at any and all stages, not to force on them anything and especially not to overwhelm them with more information than they request or are able to handle at any given time. Unless asked otherwise, the client will happily do all the above in the background and simply present the results in graphical form as accurately as possible but whenever the user wants to dig deeper and take advantage of the additional flexibility and power that a command line always offers over a graphical user interface, the option is and remains entirely available.

It's worth noting that the graphical user interface (GUI) is always and at all times a simplified and thus reduced set of options for interaction, never the reference as to what can be done or accomplished. This is simply because it can never be any other way really - the way computers work is always through a set of rules that are quite unaware and unimpended by whatever limitations are introduced by the GUI that has to pick some set of shortcuts essentially out of the full set of real possibilities. Hence, while the game can certainly be fully played through the graphical interface alone and the GUI - or at least the one I make and maintain - will at all times aim to support the user as much as possible, it's really unlikely that any implementation of a GUI will be ever able to provide in full the same flexibility as the command line offers. For this reason, my client at least has and will always have the command line integrated, alongside the GUI, as the more flexible and ultimately more powerful way to explore and act. Moreover, since the command line is always the reference as to what can be tried and attempted too, be aware that if you limit yourself to the GUI alone, you are more likely than not to miss out.

As for the syntax of world knowledge in Eulora2, the current set of node types and their possible direct children is the following:

ID Node Type Possible children types (if not leaf) Value type (if leaf)
null null4 Self, Sector, GUI, Chat N/A
0 Sector NameVal, DateTime, CSModel, Ground, Water, Sky, LightSource, PC, NPC, Item N/A
1 Self PlayerID, Salt, WOTFile N/A
2 DateTime N/A 3*uint8 for hour, day, month + 2*uint64 for year and era
3 PC NameVal, PosRot, Cal3DModel, Image, Particles, Work, Exchange, Item, Vitals N/A
4 NPC NameVal, PosRot, Cal3DModel, Image, Particles, Dialog, Item, Vitals N/A
5 Item NameVal, PosRot, Image, CSModel, Stack, Item5, Description N/A
6 Particles NameVal, Image, Light N/A
7 PlayerID N/A uint32 (4 bytes)
8 NameVal N/A Text
9 PosRot N/A 3*uint16 for position, 3*uint8 for rotation
10 Stack N/A 2*uint64 for quality and count, uint32 for slot number
11 Cal3DModel N/A text for filename, 16 octets for keccak hash of file
12 CSModel N/A text for filename, 16 octets for keccak hash of file
13 Image N/A text for filename, 16 octets for keccak hash of file
14 Ground NameVal, CSModel, PosRot N/A
15 Water NameVal, CSModel, PosRot N/A
16 Sky NameVal, CSModel, PosRot N/A
17 LightSource NameVal, Light, CSModel, PosRot N/A
18 Light N/A 3*uint8 for r,g,b colours, uint16 for radius of effect
19 Salt N/A text for filename, 16 octets for keccak hash of file
20 Dialog TextLine, Beg, Bribe, Threaten6 N/A
21 Beg N/A text
22 Bribe N/A text
23 Threaten N/A text
24 TextLine N/A text
25 Work NameVal, Percent, NPCID N/A
26 NPCID N/A uint32
27 Percent N/A uint8
28 GUI Skin, Branding N/A
29 Branding N/A text for filename, 16 octets for keccak hash of file
30 Skin N/A text for filename, 16 octets for keccak hash of file
31 Exchange PartnerID, Give, Receive N/A
31 Give Flag, Item N/A
32 Receive Flag, Item N/A
34 PartnerID N/A uint32
35 Flag N/A uint8
36 Chat NameVal, Lowest, Highest, Description N/A
37 Lowest N/A uint64
38 Highest N/A uint64
39 Description N/A Text
40 WOTFile N/A text for filename, 16 octets for keccak hash of file
41 Vitals N/A 12*uint8 for percentage of each vital
42 Skill NameVal, Description, Level N/A
43 Level N/A 2*uint8 for experience and knowledge, respectively, followed by 1*uint64 for rank
44 Category N/A Text
45 RefID N/A uint64

The above syntax has been in use successfully for a couple of years already so it's not going to change fundamentally. New node types are likely to be added as the implementation proceeds further but they will be added to this list, as needed. Even at this time, I don't quite see the case for removing some node types and even less so for altering them since that would indeed be the only way in which compatibility is broken. Nevertheless, until access to the client is widely open to everyone, consider the list above as a work in progress rather than fixed and otherwise as an invitation and opportunity to get involved perhaps, if you do your own implementation and have any questions related or otherwise relevant to it.

Any related discussion is welcome and can best happen in the comments below, as usual.

  1. Note that the server for Eulora2 serves essentially as an oracle for the game - it answers any and all queries but it never pushes information unasked. This is by design, fully intended as such and entirely unlikely to ever change. There's a wide scope for client implementations to compete though on different approaches to best make use of the server oracle for their users and thus provide an in-game advantage that players will be willing to pay for. 

  2. This as all numbers in here are given as example only and as such, rather shorter than the actual values. 

  3. Given here in hex for easy distinction of file hashes vs other values so it's easier to follow with an actual client at hand, since the client uses this hex string to name the file on disk. Otherwise the value of that keccak sum is sent from the server directly as a number as obtained from keccak, of course. 

  4. This is by convention the root of the tree and there is always at all times one single root. 

  5. This is because some items can contain other items, of course. 

  6. This was an early test and it works fine but meanwhile my design moved away from pretending to have "dialog" with bots and towards an altogether more promising quest system. For now though this node is still possible to encounter, indeed, hence it's shown in here. 

Comments feed: RSS 2.0

11 Responses to “The Syntax of World Knowledge in Eulora2”

  1. Jacob Welsh says:

    It seems the binary protocol representation is given here for the leaf value types but not for the "structure nodes" (as I believe I saw them called in earlier iterations); has that been finalized yet?

    This as all numbers in here are given as example only and as such, rather shorter than the actual values.

    Seeing that the PlayerID is a uint32, it seems to me rather the opposite as the example 10950238522 will not fit in such a field!

    This next one might be diverging from matters of syntax and getting more into semantics, but are the node IDs stable over time? In the sense that the same ID seen at multiple times can be assumed to represent "the same" object in whatever way that might be meaningful? For instance with PCs there is the internal data of the public key which is not available to other clients for direct comparison but nonetheless creates a strong notion of object identity for that case.

  2. Diana Coman says:

    Hm, not quite sure what do you have in mind as "binary protocol representation" here. All nodes in the data hierarchy have an ID (numerical, uint32) and a type from those described here. When asking about a node, this is done by asking for the "object" with the node's ID and the reply comes back as either a list of children nodes (for non-leaf nodes) or as a value of the corresponding type (for leaf nodes). In either case, the reply is a 4.8 Object Type protocol message. The "object properties" as the communication protocol calls them are exactly the list of children nodes or the value, for non-leaves and leaves, respectively.

    Heh, good catch there, I didn't bother to check the fit for this example. In practice, the machine can't send something not fitting in uint32.

    Generally speaking, IDs are meant indeed to identify and as such, once some entity is known with some ID, there isn't really any reason to randomly flip to a different ID - if that happens, the new entity is by definition different. This said though, it's important to note that *each* player sees the world differently and this means specifically seeing a *different* ID in general for the same thing. So what you see as 12345, I might see as 8742893 and so on. In other words, one can rely that their own view of the world remains consistent over time but not that their own view matches in any way another's view. And in principle there is no promise that IDs aren't recycled at some point - once an entity vanishes, its ID may pop up again at a later time for some other entity.

  3. Jacob Welsh says:

    I'll work an example then based on the tree shown here, requesting multiple objects in one packet for good measure since it seems this is possible.

    Say I've learned the two child IDs of the root node and now wish to expand them. I send a Serpent payload consisting of type ID = 10 (4.7 Object Request) (uint8) followed by count = 2 (uint8), id_0 = 23523903281 (uint32), id_1 = 513250013 (uint32), and whatever message count and padding.

    The server replies with a Serpent payload of type ID = 11 (4.8 Object Info) (uint8) followed by what exactly?

    Perhaps it's not obvious whether the following object IDs are the requested ones or their children, but it seems to me they must be the requested ones because how else can I determine which parent a given response pertains to? (Hm, I guess I could use the message count to establish ordering and keep track of what I asked for, but this doesn't seem to be in the "stateless" spirit of the protocol.) So then we have count = 2 (uint8) followed by id_0 = 23523903281 (uint32), properties_0 = ? (text), id_1 = 513250013 (uint32), properties_1 = ? (text), message count and padding.

    If I'm on track so far, the content of properties_0 will be a string representing the list of pairs ((PlayerID, 123503920), (Salt, 8908319)), and properties_1 likewise of ((NameVal, 120945693), (Ground, 935410), (Sky, 1930582), (PC, 10950238522)). So the question is simply, how are those abstract lists expressed in concrete, bytes-based terms? Spelled out with parentheses and commas and all, or something more compact?

  4. Diana Coman says:

    You are right on track there. The request is exactly as you describe and the reply is indeed Serpent with type ID =11 (4.8 Object Info) and as you describe it, with the following object IDs being the requested ones (precisely for the reasons you note, too). The properties being text type consist first of 2 byte size (including these 2 bytes) and then the value or list of children, as relevant. No commas and no parantheses included in the messages, no, I've added them here just for making it easier to follow. The payload in the actual messages though is really just plain numbers back to back, one for the ID and one for the type, in this order, both uint32, hence 8 bytes per child and 2 bytes for text/property size overall, it's for machine use after all. To answer your last question directly: as compact as possible.

    For our example, this means specifically that properties_0 will be 18 (as uint16 aka the text size in bytes) followed by 123503920 (as uint32, aka the ID of the child object) followed by 7 (as uint32 aka the PlayerID type of this child object) followed by 8908319 (as uint32) followed by 19 (as uint32 aka Salt type). Which I suppose points out that the numerical values of the node types are still to be added to the table in this article, indeed. I'll leave it for next revision of this text but added they shall be and fixed they are and remain, indeed.

    Worth noting also that if/where needed, the server will send more than one message in reply to an object info request, since a node can indeed have more children than fit in a single message. In such cases, each child (ie that properties field) fits though entirely in one single message, it's never split across two or more messages and each message contains the requested object's ID as per the format for 4.8 Object Info, of course.

  5. Jacob Welsh says:

    Thanks, those are the missing bits I was after indeed.

    a node can indeed have more children than fit in a single message. In such cases, each child (ie that properties field) fits though entirely in one single message, it's never split across two or more messages

    That's a helpful addition too; if limited to one message, I figure the fanout (branching factor / child count) would be limited to 183. Though it does bring back in the "reassembly" type difficulties e.g. can I know if I've seen the full response or I'm still missing some children?

  6. Diana Coman says:

    My pleasure.

    can I know if I've seen the full response or I'm still missing some children?

    It's a live world so note first that one can never even know if they're seeing *the current children*, strictly speaking. I fully get what you mean, yes, coming at it from the nicely ordered and precise computing/maths perspective. But that never meshes all that well with anything that is not static, meaning anything that evolves and thus changes, as e2 is. In such cases, the only possible solutions are ever of two types only: either "stop the clocks" thus blocking/denying/at-the-very-least ignoring the changing nature at work or live with uncertainty and imprecision, aiming to manage them rather than eliminate them. It's this last case I'm going with here and so the direct answer to your question is no, you can never know anyway whether you aren't missing some children but you can always re-visit the whole hierarchy and thus adjust your knowledge to try and keep up as best as possible with the world itself. And for that trickiest of parts which is knowing in fact when some children are gone rather than added, there's a baked-in straightforward solution that doesn't rely on any completeness of the children set or some such.

  7. Jacob Welsh says:

    And for that trickiest of parts which is knowing in fact when some children are gone rather than added, there's a baked-in straightforward solution that doesn't rely on any completeness of the children set or some such.

    Hmm. What happens if the client requests a nonexistent object ID? A rubbish response message?

  8. Diana Coman says:

    Heh, a non-existent object is indicated exactly as such, because note that the format permits this quite naturally. Say the client sends a 4.7 message asking for object id 1231 which doesn't exist (or doesn't exist anymore). Then the answer duly comes back as a 4.8 message with the count 1 (one object), the object id 1231 (the one requested), the properties field a text with size precisely 2 aka the 2 octets for size itself and thus ...empty properties. The very definition of non-existent being quite literally that it's a void, as simple as that. Some info on a requested object exists at all times, hence at least one 4.8 is always sent in response to a 4.7, only that info can be simply that the object is void.

  9. Jacob Welsh says:

    Ah, the text size, neat.

    So then, if we ask about a node and it comes back void, we conclude it's gone from whatever children set we previously found it in.

    Also, if we see an ID under a different parent than previously, we can conclude it's gone from the original children set (moved).

    While perhaps straightforward, I don't exactly like this solution as it seems to demand refreshing all known nodes, leaves included, if we want to be sure we've noticed all disappearances. Inefficient!!

  10. Diana Coman says:

    Exactly so. I fully get that "inefficient!!", too, I keep throwing it at the whole quite often. It comes back with a grin though - inefficient but quite powerful, flexible and as close to lifelike as it gets! Also, it tends to get tempered a tiny bit by noticing the efficiency gain in the fact that an ID that got moved from its parent to a different parent *also* moves in one go (how efficient!!!) its whole subtree.

  11. Diana Coman says:

    Updated article, adding the IDs for each type and a few new types at the end (Skill, Level, Category, RefID).

Leave a Reply