As the game‘s development is progressing beyond the emergence of time, past the addition of vitals and into enabling player activities, all the design and implementation work done so far feeds as well into clarifying and refining the game-related parts of the communication protocol that were previously tentatively and incompletely stated 1. Specifically, it’s again that ‘Character Actions’ section 7 that comes into focus but this time the changes go beyond just simple additions and move things forward from where the previous effort towards specification stopped. Consequently, I have to bring together the different parts and restate the current take on the protocol in full, in here, where it can be further iterated and discussed as needed:
This is the current take on Eulora2’s communication protocol, last revised June 13th, 2023.
1. Overall Goals:
- 1.1. All communications between clients and server to be encrypted.
- 1.2. Clients to be able to receive from server any data they lack (including maps, skins, sound or video content etcetera), on demand.
- 1.3. Clients to be able to choose and adjust both the level of security and their volume of communications with the server, as they will ultimately have to pay for the load that they generate.
2. Explicit Dependencies :
- 2.1. Eucrypt for RSA with Keccak-based OAEP and Serpent symmetric ciphering.
3. Data Structures :
- char / uint8 (1 byte) ;
- uint16 (2 byte) ;
- uint32 (4 byte) ;
- uint64 (8 byte) ;
- float 2 (4 byte) ;
- hash (128 bits) ;
- chunk [of file] (bitfield, 11760 bits) ;
- serpent-packet (1472 bytes) ;
- rsa-message 3 (1872 bits 4) ;
- rsa-packet 5 (1470 bytes) ;
- object (size of 104 bits 6: uint32 7 followed by 3 uint16s representing position 8 followed by 3 uint8s representing rotation 9 ) ;
- legacy-text (size of n+n/256+1 bytes ; where the leading byte is the bytecount of the 2nd segment and the 2nd segment is the bytecount of the third segment) 10.
- text (2 byte hearder containing the ~total~ byte length ; up to 1470 bytes of text ).
3.0. Basic types :
3.1. Special types:
4. Serpent Packets 11 :
- uint8 (type ID, =100), followed by
- uint8 (count of keys in this set, n), followed by
- n*(4*int64 + uint32) (32 bytes each key followed by a 4 byte ID calculated through crc32 12 ), followed by
- an uint8 flag (LSB bit set — keys to be used to talk to client ; MSB set — key to be used to talk to server ; client-set MSB is ignored), followed by
- uint16 (message count 13), followed by
- padding to Serpent-message length.
- uint8 (type ID, =102), followed by
- uint8 (count of server keys requested), followed by
- uint8 (count of client keys requested), followed by
- uint8 (id 14 of serpent key preferred for further inbound Serpent-messages), followed by
- uint8 (count of burned keys in this message), followed by
- n*int8 (id of burned key), followed by
- uint16 (message count), followed by
- padding to Serpent-message length.
- uint8 (type ID, =3), followed by
- hash (corresponding to the sought file 15), followed by
- uint8 (manifest packets sought count, 0=all), followed by
- n* uint16 (manifest packet index sought), followed by
- padding to Serpent-message length.
- uint8 (type ID, =4), followed by
- uint16 (count of manifest packets for this file 16), followed by
- uint16 (index of current packet in list above), followed by
- uint8 (fragment count 17), followed by
- n* uint64 (hash of the nth fragment of manifested file).
- uint16 (keccak hash of foregoing), followed by
- padding to Serpent-message length.
- uint8 (type ID, =5), followed by
- hash (corresponding to the sought file), followed by
- uint8 (file chunks sought count), followed by
- n* uint64 (the hash of fragment sought), followed by
- padding to Serpent-message length.
- uint8 (type ID, =6), followed by
- chunk.
- uint8 (type ID, =7), followed by
- uint16 (bytesize of useful part of the chunk following 18, followed by
- chunk 19.
- uint8 (type ID, =8), followed by
- text (fully specified action, see section 7), followed by
- uint16 (message count), followed by
- padding to Serpent-message length.
- uint8 (type ID, =9), followed by
- uint32 (id of top level item 22), followed by
- uint8 (count of objects), followed by
- object list 23, followed by
- uint16 (message count), followed by
- padding to Serpent-message length.
- uint8 (type ID, =10), followed by
- uint8 (count of objects), followed by
- n*int32 (id of object), followed by
- uint16 (message count), followed by
- padding to Serpent-message length.
- uint8 (type ID, =11), followed by
- uint8 (count of objects), followed by
- n times uint32 (id of object) and text (object properties, as per extant game structures, including art files needed and so on 24), followed by
- uint16 (message count), followed by
- padding to Serpent-message length.
- uint8 (type ID, =12), followed by
- uint8 (count of lines), followed by
- n times uint32 (id of chatroom), uint64 (lowest line number requested), uint64 (highest line number requested), followed by
- padding to Serpent-message length
- uint8 (type ID, =13), followed by
- text (line content), followed by
- uint32 (chatroom ID), followed by
- uint32 (speaker ID), followed by
- uint64 (line number), followed by
- uint64 (keccak hash of the line’s content and number), followed by
- padding to Serpent-message length
uint8 (type ID, =14), followed byuint8 (count of requests), followed byn times uint32 (source ID), uint32 (target ID), followed byuint16 (message count), followed bypadding to Serpent-message length
4.1. Serpent Key Set:
4.2. Serpent Keys Lifecycle Management:
4.4.a. File Request, manifest
4.4.b. File Transfer, manifest (always sent and only sent in response to ID 3)
4.4.c. File Request, chunks
4.4.d. File Transfer, non-last chunk (always sent and only sent in response to ID 5)
4.4.f. File Transfer, last chunk (sent at most once per ID 3)
4.5. Client Action 20 :
4.6. World Bulletin 21:
4.7. Object Request:
4.8. Object Info:
4.9 Line Request
4.10 Line Info
Updated 30 Oct 2024: ratings are now fully integrated and used in-game, being fully handled via the relevant Rating node in the data hierarchy. Consequently there is no need anymore to have dedicated message types for discovery and retrieval, so the “rating request” and “rating info” message types below are removed entirely. Rating someone remains done as an action (see the action types further down) and discovery happens as part of the usual game play.
4.11 Rating Request
4.12 Rating Info
5. RSA Packets 27 :
- uint8 (equal to 251 to indicate packet contains a new RSA key), followed by
- uint8 (protocol version), followed by
- uint16 (subversion), followed by
- uint32 (IP of server 29), followed by
- uint32 (IP of client 30), followed by
- uint64 (keccak hash of client binary), followed by
- uint64 (e of RSA key), followed by
- uint8*490 (N of RSA key), followed by
- uint64 (preferred padding — the magic value of 0x13370000 requests random padding ; all other values will be used as such, bitwise, ie like an infinite-length OTP consisting of the value repeated), followed by
- uint16 (message count), followed by
- padding to RSA-message length, 1424 (5616-8-8-16-32-64-64-3920-64-16) bits exactly.
- uint8 (equal to 157 to indicate packet contains new Serpent keys), followed by
- uint8 (count of keys 32 in this set, n; n<=19 33), followed by
- n*(4*int64 + uint32) (32 bytes each key followed by a 4 byte ID calculated as crc32 on the key itself), followed by
- an uint8 flag (LSB bit set — keys to be used to talk to client ; MSB set — key to be used to talk to server ; client-set MSB is ignored by server ; server will set LSB on keys requested by client for its own use thus supporting clients with no trustworthy random generators of their own), followed by
- uint16 (message count), followed by
- padding to RSA-message length.
- New client issues 5.1 packet keyed to the server’s public key, including its own RSA key.
- The client’s IP is recorded, and will have to be explicitly changed by the client later if needed. Server replies with 5.1 packet keyed to the client’s announced key, including its private RSA key for use by that client ; and with 5.2 packet containing key material for client’s use in keying messages to the server. If the client fails to provide its own set of serpent keys, the server will further issue it a set of serpent keys ; thenceforth the server will send more serpent keys mirroring the client’s supply, and will similarily mirror key burning and select operations on its own set.
- Should the client’s IP change, it will issue a 5.1 packet keyed to the server public key immediately followed by a 5.1 packet keyed to the server’s original private key. The server will then update the client’s IP accordingly (this also trashes the extant Serpent keyset and triggers 5.2).
- The bulk of communication is intended to go through the Serpent system ; outside of identification and bootstrap handshakes RSA isn’t used. Should either party believe the Serpent keysets’ve been FUBAR’d, a 5.2 packet will reset that keyset.
- uint8 (type ID, =0), followed by
- uint8 (count of objects), followed by
- n* uint32 (object ids). Defaults to currently targeted item.
- uint8 (type ID, =1), followed by
- text (title of quest), followed by
- u64 (count of how many different times can this quest be completed – note that the reward will be divided accordingly), followed by
- uint32 (arbiter id, defaults to nearest town crier), followed by
- uint32 (object id of a sample of the required item), followed by
- uint64 (wanted quality, defaults to 1), followed by
- uint64 (wanted quantity, defaults to 1), followed by
- uint8 (expiry day, defaults to 1), followed by
- uint8 (expiry month, defaults to 1), followed by
- uint64 (expiry year, defaults to next year), followed by
- uint32 (object id of reward item stack, no default)
- uint8 (type ID, =2). This is a request to start the chosen activity (see next field), followed by
- uint32 (object id, indicating the intended activity, defaults to explore 37), followed by
- uint32 (object id, defaults to current target or location, as relevant), followed by
- uint32 (object id, defaults to currently equipped method, followed by 38)
- uint32 (object id, defaults to currently equipped tool)
- uint8 (type ID, =3), followed by
- uint32 (object id, the other party), followed by
- uint32 (object id, the trade itself 39), followed by
- uint8 (count of objects), followed by
- n* uint32 (object ids) and uint64 (object count), followed by
- uint8 (flag, set to 0x10 to lock a trade and to 0x0c to approve a trade previously locked by both players).
- uint8 (type ID, =4), followed by
- uint32 (object id, the other party), followed by
- uint32 (object id, the battle itself (server set, exactly in the way trade works). This not currently implemented, except player setting itself the bomb results in instadeath.
- uint8 (type ID, =6), followed by
- uint32 (destination id, defaults to current target), followed by
- uint32 (slot id), followed by
- uint32 (object id, of the item being moved), followed by
- uint32 (quantity moved).
- uint8 (type ID, =7), followed by
- uint32 (object id, the other party), followed by
- uint32 (object id, the train session itself (server set, exactly in the way trade and battle work).
This not currently muchly implemented, except some NPCs train for money — but will get greatly expanded asap.Meanwhile, it actually got both expanded and implemented, see the relevant comment for more details. - uint8 (type ID, =8), followed by
- object type, containing new client position.
- uint8 (type ID, =9), followed by
- text (line content), followed by
- uint32 (room ID), followed by
- uint64 (keccak hash of the line’s content)
- uint8 (type ID, =10), followed by
- text (rating comment), followed by
- uint32 (target ID)
- uint8 (rating value)
- This type of iterative refinement and interaction between design and implementation is neither new nor surprising to me, really. The only difference is that I’m doing now out of necessity all sides of this. Experience helps *a lot*, of course, even or perhaps especially beyond and above what one ‘expects’ or what one ‘wants’ or what one ‘signed up for’ and so on and so forth. It might even help in the sense that it makes it possible at all, at that, even if it doesn’t make it easy by any definition of the word.[↩]
- Floating point item deliberately not specified[↩]
- Each such message is OAEP-padded and then encrypted with a (3920 bit) RSA key. Three such messages are strung together to form a RSA packet. Because of the significant overhead involved (both in terms of space and time), Serpent-encrypted comms are preferred whenever feasible. [↩]
- See TMSR-RSA OAEP padding for the principle and this discussion for details.[↩]
- This is the total size of a packet containing RSA-encrypted material. The useful size (ie payload) of such a packet is merely 702 bytes.[↩]
- We really really want to keep this down. 13 bytes is the lowest I can conceive of, but I would so not mind halving it.[↩]
- Representing the identifying hash of the object in question.
We’re using the narrower size to save on network traffic — all the expenditure of another 32 bits here would buy us is de-ambiguation for cases where the count of objects around makes 1 in 2 billion collisions relevant. It doesn’t seem likely a client could support such abundance of objects.
Note that the hashes used here are client-specific, the server doesn’t leak its own internal representation of objects to the clients.[↩]
- Coordinates X, Y and Z in that order. Because the map goes from -512 to +512, the relationship between the given figure (GF) and map coordinates (MC) is GF / factor_coord – max_coord = MC, where max_coord is 512 and factor_coord is 65535/(2*max_coord).[↩]
- As a full rotation is 2 pi, the relationship between the given figure (GF) and object rotation (OR) is GF / 128 * pi = OR.[↩]
- This arrangement permits the representation of arbitrarily large textfields (2nd segment can represent up to 115`792`089`237`316`195`423`570`985`008`687`907`853`269`984`665`640`564`039`457`584`007`913`129`639`936 bytes, which is more than enough space for all the text ever produced — or likely to ever be produced — by humanity) at the modest cost of a fixed 3 byte header.
Unfortunately, it has no longer any utility for Eulora, since we’ve moved to fixed packets. I’m preserving it here because I really like it in the abstract and it has no other place to go.[↩]
- These packets consist of 92 successive 128 bit chunks, Serpent-enciphered individually. To extract the payload one splits the message into 92 16-byte chunks, deciphers them then collates the output into a final result. To produce the packet one cuts a 11`776 bit payload into 92 128-bit chunks, Serpent-enciphers them, and collates the results into the outbound packet.[↩]
- Polynomial generator 0x04c11db7. Keys with null IDs are discarded and regenerated.[↩]
- Each client and the server will keep a count of messages they sent each other. This value must be incremented on each subsequent message sent by no less than 1 and no more than 255.[↩]
- Keys are maintained by both client and server in an ordered ring buffer 256 elements long. The server will not send more keys than the total count of 0(absent)-keys in the respective buffer, irrespective of request count. If the message contains an unknown ID or otherwise is unprocessable, the issuance of a 5.2 packet is adequate response.[↩]
- This is the keccak hash of the actual file contents. By convention this hash rendered as a 32 alphanumeric character string is also used as the filename for the file in question.[↩]
- This system allows up to 65`536 manifest packets, adding up to potentially 11`993`088 (65`536 * 183) fragments representing a file of up to about 140 Gb (141`038`726`624 = 11`993`088 * 11`760 + 11`744 bits exactly). This will have to be sufficient.[↩]
- From 1 to 146 inclusive.[↩]
- This also means the protocol does not allow the transfer of files of certain sizes (within 8 bits of a multiple of 11760), which is fine with me.[↩]
- The final fragment of the file will have to be padded to length as per this spec. [↩]
- This is never issued by the server.[↩]
- This is never issued by the client.[↩]
- As discussed in comments, the world is a hierarchical structure of objects within objects.[↩]
- This portion will get more clarification later on! [↩]
- The complete list of these is currently exposed by the extant client, but in any case we’ll publish a complete schematic. The server will set the “target” of the player on the last object in the list.[↩]
- Limited to at most 1445 characters, matching the maximum length of a chat line.[↩]
- Valid rating values are integers between -10 and 10. For a given rating value here GR, the corresponding actual rating value RV is therefore calculated as: Min(GR, 20) – 10.[↩]
- These packets consist of three 490 byte successive chunks RSA-encrypted individually. To extract the payload one splits the message into three 490 byte chunks, RSA-decrypts and de-OAEP-pads each one, the collates the results into a final result. To produce the packet one cuts a 5`616 bit payload into three 1`872 bit chunks, OAEP-pads and encrypts them, and collates the results into the outbound packet.[↩]
- This is the manner in which new clients register their RSA key with the server (thereby opening a new game account). Later replacement of a registered key IS NOT POSSIBLE. Keep your client’s RSA key safe.
This is also the manner through which IP changes for an account are registered with the server. See the Protocol Mechanics heading for details.[↩]
- This is used by the server when signalling to the client to talk to a different server (which is a thing for scaling, because different sectors will be handled by different servers).[↩]
- If the client doesn’t know its own IP, it’s acceptable for this to be zero. [↩]
- This permits either client or server to declare Serpent keys via RSA. It is not mandatory (as there exists a Serpent-encapsulated mechanism for the same end) but entirely legal. The server will always respond with at least one 5.2 packet after an accepted 5.1 packet creates a new player account, consisting of 40 Serpent keys to be used to talk to the server. Should the client respond with any other packet than 5.2 or 4.1, the server will send a 2nd 5.2 packet, containing 40 Serpent keys for the client’s use. [↩]
- Keys obtained through a 5.2 packet are always indexed in the client’s buffer in the order they were found in that packet, starting with the first position.[↩]
- A RSA packet has 702 total bytes available, of which 5 are used otherwise and the remainder of 697 are available for packing serpent keys, which take 36 bytes each (crc32 id inclusive).[↩]
- Length being actually how they’re sorted on the server side.[↩]
This is meant as a production order, essentially. I expect it will get further refined as its implementation gets nearer.It already got further refined and more specifically defined as a way to issue/create Quests or “wanted” posters fundamentally, see the relevant comment on quests.[↩]- This is a generic initiation of player activity and as such literally an attempt, with the contents defining the exact activity. It replaces both Explore and Repair from earlier versions of this protocol but it further stands in for *any* game-defined activity even ones that may appear only at a later time. It’s as generic as it gets and quite on purpose, since this is the network layer, not the game layer. For more details on the game-defined activities, see the description of the data hierarchy.[↩]
- Known activities are obtained like everything else in game as part of the data hierachy representing the client’s view of the world at any given time. The relevant grammar for it will be made public as it serves as the in-game protocol specification just as this serves as the network-level client-server protocol specification.[↩]
- Known in Eulora 1 as ‘recipe’ and ‘equipped in mind’. At this level, the protocol for Eulora2 is really more generic than that, without any loss otherwise whatsoever.[↩]
- This is set by server through a type 6 message for both players involved, the trade is an object like any other that the OP has to request. The server will also expire trades, enforce them etc.[↩]
- Note that this is an action and as such just a part of a 4.5 message, Client Action, where the overall message structure is specified.[↩]
5.1. RSA key set 28.
5.2. Serpent key set 31:
6. Protocol Mechanics :
6.0. All communications between server and client will consist of messages. These messages may be encrypted either via eucrypt.RSA or eucrypt.Serpent. All RSA-encrypted messages will be exactly 1`470 bytes in length ; all Serpent messages will be exactly 1`472 bytes in length 34 . The server will handle Serpent messages in preference of RSA messages (which are processed on an as-available basis). Clients that send garbage will be punished ; the costs involved (encryption/decryption ; generating entropy ; lookups and whatnots) will be pushed onto the client, for which reason writing the clients lightly pays off.
6.1 The handshake works as follows:
6.2. The server will issue type 4.6 packets in response to relevant type 4.5 packets received — these can either signify the acceptance or the rejection of the client action, and the client must adjust its internal state accordingly.
7. Character Actions :
7.0. Lock:
7.1. Make 35:
7.2. Attempt 36:
7.3. Exchange:
7.4. Attack:
7.6. Move:
7.7. Train:
7.8. Relocate:
7.9 Say 40:
7.10 Rate:
Please leave your comments below.