Ossasepia

November 4, 2018

SMG Comms Chapter 6: Packing and Unpacking RSA

Filed under: Coding, SMG_Comms — Diana Coman @ 1:38 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

Building on the new convenient way to simply call rsa+oaep encrypt/decrypt from Ada, this chapter provides the Pack/Unpack functions that convert between SMG Comms RSA packets (each of them 1470 octets long, effectively several RSA-encrypted chunks glued together) and SMG Comms messages aka the actual content of the packet from the game's point of view. To do this, the Pack/Unpack functions simply chop the input into fixed-size parts, apply rsa+oaep encrypt/decrypt as suitable and then glue back the results in order, providing the whole thing as final output. Note that from the point of view of Pack/Unpack, both packets and messages have fixed lengths, as specified! In other words: the padding that might be added to data structures to get the whole thing to the specified length of a RSA message is preserved - that is no concern at this layer at all1. In a nutshell: Unpack removes any padding that Pack added but preserves any other padding that was added *before* calling Pack. In turn, Pack expects as input a fixed-size array of octets aka a RSA message as defined in Raw_Types - and the size of this is simply a multiple of the maximum size that one single RSA-encrypted operation can handle.

While the above fixed-size helps considerably to keep the whole thing quite easy to follow and clear, things are still a bit more gnarly than with Serpent, there's no escaping that. In particular, the RSA Decrypt operation CAN fail visibly - when the decoded "length" of the actual content turns out to be larger than maximum/provided space. For this reason, the Unpack function has a "Success" flag that gets set to false when output is essentially undefined. Note that the Unpack will fail in this way as soon as one of the chunks fails the RSA+OAEP decrypt in any way - there is no point in even attempting to recover some "partial" thing. Even more importantly: a True value for the Success flag does NOT indicate in any way that the result obtained is valid - all it says is that the RSA+OAEP decrypt operations completed successfully and the result is the one returned but whether that's a valid data structure or not is not a concern at this point (nor could it be since Unpack can't check such a thing in any way).

The .vpatch for this chapter includes the following main changes/additions:

  • A new note is added next to the offending Keccak line, describing the proposed way2 to preserve the No_Implicit_Conditionals restriction if desired. As mentioned in the forum, I do not quite see the need for this change at this moment but perhaps it will be needed at a later time, so there it is, right where it'll come in handy.
  • There's a correction of the value for the constant RSA_MSG_OCTETS. The new, correct value matches the fact that this is meant to be the fixed-size length of a SMG Comms RSA message rather than (as it was before) the max length that can be encrypted with RSA in one operation. So the RSA_MSG_OCTETS is calculated based on other specification-defined constants and it is a multiple of the max length that can be encrypted with RSA in one operation.
  • As part of doing the above correction of RSA_MSG_OCTETS, I also moved the initialization of some of the OAEP constants into Raw_Types (from OAEP packet where they were before). More precisely, the OAEP packet still uses its own internal constants (e.g. the "reserved" octets spelling "TMSR-RSA") but it initialises those (at elaboration time) with the values that it reads from Raw_Types. This reflects the fact that the OAEP packet in this project is the one using the Raw_Types that the protocol defines, not the other way around (in other words: OAEP does not define/impose those values - it's SMG Comms that does!).
  • The Packing package lost its "Pure" status (i.e. the compiler can't rely on it having no internal state and can't therefore cache calls to its method). The reason for this is at root the need for random padding for RSA since yes, can't cache the call to a method that has to use different, random padding at every call. In any case, for anyone interested, the specific route to follow this is that the Pack method for RSA calls OAEP+RSA which in turn use the non-pure Sequential_IO (via the RNG package) to read random octets from a Fuckgoats.
  • The new Pack and Unpack methods for RSA messages <-> packets conversions are in packing.ads/adb.
  • Corresponding tests for the Pack/Unpack methods are added to test_packing.ads/adb and called from testall.adb.

The signatures of the new Pack/Unpack methods in packing.ads are the following:

  -- Packing a RSA message into RSA packet, using the given key
  function Pack( Msg : in Raw_Types.RSA_Msg;
                 K   : in RSA_OAEP.RSA_pkey)
                return Raw_Types.RSA_Pkt;

  -- Unpacking a RSA packet into contained message, using the given key
  function Unpack( Pkt     : in Raw_Types.RSA_Pkt;
                   K       : in RSA_OAEP.RSA_skey;
                   Success : out Boolean)
                  return Raw_Types.RSA_Msg;

And the corresponding code for the new Pack/Unpack in packing.adb:

  -- Packing a RSA message into RSA packet, using the given key
  function Pack( Msg : in Raw_Types.RSA_Msg;
                 K   : in RSA_OAEP.RSA_pkey)
                return Raw_Types.RSA_Pkt is

  -- a chunk that can be processed via rsa+oaep at any given time
    Chunk: Raw_Types.Octets(1..Raw_Types.OAEP_MAX_LEN) := (others => 0);

  -- number of chunks in the message to process
  -- NO incomplete chunks will be processed!
  -- NB: values are set so that there are no incomplete blocks here
    N   : constant Natural := Msg'Length / Chunk'Length;

  -- intermediate result, as obtained from rsa_oaep
    Encr: Raw_Types.RSA_len := (others => 0);

  -- final resulting RSA Packet
    Pkt : Raw_Types.RSA_Pkt := (others => 0);
  begin
    -- there should ALWAYS be precisely N chunks in Msg to feed to rsa_oaep
    -- process chunks of Msg one at a time
    for I in 1..N loop
      -- get current chunk
      Chunk := Msg(Msg'First + (I-1) * Chunk'Length ..
                   Msg'First + I * Chunk'Length - 1 );
      -- call rsa oaep encrypt on current chunk
      RSA_OAEP.Encrypt( Chunk, K, Encr );
      -- copy result to its place in final packet
      Pkt( Pkt'First + (I-1) * Encr'Length ..
           Pkt'First + I * Encr'Length - 1 ) := Encr;
    end loop;
    -- return final result
    return Pkt;
  end Pack;

  -- Unpacking a RSA packet into contained message, using the given key
  function Unpack( Pkt     : in Raw_Types.RSA_Pkt;
                   K       : in RSA_OAEP.RSA_skey;
                   Success : out Boolean)
                  return Raw_Types.RSA_Msg is
    -- a chunk - basically input for RSA_OAEP.Decrypt
    Chunk : Raw_Types.RSA_len := (others => 0);

    -- intermediate result of rsa_oaep decrypt
    Decr  : Raw_Types.Octets( 1..Raw_Types.OAEP_MAX_LEN ) := (others => 0);
    Len   : Natural;
    Flag  : Boolean;

    -- number of chunks in the packet
    -- NB: there should be only FULL chunks! otherwise -> fail
    N   : constant Natural := Pkt'Length / Chunk'Length;

    -- final resulting message content of the given RSA packet
    Msg : Raw_Types.RSA_Msg := (others => 0);
  begin
    -- initialize Success flag
    Success := True;

    -- process given packet, chunk by chunk
    for I in 1..N loop
      -- get current chunk
      Chunk := Pkt( Pkt'First + (I-1) * Chunk'Length ..
                    Pkt'First + I * Chunk'Length - 1 );
      -- decrypt it via rsa+oaep
      RSA_OAEP.Decrypt( Chunk, K, Decr, Len, Flag );
      -- check result and if ok then copy it to final result at its place
      -- NB: if returned length is EVER less than OAEP_MAX_LEN then -> fail!
      -- the reason for above: there will be undefined bits in the output!
      if Len /= Raw_Types.OAEP_MAX_LEN or (not Flag) then
        Success := False;
        return Msg;
      else
        Msg( Msg'First + (I-1) * Decr'Length ..
             Msg'First + I * Decr'Length - 1 ) := Decr;
      end if;
    end loop;

    -- return obtained message
    return Msg;
  end Unpack;

The new tests for the above, in test_packing.adb:

  procedure Test_Pack_Unpack_RSA is
    Msg       : RSA_Msg;
    Decr_Msg  : RSA_Msg;
    PKey      : RSA_pkey;
    SKey      : RSA_skey;
    Success   : Boolean;
    Pkt       : RSA_Pkt;
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
  begin
    -- initialize RSA key with values previously obtained from EuCrypt
    Hex2Octets( n, SKey.n );
    Hex2Octets( e, SKey.e );
    Hex2Octets( d, SKey.d );
    Hex2Octets( p, SKey.p );
    Hex2Octets( q, SKey.q );
    Hex2Octets( u, SKey.u );
    -- copy n and e for public key
    PKey.n := SKey.n;
    PKey.e := SKey.e;
    -- get random data for "message"
    RNG.Get_Octets(Msg);

    -- pack the message into corresponding packet
    Pkt := Packing.Pack( Msg, PKey );
    -- unpack and check the result
    Decr_Msg := Packing.Unpack( Pkt, SKey, Success );
    if (not Success) or (Decr_Msg /= Msg) then
      Put_Line("FAIL: pack/unpack with RSA.");
    else
      Put_Line("PASS: pack/unpack with RSA.");
    end if;

    -- try to unpack a mangled package
    Pkt(Pkt'First + 3) := Pkt(Pkt'First + 3) and 16#AB#;
    Decr_Msg := (others => 0);
    Decr_Msg := Packing.Unpack( Pkt, SKey, Success );
    if Success then
      Put_Line("FAIL: pack/unpack with RSA on mangled package.");
    else
      Put_Line("PASS: pack/unpack with RSA on mangled package.");
    end if;

  end Test_Pack_Unpack_RSA;

The .vpatch with all the above is on my Reference Code Shelf and linked below for your convenience:

UPDATE (5 November 2018): I've changed the tests so that they read the RSA keypair from a file rather than using a hard-coded key; as discussed in the logs, this is the only way I can see to keep all lines in code files under 80 columns while also keeping RSA components as they are i.e. well longer than 80 characters in any sane format. The added procedure simply reads key components in a rather blunt way, character by character and demanding *exactly* as many characters as needed to fill the length of key it expects. This is NOT a generic "key reading" but simply a quick switch of input for the basic tests:

  procedure ReadRSAKey( Filename: in String; Key: out RSA_OAEP.RSA_skey ) is
    package Char_IO is new Ada.Sequential_IO(Character);
    use Char_IO;
    Full : String(1..RSA_len'Length*2) := (others => '0');
    Half : String(1..RSA_half'Length*2) := (others => '0');
    F    : Char_IO.File_Type;
    C    : Character;
  begin
    Open( File => F, Mode => In_File, Name => Filename );

    -- read n
    for I in Full'Range loop
      Read(F, Full(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Full, Key.n);

    -- read e
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.e);

    -- read d
    for I in Full'Range loop
      Read(F, Full(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Full, Key.d);

    -- read p
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.p);

    -- read q
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.q);

    -- read u
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    Hex2Octets(Half, Key.u);

    -- Close file
    Close( F );

    exception
      when Char_IO.End_Error =>
        Put_Line("ReadRSAKey ERROR: Unexpected end of file in " & Filename);
      when others =>
        Put_Line("ReadRSAKey ERROR: can not open file " & Filename);

  end ReadRSAKey;

Since this change of inputs for tests has nothing to do with either protocol code per se or with advancing the protocol implementation, I'm considering it more of a fix of sorts to this chapter's code and so it'll be a .vpatch by itself, to be found both on the Reference Code Shelf and linked from here:

UPDATE (12 November 2018): I've refactored the tests to extract the reading of a RSA key in a package of its own and thus use it afterwards from everywhere it's needed (i.e. not only in testing oaep_rsa but also in testing the packing for instance). Since this happened after I published Chapter 7, it's a new .vpatch on top:


  1. It will be handled at a higher level, where data structures are populated. 

  2. Thank you, asciilifeform! 

October 31, 2018

SMG Comms Chapter 5: RSA with OAEP from Ada

Filed under: Coding, SMG_Comms — Diana Coman @ 5:35 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

This brings together RSA and OAEP as well as the raw representation as octets of all the things involved: true random padding, RSA key components, plain messages, encrypted outputs, OAEP blocks or Keccak hashes. The main goal was to have some clear and easy to use methods in Ada for doing the whole RSA+OAEP dance from start to finish and then back as needed. Due to the current situation re MPI and RSA, a significant part of this goal involves keeping the C swamp at bay: calling C methods only when there really is at the moment no alternative and in general as few times as possible, making sure that anything built on top of this part of smg_comms can effectively ignore that there is any C code involved at a lower level at all. The reason for insulating the C code away like this is of course that I hope to be able to get rid of it entirely at some point and have everything in Ada without having to touch anything higher up than this part right here. At some point, but not at this point here, not just yet.

At first glance, this chapter was meant to be quite easy and straightforward: glue the different parts together and that's it! As usual, the details were more gnarly and as a result there were a few things I had to change/re-implement in order to get this part into some sort of reasonable shape:

  • I've added a few new subtypes of Octets to Raw_Types: RSA_len and RSA_half to hold a component of precisely the length or, respectively, half of the length set for the modulus of a RSA key. Those are needed to store the RSA keys in raw format aka arrays of octets. While I could also use the generic Octets type, the clear specification of length reflects the expected lengths of each RSA component and brings clarity so it's well worth it in my books.
  • The random padding is now obtained directly from Ada code rather than via C_wrappers and truerandom.c. The new Ada code simply reads from a Fuckgoats that it expects to be available at a pre-set path, *already initialised*1 and ready to read from. This means that random octets do not require anymore a call to C and the corresponding conversions from array of octets to char * and back.
  • The comparison between an OAEP block and the RSA key's modulus is simplified: it's only the top octet of both that is compared. This was discussed previously in the logs but at any rate, the reason for this approach is that it greatly reduces the complexity of the code since it's really a one line compare of 2 8-bit values that are already available as such at the very spot where they are needed! So no more calling the C swamp, converting to char* and then to MPIs and then comparing the full MPIs and returning the result through to Ada. The potential additional OAEP encryption performed at times is considered a price well worth paying in order to avoid the whole Ada->C->MPI->ADA dance that needs to be performed otherwise *each and every time*.
  • The conversions from Ada's array of octets to C's char* are done in local code, octet by octet and strictly on arrays that are precisely the same size. I am aware that GNAT provides in principle methods for such conversions, namely To_Ada and To_C but they still fail to convince me that they provide a better service really. If you think they do though, feel free to state your case in the comments below.
  • The OAEP implementation is the one from EuCrypt but with different access/interface methods: instead of accepting and producing Strings like the EuCrypt version, the OAEP package in SMG COMMS accepts and produces arrays of octets (i.e. Raw_Types.Octets type) since this is effectively what is needed here. Not to mention that I'm not particularly sad to get rid of Strings even when they are fixed size. Obviously, this means that I also replaced the conversions String <-> Bitstream with Octets <-> Bitstream. The rest however stays the same (including the underlying Keccak for hashes).
  • Because of moving all the OAEP+RSA part to Ada and imposing only one-way calls from Ada to C, there was no need anymore for some of the wrappers that were provided in EuCrypt's OAEP and rsa_oaep code. Those were simply discarded so that the code you get by pressing this patch includes simply what is needed for SMG Comms and nothing more.
  • Whenever the Ada code (specifically the rsa+oaep code) retrieves an array of octets from C, it will basically reverse the "normalization" that the MPI lib performs so that one doesn't end up in the idiotic situation where the retrieved value is actually different than the one given to MPI just because of leading zeroes. This is part of the price paid for using the MPI lib and at any rate it should not be the concern of any of the calling code so it's kept well separated so that any callers of RSA_OAEP.Encrypt or RSA_OAEP.Decrypt don't have to deal with it directly: the encryption uses at most msg_max_len octets and produces a fixed-size array while the decrypt similarly requires the same fixed-size array as input and will produce as output the actual length of the plain message and the plain message itself.
  • The "No Implicit Conditionals" restriction for the whole smg comms had to be discarded because of the Keccak implementation - with the restriction applied, compilation of Keccak fails2. At the moment I don't consider this to be a big issue here but feel free to loudly disagree in the comments below - if you have a solid case, I can change my mind.
  • I've updated the relevant tests for RSA (including those from the C_Wrappers package previously published) to use RSA keys of the new size that SMG Comms uses (as opposed to the EuCrypt size). I've added also new tests for the new RSA+OAEP Ada code, of course.

The new Ada code in rng.ads for reading random octets directly from an FG that is already initialised:

 -- True Random octets generator reading from an FG and using Raw_Types.
 -- S.MG, 2018

with Ada.Sequential_IO;
with Interfaces;
with Raw_Types;

package RNG is

  -- for reading from the FG one Octet at a time
  package Octet_IO is new Ada.Sequential_IO(Element_Type =>
                                              Interfaces.Unsigned_8);

  -- path to the FG; NB: the FG has to be initialized prior to using SMG Comms!
  RNG_PATH: constant String := "/dev/ttyUSB0";

  -- exception raised when FG is not accessible / read fails
  FG_Failure : exception;

  -- reads O'Length octets from the FG at RNG_PATH
  procedure Get_Octets( O: out Raw_Types.Octets );
end RNG;

And the corresponding rng.adb, short and sweet:

 -- S.MG, 2018

with Ada.Exceptions; use Ada.Exceptions;

package body RNG is

  procedure Get_Octets( O: out Raw_Types.Octets ) is
    F : Octet_IO.File_Type;
  begin
    begin
      Octet_IO.Open( File => F, Mode => Octet_IO.In_File, Name => RNG_PATH );
      for I in O'Range loop
        Octet_IO.Read( F, O(I) );
      end loop;
      Octet_IO.Close(F);
    exception
      when others => Raise_Exception(FG_Failure'Identity,
                                     "Failed to access default RNG source!");
    end;
  end Get_Octets;

end RNG;

The code for doing oaep+rsa encrypt from Ada, in rsa_oaep.ads:

 -- Ada implementation of RSA with OAEP according to TMSR and Eulora spec
 -- Uses:
 --   - Eulora's raw types (Ada, raw_types.ads)
 --   - Keccak hashes (Ada, keccak.ads/adb)
 --   - OAEP schema (Ada, oaep.ads/adb)
 --   - RNG (Ada, rng.ads/adb) for true random padding
 --   - C wrappers lib (C, c_wrappers/) for:
 --       - MPI (C, mpi/)
 --       - RSA (C, rsa/rsa.c)
 --
 -- S.MG, 2018

with Raw_Types;
with Interfaces.C; use Interfaces.C;

package RSA_OAEP is
  -- exception for mismatched lengths when converting octets <-> char arrays
  Mismatched_Lengths_Error: exception;

  -- public RSA key with n,e stored as raw octets
  type RSA_pkey is
    record
      n : Raw_Types.RSA_len; --public modulus
      e : Raw_Types.RSA_half; --public exponent
    end record; --RSA_pkey

  -- private (secret) RSA key with components stored as raw octets
  type RSA_skey is
    record
      n : Raw_Types.RSA_len;  --public modulus
      e : Raw_Types.RSA_half; --public exponent
      d : Raw_Types.RSA_len;  --secret exponent e*d=1 mod phi; phi=(p-1)*(q-1)
      p : Raw_Types.RSA_half; --prime p
      q : Raw_Types.RSA_half; --prime q
      u : Raw_Types.RSA_half;  --inverse of p mod q; for faster calculations
    end record; --RSA_skey

  -- Encryption RSA+OAEP (i.e. using public key)
  -- NB: at most OAEP.MAX_LEN_MSG octets from Plain will be encrypted!
  -- Relies directly on:
  --   oaep.adb/ads
  --   c_wrappers.c ( RSA )
  --   rng.adb/ads ( random padding )
  procedure Encrypt( Plain: in Raw_Types.Octets;
                     Key  : in RSA_pkey;
                     Encr : out Raw_Types.RSA_len );

  -- Decryption RSA+OAEP (i.e. using private/secret key)
  -- The opposite of Encrypt above.
  -- NB: Plain has to have ENOUGH space for result!
  -- Result can be at most OAEP.MAX_LEN_MSG octets.
  -- Relies directly on:
  --   oaep.adb/ads
  --   c_wrappers.c (RSA)
  -- Plain_Len gives the length in OCTETS of the decrypted message.
  procedure Decrypt( Encr      : in Raw_Types.RSA_len;
                     Key       : in RSA_skey;
                     Plain     : out Raw_Types.Octets;
                     Plain_Len : out Natural;
                     Success   : out Boolean);

 --helper methods:
 -- mainly conversions to/from C's char* and imports from C_wrappers
  -- encrypt with public RSA key given as struct, Ada style
  -- NB: result is potentially 0-led (i.e. at end of Encr not at start!)
  procedure Public_RSA( Plain: in Raw_Types.Octets;
                        Key  : in RSA_pkey;
                        Encr : out Raw_Types.RSA_len);

  -- encrypt with public RSA key given as char arrays, via C_Wrappers
  -- this returns the length of result because C functions trim leading 0s!
  function Public_RSA_C( Encr      : out Interfaces.C.char_array;
                         Encr_Len  : in Integer;
                         Plain     : in Interfaces.C.char_array;
                         Plain_Len : in Integer;
                         RSA_N     : in Interfaces.C.char_array;
                         N_Len     : in Integer;
                         RSA_E     : in Interfaces.C.char_array;
                         E_Len     : in Integer)
                     return Integer;
   pragma Import(C, Public_RSA_C, "public_rsa_octets"); 

  -- decrypt with private RSA key given as struct, Ada style
  -- NB: Plain has to have ENOUGH space!
  -- NB: Result is potentially 0-led (i.e. at the end of Plain, not at start!)
  -- @return actual length of result
  procedure Private_RSA( Encr  : in Raw_Types.RSA_len;
                         Key   : in RSA_skey;
                         Plain : out Raw_Types.Octets);

  -- encrypt with private/secret RSA key given as char arrays (via C_wrappers)
  -- this returns length because C methods trim leading 0s
  function Private_RSA_C( Plain     : out Interfaces.C.char_array;
                          Plain_Len : in Integer;
                          Encr      : in Interfaces.C.char_array;
                          Encr_Len  : in Integer;
                          RSA_N     : in Interfaces.C.char_array;
                          N_Len     : in Integer;
                          RSA_E     : in Interfaces.C.char_array;
                          E_Len     : in Integer;
                          RSA_D     : in Interfaces.C.char_array;
                          D_Len     : in Integer;
                          RSA_P     : in Interfaces.C.char_array;
                          P_Len     : in Integer;
                          RSA_Q     : in Interfaces.C.char_array;
                          Q_Len     : in Integer;
                          RSA_U     : in Interfaces.C.char_array;
                          U_Len     : in Integer)
                      return Integer;
  pragma Import( C, Private_RSA_C, "private_rsa_octets" );

  -- convert from Ada's Octets (array of octets) to C's char* (char_array)

  -- This copies the octets from O to the beginning of A
  -- NB: there are NO checks or memory allocations here!
  -- Caller has to make sure that:
  --   A has allocated space for at least O'Length octets!
  procedure Octets_To_Char_Array( O   : in Raw_Types.Octets;
                                  A   : out Interfaces.C.char_array);

  -- This copies first O'Length characters from A to O
  -- NB: this does NOT allocate /check memory!
  -- Caller has to ensure that:
  --    A has space at least O'Length characters
  procedure Char_Array_To_Octets( A   : in Interfaces.C.char_array;
                                  O   : out Raw_Types.Octets);

end RSA_OAEP;

With the corresponding implementation in rsa_oaep.adb:

 -- S.MG, 2018

with Interfaces; use Interfaces;
with OAEP;
with RNG;

package body RSA_OAEP is

  -- OAEP + RSA with given public key.
  -- Steps:
  -- 1. repeat oaep.encrypt of Plain
  --      until the result has first octet < than first octet of Key's modulus.
  -- 2. RSA.encrypt with Key on the result from 1.
  procedure Encrypt( Plain: in Raw_Types.Octets;
                     Key  : in RSA_pkey;
                     Encr : out Raw_Types.RSA_len ) is
    -- block obtained through OAEP padding of given input Plain
    Blk     : OAEP.OAEP_Block := ( others => 0 );
    Entropy : OAEP.OAEP_Block;
    Len     : constant Natural := entropy'Length;
  begin
    -- loop cond: 1st octet of oaep block is < 1st octet of key's modulus
    loop
      -- get random bits via RNG (FG)
      RNG.Get_Octets( Entropy );

      -- oaep encrypt
      OAEP.OAEP_Encrypt( Plain, Entropy, Blk );

      -- check that the oaep block is suitable i.e. comparison of 1st octet
      if Blk(Blk'First) < Key.n(Key.n'First) then exit; end if;

    end loop; -- oaep block < modulus

    -- RSA encrypt
    Public_RSA( Blk, Key, Encr );
  end; --Encrypt (OAEP+RSA)

  -- RSA+OAEP Decrypt
  -- Steps:
  -- 1. RSA Decrypt (secret key)
  -- 2. OAEP Decrypt
  procedure Decrypt( Encr : in Raw_Types.RSA_len;
                     Key  : in RSA_skey;
                     Plain: out Raw_Types.Octets;
                     Plain_Len: out Natural;
                     Success: out Boolean) is
    Blk     : OAEP.OAEP_Block;
    Msg     : OAEP.OAEP_HALF;
    Msg_Len : Natural;
  begin
    -- RSA decrypt with provided secret key
    -- NB: result HAS TO BE AN OAEP BLOCK HERE! Even if - potentially - 0 led!
    Private_RSA( Encr, Key, Blk );

    -- OAEP decrypt
    OAEP.OAEP_Decrypt(Blk, Msg_Len, Msg, Success);
    -- switch to Length in OCTETS!
    Msg_Len := Msg_Len / 8;

    -- check that result FITS in given output array - otherwise, fail
    if Msg_Len > Plain'Length then
      Success := FALSE;
      Plain_Len := Msg_Len; --to give a clue to caller
    else
      Plain_Len := Msg_Len;
      Plain( Plain'First ..
             Plain'First + Plain_Len-1) := Msg( Msg'First ..
                                                Msg'First + Plain_Len -1 );
    end if;
  end Decrypt;

  -- helper methods
  -- encrypt with public RSA key given as struct, Ada style
  procedure Public_RSA( Plain: in Raw_Types.Octets;
                        Key  : in RSA_pkey;
                        Encr : out Raw_Types.RSA_len) is
    Encr_char  : char_array( size_t(Encr'First) .. size_t(Encr'Last));
    Plain_char : char_array( size_t(Plain'First) .. size_t(Plain'Last));
    N_char     : char_array( size_t(Key.n'First) .. size_t(Key.n'Last));
    E_char     : char_array( size_t(Key.e'First) .. size_t(Key.e'Last));
    out_len    : Integer;
  begin
    -- convert to char array
    Octets_To_Char_Array( Plain, Plain_char );
    Octets_To_Char_Array( Key.n, N_char );
    Octets_To_Char_Array( Key.e, E_char );
    -- call C imported function
    out_len := Public_RSA_C( Encr_char  , Encr'Length,
                             Plain_char , Plain'Length,
                             N_char     , Key.n'Length,
                             E_char     , Key.e'Length);
    -- convert back to octets
    Char_Array_To_Octets( Encr_char, Encr );
    -- C code trims leading 0s -> need to move octets if out_len0);
    end if;
    -- no need to return anything!
  end Public_RSA;

  procedure Private_RSA( Encr  : in Raw_Types.RSA_len;
                         Key   : in RSA_skey;
                         Plain : out Raw_Types.Octets) is
    Plain_Char : char_array( size_t(Plain'First) .. size_t(Plain'Last) );
    Plain_Len  : Integer;
    Encr_Char  : char_array( size_t(Encr'First)  .. size_t(Encr'Last) );
    N_Char     : char_array( size_t(Key.n'First) .. size_t(Key.n'Last) );
    E_Char     : char_array( size_t(Key.e'First) .. size_t(Key.e'Last) );
    D_Char     : char_array( size_t(Key.d'First) .. size_t(Key.d'Last) );
    P_Char     : char_array( size_t(Key.p'First) .. size_t(Key.p'Last) );
    Q_Char     : char_array( size_t(Key.q'First) .. size_t(Key.q'Last) );
    U_Char     : char_array( size_t(Key.u'First) .. size_t(Key.u'Last) );
  begin
    -- convert key and encrypted message to C char_arrays
    Octets_To_Char_Array( Encr, Encr_Char );
    Octets_To_Char_Array( Key.n, N_Char );
    Octets_To_Char_Array( Key.e, E_Char );
    Octets_To_Char_Array( Key.d, D_Char );
    Octets_To_Char_Array( Key.p, P_Char );
    Octets_To_Char_Array( Key.q, Q_Char );
    Octets_To_Char_Array( Key.u, U_Char );
    -- call RSA decrypt via C_Wrappers
    Plain_Len := Private_RSA_C( Plain_Char, Plain'Length,
                                Encr_Char , Encr'Length,
                                N_Char    , Key.n'Length,
                                E_Char    , Key.e'Length,
                                D_Char    , Key.d'Length,
                                P_Char    , Key.p'Length,
                                Q_Char    , Key.q'Length,
                                U_Char    , Key.u'Length);
    -- convert result back to Octets
    Char_Array_To_Octets( Plain_Char, Plain );
    -- if length < OAEP_Block'Length,it's 0-led and got trimmed, so move it
    if Plain_Len < Plain'Length then
      Plain(Plain'Last-Plain_Len+1..Plain'Last):= Plain(Plain'First ..
                                                   Plain'First + Plain_Len -1);
      Plain(Plain'First .. Plain'Last-Plain_Len) := (others => 0);
    end if;
    -- no need to return anything!
  end Private_RSA;

  procedure Octets_To_Char_Array( O   : in Raw_Types.Octets;
                                  A   : out Interfaces.C.char_array) is
  begin
    -- check that lengths ARE THE SAME!
    if A'Length /= O'Length then
      raise Mismatched_Lengths_Error;
    end if;
    -- copy values over octet by octet
    for I in 0 .. O'Length-1 loop
      A( A'First + Interfaces.C.size_t( I )) :=
                     Interfaces.C.Char( Character'Val(O(O'First + I)));
    end loop;
  end Octets_To_Char_Array;

  procedure Char_Array_To_Octets( A   : in Interfaces.C.char_array;
                                  O   : out Raw_Types.Octets) is
  begin
    -- check that lengths ARE THE SAME!
    if A'Length /= O'Length then
      raise Mismatched_Lengths_Error;
    end if;
    -- copy values over octet by octet
    for I in 0..O'Length -1 loop
      O( O'First + I ) := Character'Pos( Character(
                             A( A'First + Interfaces.C.size_t( I )) ));
    end loop;
  end Char_Array_To_Octets;

end RSA_OAEP;

The new oaep.ads using Raw_Types.Octets:

-- Implementation of TMSR's OAEP with Keccak as hash function
-- NB: this uses Eulora's protocol constants (esp. RSA key length) and types.
--
-- S.MG, 2018

with Keccak;    -- Keccak is used as hash function
with Raw_Types; -- Eulora's protocol raw types and constant values

with Interfaces; use Interfaces; -- for Unsigned_8 type and bit-level ops

package OAEP is
  pragma Pure( OAEP ); -- stateless, no side effects -> can cache calls

  -- constants for OAEP
  OAEP_LENGTH_OCTETS : constant := Raw_Types.RSA_KEY_OCTETS;
  OAEP_LENGTH_BITS   : constant := OAEP_LENGTH_OCTETS * 8;
  OAEP_HALF_OCTETS   : constant := OAEP_LENGTH_OCTETS / 2;
  TMSR_STR           : constant String := "TMSR-RSA";
    -- "TMSR-RSA" as unsigned_8 values:
  TMSR               : constant Raw_Types.Octets := (84,77,83,82,45,82,83,65);
  MAX_LEN_MSG        : constant Natural := OAEP_HALF_OCTETS -
                                           TMSR_STR'Length - 3;

  -- subtypes for OAEP encrypt/decrypt
  subtype OAEP_Block is Raw_Types.Octets( 1 .. OAEP_LENGTH_OCTETS );
  subtype OAEP_HALF is Raw_Types.Octets( 1 .. OAEP_HALF_OCTETS );

  -- padding & formatting of maximum MAX_LEN_MSG octets of the given input
  -- uses TMSR's OAEP schema:
  -- 1.format M00 as: [random octet][sz1][sz2]"TMSR-RSA"[random]*Message
  --    where sz1 and sz2 store the length of the message in bits
  --    the random octets before message are padding to make OAEP_LENGTH_OCTETS
  -- 2. R = OAEP_HALF_OCTETS random bits
  -- 3. X = M00 xor hash(R)
  -- 4. Y = R xor hash(X)
  -- 5. Result is X || Y
  -- NB: the Entropy parameter should be random octets from which this method
  -- will use as many as required for the OAEP encryption of given Msg
  -- NB: at MOST MAX_LEN_MSG octets of Msg! (Msg at most MAX_LEN_MSG*8 bits!)
  procedure OAEP_Encrypt( Msg      : in Raw_Types.Octets;
                          Entropy  : in OAEP_Block;
                          Output   : out OAEP_Block);

  -- This is the opposite of OAEP_Encrypt above.
  -- @param Encr - an OAEP block previously obtained from OAEP_Encrypt
  -- @param Len - this will hold the length of the obtained message (in bits!)
  -- @param Output - the first Len octets of this are the recovered message
  -- @param Success - set to TRUE if message was recovered, false otherwise
  -- NB: when Success is FALSE, both Len and Output have undefined values
  procedure OAEP_Decrypt( Encr    : in OAEP_Block;
                          Len     : out Natural;
                          Output  : out OAEP_HALF;
                          Success : out Boolean);

private
  -- gnat-specific methods for bit-level operations
	function Shift_Right( Value  : Unsigned_8;
                        Amount : Natural )
                        return Unsigned_8;
  pragma Import(Intrinsic, Shift_Right); 

	function Shift_Left( Value  : Unsigned_8;
                        Amount : Natural )
                        return Unsigned_8;
  pragma Import(Intrinsic, Shift_Left); 

  -- helper method: xor 2 half-oaep blocks
  function XOR_Octets(A : in OAEP_HALF;
                      B : in OAEP_HALF)
                     return OAEP_HALF;

  -- conversions between bitstream and string
  -- NB: caller has to ensure correct size of output parameter! no checks here.
  procedure ToOctets   ( B : in Keccak.Bitstream;
                         O : out Raw_Types.Octets );

  procedure ToBitstream( O : in Raw_Types.Octets;
                         B : out Keccak.Bitstream );

  -- wrapper for Sponge to use Octets for input/output
  procedure HashKeccak( Input     : in Raw_Types.Octets;
                        Output    : out Raw_Types.Octets;
                        Block_Len : in Keccak.Keccak_Rate :=
                                       Keccak.Default_Bitrate);

end OAEP;

And the implementation in oaep.adb:

-- S.MG, 2018

package body OAEP is

  -- padding & formatting of maximum MAX_LEN_MSG*8 bits of the given input
  -- uses TMSR's OAEP schema:
  -- 1.format M00 as: [random octet][sz1][sz2]"TMSR-RSA"[random]*Message
  --    where sz1 and sz2 store the length of the message in bits
  --    the random octets before message are padding to make OAEP_LENGTH_OCTETS
  -- 2. R = OAEP_HALF_OCTETS random bits
  -- 3. X = M00 xor hash(R)
  -- 4. Y = R xor hash(X)
  -- 5. Result is X || Y
  -- NB: the Entropy parameter should be random octets from which this method
  -- will use as many as required for the OAEP encryption of given Msg
  -- NB: at MOST MAX_LEN_MSG octets of Msg! (Msg at most 1960 bits)
  procedure OAEP_Encrypt( Msg     : in Raw_Types.Octets;
                          Entropy : in OAEP_Block;
                          Output  : out OAEP_Block) is
    M00    : OAEP_HALF;
    R      : OAEP_HALF;
    HashR  : OAEP_HALF;
    X      : OAEP_HALF;
    HashX  : OAEP_HALF;
    Y      : OAEP_HALF;
    MsgLen : Natural;
    PadLen : Natural;
  begin
    -- calculate maximum length of msg and needed amount of padding
    -- make sure also that only MAX_LEN_MSG octets at most are used from Msg
    MsgLen := Msg'Length;                          -- real msg length
    if MsgLen > MAX_LEN_MSG then
      MsgLen := MAX_LEN_MSG;  --only first MAX_LEN_MSG octets used
      PadLen := 0;            --no padding needed
    else
      PadLen := MAX_LEN_MSG - MsgLen; -- add padding as needed
    end if;

    -- step 1: header and format to obtain M00
      -- first octet is random bits
    M00( M00'First ) := Entropy( Entropy'First );

      -- next 2 octets hold the used length of Msg (number of octets)
    M00( M00'First + 2) := Unsigned_8( ( MsgLen * 8 ) mod 256 );
    M00( M00'First + 1) := Unsigned_8( ( (MsgLen * 8 ) / 256 ) mod 256 );

      -- next 8 octets are reserved for later use, currently "TMSR-RSA"
    M00( M00'First + 3 .. M00'First + 10 ) := TMSR;

      -- random bits for padding, if Msg is less than maximum length
    for I in 1 .. PadLen loop
      M00( M00'First + 10 + I ) := Entropy( Entropy'First + I );
    end loop;

      -- the message itself
    M00( M00'Last - MsgLen + 1 .. M00'Last ) :=
                               Msg( Msg'First .. Msg'First + MsgLen - 1 );

    -- step 2: R = Raw_Types.OAEP_HALF_OCTETS random octets
    -- can take LAST octets from given entropy as they are NOT used before
    -- (even if original message was empty, padding uses at most half - 10
    --   while entropy has full block length)
    R := Entropy( Entropy'Last - OAEP_HALF_OCTETS + 1 .. Entropy'Last );

    -- step 3: X = M00 xor hash(R)
    HashKeccak( R, HashR );
    X := XOR_Octets(M00, HashR);

    -- step 4: Y = R xor hash(X)
    HashKeccak( X, HashX );
    Y := XOR_Octets(R, HashX);

    -- step 5: Output is X || Y
    Output( Output'First .. Output'First + X'Length - 1 ) := X;
    Output( Output'Last - Y'Length + 1 .. Output'Last )   := Y;

  end OAEP_Encrypt;

  procedure OAEP_Decrypt( Encr    : in OAEP_Block;
                          Len     : out Natural;
                          Output  : out OAEP_HALF;
                          Success : out Boolean ) is
    X, Y, M, R   : OAEP_HALF;
    HashX, HashR : OAEP_HALF;
    LenOctets    : Natural;
  begin
    -- step 1: separate X and Y
    X := Encr( Encr'First .. Encr'First + X'Length - 1 );
    Y := Encr( Encr'Last - Y'Length + 1 .. Encr'Last );

    -- step 2: R = Y xor hash(X)
    HashKeccak( X, HashX );
    R := XOR_Octets(Y, HashX);

    -- step 3: M = X xor hash(R)
    HashKeccak( R, HashR );
    M := XOR_Octets(X, HashR);

    -- step 4: extract length and message
    Len := Natural(M( M'First + 1 )) * 256 +
           Natural(M( M'First + 2 ));
    LenOctets := Len / 8; 

    if LenOctets > MAX_LEN_MSG or LenOctets < 0 then
      Success := False;  -- error, failed to retrieve message
    else
      Success := True;
      Output( Output'First .. Output'First + LenOctets - 1 ) :=
        M( M'Last - LenOctets + 1 .. M'Last );
    end if;

  end OAEP_Decrypt;

-- private, helper methods
  procedure HashKeccak(Input     : in Raw_Types.Octets;
                       Output    : out Raw_Types.Octets;
                       Block_Len : in Keccak.Keccak_Rate :=
                                      Keccak.Default_Bitrate) is
    BIn  : Keccak.Bitstream( 0 .. Input'Length * 8 - 1 );
    BOut : Keccak.Bitstream( 0 .. Output'Length * 8 - 1 );
  begin
    ToBitstream( Input, BIn );
    Keccak.Sponge( BIn, BOut, Block_Len );
    ToOctets( BOut, Output );
  end HashKeccak;

  function XOR_Octets(A : in OAEP_HALF;
                      B : in OAEP_HALF)
                     return OAEP_HALF is
    R : OAEP_HALF;
  begin
    for I in R'Range loop
      R(I) := A(I) xor B(I);
    end loop;
    return R;
  end XOR_Octets;

  -- conversion between types
  procedure ToOctets(B: in Keccak.Bitstream; O: out Raw_Types.Octets ) is
    Pos : Natural;
  begin
    Pos := B'First;
    for I in O'Range loop
      O(I) := Unsigned_8( B( Pos     ) ) +
              Unsigned_8( B( Pos + 1 ) ) * 2 +
              Unsigned_8( B( Pos + 2 ) ) * 4 +
              Unsigned_8( B( Pos + 3 ) ) * 8 +
              Unsigned_8( B( Pos + 4 ) ) * 16 +
              Unsigned_8( B( Pos + 5 ) ) * 32 +
              Unsigned_8( B( Pos + 6 ) ) * 64 +
              Unsigned_8( B( Pos + 7 ) ) * 128;
      Pos := Pos + 8;
    end loop;
  end ToOctets;

  procedure ToBitstream(O: in Raw_Types.Octets; B: out Keccak.Bitstream ) is
    V   : Unsigned_8;
    Pos : Natural;
  begin
    Pos := B'First;
    for I in O'Range loop
      V := O( I );
      B( Pos     ) := Keccak.Bit( V and 1 );
      B( Pos + 1 ) := Keccak.Bit( Shift_Right( V, 1 ) and 1 );
      B( Pos + 2 ) := Keccak.Bit( Shift_Right( V, 2 ) and 1 );
      B( Pos + 3 ) := Keccak.Bit( Shift_Right( V, 3 ) and 1 );
      B( Pos + 4 ) := Keccak.Bit( Shift_Right( V, 4 ) and 1 );
      B( Pos + 5 ) := Keccak.Bit( Shift_Right( V, 5 ) and 1 );
      B( Pos + 6 ) := Keccak.Bit( Shift_Right( V, 6 ) and 1 );
      B( Pos + 7 ) := Keccak.Bit( Shift_Right( V, 7 ) and 1 );

      Pos := Pos + 8;
    end loop;
  end ToBitstream;

end OAEP;

The new tests added to tests/test_rsa_oaep.adb and called from tests/testall.adb, basically calling naked rsa from Ada, OAEP by itself and then rsa+oaep on a few different inputs:

 --S.MG, 2018

with Interfaces; use Interfaces;
with Interfaces.C; use Interfaces.C;
with RSA_OAEP; use RSA_OAEP;
with OAEP; use OAEP;
with Raw_Types; use Raw_Types;
with RNG; use RNG;
with Keccak; use Keccak;

with Ada.Text_IO; use Ada.Text_IO;

package body Test_RSA_OAEP is

  procedure test_char_array is
    S : String := OAEP.TMSR_STR;
    O : Octets := OAEP.TMSR;
    A : char_array(0..O'Length-1) := (others => '0');
    B : Octets(0..O'Length -1) := (others => 0);
    Fail : Boolean := FALSE;
  begin
    Octets_To_Char_Array(O, A);
    Char_Array_To_Octets(A, B);

    if B /= O then
      Put_Line("FAIL: char_array_to_octets");
    else
      Put_Line("PASS: char_array_to_octets");
    end if;

    for I in 0..S'Length-1 loop
      declare
        C : Character := Character(A(A'First + size_t(I)));
        E : Character := S(S'First + I);
      begin
        if C /= E then
          Fail := TRUE;
          Put("Error at pos " & Integer'Image(I) & ": ");
          Put(Integer'Image(Character'Pos(C)));
          Put_Line(" instead of " & Integer'Image(Character'Pos(E)));
        end if;
      end;
    end loop;
    if FAIL then
      Put_Line("FAIL: test octets_to_char_array");
    else
      Put_Line("PASS: test octets_to_char_array");
    end if;
  end test_char_array;

  -- test OAEP encrypt + decrypt
  procedure test_oaep is
    Plain: Octets(1..MAX_LEN_MSG);
    Short: Octets(0..10);
    Encr : OAEP_Block;
    Decr : OAEP_HALF;
    Len  : Natural;
    Entropy: OAEP_Block;
    Success : Boolean;
  begin
    RNG.Get_Octets(Plain);
    RNG.Get_Octets(Entropy);
    RNG.Get_Octets(Short);

    -- test full length message
    OAEP_Encrypt(Plain, Entropy, Encr);
    OAEP_Decrypt(Encr, Len, Decr, Success);

    if not Success or Len/8 /= Plain'Length then
      Put_Line("FAIL: oaep encrypt/decrypt on max len message.");
    else
      if Decr(Decr'First..Decr'First+Len/8-1) /=
              Plain(Plain'First..Plain'First+Len/8-1) then
        Put_Line("FAIL: oaep encrypt/decrypt on max len message - " &
                 "result different from expected.");
      else
        Put_Line("PASS: oaep encrypt/decrypt on max len message.");
      end if;
    end if;

    -- test short message
    OAEP_Encrypt(Short, Entropy, Encr);
    OAEP_Decrypt(Encr, Len, Decr, Success);
    if not Success or Len/8 /= Short'Length then
      Put_Line("FAIL: oaep encrypt/decrypt on short message.");
    else
      if Decr(Decr'First..Decr'First+Len/8-1) /=
             Short(Short'First..Short'First+Len/8-1) then
        Put_Line("FAIL: oaep encrypt/decrypt on short message - " &
                 "result different from expected.");
      else
        Put_Line("PASS: oaep encrypt/decrypt on short message.");
      end if;
    end if;

  end test_oaep;

  -- test JUST RSA (i.e. without oaep) with RSA key pair previously generated
  procedure test_rsa is
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
    Plain: OAEP_Block := (others => 0);
    Decr : OAEP_Block := (others => 0);
    Encr : RSA_len;
    pkey: RSA_pkey;
    skey: RSA_skey;
  begin
    -- initialize with RSA pair previously generated
    Hex2Octets( n, skey.n );
    Hex2Octets( e, skey.e );
    Hex2Octets( d, skey.d );
    Hex2Octets( p, skey.p );
    Hex2Octets( q, skey.q );
    Hex2Octets( u, skey.u );
    -- copy n and e for public key
    pkey.n := skey.n;
    pkey.e := skey.e;
    -- get random data
    RNG.Get_Octets(Plain);
    -- make first octet < RSA key's modulus first octet
    Plain(Plain'First) := 16#00#;
    -- naked rsa encrypt/decrypt
Put_Line("Encrypting with RSA public key...");
    Public_RSA( Plain, pkey, Encr );
Put_Line("Decrypting with RSA private key...");
    Private_RSA( Encr, skey, Decr );
Put_Line("Checking...");

    -- check result
    if Decr /= Plain then
      Put_Line("FAIL: RSA encrypt/decrypt result doesn't match plain.");
    else
      Put_Line("PASS: RSA encrypt/decrypt");
    end if;
  end test_rsa;

  -- test rsa+oaep with RSA key pair previously generated
  procedure test_rsa_oaep is
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
    Plain: Octets(1..MAX_LEN_MSG) := (others=>20);
    Short: Octets(1..10);
    Decr : RSA_len;
    Encr : RSA_len;
    pkey: RSA_pkey;
    skey: RSA_skey;
    Success: Boolean;
    Len : Natural;
  begin
    -- initialize with RSA pair previously generated
    Hex2Octets( n, skey.n );
    Hex2Octets( e, skey.e );
    Hex2Octets( d, skey.d );
    Hex2Octets( p, skey.p );
    Hex2Octets( q, skey.q );
    Hex2Octets( u, skey.u );
    -- copy n and e for public key
    pkey.n := skey.n;
    pkey.e := skey.e;

    -- test with 0 message of length Plain'Length
    RSA_OAEP.Encrypt(Plain, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Plain'Length
         or Plain /= Decr(Decr'First..Decr'First+Plain'Length-1) then
      Put_Line("FAIL: RSA_OAEP on max len message 20-filled.");
    else
      Put_Line("PASS: RSA_OAEP on max len message 20-filled.");
    end if;

    -- get random data for "plain" message
    RNG.Get_Octets(Plain);
    RSA_OAEP.Encrypt(Plain, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Plain'Length
         or Plain /= Decr(Decr'First..Decr'First+Plain'Length-1) then
      Put_Line("FAIL: RSA_OAEP on random data of max length.");
    else
      Put_Line("PASS: RSA_OAEP on random data of max length.");
    end if;

    -- get random data for "short" message
    RNG.Get_Octets(Short);
    RSA_OAEP.Encrypt(Short, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Short'Length
         or Short /= Decr(Decr'First..Decr'First+Short'Length-1) then
      Put_Line("FAIL: RSA_OAEP on random data of short length.");
    else
      Put_Line("PASS: RSA_OAEP on random data of short length.");
    end if; 

  end test_rsa_oaep;

-- helper methods
  procedure Hex2Octets( Hex: in String; O: out Raw_Types.Octets ) is
    S : String := "16#AA#";
    -- to make sure that input String has EVEN number of chars (ie full octets)
    H : String(1..Hex'Length+Hex'Length mod 2) := (others=>'0');
  begin
    -- first char is 0 if needed to cover full octet...
    H(H'Length-Hex'Length+1..H'Length) := Hex;
    O := (others => 0);
    for I in 0 .. H'Length/2-1 loop
      S := "16#" & H(H'First + I*2 .. H'First + I*2 + 1) & "#";
      O(O'Last - H'Length/2 + 1 + I) := Unsigned_8'Value(S);
    end loop;
  end Hex2Octets;

  procedure PrintOctets( O: in Raw_Types.Octets; Title: in String ) is
  begin
    Put_Line(Title);
    for V of O loop
      Put(Unsigned_8'Image(V) & " ");
    end loop;
    New_Line;
  end PrintOctets;

end Test_RSA_OAEP;

The full .vpatch for this chapter is quite long at 1714 lines but a significant part of that is essentially imported from EuCrypt (Keccak most notably but also the main part of OAEP). As usual, the .vpatch and my signature for it can be found on my Reference Code Shelf or directly from the links below:


  1. Yes, I adopted Stanislav's approach to reading from the FG. I also had a look at his code at www.loper-os.org/?p=2175  

  2. The exact line is this one: http://www.dianacoman.com/2018/01/18/eucrypt-chapter-6-keccak-transformations/#selection-159.735-159.812 and it's been pointed out already (in the logs, where else?) that there is an implicit check against div by 0 which could be avoided simply by rewriting the line as Output(X, Y) := Rotate_Left(Input(X,Y), ( (T+1)*(T+2)/2) and (Z_Length -1) );  

October 25, 2018

SMG Comms Chapter 4: C Wrappers for RSA and MPI

Filed under: Coding, SMG_Comms — Diana Coman @ 7:28 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

The C part of SMG Comms is - unsurprisingly - a pain in the ass and a three-pointed thorn in the backside:
1. All the RSA operations are coded in C because of the heavy dependency on the MPI lib. This is not something I can change at the moment since I don't yet have an Ada implementation of long integer arithmetics. Sigh.
2. The OAEP padding schema has to be called repeatedly until the result is indeed (when considered as a very large number) a value smaller than the RSA key's modulus. So although the OAEP padding itself (including underlying Keccak are implemented in Ada), the comparison between obtained value and the modulus of the key goes through the MPI swamp so it forces a call to C. Nevertheless, the "implemented in Ada" and "repeat until the result CAN be encrypted with given RSA key" are both absolutely right and proper and as they should be so there's nothing to change here. Hooray.
3. As a result of 1 and 2 above, the code flow can easily get ugly because one ends up with calls from Ada to C (to do RSA operations for instance) but *also* from C to Ada (to do Keccak, if one aims to provide oaep+rsa from C directly). This is currently the sad state of the wrappers provided as first attempt in EuCrypt for rsa-oaep. Nevertheless, it's horribly ugly, difficult to follow and overall unsanitary so it has to go.

My current solution to the above issue with rsa-oaep is to isolate properly the C code and stick to calls in only one direction, namely from Ada to C. To do this however, I first need to write some more ...C code. Eurgh. The new C code is a set of wrappers for RSA and MPI methods that I need to call from Ada (essentially from what will be Ada implementation of the rsa-with-oaep part): those wrappers provide a sane interface that matches SMG Comms' raw types layer - basically those methods can be called with arrays of octets as input/output without forcing onto the rest of the code MPI structures, unpredictable memory allocation and other abominations.

The above approach has two very important benefits: the code is much easier to follow since there is no ugly back and forth between Ada and C; the C part (MPI especially but also the RSA layer) is properly separated from the Ada part so that it should be relatively easy to simply replace it with a sane Ada implementation whenever such a wonder becomes available. The main downside to this is that it adds conversions from arrays of octets to MPIs and back as part and parcel of using the wrappers but I still think it's the better option available at this moment.

The C wrappers that I need for SMG Comms include the comparison of 2 MPI values (so mpi_cmp from MPI lib), the encryption and decryption with given RSA key as well as a procedure to generate a RSA key (from rsa lib). In addition, I wrote an mpi_to_octets function that does as the name suggests: reads the raw octets stored in the MPI and returns them to the caller. The reason for this to feature as a separate function is the sad fact that MPI's own "get buffer" allocates memory and then expects the caller to de-allocate it. Rather than import this headache and propagate it all through the new code, I wrapped it up in here and dealt with it: callers of mpi_to_octets should allocate their own memory and de-allocate it accordingly. The signatures of those wrapper methods are in all their enumerated-arrays-of-octets glory (RSA key components are passed as array of octets just as well as anything else) in c_wrappers/c_wrappers.c:

//Wrapper methods for C implementations of RSA and MPI.
//To be used / called from Ada so that the C part remains well separated and
//  can therefore be swapped easily at a later stage for something sane.
//S.MG, 2018

#include "mpi.h"

//Comparison of 2 arrays of octets interpreted as MPIs.
//This method creates 2 MPIs out of the given arrays of octes and then
//  calls the mpi_cmp method from mpi/mpi-cmp.c, returning its result.
// ************************************************************************
// ***NB: Make SURE that a and b have indeed allocated at least len_a and
// *****    len_b octets respectively! NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param a An array of octets representing the first MPI.
//@param len_a The length of the first array (number of octets).
//@param b An array of octets representing the second MPI.
//@param len_b The length of the second array (number of octets).
//@return 0 when a=b, -1 when ab
int mpi_cmp_octets(char *a, unsigned int len_a, char *b, unsigned int len_b);

//Encryption of given octets with public RSA key: n and e given as octets too.
// ************************************************************************
// ***Length of output is KEY_LENGTH_OCTETS.
// ***NB: Make SURE that out, input, n and e have enough space allocated!!
// ***NB: NO MEMORY ALLOCATED for its parameters by this method and
// *****    NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param out Pointer to ALREADY ALLOCATED space for the encrypted data.
//@param len_out Length of the allocated space for out (in octets).
//@param input Pointer to the data to be encrypted.
//@param len_input Length of the allocated space for input (in octets).
//@param n Pointer to the public RSA modulus to use for encryption.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent to use for encryption.
//@param len_e Length of e (in octets).
//@return The actual length of the output i.e. number of chars written to out.
int public_rsa_octets( char *out  , unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e);

//Encryption of given octets with *private* RSA key given as octets.
// ************************************************************************
// ***Length of output is KEY_LENGTH_OCTETS.
// ***NB: Make SURE that ALL pointers have enough space allocated!!
// ***NB: NO MEMORY ALLOCATED for its parameters by this method and
// *****    NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param out Pointer to ALREADY ALLOCATED space for the encrypted data.
//@param len_out Length of the allocated space for out (in octets).
//@param input Pointer to the data to be encrypted.
//@param len_input Length of the allocated space for input (in octets).
//@param n Pointer to the public RSA modulus of the given key.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent of the given key.
//@param len_e Length of e (in octets).
//@param d Pointer to the private RSA exponent of the given key.
//@param len_d Length of d (in octets).
//@param p Pointer to the prime p of the given key.
//@param len_p Length of p (in octets).
//@param q Pointer to the prime q of the given key.
//@param len_q Length of q (in octets).
//@param u Pointer to the inverse of p mod q for the given key.
//@param len_u Length of u (in octets).
//@return The actual length of the output i.e. number of chars written to out.
int private_rsa_octets( char *out, unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e,
                        char *d    , unsigned int len_d,
                        char *p    , unsigned int len_p,
                        char *q    , unsigned int len_q,
                        char *u    , unsigned int len_u);

//Generates a new RSA key and stores its components at the specified locations.
//@param n Pointer to the public RSA modulus of the given key.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent of the given key.
//@param len_e Length of e (in octets).
//@param d Pointer to the private RSA exponent of the given key.
//@param len_d Length of d (in octets).
//@param p Pointer to the prime p of the given key.
//@param len_p Length of p (in octets).
//@param q Pointer to the prime q of the given key.
//@param len_q Length of q (in octets).
//@param u Pointer to the inverse of p mod q for the given key.
//@param len_u Length of u (in octets).
void gen_rsa_octets( char *n, unsigned int *len_n,
                     char *e, unsigned int *len_e,
                     char *d, unsigned int *len_d,
                     char *p, unsigned int *len_p,
                     char *q, unsigned int *len_q,
                     char *u, unsigned int *len_u);

//Copies the buffer of m to the location to which out points.
//*****************************************************************
//*** This method does NOT allocate memory for out!
//*** NB: caller should allocate enough memory!
//*****************************************************************
//@param out pointer to allocated memory where to copy the MPI's octets
//@param len_out size of out; will be replaced by actual number of octets copied
//@param m The MPI whose octets are to be retrieved
void mpi_to_octets( char *out, unsigned int *len_out, MPI m);

Corresponding code for the wrapper methods:

//Wrapper methods for C implementations of RSA and MPI.
//To be used / called from Ada so that the C part remains well separated.
//S.MG, 2018

#include "c_wrappers.h"
#include "mpi.h"
#include "smg_rsa.h"
#include <assert.h>
#include <string.h> //for memmove...

//Wrapper for comparing the given arrays of octets as MPIs
//Steps:
//1. allocate space for 2 MPIs u and v
//2. set buffer of u to content of a, buffer of v to content of b
//3. call mpi_cmp(u,v) and store result
//4. de-allocate u and v
//5. return result
int mpi_cmp_octets(char *a, unsigned int len_a, char *b, unsigned int len_b) {
  //variable to hold the final result of comparing a to b as MPIs
  int result;

  //calculate how much space is needed for each MPI
	unsigned int nlimbs_a = mpi_nlimb_hint_from_nbytes( len_a );
	unsigned int nlimbs_b = mpi_nlimb_hint_from_nbytes( len_b );

  //allocate space for the 2 MPIs
	MPI u = mpi_alloc(nlimbs_a);
	MPI v = mpi_alloc(nlimbs_b);

  //set the given octets as the values of the 2 MPIs
  //the sign is set to 0 (last parameter).
  mpi_set_buffer(u, a, len_a, 0);
  mpi_set_buffer(v, b, len_b, 0);

  //compare the MPIs as numbers and store the result
  result = mpi_cmp(u, v);

  //tidy up: free the memory allocated for the 2 MPIs
  mpi_free(u);
  mpi_free(v);

  //return the result comparing a to b as MPIs
  return result;
}

//Encryption of given input with a public RSA key: n and e given as octets too.
//Steps:
//1. create (allocate memory for) MPIs for out and input;
//2. set the input as buffer for the corresponding MPI;
//3. create and set the public key structure with n,e as contents;
//4. call rsa/public_rsa and retrieve the result storing it in out;
//5. free allocated memory for all the MPIs.
//6. return the actual length of the encrypted result
int public_rsa_octets( char *out  , unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e) {

  // precondition: output has enough memory allocated
  assert( len_out >= KEY_LENGTH_OCTETS );

  //allocate memory for input and output MPIs
	unsigned int nlimbs_in  = mpi_nlimb_hint_from_nbytes( len_input );
	unsigned int nlimbs_out = mpi_nlimb_hint_from_nbytes( len_out   );
	MPI in_mpi  = mpi_alloc(nlimbs_in);
	MPI out_mpi = mpi_alloc(nlimbs_out);

  //set input as buffer for in_mpi
  mpi_set_buffer(in_mpi, input, len_input, 0);

  //create public key structure and set its contents to given n, e
  RSA_public_key pk;
	unsigned int nlimbs_n = mpi_nlimb_hint_from_nbytes( len_n );
	unsigned int nlimbs_e = mpi_nlimb_hint_from_nbytes( len_e );
  pk.n = mpi_alloc(nlimbs_n);
  pk.e = mpi_alloc(nlimbs_e);
  mpi_set_buffer(pk.n, n, len_n, 0);
  mpi_set_buffer(pk.e, e, len_e, 0);

  //call rsa public_key encryption and retrieve the result, storing it in out
  public_rsa( out_mpi, in_mpi, &pk);
  int len = len_out;
  mpi_to_octets( out, &len, out_mpi );

  //tidy up: free allocated memory for ALL MPIs.
  mpi_free(in_mpi);
  mpi_free(out_mpi);
  mpi_free(pk.n);
  mpi_free(pk.e);

  //return actual length
  return len;
}

//Decryption of given input with the private key given through its components.
//Steps:
//1. create (allocate memory for) MPIs for out and input;
//2. set the input as buffer for the corresponding MPI;
//3. create and set the private key structure with n,e,d,p,q,u as contents;
//4. call rsa/private_rsa and retrieve the result storing it in out;
//5. free allocated memory for all the MPIs.
//6. return the actual length of the result
int private_rsa_octets( char *out, unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e,
                        char *d    , unsigned int len_d,
                        char *p    , unsigned int len_p,
                        char *q    , unsigned int len_q,
                        char *u    , unsigned int len_u) {
  // precondition: output has enough memory allocated
  assert( len_out >= KEY_LENGTH_OCTETS );

  //allocate memory for input and output MPIs
	unsigned int nlimbs_in  = mpi_nlimb_hint_from_nbytes( len_input );
	unsigned int nlimbs_out = mpi_nlimb_hint_from_nbytes( len_out   );
	MPI in_mpi  = mpi_alloc(nlimbs_in);
	MPI out_mpi = mpi_alloc(nlimbs_out);

  //set input as buffer for in_mpi
  mpi_set_buffer(in_mpi, input, len_input, 0);

  //create private key structure and set its contents to given n,e,d,p,q,u
  RSA_secret_key sk;
	unsigned int nlimbs_n = mpi_nlimb_hint_from_nbytes( len_n );
	unsigned int nlimbs_e = mpi_nlimb_hint_from_nbytes( len_e );
	unsigned int nlimbs_d = mpi_nlimb_hint_from_nbytes( len_d );
	unsigned int nlimbs_p = mpi_nlimb_hint_from_nbytes( len_p );
	unsigned int nlimbs_q = mpi_nlimb_hint_from_nbytes( len_q );
	unsigned int nlimbs_u = mpi_nlimb_hint_from_nbytes( len_u );
  sk.n = mpi_alloc(nlimbs_n);
  sk.e = mpi_alloc(nlimbs_e);
  sk.d = mpi_alloc(nlimbs_d);
  sk.p = mpi_alloc(nlimbs_p);
  sk.q = mpi_alloc(nlimbs_q);
  sk.u = mpi_alloc(nlimbs_u);
  mpi_set_buffer(sk.n, n, len_n, 0);
  mpi_set_buffer(sk.e, e, len_e, 0);
  mpi_set_buffer(sk.d, d, len_d, 0);
  mpi_set_buffer(sk.p, p, len_p, 0);
  mpi_set_buffer(sk.q, q, len_q, 0);
  mpi_set_buffer(sk.u, u, len_u, 0);

  //call rsa secret_key encryption and retrieve the result, storing it in out
  secret_rsa( out_mpi, in_mpi, &sk );
  int len = len_out;
  mpi_to_octets( out, &len, out_mpi );

  //tidy up: free memory previously allocated for MPIs
  mpi_free(in_mpi);
  mpi_free(out_mpi);
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  //return number of octets copied in out - real length of result
  return len;
}

//Generates a new RSA key and stores its components at the specified locations.
void gen_rsa_octets( char *n, unsigned int *len_n,
                     char *e, unsigned int *len_e,
                     char *d, unsigned int *len_d,
                     char *p, unsigned int *len_p,
                     char *q, unsigned int *len_q,
                     char *u, unsigned int *len_u) {
  // precondition: all pointers have enough memory allocated
  assert( *len_n >= KEY_LENGTH_OCTETS );
  assert( *len_e >= KEY_LENGTH_OCTETS );
  assert( *len_d >= KEY_LENGTH_OCTETS );
  assert( *len_p >= KEY_LENGTH_OCTETS / 2);
  assert( *len_q >= KEY_LENGTH_OCTETS / 2);
  assert( *len_u >= KEY_LENGTH_OCTETS / 2);

  //the secret key structure that will hold generated key components
  RSA_secret_key sk;
	int nlimbs = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS );
	int nlimbs_pq = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS / 2 );

	sk.n = mpi_alloc(nlimbs);
	sk.e = mpi_alloc(nlimbs);
	sk.d = mpi_alloc(nlimbs);
	sk.p = mpi_alloc(nlimbs_pq);
	sk.q = mpi_alloc(nlimbs_pq);
	sk.u = mpi_alloc(nlimbs_pq);

  //generate the rsa key pair - this may take a while!
  gen_keypair(&sk);

  //copy components to their place
  mpi_to_octets( n, len_n, sk.n );
  mpi_to_octets( e, len_e, sk.e );
  mpi_to_octets( d, len_d, sk.d );
  mpi_to_octets( p, len_p, sk.p );
  mpi_to_octets( q, len_q, sk.q );
  mpi_to_octets( u, len_u, sk.u );

  //tidy up: free ALL MPIs
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);
}

void mpi_to_octets( char *out, unsigned int *len_out, MPI m) {
  //copy the components as raw octets to the given pointers
  int len = 0;
  int sign;
  unsigned char * buffer = mpi_get_buffer( m, &len, &sign );

  //check and don't copy MORE than there is allocated space in out!
  assert( len <= *len_out );
  memmove( out, buffer, len );
  *len_out = len;    //save actual length of the component

  xfree( buffer ); //free the buffer that was allocated by mpi_get_buffer
}

The basic tests for the new C wrappers are the following:

//S.MG, 2018

#include "mpi.h"
#include "smg_rsa.h"

#include "c_wrappers.h"

void test_mpi_cmp() {
  int result;
  int i;
  char a[KEY_LENGTH_OCTETS];
  char b[KEY_LENGTH_OCTETS];

  //initialize mpis
  for (i=0;i b
  a[240] = 241;
  result = mpi_cmp_octets(a, KEY_LENGTH_OCTETS, b, KEY_LENGTH_OCTETS);
  if (result == 1)
    printf("PASS: mpi_cmp_octets on a > b.\n");
  else {
    printf("FAIL: mpi_cmp_octets on a > b ");
    printf("returned %d instead of 1.\n", result);
  }

  //tidy up
}

void test_gen_rsa_octets() {
  RSA_secret_key sk;
  RSA_public_key pk;
	int nlimbs = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS );
	int nlimbs_pq = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS / 2 );

  //allocate memory
	sk.n = mpi_alloc(nlimbs);
	sk.e = mpi_alloc(nlimbs);
	sk.d = mpi_alloc(nlimbs);
	sk.p = mpi_alloc(nlimbs_pq);
	sk.q = mpi_alloc(nlimbs_pq);
	sk.u = mpi_alloc(nlimbs_pq);

	pk.n = mpi_alloc(nlimbs);
	pk.e = mpi_alloc(nlimbs);

  //generate key pair
  int len_n = KEY_LENGTH_OCTETS;
  int len_e = len_n;
  int len_d = len_n;
  int len_p = KEY_LENGTH_OCTETS / 2;
  int len_q = len_p;
  int len_u = len_p;
  char n[KEY_LENGTH_OCTETS];
  char e[KEY_LENGTH_OCTETS];
  char d[KEY_LENGTH_OCTETS];
  char p[KEY_LENGTH_OCTETS / 2];
  char q[KEY_LENGTH_OCTETS / 2];
  char u[KEY_LENGTH_OCTETS / 2];
  gen_rsa_octets(n, &len_n,
                 e, &len_e,
                 d, &len_d,
                 p, &len_p,
                 q, &len_q,
                 u, &len_u);

  //check encryption/decr works
  mpi_set_buffer(sk.n, n, len_n, 0);
  mpi_set_buffer(sk.e, e, len_e, 0);
  mpi_set_buffer(sk.d, d, len_d, 0);
  mpi_set_buffer(sk.p, p, len_p, 0);
  mpi_set_buffer(sk.q, q, len_q, 0);
  mpi_set_buffer(sk.u, u, len_u, 0);

  mpi_set_buffer(pk.n, n, len_n, 0);
  mpi_set_buffer(pk.e, e, len_e, 0);

  MPI encr = mpi_alloc(0);
	MPI plain = mpi_alloc(0);
  MPI out = mpi_alloc(0);
	mpi_fromstr(plain, "0x\
5B6A8A0ACF4F4DB3F82EAC2D20255E4DF3E4B7C799603210766F26EF87C8980E737579\
EC08E6505A51D19654C26D806BAF1B62F9C032E0B13D02AF99F7313BFCFD68DA46836E\
CA529D7360948550F982C6476C054A97FD01635AB44BFBDBE2A90BE06F7984AC8534C3\
28097EF92F6E78CAE0CB97");
  public_rsa(encr, plain, &pk);
  secret_rsa(out, encr, &sk);

  if (mpi_cmp(out, plain) != 0)
    printf("FAIL: test_gen_rsa encr/decr failed.\n");
  else
    printf("PASS: test_gen_rsa encr/decr passed.\n");

  //tidy up
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  mpi_free(pk.n);
  mpi_free(pk.e);

  mpi_free(plain);
  mpi_free(encr);
  mpi_free(out);
}

void test_rsa_octets() {
  int noctets = 512;
	RSA_public_key pk;
	pk.n = mpi_alloc(0);
	pk.e = mpi_alloc(0);

  RSA_secret_key sk;
  sk.n = mpi_alloc(0);
  sk.e = mpi_alloc(0);
  sk.d = mpi_alloc(0);
  sk.p = mpi_alloc(0);
  sk.q = mpi_alloc(0);
  sk.u = mpi_alloc(0);

//key pair previously generated with EuCrypt
  mpi_fromstr(sk.n, "0x\
CD2C025323BEA46FFF2FA8D7A9D39817EA713421F4AE03FA8120641193892A70BFECF5\
83101635A432110D3DDE6339E3CC7ECC0AD91C026FCACE832DD3888A6FCA7BCE56C390\
5A5AC8C7BC921DA675E4B62489B254EB34659D547D71165BC998983A81937BD251AEE1\
2D985EC387D5376F5DCC5EF7EC530FBD6FD2AA7285EE1AF3335EA73163F0954F30402E\
D7B374EE84A97B1849B0674B0DA0A2050BD79B71ABB1559F3A9CFDB8557DED7BC90CF2\
09E8A847E9C226140845B7D03842162E7DA5DD16326CB1F71A248D841FE9076A09911F\
2F4F5E3EA44EA8DE40332BF00406990BCCF61C322A03C456EF3A98B341E0BDBC1088CE\
683E78510E76B72C2BCC1EE9AEDD80FFF18ABFC5923B2F36B581C25114AB2DF9F6C2B1\
9481703FD19E313DCD7ACE15FA11B27D25BCE5388C180A7E21167FB87750599E1ED7C7\
50F4A844E1DC2270C62D19671CF8F4C25B81E366B09FC850AE642136D204A9160AEECE\
575B57378AA439E9DD46DC990288CD54BAA35EEE1C02456CD39458A6F1CBF012DCEDF4\
27CCF3F3F53645658FC49C9C9D7F2856DB571D92B967AB5845514E0054DDB49099F5DD\
04A6F6F5C5CE642276834B932881AEB648D1F25E9223971F56E249EF40CF7D80F22621\
CDD0260E9E7D23746960ADB52CF2987584FB1DE95A69A39E5CB12B76E0F5C1A0529C0C\
065D2E35720810F7C7983180B9A9EA0E00C11B79DC3D");

  mpi_fromstr(sk.e, "0x\
DD4856B4EE3D099A8604AE392D8EFEC094CDF01546A28BE87CB484F999E8E75CDFCD01\
D04D455A6A9254C60BD28C0B03611FC3E751CC27EF768C0B401C4FD2B27C092834A6F2\
49A145C4EDC47A3B3D363EC352462C945334D160AF9AA72202862912493AC6190AA3A6\
149D4D8B9996BA7927D3D0D2AD00D30FD630CF464E6CAF9CF49355B9A70E05DB7AE915\
F9F602772F8D11E5FCDFC7709210F248052615967090CC1F43D410C83724AA5912B2F0\
52E6B39449A89A97C79C92DC8CB8DEEFCF248C1E1D2FC5BFE85165ECA31839CAA9CEB3\
3A92EBDC0EB3BAC0F810938BB173C7DA21DCBB2220D44CBA0FD40A2C868FC93AC5243E\
C137C27B0A76D65634EBB3");

  mpi_fromstr(sk.d, "0x\
7C8A6FA1199D99DCA45E9BDF567CA49D02B237340D7E999150BC4883AE29DEC5158521\
B338F35DC883792356BDDBB3C8B3030A6DD4C6522599A3254E751F9BA1CB1061C5633C\
81BBFACF6FCD64502614102DFED3F3FA284066C342D5E00953B415915331E30812E5FB\
CD6680ADCCDEE40B8376A3A225F2E160EA59C7566804526D73BB660A648A3EF9802313\
B2F841E8458B2AAACE7AACF31083E8F3F630298138393BC88BBD7D4AA4334949651D25\
365B10DBF4A4A08E20A6CC74BFDD37C1C38E2ADC2A283DF06590DF06B46F67F6ACA67F\
AC464C795261659A2F9558802D0BBAA05FD1E1AF2CDC70654723DF7EFAEA148B8CDBEB\
C89EA2320AB9BBB1BC4311475DF3D91446F02EF192368DFEBAC598CCFD4407DEC58FDC\
1A94CCDD6E5FBA9C52164ACEA8AEE633E557BCCEACB7A1AF656C379482D784A120A725\
32F9B2B35173D505F21D5AD4CB9511BC836DC923730B70291B70290A216CA3B21CFF79\
E895C35F4F7AF80E1BD9ED2773BD26919A76E4298D169160593E0335BE2A2A2D2E8516\
948F657E1B1260E18808A9D463C108535FB60B3B28F711C81E5DE24F40214134A53CE5\
9A952C8970A1D771EBEFFA2F4359DCF157995B3F1950DE3C6EC41B7FF837148F55F323\
372AF3F20CE8B8038E750C23D8F5041FA951327859B0E47483F0A47103EF808C72C251\
006FA526245291C8C84C12D2EF63FB2301EA3EEDA42B");

  mpi_fromstr(sk.p, "0x\
E236732452039C14EC1D3B8095BDDCFB7625CE27B1EA5394CF4ED09D3CEECAA4FC0BF6\
2F7CE975E0C8929CE84B0259D773EA038396479BF15DA065BA70E549B248D77B4B23ED\
A267308510DBEE2FD44E35D880EE7CFB81E0646AA8630165BD8988C3A8776D9E704C20\
AA25CA0A3C32F27F592D5FD363B04DD57D8C61FFDCDFCCC59E2913DE0EE47769180340\
E1EA5A803AA2301A010FF553A380F002601F0853FCACDB82D76FE2FACBCD6E5F294439\
0799EA5AE9D7880D4E1D4AE146DC1D4E8495B9DD30E57E883923C5FC26682B7142D35C\
D8A0FC561FE725A6CF419B15341F40FE0C31132CBD81DD8E50697BD1EBFFA16B522E16\
F5B49A03B707218C7DA60B");

  mpi_fromstr(sk.q, "0x\
E830482A3C4F5C3A7E59C10FF8BA760DB1C6D55880B796FFDA4A82E0B60E974E81D04B\
2A4AD417823EBFB4E8EFB13782943562B19B6C4A680E3BA0C8E37B5023470F4F1AC1F8\
A0B10672EF75CD58BCD45E6B14503B8A6A70AFE79F6201AF56E7364A1C742BE1453FD2\
24FDC9D66522EAF4466A084BCB9E46D455A2946E94CBF028770F38D0B741C2CC59308F\
71D8C2B4B9C928E0AE8D68DEB48A3E9EFD84A10301EBD55F8221CA32FC567B306B2A8E\
116350AFB995859FDF4378C5CFD06901494E8CFA5D8FAC564D6531FA8A2E4761F5EFBA\
F78750B6F4662BE9EA4C2FAD67AF73EEB36B41FC15CB678810C19A51DF23555695C4C1\
546F3FACA39CAA7BB8DBD7");

  mpi_fromstr(sk.u, "0x\
846232322775C1CD7D5569DC59E2F3E61A885AE2E9C4A4F8CB3ACBE8C3A5441E5FE348\
A2A8AC9C2998FBF282222BF508AA1ECF66A76AEDD2D9C97028BFD3F6CA0542E38A5312\
603C70B95650CE73F80FDD729988FBDB5595A5BF8A007EA34E54994A697906CE56354C\
E00DF10EB711DEC274A62494E3D350D88736CF67A477FB600AC9F1D6580727585092BF\
5EBC092CC4D6CF75769051033A1197103BE269942F372168A53771746FBA18ED6972D5\
0B935A9B1D6B5B3DD50CD89A27FE93C10924E9103FACF7B4C5724A046C3D3B50CC1C78\
5F5C8E00DBE1D6561F120F5294C170914BC10F978ED4356EED67A9F3A60D70AFE540FC\
5373CBAE3D0A7FD1C87273");

// copy the public key components
  pk.n = mpi_copy( sk.n );
  pk.e = mpi_copy( sk.e );

// some plain text message
	MPI plain = mpi_alloc(0);
	mpi_fromstr(plain, "0x\
5B6A8A0ACF4F4DB3F82EAC2D20255E4DF3E4B7C799603210766F26EF87C8980E737579\
EC08E6505A51D19654C26D806BAF1B62F9C032E0B13D02AF99F7313BFCFD68DA46836E\
CA529D7360948550F982C6476C054A97FD01635AB44BFBDBE2A90BE06F7984AC8534C3\
28097EF92F6E78CAE0CB97");

// expected encrypted MPI (via rsa.c directly)
  MPI encr = mpi_alloc(0);
  public_rsa( encr, plain, &pk);
  MPI decr = mpi_alloc(0);
  secret_rsa( decr, encr, &sk);
  if (mpi_cmp(decr, plain) != 0)
    printf("FAIL: decrypted != plain in test_rsa_octets (MPI call)\n");

//allocate the char arrays for _octets rsa
  int len_n = noctets;
  int len_e = len_n;
  int len_d = len_n;
  int len_p = noctets / 2;
  int len_q = len_p;
  int len_u = len_p;
  char n[noctets];
  char e[noctets];
  char d[noctets];
  char p[noctets / 2];
  char q[noctets / 2];
  char u[noctets / 2];

//copy the key components into char arrays
  mpi_to_octets(n, &len_n, sk.n);
  mpi_to_octets(e, &len_e, sk.e);
  mpi_to_octets(d, &len_d, sk.d);
  mpi_to_octets(p, &len_p, sk.p);
  mpi_to_octets(q, &len_q, sk.q);
  mpi_to_octets(u, &len_u, sk.u);

//call _octets rsa and check results
  int len_encr = noctets;
  int len_decr = noctets;
  int len_plain = noctets;
  char plain_o[noctets];
  char encr_o[noctets];
  char expected_encr_o[noctets];
  char decr_o[noctets];
  char expected_decr_o[noctets];

  mpi_to_octets(plain_o, &len_plain, plain);
  mpi_to_octets(expected_encr_o, &len_encr, encr);
  mpi_to_octets(expected_decr_o, &len_decr, decr);
  len_decr = noctets;

  int len;
  len = public_rsa_octets( encr_o, len_encr, plain_o, len_plain,
                           n, len_n, e, len_e);
  if (len != len_encr)
    printf("FAIL: actual len of encr is %d; expected %d\n", len, len_encr);
  else
    printf("PASS: actual len of encr matches expected: %d\n", len);
  int errors= 0;
  int i;
  for (i=0;i0)
    printf("FAIL: found %d errors in public_rsa_octets output\n", errors);
  else
    printf("PASS: no errors found in public_rsa_octets output\n");

  len_encr = len;
  len = private_rsa_octets( decr_o, len_decr, encr_o, len_encr,
                            n, len_n, e, len_e,
                            d, len_d, p, len_p,
                            q, len_q, u, len_u);
  if (len != len_plain)
    printf("FAIL: actual len of decr is %d; expected %d\n", len, len_plain);
  else
    printf("PASS: actual len of decr matches expected: %d\n", len);

  errors = 0;
  for (i=0;i0)
    printf("FAIL: found %d errors in private_rsa_octets output\n", errors);
  else printf("PASS: no errors found in private_rsa_octets_output\n");

  //tidy up
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  mpi_free(pk.n);
  mpi_free(pk.e);

  mpi_free(plain);
  mpi_free(encr);
  mpi_free(decr);
}

int main(int ac, char **av) {
  if (ac < 2) {
    printf("Usage: %s testID\n", av[0]);
    return -1;
  }

  int id = atoi(av[1]);
  switch (id) {
    case 1:
      test_mpi_cmp();
      break;
    case 2:
      test_gen_rsa_octets();
      break;
    case 3:
      test_rsa_octets();
      break;
    default:
      printf("Current test ids:\n");
      printf("1 test of mpi_cmp_octets\n");
      printf("2 test of gen_rsa_octets (can be very SLOW!)\n");
      printf("3 test of rsa_octets (can take a few minutes)\n");
  }
  return 0;
}

The rest (and the longest part by far) of the .vpatch for this chapter includes the MPI lib from EuCrypt and the RSA lib from the same EuCrypt minus the oaep parts that are not going to be used by SMG Comms. Specifically, the changes to the RSA lib from EuCrypt are the following:

  • The constant KEY_LENGTH_OCTETS is changed to the value used by Eulora: 490. I've updated also all the references to the previously fixed value 4096 throughout the comments.
  • I discarded two methods from rsa.c/h: rsa_oaep_encrypt, rsa_oaep_decrypt, oaep_encrypt_c, oaep_decrypt_c.

In addition to the above, the .vpatch also makes a small change to the smg_comms.gpr file reflecting the fact that SMG Comms does not touch mpi and rsa directly but only through the new C wrappers so that the .gpr file uses "c_wrappers.gpr" and not rsa.gpr + mpi.gpr.

The .vpatch is as usual on my Reference Code Shelf and linked also below for your convenience:

October 18, 2018

SMG Comms Chapter 3: Packing Serpent

Filed under: Coding, SMG_Comms — Diana Coman @ 10:12 a.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

This chapter uses the raw types of the protocol as defined in the previous chapter and adds two methods that are still at layer 0 of the protocol: one method for packing Serpent messages into corresponding Serpent packets and one method for unpacking such Serpent packets to extract their contained Serpent message. It is a single small step forward but the corresponding .vpatch is still quite large as it contains the Serpent code in addition to the packing/unpacking methods. The packing is effectively an encryption with a given Serpent key while the unpacking is the corresponding decryption. Nevertheless, there are a few bits that are specific to this implementation as they reflect the requirements of Eulora's protocol:

  • Packing receives as input a Serpent_Msg and produced as output a Serpent_Pkt. Symmetrically, unpacking receives as input a Serpent_Pkt and produces a Serpent_Msg. Both Serpent_Pkt and Serpent_Msg are arrays of octets but of fixed, pre-defined size: 1472 octets and nothing else.
  • Both packing and unpacking will split their input into blocks of the size that Serpent can handle, encrypt/decrypt them and then glue together the results to produce the output. So there are effectively 921 encrypting / decrypting operations with the same, given Serpent key, for one single pack / unpack call.

I've adapted the Serpent implementation that I previously published as part of EuCrypt, effectively integrating it into SMG Comms and stripping away anything that isn't directly needed by SMG Comms:

  • Serpent is now simply a package like all the others rather than a stand-alone library. While it is true that any changes to the original will have to be manually ported to this one as well, that was always going to be the case anyway. So I don't really see much point in carrying about all the glue and additional files to make a library out of only 2 files. Hence, Serpent in SMG Comms has 2 files and nothing more: serpent.ads and serpent.adb. Short and clear.
  • Since this is production use already, testing parts such as the "Selftest" method don't really have any business in the code itself. I've moved this method where it belongs, namely with the tests for all the code, in the tests directory (test_serpent.ads/.adb)
  • Since Serpent here becomes part of SMG Comms it follows that it should also use the raw types of the protocol - it will anyway be called to use variables of those types rather than anything else. There is no point in forcing back and forth conversions between SMG Comms' "Octets" and Serpent's "Bytes" types that are both arrays of octets anyway. So I've changed the definition of the "Bytes" type in Serpent so that it is here simply a subtype of the "Octets" type. This has the advantage that it allows smooth calls to Serpent from SMG Comms while being a small, easily-reversible change that also maintains otherwise the clarity of Serpent's code as it is. Basically SMG Comms gets to call Serpent without having to do explicit conversions between types that are anyway the same thing and Serpent gets to keep calling arrays of octets Bytes internally as it does in its stand-alone lib version.

In addition to the above, the .vpatch for this chapter also adds tests for the packing/unpacking methods2. I've also made a small change to raw_types.ads so that there is now only one variable for Serpent length: SERPENT_OCTETS. This reflects better the fact that there really is only one length for Serpent and it still allows to keep code clear by having the two array types Serpent_Pkt and Serpent_Msg - they just use the same length. Clarity of code is a tricky choice, what more can I say. Here's the updated code in raw_types:

  -- constants from SMG.COMMS standard specification
    -- size of a serpent-encrypted packet and message, in octets
    -- note that this corresponds to 1472/16 = 92 Serpent blocks
    -- NB: lengths are the same!
  SERPENT_OCTETS : constant Positive := 1472;

    -- size of a RSA-encrypted packet and message in octets and bits
  RSA_PKT_OCTETS     : constant Positive := 1470;
  RSA_MSG_OCTETS     : constant Positive := 234;
  RSA_MSG_BITS       : constant Positive := RSA_MSG_OCTETS * 8; --1872

  -- raw, low-level types
  -- all messages and packets are simply arrays of octets at low level/raw
  type Octets is array( Natural range <> ) of Interfaces.Unsigned_8;

  -- raw representations of basic types (with fixed, well-defined sizes)
  subtype Octets_1 is Octets( 1 .. 1 );
  subtype Octets_2 is Octets( 1 .. 2 );
  subtype Octets_4 is Octets( 1 .. 4 );
  subtype Octets_8 is Octets( 1 .. 8 );

  -- RSA packets and contained raw messages
  subtype RSA_Pkt is Octets( 1 .. RSA_PKT_OCTETS );
  subtype RSA_Msg is Octets( 1 .. RSA_MSG_OCTETS );

  -- Serpent packets and contained raw messages
  -- NB: length is the same but the distinction makes the code clearer
  subtype Serpent_Pkt is Octets( 1 .. SERPENT_OCTETS );
  subtype Serpent_Msg is Octets( 1 .. SERPENT_OCTETS );

And the new code in packing.ads:

  -- Packing/unpacking for Eulora's communication protocol:
  -- Serpent Message to/from Serpent Packet
  -- RSA Message to/from RSA Packet
  -- S.MG, 2018

with Raw_Types;
with Serpent;

package Packing is
  -- no side effects or internal state
  Pragma Pure(Packing);

  -- Packing a Serpent message into Serpent package, using the given key
  function Pack( Msg : in Raw_Types.Serpent_Msg;
                    K   : in Serpent.Key )
                  return Raw_Types.Serpent_Pkt;

  -- Unpacking a Serpent packet into contained message, using the given key
  function Unpack( Pkt : in Raw_Types.Serpent_Pkt;
                    K   : in Serpent.Key)
                  return Raw_Types.Serpent_Msg;

  -- internals of this package, NOT for outside use
private
  -- length of 1 Serpent block
  Block_Len: constant Natural := Serpent.Block'Length;

  -- number of Serpent blocks in one single Serpent message/packet
  S_Blocks : constant Natural := Raw_Types.SERPENT_OCTETS / Block_Len;

end Packing;

The new code in packing.adb:

  -- Packing/unpacking for Eulora's communication protocol:
  -- Serpent Message to/from Serpent Packet
  -- RSA Message to/from RSA Packet
  -- S.MG, 2018

package body Packing is

  -- Packing a Serpent message into Serpent package, using the given key
  function Pack( Msg : in Raw_Types.Serpent_Msg;
                 K   : in Serpent.Key )
               return Raw_Types.Serpent_Pkt is

    -- single Serpent blocks containing plain / encrypted data
    Plain    : Serpent.Block;
    Encr     : Serpent.Block;

    -- Serpent Key Schedule - needed for direct encr/decr calls
    KS       : Serpent.Key_Schedule;

    -- final resulting Serpent package
    Pkt      : Raw_Types.Serpent_Pkt := (others => 0);
  begin
    -- prepare the Serpent key schedule based on given key
    Serpent.Prepare_Key( K, KS );

    -- encrypt message block by block and copy result in packet
    for I in 1 .. S_Blocks loop
      -- get current block to encrypt
      Plain := Msg( Msg'First + (I-1) * Block_Len ..
                    Msg'First +  I    * Block_Len - 1 );
      -- encrypt with Serpent
      Serpent.Encrypt( KS, Plain, Encr );
      -- copy result to output packet
      Pkt( Pkt'First + (I-1) * Block_Len ..
           Pkt'First +  I    * Block_Len - 1 )
         := Encr;
    end loop;

    -- return result
    return Pkt;
  end Pack;

  -- Unpacking a Serpent packet into contained message, using the given key
  function Unpack( Pkt : in Raw_Types.Serpent_Pkt;
                   K   : in Serpent.Key)
                 return Raw_Types.Serpent_Msg is
    -- single Serpent blocks containing plain / encrypted data
    Plain    : Serpent.Block;
    Encr     : Serpent.Block;

    -- Serpent Key Schedule - needed for direct encr/decr calls
    KS       : Serpent.Key_Schedule;

    -- the message extracted from the given packet
    Msg : Raw_Types.Serpent_Msg := (others => 0);
  begin
    -- prepare the Serpent key for use
    Serpent.Prepare_Key( K, KS );

    -- decrypt the Serpent packet block by block
    for I in 1 .. S_Blocks loop
      -- get current block from input and decrypt
      Encr := Pkt( Pkt'First + (I-1) * Block_Len ..
                   Pkt'First +  I    * Block_Len - 1 );
      Serpent.Decrypt( KS, Encr, Plain );

      -- copy result to its correct position in final output
      Msg( Msg'First + (I-1) * Block_Len ..
           Msg'First +  I    * Block_Len - 1 )
         := Plain;
    end loop;

    -- return the result - the message content of given package
    return Msg;
  end Unpack;

end Packing;

The .vpatch and my signature for it are as usual on my Reference Code Shelf as well as linked here for your convenience:


  1. 1472 / 16 = 92 

  2. And needed they were too for they actually caught an error that had survived somehow several re-readings of the code to the point that I was totally surprised when the tests first...failed. Never underestimate your own capacity of introducing idiotic errors in the simplest of things! 

October 16, 2018

SMG Comms Chapter 2: Raw Types

Filed under: Coding, SMG_Comms — Diana Coman @ 11:20 a.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

My previous Chapter 1 with its assorted warts and lamentations sparked a rather productive discussion in the forum that ended up with a significant revision of the protocol specification. The main change has to do with packet sizes and types: ALL packets will be either 1470 octets long and RSA encrypted OR 1472 octets long and Serpent encrypted. Those sizes are chosen to ensure that there is no fragmenting of UDP frames sent on the wire since such fragmenting increases significantly the chance of packet loss. A test with those UDP packet sizes sent back and forth between UK and UY did not uncover any trouble - if anything, the results were actually better than expected. In any case, increased chances of reliable communications aside, the added benefit of those fixed, unique sizes is a clearer protocol alltogether. So after a bit of digesting the new specification, I can finally say that I have some sort of plan for its implementation rather than grasping at it from any thread that I could perceive at all (as it was pretty much the case in Chapter 1). Specifically, my current implementation plan is made of 3 layers that build on one another and give me a structure to rely on:

  1. Layer 0 - Raw types
  2. Layer 1 - Sender/Receiver
  3. Layer 2 - Data structures

1. Layer 0 - Raw types

The raw types for Eulora's communication protocol are essentially two: octet (the basic unit, effectively 8 bits) and arrays of octets (various fixed sizes matching the basic types in the protocol's specification). However, for the arrays of octets, there is a useful distinction to be made between packets (i.e. the received array of octets of fixed length, either 1470 or 1472 octets) and messages (i.e. the content of a packet). Essentially messages are obtained from packets in Eulora's communication protocol through decryption using either Serpent (for 1472 octets long packets) or RSA (for 1470 octets long packet). Symmetrically, packets are obtained by encrypting messages with either Serpent (for 1472 octets long messages) or RSA (for 234 octets long messages). Consequently, this layer 0 offers the definitions of types that cover the full set of possible messages and packets together with the relevant conversion methods between packets and messages as well as between raw octets and int/unsigned/float data types. Note that a "message" here is simply the decrypted content of a packet, nothing more. The interpretation of the octets in a message and/or their use to fill some data structure that the protocol defines happens at a higher layer, not here.

2. Layer 1 - Sender/Receiver

Layer 1 is concerned with sending and receiving packets using the types and conversion methods provided by layer 0 together with the UDP lib for actual network communication. So far this seems to me at the moment a very thin layer really since it's little more than a basic receiver and sender: the receiver simply uses UDP lib to endlessly read all incoming packets and store them in a queue for later processing by consumer(s) external to this layer; the sender may have perhaps an input queue from which it keeps reading packets together with their intended destination and then sends them along using the UDP lib. This layer comes right after the raw types because it needs and uses those raw types but at the moment it's unclear to me whether it belongs as part of a protocol implementation since I can see the case for it being left to the user application of the protocol. At any rate, the protocol specification is not really concerned with this part and this description of the sender/receiver is not a given - it may change as required by each application.

4. Layer 2 - Data Structures

This layer is on top of Layer 0 - it provides the data structures for the different types of messages in Eulora's communication protocol. While any message in Eulora's protocol is just an array of octets at layer 0, the information contained in any message has to be structured according to the protocol's rules described in sections 4, 5 and 7 of the specification. This layer provides the definitions of those data structures as well as the conversion methods from messages to data structures and back.

Note that those 3 layers do not include protocol mechanics since those are really part of the server or client applications. Perhaps a demo / test application implementing the specified protocol mechanics would be useful to include as part of this implementation but I'd rather not specify it as such at this point. For now I'm fine with those 3 layers but if you see any holes in them or problems with them, let me know in the comments below!

Implementation

To start it all off, here's the first part of layer 0 - raw types, including the definitions of types and the most basic conversions via Ada.Unchecked_Conversion:

 -- raw types for the communication protocol
 -- these are used throughout at the lowest level of the protocol
 -- essentially they are the units of packets and of messages
 -- SMG.Comms has only 2 types of packets: RSA and Serpent
 -- a message is the decrypted content of a packet
 -- S.MG, 2018

with Interfaces; use Interfaces; -- Unsigned_n and Integer_n
with Ada.Unchecked_Conversion;

package Raw_Types is

  -- constants from SMG.COMMS standard specification
    -- size of a serpent-encrypted packet and message, in octets
    -- note that this corresponds to 1472/16 = 92 Serpent blocks
    -- NB: lengths are the same but the distinction makes the code clearer
  SERPENT_PKT_OCTETS : constant Positive := 1472;
  SERPENT_MSG_OCTETS : constant Positive := SERPENT_PKT_OCTETS;

    -- size of a RSA-encrypted packet and message in octets and bits
  RSA_PKT_OCTETS     : constant Positive := 1470;
  RSA_MSG_OCTETS     : constant Positive := 234;
  RSA_MSG_BITS       : constant Positive := RSA_MSG_OCTETS * 8; --1872

  -- raw, low-level types
  -- all messages and packets are simply arrays of octets at low level/raw
  type Octets is array( Natural range <> ) of Interfaces.Unsigned_8;

  -- raw representations of basic types (with fixed, well-defined sizes)
  subtype Octets_1 is Octets( 1 .. 1 );
  subtype Octets_2 is Octets( 1 .. 2 );
  subtype Octets_4 is Octets( 1 .. 4 );
  subtype Octets_8 is Octets( 1 .. 8 );

  -- RSA packets and contained raw messages
  subtype RSA_Pkt is Octets( 1 .. RSA_PKT_OCTETS );
  subtype RSA_Msg is Octets( 1 .. RSA_MSG_OCTETS );

  -- Serpent packets and contained raw messages
  -- NB: length is the same but the distinction makes the code clearer
  subtype Serpent_Pkt is Octets( 1 .. SERPENT_PKT_OCTETS );
  subtype Serpent_Msg is Octets( 1 .. SERPENT_MSG_OCTETS );

  -- blind, unchecked casts ( memcpy style )
  function Cast is new Ada.Unchecked_Conversion( Integer_8  , Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1   , Integer_8 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_8 , Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1   , Unsigned_8 );

  function Cast is new Ada.Unchecked_Conversion( Integer_16 , Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2   , Integer_16 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_16, Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2   , Unsigned_16 );

  function Cast is new Ada.Unchecked_Conversion( Integer_32 , Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4   , Integer_32 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_32, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4   , Unsigned_32 );

  -- Gnat's Float has 32 bits but this might be different with other compilers
  function Cast is new Ada.Unchecked_Conversion( Float, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4, Float );

  function Cast is new Ada.Unchecked_Conversion( Integer_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Integer_64 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Unsigned_64 );

end Raw_Types;

I packed the above code as a .vpatch on top of the previous genesis .vpatch mainly for keeping my original promise of showing this being built as it is with detours and corrections on the way. Otherwise this could equally well be a genesis in itself given how radically it changes pretty much everything in there. Nevertheless, at least for now - meaning until a final version is achieved and a regrind makes perhaps sense - I'll keep growing the smg_comms tree. I'll link all smg_comms .vpatches and my signatures for them from the Code Shelf page as usual, while linking the current ones here as well:

The next chapter will add to the raw types layer, providing conversions from packets to messages and back. That will bring in parts of EuCrypt of course, namely the Serpent and RSA implementations. For this reason, I aim to do this perhaps in two steps (first Serpent and only then RSA) in order to keep patches as clear and easy to follow as possible. And this is of course a very good opportunity to re-read the EuCrypt implementation, to re-write it on one level or another as I bring it over to SMG_Comms and to see first-hand how it is to use it!

October 13, 2018

Results of Testing UDP - Take 2

Filed under: Coding, UDP — Diana Coman @ 8:32 p.m.

As the first test revealed that most UDP packets make it safely at least when sent 1 second apart, I decided that the next week-long test should run with a substantially smaller delay: 0.01 seconds between each 2 packets. Moreover, I've changed the sender to simply send 10000 packets at each run, alternating between 1472 and 1470 octets in size. The focus on those 2 sizes reflects the fact that ALL packets are either 14721 or 14702 octets long in the newest version of Eulora's communication protocol. Here's the summary of the data I collected:

UY UK
Sent: 21400003 2140000
Received: 2134569 (99.74%) 2139788 (99.99%)
Errors received: 0 0
1470-size sent: 1070000 1070000
1470-size received: 1067322 (99.75%) 1069901 (99.99%)
1472-size sent: 1070000 1070000
1472-size received: 1067247 (99.74%) 1069887 (99.99%)

From the above, it would seem that even 0.01 delay between any two packets being sent is not causing any problem at all - at least not for the sizes considered4. If anything, the % of received packets increased slightly compared to the previous test. Of the packets that made it to the other side, none were mangled, hence no errors received5. Unsurprising perhaps, the 2 octets difference in size between the 2 types of packets does not seem to have the slightest impact on % of packets that make it to the other side - the route itself is likely to be more important in this respect.

The timings remain iffier to investigate as I did not change the tester specifically for this. Applying the same correction as in the previous test (i.e. calculating first the difference between the logged local times and then simply adjusting so that the minimum difference is 0) yields again the same stats on both sides (if slightly different values than during the last test):

Data Min 1st Q Median Mean 3rd Q Max
UK -> UY 0 7 13 13.09 20 27
UY -> UK 0 7 13 13.19 20 27

As before, I would not suggest taking the above timings for anything really. The only thing that can perhaps be said is that packets don't seem to take all that much to arrive at their destination at least on those routes. This is of course no guarantee at all, just a limited observation of this particular data set.


  1. Serpent encrypted packets that are meant for most communications in Eulora. 

  2. RSA encrypted packets that are meant only for new accounts in Eulora. 

  3. I've ran this for almost 9 days so more than the previous 1 week for the first test. 

  4. Arguably larger sizes that would get fragmented on the way might have higher chances of becoming lost. 

  5. There was however one spam packet received by the node in UY from an entirely unrelated IP but I removed that before making the summary above since it has nothing to do with the test itself. Also, the fact that one will receive spam is not exactly news otherwise so there isn't much to say about it at this moment. 

October 10, 2018

EuCrypt Chapter 14: CRC32 Implementation with Lookup Table

Filed under: Coding, EuCrypt — Diana Coman @ 1:51 p.m.

~ This is part of the EuCrypt series. Start with Introducing EuCrypt. ~

The communication protocol for Eulora uses CRC32 as checksum for its packages and so I found myself looking in disbelief at the fact that GNAT imposes the use of Streams for something as simple as calculating the checksum of anything at all, no matter how small no matter what use, no matter what you might need or want or indeed find useful, no matter what. No matter! As usual, the forum quickly pointed me in the right direction - thank you ave1! - namely looking under the hood of course, in this case GNAT's own hood, the Systems.CRC32 package. Still, this package makes a whole dance of eating one single character at a time since it is written precisely to support the stream monstrosity on top rather than to support the user with what they might indeed need. Happily though, CRC32 is a very simple thing and absolutely easy to lift and package into 52 lines of code in the .adb file + 130 in the .ads file so 182 all in total1, comments and two types of input (string or raw array of octets) included. And since a CRC32 implementation is anyway likely to be useful outside of Eulora's communication protocol too, I'm adding it as a .vpatch on the EuCrypt tree where it seems to fit best at the moment. It's a lib on its own as "CRC32" when compiled via its own crc32.gpr or otherwise part of the aggregate EuCrypt project if you'd rather compile all of EuCrypt via its eucrypt.gpr.

My CRC32 lib uses the 0x04C11DB7 generator polynomial and a lookup table with pre-calculated values for faster execution. As Stanislav points out, implementing the actual division and living with the slow-down incurred is not a huge problem but at least for now I did not bother with it. The CRC32 lib provides as output 32 bits of checksum for either a string or an array of octets. At the moment at least I really don't see the need for anything more complicated than this - even the string-input method is more for the convenience of other/later uses than anything else. For my own current work on Eulora's protocol I expect to use CRC32 on arrays of octets directly.

The .vpatch adding CRC32 to EuCrypt and my signature for it are as usual on my Reference Code Shelf with additional links here for your convenience:


  1. Specifically: 52 lines is the count of crc32.adb that does the work. The .ads file brings in another 130 lines that are mostly the lookup table with pre-calculated values. The .gpr file has another 61 lines and the restrict.adc another 80 lines. 

October 4, 2018

Results of Testing UDP - Take 1

Filed under: Coding, UDP — Diana Coman @ 6:09 p.m.

For one full week, between 26 September 2018 and 3 October 2018, my UDP Tester ran on 2 computers, one in the UK and one in Uruguay (UY), sending and receiving UDP messages in both directions. On each side, the receiver ran continuously, logging all UDP messages that it received during the whole interval. By contrast, the sender ran on both sides hourly but at different times so that the communications did not overlap. I don’t expect it would have been any trouble even if they did overlap but this was meant to be a test of UDP under best conditions and for this reason I set the times so that the sender on one end always finished a full run before the one on the other end started. For the same reason, the messages were sent at a rate of at most 1 message per second1.

At each run, the sender sent exactly 2043 UDP messages with lengths between 6 and 2048, each message having a different length. The order of messages was pseudo-random, relying on the Mersenne Twister prng using as seed the local time at the start of the run (in unix format). The sender kept a log of all messages it sent, including destination IP and port, seed used for MT, local time when message was sent and size of message. The receiver also logged basic information about each message: source IP and port, local time and message size as observed by receiver as well as those contained in the message’s own header, number of observed incorrect bits in the message’s payload as well as the expected and actual values of incorrect octets.

A first look at the week-long data yielded a bit of a surprise in that the UY receiver had actually received *more* messages than were sent from the UK! At a closer look, it turned out that 4933 UDP messages arriving at the UY node were actually sent by its own local switch! And moreover, they were all, without exception, recorded as corrupted since neither size nor payload matched the expected values2. At the moment those switch-generated messages are a bit of a mystery - it’s unclear what they are exactly or why and how they appeared. Working hypotheses would be that they are either local dhcp messages (although the port number would be a weird choice for those) or stray frags of bigger UDP messages. My one single attempt to replicate this behaviour while simultaneously capturing everything with tcpdump has so far failed - there were no such unexpected messages at all over several hours of UK sender at work. I might perhaps try again at a later date after I’m done with the more pressing tests that I need for SMG comms or simply process the existing error log and reconstitute the already observed weird messages from there. Anyway, for now I put those anomalous messages to the side and focus instead on the rest of the messages (which were sent as expected either by the node in the UK or by the one in UY). Here’s a summary of the data thus cleaned:

UK node UY node
Total sent: 3459283 3452674
Total received: 3447175 3451836
% received: 99.84%7 99.78%8
Errors received9: 0 0

Arguably the lost messages are of most interest in all the above: can one say perhaps that the largest messages10 get lost more often? Not really or at least not based on this little set of data. Compare the summary stats for three groups of messages: all messages sent from the UK (reflecting as expected the sizes sent and the fact that the same number of messages of each size are sent), all messages lost at UY (i.e. did not make it on the way from UK to UY) and all messages lost at UK (i.e. did not make it on the way from UY to UK):

Data Min 1st Q Median Mean 3rd Q Max
All sent from UK 6 516 1027 1027 1538 2048
Lost at UY (UK->UY) 13 513 1049 1051 1602 2045
Lost at UK (UY->UK) 16 553.5 1072 1061 1576 2047

While the data set of lost messages is quite small (550 messages lost at UK and 745 at UY), note that this is mainly due to the fact that there are relatively few losses overall: less than 0.4% of messages sent got lost on the way. So it would seem that at least under the conditions and on the routes considered11, UDP is not all that unreliable anyway. In any case, those summaries above seem to me remarkably close to one another - meaning that there isn’t any visible evidence that some sizes would get lost more than others, at least not for the set of sizes considered. Arguably sizes of up to 2048 octets of message are quite fine for communications over UDP - or at any rate, just as fine as smaller sizes.

In terms of order of received messages, the UY node received ALL messages precisely in the order in which they were sent but the UK node reported 66 messages in total that arrived out of order. Although this is a tiny number, it is perhaps reasonable to assume that it might increase in worse conditions (e.g. significantly less than 1 second between sending messages).

The actual timings are a bit iffier to investigate since the precision of UDP Tester turns out to be less than what would be needed for such task. Moreover, there is something weird going on with the way I recorded the time because the difference between the two nodes should be of ~34 seconds (UY node local time = UK node local time + 34) but this doesn't quite square with all the data especially at the UY receiver end12. On the more positive side though, at least the measurement bias there is constant for all the data and it doesn't introduce any weird effects so I can still attempt to infer something considering also that observed behaviour suggests that most UDP messages really make it to the other end within 1 second. Consequently, I calculated the delta on both sides as TR - TS at first and then I added (on UK side) respectively subtracted (on UY side) the quantity needed to make the lowest delta 0. So at the UY receiver, delta = TR - TS - 11 while at the UK receiver, delta = TR - TS + 32. With this correction, the summary stats for the delta on both sides turn out to be remarkably similar:

Data Min 1st Q Median Mean 3rd Q Max
Deltas at UK node: 0 5 11 10.60 16 21
Deltas at UY node: 0 5 11 10.62 16 21

Note that I do *not* recommend taking the above delta values for anything really, as the tester's precision in recording time is just not enough for this.

You are of course warmly invited to run your own tests and to play with this dataset in any way you find fit. So here's the data from both nodes, including the additional 4933 messages that the UY node received from its own switch:

  • udp_test_take1_data.zip (~10MB)
  • SHA512SUM: 963b8a1467630eea35532122ab7c2d25cb8741001808841f7cf02b34abb6ad5300adcb1d667dd902b4278dd2b373dc46427b0b0bbc918ee52f326456535a4114 udp_test_take1_data.zip

Have fun!


  1. Specifically: the sender had a delay of 1 second between any two consecutive messages. 

  2. The UDP tester simply fills the message up to any length with values calculated as Pos mod 256 where Pos is the position of the respective octet in the full message. 

  3. This is precisely 345928/2043=169 runs. 

  4. This includes a partial 170th run since I stopped the whole test while the UY sender was running already its 170th run. 

  5. 550 messages lost in total. 

  6. 745 messages lost in total. 

  7. 344717 / 345267 * 100 

  8. 345183 / 345928 * 100 

  9. This refers to messages received but with payloads that don’t match the expected values. 

  10. Note that this test capped the messages at 2048 so “largest” here means strictly < 2049 octets. 

  11. The UK node is a “consumer” node i.e. behind a router and on a residential connection; the UY node is S.MG’s test server with Pizarro. 

  12. Considering TR as TimeReceived and TS as TimeSent, at UY receiver the delta should be calculated as TR - (TS + 34) = TR - TS - 34; however, there are entries with TR-TS as low as 11 so basically it would seem that messages arrived before they were even sent. 

September 27, 2018

Tester for UDP Communications

Filed under: Coding, UDP — Diana Coman @ 8:33 p.m.

This code builds on Stanislav Datskovskiy's minimal UDP lib to provide a convenient way of gathering data to evaluate UDP communications between any desired two nodes. The initial specification was to provide a reliable way to "send a soup of all packets lengths from 1 to 65536 bytes each hour back and forth" but this was refined further down the line1 to reduce the maximum length of the payload to 2048 and to include a Mersenne-Twister (MT) pseudorandom number generator (prng) for scrambling the various message sizes. To accomplish that, there are two main parts that I'm adding to the original UDP library:

  1. The MT lib that is simply an Ada implementation of the MT prng algorithm.
  2. The UDP_Tester package that uses a slightly adapted UDP lib and the MT lib above to provide a UDP sender and a corresponding UDP receiver that implement the testing specification and log the relevant data.

MT lib
The MT lib is a standalone Ada library implementing the well-known Mersenne Twister algorithm for pseudorandom number generation2. I am not aware of any other Ada implementation of MT and since I really don't want to add C code to my plate unless I absolutely have to, I simply ported to Ada the reference C implementation provided by the original authors of MT. As such, there isn't much to discuss about the implementation itself - I'll point however the changes I made with respect to the original implementation, namely:

  • In my Ada version of MT, there is no default seeding of the MT generator. The C version allows the caller to ask the generator for numbers without having seeded it - in this case, the C implementation seeds itself with a magic default value that is hardcoded at the checking spot and then just proceeds as if nothing was wrong. I find this approach abhorent because it effectively hides an error (calling the generator without seeding it first) rather than complaining about it and forcing its correction. Consequently, my MT lib will simply abort if the generator is asked for numbers without having been seeded first.3
  • Because Ada is most pointedly not a sort of C, there was no need for the various types of hacks added to the C version to ensure that it still worked correctly on 64-bit machines and not only on the 32-bit machines for which the code was initially written. In Ada it's enough to specify one's types correctly to be exactly and guaranteed 32-bits and then proceed to work with them as such, regardless of whether the processor running the code has 32-bit or 64-bit or x-bit registers.
  • My MT lib provides just one type of pseudorandom numbers, namely unsigned integers on 32 bits. The original C implementation had several wrappers around this main function to map the 32 bits pseudorandom numbers to the interval (0,1) for instance and similar. All the mappings were trivial and they can be easily done by any caller that requires them - precisely in the way they require them. At this stage at least I don't quite see the need for those as part of MT lib and so I left them out - there is only the core offering of 32 pseudorandom bits as it were and the caller can then use or interpret them as they see fit.
  • To check that my implementation results in precisely the same sequence of numbers as the original C MT, I added an automated test (still in Ada) that uses the reference seed and checks the output of MT lib (number by number) against the reference output from the original C MT, reporting any mismatches.

The code of MT lib is self-contained with declarations in mt.ads:

 -- Ada implementation of the Mersenne Twister Pseudo-Random number generator
 -- S.MG, 2018

with Interfaces; use Interfaces;

package MT is
  -- Interfaces.Unsigned_32 in GNAT is mod 2**32 and has bitwise shifts defined
  subtype U32 is Interfaces.Unsigned_32;

  -- period parameters
  N          : constant := 624;
  M          : constant := 397;
  MATRIX_MASK: constant U32 := 16#9908_b0df#;
  UPPER_MASK : constant U32 := 16#8000_0000#;
  LOWER_MASK : constant U32 := 16#7fff_ffff#;

  -- array type for storing the state vector of the generator
  type State_Type is Array( 0 .. N-1 ) of U32;

  -- array type for initialization by array - change key len here if needed
  KEY_LEN    : constant := 4;
  type Init_Array_Type is Array( 0 .. KEY_LEN - 1 ) of U32;

  -- exception raised by a call to generator before initializing it
  No_Init_Exception : exception;

  -- initialize the generator with a seed (number)
  procedure Init_Genrand(Seed : in U32);

  -- initialize the generator with array of 8-octets elements
  procedure Init_Genrand(Seed : in Init_Array_Type);

  -- generate the next pseudo-random 32 bits number in the sequence
  function Gen_U32 return U32;

  -- for testing
  function Get_State return State_Type;

  -- internals of the generator, NOT for direct access
private
  -- actual state of the generator
  State      : State_Type;

  -- flag for generator routine
  Mti_Flag : U32 := N + 1;  -- default value -> state(N) is not initialised

end MT;

And implementation in mt.adb:

 -- Ada implementation of the Mersenne Twister Pseudo-Random number generator
 -- S.MG, 2018

package body MT is

  procedure Init_Genrand(Seed : in U32) is
  begin
    State(0) := Seed;
    for I in State'First + 1 .. State'Last loop
      State(I) := U32(1812433253) *
                  ( State(I - 1) xor
                    ( Shift_Right(State(I - 1), 30) )
                  ) + U32(I) ;
    end loop;
    Mti_Flag := N;
  end Init_Genrand;

  procedure Init_Genrand(Seed : in Init_Array_Type) is
    Default_Seed: constant U32 := U32(19650218); -- magic value!
    I, J, K : Integer;
  begin
    Init_Genrand(Default_Seed);
    I := 1;
    J := 0;
    if N > Seed'Length then
      K := N;
    else
      K := Seed'Length;
    end if;

    while K > 0 loop
      State(I) := (State(I) xor
                  ( (State(I-1) xor
                     Shift_Right(State(I-1), 30)
                    ) * U32(1664525)
                  )) + Seed(J) + U32(J);
      I := I + 1;
      J := J + 1;
      if I >= N then
        State(0) := State(N-1);
        I := 1;
      end if;
      if J >= Seed'Length then
        J := 0;
      end if;
      K := K - 1;
    end loop;

    K := N -1;
    while K > 0 loop
      State(I) := (State(I) xor
                  ( (State(I-1) xor
                     Shift_Right(State(I-1), 30)
                    ) * U32(1566083941)
                  )) - U32(I);
      I := I + 1;
      if I >= N then
        State(0) := State(N-1);
        I := 1;
      end if;
      K := K - 1;
    end loop;
    State(0) := 16#8000_0000#; -- MSB is 1 to ensure non-zero initial state
  end Init_Genrand;

  function Gen_U32 return U32 is
    Y     : U32;
    MASK1 : constant U32 := U32(1);
    Mag01 : Array ( 0 .. 1 ) of U32;
  begin
    -- Mag01[x] is x * Matrix_A of the algorithm for x 0 or 1
    Mag01(0) := U32(0);
    Mag01(1) := MATRIX_MASK;

    -- if no numbers available, generate another set of N words
    if Mti_Flag >= N then

      -- check it's not a non-initialised generator
      if Mti_Flag = (N + 1) then
         -- Generator was NOT initialised!
         -- Original C code initialises with default seed 5489
         -- This code will simply raise exception and abort
         raise No_Init_Exception;
      end if;

      for K in 0 .. N - M - 1 loop
        Y := ( State(K)   and UPPER_MASK ) or
             ( State(K+1) and LOWER_MASK );
        State(K) := State(K+M) xor
                      Shift_Right(Y, 1) xor
                        Mag01(Integer(Y and MASK1));
      end loop;
      for K in N-M .. N - 2 loop
        Y := ( State(K)   and UPPER_MASK  ) or
             ( State(K+1) and LOWER_MASK);
        State(K) := State(K + M - N) xor
                      Shift_Right(Y, 1) xor
                        Mag01(Integer(Y and MASK1));
      end loop;
      Y := (State(N-1) and UPPER_MASK ) or
             (State(0) and LOWER_MASK );
      State(N - 1) := State(M-1) xor
                        Shift_Right(Y, 1) xor
                          Mag01(Integer(Y and MASK1));
      Mti_Flag := 0;
    end if;

    -- retrieve next available number
    Y        := State(Integer(Mti_Flag));
    Mti_Flag := Mti_Flag + 1;

    -- tempering
    Y := Y xor Shift_Right(Y, 11);
    Y := Y xor (Shift_Left(Y, 7) and 16#9d2c_5680#);
    Y := Y xor (Shift_Left(Y, 15) and 16#efc6_0000#);
    Y := Y xor Shift_Right(Y, 18);

    -- return tempered number
    return Y;
  end Gen_U32;

  function Get_State return State_Type is
  begin
    return State;
  end Get_State;

end MT;

The test for the above MT lib is in its own testmt/test_mt.adb, including the reference output (that really is the largest part of the whole file):

  --S.MG, 2018

with Ada.Text_IO; use Ada.Text_IO;
with Interfaces; use Interfaces;
with MT;

procedure Tests_MT is
  Seeds : MT.Init_Array_Type;
  X     : MT.U32;
  No    : constant Integer := 1000;
  Result: Array(0..No-1) of MT.U32 := (
    1067595299,  955945823,  477289528, 4107218783, 4228976476,
    3344332714, 3355579695,  227628506,  810200273, 2591290167,
    2560260675, 3242736208,  646746669, 1479517882, 4245472273,
    1143372638, 3863670494, 3221021970, 1773610557, 1138697238,
    1421897700, 1269916527, 2859934041, 1764463362, 3874892047,
    3965319921,   72549643, 2383988930, 2600218693, 3237492380,
    2792901476,  725331109,  605841842,  271258942,  715137098,
    3297999536, 1322965544, 4229579109, 1395091102, 3735697720,
    2101727825, 3730287744, 2950434330, 1661921839, 2895579582,
    2370511479, 1004092106, 2247096681, 2111242379, 3237345263,
    4082424759,  219785033, 2454039889, 3709582971,  835606218,
    2411949883, 2735205030,  756421180, 2175209704, 1873865952,
    2762534237, 4161807854, 3351099340,  181129879, 3269891896,
     776029799, 2218161979, 3001745796, 1866825872, 2133627728,
      34862734, 1191934573, 3102311354, 2916517763, 1012402762,
    2184831317, 4257399449, 2899497138, 3818095062, 3030756734,
    1282161629,  420003642, 2326421477, 2741455717, 1278020671,
    3744179621,  271777016, 2626330018, 2560563991, 3055977700,
    4233527566, 1228397661, 3595579322, 1077915006, 2395931898,
    1851927286, 3013683506, 1999971931, 3006888962, 1049781534,
    1488758959, 3491776230,  104418065, 2448267297, 3075614115,
    3872332600,  891912190, 3936547759, 2269180963, 2633455084,
    1047636807, 2604612377, 2709305729, 1952216715,  207593580,
    2849898034,  670771757, 2210471108,  467711165,  263046873,
    3569667915, 1042291111, 3863517079, 1464270005, 2758321352,
    3790799816, 2301278724, 3106281430,    7974801, 2792461636,
     555991332,  621766759, 1322453093,  853629228,  686962251,
    1455120532,  957753161, 1802033300, 1021534190, 3486047311,
    1902128914, 3701138056, 4176424663, 1795608698,  560858864,
    3737752754, 3141170998, 1553553385, 3367807274,  711546358,
    2475125503,  262969859,  251416325, 2980076994, 1806565895,
     969527843, 3529327173, 2736343040, 2987196734, 1649016367,
    2206175811, 3048174801, 3662503553, 3138851612, 2660143804,
    1663017612, 1816683231,  411916003, 3887461314, 2347044079,
    1015311755, 1203592432, 2170947766, 2569420716,  813872093,
    1105387678, 1431142475,  220570551, 4243632715, 4179591855,
    2607469131, 3090613241,  282341803, 1734241730, 1391822177,
    1001254810,  827927915, 1886687171, 3935097347, 2631788714,
    3905163266,  110554195, 2447955646, 3717202975, 3304793075,
    3739614479, 3059127468,  953919171, 2590123714, 1132511021,
    3795593679, 2788030429,  982155079, 3472349556,  859942552,
    2681007391, 2299624053,  647443547,  233600422,  608168955,
    3689327453, 1849778220, 1608438222, 3968158357, 2692977776,
    2851872572,  246750393, 3582818628, 3329652309, 4036366910,
    1012970930,  950780808, 3959768744, 2538550045,  191422718,
    2658142375, 3276369011, 2927737484, 1234200027, 1920815603,
    3536074689, 1535612501, 2184142071, 3276955054,  428488088,
    2378411984, 4059769550, 3913744741, 2732139246,   64369859,
    3755670074,  842839565, 2819894466, 2414718973, 1010060670,
    1839715346, 2410311136,  152774329, 3485009480, 4102101512,
    2852724304,  879944024, 1785007662, 2748284463, 1354768064,
    3267784736, 2269127717, 3001240761, 3179796763,  895723219,
     865924942, 4291570937,   89355264, 1471026971, 4114180745,
    3201939751, 2867476999, 2460866060, 3603874571, 2238880432,
    3308416168, 2072246611, 2755653839, 3773737248, 1709066580,
    4282731467, 2746170170, 2832568330,  433439009, 3175778732,
      26248366, 2551382801,  183214346, 3893339516, 1928168445,
    1337157619, 3429096554, 3275170900, 1782047316, 4264403756,
    1876594403, 4289659572, 3223834894, 1728705513, 4068244734,
    2867840287, 1147798696,  302879820, 1730407747, 1923824407,
    1180597908, 1569786639,  198796327,  560793173, 2107345620,
    2705990316, 3448772106, 3678374155,  758635715,  884524671,
     486356516, 1774865603, 3881226226, 2635213607, 1181121587,
    1508809820, 3178988241, 1594193633, 1235154121,  326117244,
    2304031425,  937054774, 2687415945, 3192389340, 2003740439,
    1823766188, 2759543402,   10067710, 1533252662, 4132494984,
      82378136,  420615890, 3467563163,  541562091, 3535949864,
    2277319197, 3330822853, 3215654174, 4113831979, 4204996991,
    2162248333, 3255093522, 2219088909, 2978279037,  255818579,
    2859348628, 3097280311, 2569721123, 1861951120, 2907080079,
    2719467166,  998319094, 2521935127, 2404125338,  259456032,
    2086860995, 1839848496, 1893547357, 2527997525, 1489393124,
    2860855349,   76448234, 2264934035,  744914583, 2586791259,
    1385380501,   66529922, 1819103258, 1899300332, 2098173828,
    1793831094,  276463159,  360132945, 4178212058,  595015228,
     177071838, 2800080290, 1573557746, 1548998935,  378454223,
    1460534296, 1116274283, 3112385063, 3709761796,  827999348,
    3580042847, 1913901014,  614021289, 4278528023, 1905177404,
      45407939, 3298183234, 1184848810, 3644926330, 3923635459,
    1627046213, 3677876759,  969772772, 1160524753, 1522441192,
     452369933, 1527502551,  832490847, 1003299676, 1071381111,
    2891255476,  973747308, 4086897108, 1847554542, 3895651598,
    2227820339, 1621250941, 2881344691, 3583565821, 3510404498,
     849362119,  862871471,  797858058, 2867774932, 2821282612,
    3272403146, 3997979905,  209178708, 1805135652,    6783381,
    2823361423,  792580494, 4263749770,  776439581, 3798193823,
    2853444094, 2729507474, 1071873341, 1329010206, 1289336450,
    3327680758, 2011491779,   80157208,  922428856, 1158943220,
    1667230961, 2461022820, 2608845159,  387516115, 3345351910,
    1495629111, 4098154157, 3156649613, 3525698599, 4134908037,
     446713264, 2137537399, 3617403512,  813966752, 1157943946,
    3734692965, 1680301658, 3180398473, 3509854711, 2228114612,
    1008102291,  486805123,  863791847, 3189125290, 1050308116,
    3777341526, 4291726501,  844061465, 1347461791, 2826481581,
     745465012, 2055805750, 4260209475, 2386693097, 2980646741,
     447229436, 2077782664, 1232942813, 4023002732, 1399011509,
    3140569849, 2579909222, 3794857471,  900758066, 2887199683,
    1720257997, 3367494931, 2668921229,  955539029, 3818726432,
    1105704962, 3889207255, 2277369307, 2746484505, 1761846513,
    2413916784, 2685127085, 4240257943, 1166726899, 4215215715,
    3082092067, 3960461946, 1663304043, 2087473241, 4162589986,
    2507310778, 1579665506,  767234210,  970676017,  492207530,
    1441679602, 1314785090, 3262202570, 3417091742, 1561989210,
    3011406780, 1146609202, 3262321040, 1374872171, 1634688712,
    1280458888, 2230023982,  419323804, 3262899800,   39783310,
    1641619040, 1700368658, 2207946628, 2571300939, 2424079766,
     780290914, 2715195096, 3390957695,  163151474, 2309534542,
    1860018424,  555755123,  280320104, 1604831083, 2713022383,
    1728987441, 3639955502,  623065489, 3828630947, 4275479050,
    3516347383, 2343951195, 2430677756,  635534992, 3868699749,
     808442435, 3070644069, 4282166003, 2093181383, 2023555632,
    1568662086, 3422372620, 4134522350, 3016979543, 3259320234,
    2888030729, 3185253876, 4258779643, 1267304371, 1022517473,
     815943045,  929020012, 2995251018, 3371283296, 3608029049,
    2018485115,  122123397, 2810669150, 1411365618, 1238391329,
    1186786476, 3155969091, 2242941310, 1765554882,  279121160,
    4279838515, 1641578514, 3796324015,   13351065,  103516986,
    1609694427,  551411743, 2493771609, 1316337047, 3932650856,
    4189700203,  463397996, 2937735066, 1855616529, 2626847990,
      55091862, 3823351211,  753448970, 4045045500, 1274127772,
    1124182256,   92039808, 2126345552,  425973257,  386287896,
    2589870191, 1987762798, 4084826973, 2172456685, 3366583455,
    3602966653, 2378803535, 2901764433, 3716929006, 3710159000,
    2653449155, 3469742630, 3096444476, 3932564653, 2595257433,
     318974657, 3146202484,  853571438,  144400272, 3768408841,
     782634401, 2161109003,  570039522, 1886241521,   14249488,
    2230804228, 1604941699, 3928713335, 3921942509, 2155806892,
     134366254,  430507376, 1924011722,  276713377,  196481886,
    3614810992, 1610021185, 1785757066,  851346168, 3761148643,
    2918835642, 3364422385, 3012284466, 3735958851, 2643153892,
    3778608231, 1164289832,  205853021, 2876112231, 3503398282,
    3078397001, 3472037921, 1748894853, 2740861475,  316056182,
    1660426908,  168885906,  956005527, 3984354789,  566521563,
    1001109523, 1216710575, 2952284757, 3834433081, 3842608301,
    2467352408, 3974441264, 3256601745, 1409353924, 1329904859,
    2307560293, 3125217879, 3622920184, 3832785684, 3882365951,
    2308537115, 2659155028, 1450441945, 3532257603, 3186324194,
    1225603425, 1124246549,  175808705, 3009142319, 2796710159,
    3651990107,  160762750, 1902254979, 1698648476, 1134980669,
     497144426, 3302689335, 4057485630, 3603530763, 4087252587,
     427812652,  286876201,  823134128, 1627554964, 3745564327,
    2589226092, 4202024494,   62878473, 3275585894, 3987124064,
    2791777159, 1916869511, 2585861905, 1375038919, 1403421920,
      60249114, 3811870450, 3021498009, 2612993202,  528933105,
    2757361321, 3341402964, 2621861700,  273128190, 4015252178,
    3094781002, 1621621288, 2337611177, 1796718448, 1258965619,
    4241913140, 2138560392, 3022190223, 4174180924,  450094611,
    3274724580,  617150026, 2704660665, 1469700689, 1341616587,
     356715071, 1188789960, 2278869135, 1766569160, 2795896635,
      57824704, 2893496380, 1235723989, 1630694347, 3927960522,
     428891364, 1814070806, 2287999787, 4125941184, 3968103889,
    3548724050, 1025597707, 1404281500, 2002212197,   92429143,
    2313943944, 2403086080, 3006180634, 3561981764, 1671860914,
    1768520622, 1803542985,  844848113, 3006139921, 1410888995,
    1157749833, 2125704913, 1789979528, 1799263423,  741157179,
    2405862309,  767040434, 2655241390, 3663420179, 2172009096,
    2511931187, 1680542666,  231857466, 1154981000,  157168255,
    1454112128, 3505872099, 1929775046, 2309422350, 2143329496,
    2960716902,  407610648, 2938108129, 2581749599,  538837155,
    2342628867,  430543915,  740188568, 1937713272, 3315215132,
    2085587024, 4030765687,  766054429, 3517641839,  689721775,
    1294158986, 1753287754, 4202601348, 1974852792,   33459103,
    3568087535, 3144677435, 1686130825, 4134943013, 3005738435,
    3599293386,  426570142,  754104406, 3660892564, 1964545167,
     829466833,  821587464, 1746693036, 1006492428, 1595312919,
    1256599985, 1024482560, 1897312280, 2902903201,  691790057,
    1037515867, 3176831208, 1968401055, 2173506824, 1089055278,
    1748401123, 2941380082,  968412354, 1818753861, 2973200866,
    3875951774, 1119354008, 3988604139, 1647155589, 2232450826,
    3486058011, 3655784043, 3759258462,  847163678, 1082052057,
     989516446, 2871541755, 3196311070, 3929963078,  658187585,
    3664944641, 2175149170, 2203709147, 2756014689, 2456473919,
    3890267390, 1293787864, 2830347984, 3059280931, 4158802520,
    1561677400, 2586570938,  783570352, 1355506163,   31495586,
    3789437343, 3340549429, 2092501630,  896419368,  671715824,
    3530450081, 3603554138, 1055991716, 3442308219, 1499434728,
    3130288473, 3639507000,   17769680, 2259741420,  487032199,
    4227143402, 3693771256, 1880482820, 3924810796,  381462353,
    4017855991, 2452034943, 2736680833, 2209866385, 2128986379,
     437874044,  595759426,  641721026, 1636065708, 3899136933,
     629879088, 3591174506,  351984326, 2638783544, 2348444281,
    2341604660, 2123933692,  143443325, 1525942256,  364660499,
     599149312,  939093251, 1523003209,  106601097,  376589484,
    1346282236, 1297387043,  764598052, 3741218111,  933457002,
    1886424424, 3219631016,  525405256, 3014235619,  323149677,
    2038881721, 4100129043, 2851715101, 2984028078, 1888574695,
    2014194741, 3515193880, 4180573530, 3461824363, 2641995497,
    3179230245, 2902294983, 2217320456, 4040852155, 1784656905,
    3311906931,   87498458, 2752971818, 2635474297, 2831215366,
    3682231106, 2920043893, 3772929704, 2816374944,  309949752,
    2383758854,  154870719,  385111597, 1191604312, 1840700563,
     872191186, 2925548701, 1310412747, 2102066999, 1504727249,
    3574298750, 1191230036, 3330575266, 3180292097, 3539347721,
     681369118, 3305125752, 3648233597,  950049240, 4173257693,
    1760124957,  512151405,  681175196,  580563018, 1169662867,
    4015033554, 2687781101,  699691603, 2673494188, 1137221356,
     123599888,  472658308, 1053598179, 1012713758, 3481064843,
    3759461013, 3981457956, 3830587662, 1877191791, 3650996736,
     988064871, 3515461600, 4089077232, 2225147448, 1249609188,
    2643151863, 3896204135, 2416995901, 1397735321, 3460025646);

begin
  Seeds(0) := MT.U32(16#123#);
  Seeds(1) := MT.U32(16#234#);
  Seeds(2) := MT.U32(16#345#);
  Seeds(3) := MT.U32(16#456#);
  MT.Init_Genrand(Seeds);

  Put_Line("Generating numbers...");
  for I in 0 .. No-1 loop
    X := MT.Gen_U32;
    Put(".");
    if X /= Result(I) then
      Put_Line("");
      Put_Line("ERROR at position " & Integer'Image(I) &
               "; expected " & MT.U32'Image(Result(I)) &
               " but result is " & MT.U32'Image(X));
    end if;
  end loop;
  Put_Line("");
end Tests_MT;

UDP_Tester
The UDP_Tester has two main components: the sender and the receiver. For convenience, there are simple wrappers for those components - udp_sender.adb and udp_receiver.adb respectively - so that a build of UDP_Tester will directly provide two executables - sender and receiver. The IP and port numbers are knobs that can be easily changed from those wrappers directly since they are passed as parameters to the UDP_tester methods themselves.
udp_sender.adb:

  -- S.MG, 2018

with UDP_Tester;

procedure UDP_Sender is
  -- Sender and Receiver addresses + ports
  Receiver_IP   : constant String   := "161.0.121.202"; -- smg.test machine
  Receiver_Port : constant          := 7000;

  Sender_Port   : constant          := 5000;

begin

  UDP_Tester.Sender( Receiver_IP, Receiver_Port, Sender_Port) ;

end UDP_Sender;

udp_receiver.adb:

  -- S.MG, 2018

with UDP_Tester;

procedure UDP_Receiver is
  -- Port to listen on
  Receiver_Port : constant          := 7000;
begin

  UDP_Tester.Receiver( Receiver_Port );

end UDP_Receiver;

The UDP_Tester.Receiver method simply runs an endless loop in which it listens for UDP packages on a given port number and logs basic information about each pacakge it receives. The UDP_Tester.Sender method assembles and sends in a pseudorandom order all packages with length between 6 (size of own header) and maximum length. Note that the sender will send each size of package *only once* and it will simply finish once it sent one package of each size. Consequently, for data collection purposes, the sender will have to be started repeatedly from outside - most conveniently through a cron task4. Because the workings of both sender and receiver are quite basic, I won't cover them here in more detail - the code and comments should make it clear enough and if that's not the case, you can always ask your questions here in the comments. However, it's worth noting instead a few important points about the tester as a whole and the changes it imposes on the UDP lib:

  • To enable sending of packets with various sizes, the UDP lib had to be made generic. In Ada, this means that a calling packet can then instantiate the UDP type with its desired parameters (in this case the size of the UDP package). Specifically, from the UDP_Tester.Sender code, this reads:
          declare
            K : constant Positive := Packet_Size;
            -- sender will have *current* size of UDP packet
            package UDP_Sdr is new UDP( Payload_Size => K );
    

    Essentially the sender runs a loop and at each iteration it selects one of the remaining packet sizes, it instantiates UDP_Sdr as a parametrized UDP package and only then proceeds to use it to send the actual data.

    Note that this change is *not* required in production - it's simply needed for testing purposes. It's for this reason that it's part of the .vpatch for UDP_Tester rather than a .vpatch in itself on the UDP lib directly.

  • The sender will wait 1 second between any 2 packages it sends. This is mainly because a first version without this delay proved to result in significant packet loss that was clearly due to the burst mode (i.e. more than 2000 packages sent in 1 second) rather than anything else. Since the point of the test was to gather some data on UDP communication in relaxed rather than stress conditions, the delay of 1 second was introduced. However, should you wish to stress-test for any reason, you can easily remove this delay - simply comment out the "delay 1.0" line in the sender and that's all.
  • The UDP_Tester currently logs some information on both sides. Both sender and receiver will first check for a fixed-name file and if it doesn't exist they will create it, writing the header to it as well. After that, all new data will simply be appended to the file. The sender logs the size, local time, destination IP, destination port and seed used for the MT prng - this seed is the local time (unix style) when the sender is started. The receiver logs the size (both as received and as claimed in the packet's own header), the time (local time when sent and local time when received), the source IP, the source port and the number of octets that are different from expected. In addition, the receiver will further log in a separate file the actual octets that are found to contain errors, if any (together with information to easily link them back to the exact package they were part of).
  • There is no attempt to match directlty the times at the sender and at the receiver - any difference between the two local times should be noted when the tests are run and then provided together with the data so that the analysis can be done in a meaningful way. Specifically, the local times used by sender and receiver are simply retrieved with Ada.Calendar.Clock and then converted to unix style by subtracting from them the "epoch" (0 on 1/1/1970).
  • The use of generic packages and of Ada.Calendar in UDP_Tester imposed a relaxation of restrictions upstream i.e. in the UDP and MT libs. Since the relaxation of those restrictions is again (just like the generics change) required for testing *only*, they are simply commented out as part of the changes included in the UDP_Tester .vpatch.

To obtain a working UDP_Tester, simply download the relevant set of .vpatch and .sig files and press with your favourite V5. The patches are available as usual on my Reference Shelf and linked from here as well, for your convenience:

At the moment I am running the above tester in both directions between a node in the UK and one in Uruguay so hopefully I'll soon have some data to look at!


  1. Follow the discussion in the logs as there is no better way to get the details than to actually read through the log. 

  2. M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-dimensionally equidistributed uniform pseudorandom number generator", ACM Transactions on Modelling and Computer Simulation Vol. 8, No. 1, January, pp.3-30, 1998. 

  3. If one really wants a default value, this could be perhaps added as a constant in the package and then used as such by the caller but I must say that I don't quite see the point of this. 

  4. It is of course possible to change the code so that the sender also runs in a loop and simply wakes up at given intervals and resends packages but I don't quite see any real benefit to doing it this way - on the contrary, I see benefit to *not* doing it this way, since cron tasks are quite perfect for this and even more robust (no idle time and guaranteed recovery even in more extreme cases such as computer reset). 

  5. NB: Using Keccak hashes, as per current republican standard for V-trees. 

September 14, 2018

SMG.Comms Implementation - Chapter 1

Filed under: Coding, SMG_Comms — Diana Coman @ 11:02 a.m.

~ This is a work in progress towards an Ada implementation of the communication protocol discussed on Trilema four months ago. ~

As far as I know, there isn't any more recent discussion of the above specification, nor any other attempt at all at any sort of implementation. Consequently, this is the first ever attempt - a prototype at this stage rather than a reference implementation. Moreover, it's also a sort of double first, since it clearly requires a deeper knowledge of Ada than I ever needed before. My approach to this pile of unknown here is to simply start working on it and expose the path travelled from this very first attempt to the final product, mistakes and detours and pitfalls to come included. You are welcome to follow along, to help if you can, to question if you don't understand, to extract perhaps in doing so some benefit for yourself.

The first decision I had to make before I could even really attempt any sort of prototype implementation at all concerned the library to use for the actual network communication. Since GNAT is the de-facto republican Ada compiler1, the logical decision is to simply use GNAT's own library, GNAT.Sockets and avoid otherwise as much as possible introducing additional, external dependencies - I really can't see any reason to add even more code even indirectly. So then GNAT.Sockets it is and hooray for that - except that there doesn't seem to be much documentation about it other than the comments in g-socket.ads and g-socket.adb! Still, the .ads file has a rather detailed introduction of the package with some commented examples in there so definitely worth reading as a starting point especially since... there isn't any other starting point really.

The g-socket files reveal essentially that any data to be sent or received through GNAT.Sockets will have to be stored at one point or another in an entity of Ada.Streams.Stream_Element_Array type. The examples in the file (and most everything else I could find on the topic) really focus almost exclusively2 on TCP connections - I suspect one "should prefer" TCP over UDP simply like that and we are even past the point of discussing therefore UDP in any scant docs or something. Nevertheless, I don't prefer it: Eulora's protocol is rather specifically designed to be stateless and to keep communications as simple and clear as possible - for as long as UDP is enough for the job, I'd rather use it!

Rummaging a bit in the g-sockets files reveals happily that UDP with GNAT.Sockets is in fact quite straightforward: one simply needs to specify "Socket_Datagram" as mode for the socket when calling Create_Socket and then use directly the Send_Socket( socket, data, last, to_address) and Receive_Socket(socket, data, last, from_address) methods. As one would expect, there is no reliable connection established, no "stream" of send and receive but only sockets that allow independent "send" and "receive" calls - each of those could in principle use a different address even. A quick example - keep reading for a a full, working example - of a server and client working with a UDP socket looks like this:

 -- on server:
     -- create UDP socket
    Create_Socket( Sock, Family_Inet, Socket_Datagram );

    -- set options on UDP socket
    Set_Socket_Option( Sock, Socket_Level, (Reuse_Address, True));

    -- set address and bind
    Address.Addr := Any_Inet_Addr;
    Address.Port := Port_No; -- the port number that server is listening on
    Bind_Socket( Sock, Address );
    Receive_Socket( Sock, Data, Last, From );

 -- on client:
    Address.Port := Port_No; -- server's port number
    Address.Addr := Inet_Addr("127.0.0.1"); -- this should be server's address here!
    Create_Socket(Sock, Family_Inet, Socket_Datagram);

    Send_Socket(Sock, Data, Last, Address);

With the very basics of the sockets part at least in place, the next part to decide on is how to send and receive over those sockets the actual data types defined by the protocol. After reading quite a bit on the Systems Programming (an annex of the Ada standard and therefore compiler-dependent) part of Ada (that is relevant for data representation) and on streams and on representation clauses and pragmas supported by GNAT and records and discriminants for records and everything else from the Ada reference book3 that I thought could help with this, I got to the conclusion that I'll keep it simple and clear especially for now, while I'm just starting to figure it all out! So I'll simply use GNAT's fixed width types defined in the Interfaces package and I'll copy otherwise the raw octets from and to those types using Ada.Unchecked_Conversion that works as far as I understand it precisely as a raw copy from memory. Once the data is simply obtained as a vector of raw octets, I can implement two simple functions to copy them into the Stream_Element_Array structure that can be sent directly through the UDP socket. Moreover, at this stage - and at this stage *only* - I'll worry about potentially different endianness of the local machine and the network: if local environment is little endian, the methods converting to and from network format will simply read the octets in reverse order. The relevant basic types and conversion methods are defined in the smg_comms_types.ads file:

 -- S.MG, 2018
 -- prototype implementation of S.MG communication protocol

with Ada.Streams; use Ada.Streams;
with Interfaces; use Interfaces; -- Integer_n and Unsigned_n
with Ada.Unchecked_Conversion; -- converting int/uint to array of octets

package SMG_comms_types is
  -- basic types with guaranteed lengths
  type Octet_Array is array(Natural range <>) of Unsigned_8;

  subtype Octets_1 is Octet_Array( 1 .. 1 );
  subtype Octets_2 is Octet_Array( 1 .. 2 );
  subtype Octets_4 is Octet_Array( 1 .. 4 );
  subtype Octets_8 is Octet_Array( 1 .. 8 );

  subtype Message is Octet_Array( 1 .. 512 );
  subtype RSAMessage is Octet_Array( 1 .. 245 );

  -- blind, unchecked casts ( memcpy style )
  function Cast is new Ada.Unchecked_Conversion( Integer_8, Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1, Integer_8 );
  function Cast is new Ada.Unchecked_Conversion( Integer_16, Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2, Integer_16 );

  function Cast is new Ada.Unchecked_Conversion( Integer_32, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4, Integer_32 );

  function Cast is new Ada.Unchecked_Conversion( Integer_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Integer_64 );

  -- to and from streams for network communications - general
  procedure ToNetworkFormat(
      Item   : in Octet_Array;
      Buffer : out Stream_Element_Array);

  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Octet_Array);

  -- specific, convenience methods for the basic types
    -- Integer_8
  procedure ToNetworkFormat(
      Item   : in Integer_8;
      Buffer : out Stream_Element_Array);

  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Integer_8);

end SMG_comms_types;

As you can easily notice, the above does not yet cover fully even the 3.0 "Basic types" part of the protocol specification. It's all right for now though - there is quite enough there for the first basic tests and once those are fine, I'll add gradually the rest of types too. There is little point in spending the time now to implement them all before I even got the chance to change my mind regarding *how* to implement them! So if you have a better implementation solution than the above, speak up in the comments section below and save me some time and a lot of headache! Note however that simplicity, clarity and hard guarantees are rather important here.

One small point on which I'm already rather undecided is whether to continue implementing the convenience methods for different types so that one can simply call ToNetworkFormat and FromNetworkFormat for anything or to leave only the generic methods at least for basic types. At the moment I incline towards providing all those methods (i.e. adding to the single pairof methods for the Integer_8 type defined above) because more complex types will likely need such methods anyway and moreover, this approach helps to keep all those casts (unchecked_conversion) in one place rather than scattered all through the rest of the code. However, it does add to the LOC count of this package. Anyway, moving further for now, the corresponding .adb file with the implementation of smg_comms_types:

  -- S.MG, 2018
  -- prototype implementation of S.MG communication protocol

with SMG_comms_types; use SMG_comms_types;
with System; use System; -- endianness
with Ada.Exceptions;
with Ada.Streams; use Ada.Streams;

package body SMG_comms_types is

  -- to and from network format (i.e. big endian, stream_element_array)
  procedure ToNetworkFormat(
      Item   : in Octet_Array;
      Buffer : out Stream_Element_Array) is
  begin
    if Item'Length /= Buffer'Length then
      raise Constraint_Error with "Item and Buffer lengths do NOT match!";
    end if;

    if Default_Bit_Order = Low_Order_First then
      for I in 0 .. Item'Length - 1 loop
        Buffer( Buffer'Last - Stream_Element_Offset(I) ) := Stream_Element(Item(Item'First + I));
      end loop;
    else
      for I in 0 .. Item'Length - 1 loop
        Buffer( Buffer'First + Stream_Element_Offset(I) ) := Stream_Element(Item(Item'First + I));
      end loop;
    end if;
  end ToNetworkFormat;

  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Octet_Array) is
  begin
    if Item'Length /= Buffer'Length then
      raise Constraint_Error with "Buffer and Item length do NOT match!";
    end if;

    if Default_Bit_Order = Low_Order_First then
      for I in 0 .. Buffer'Length - 1 loop
        Item( Item'Last - I ) :=
          Unsigned_8( Buffer( Buffer'First + Stream_Element_Offset( I ) ) );
      end loop;
    else
      for I in 0 .. Buffer'Length - 1 loop
        Item( Item'First + I ) :=
          Unsigned_8( Buffer( Buffer'First + Stream_Element_Offset( I ) ) );
      end loop;
    end if;
  end FromNetworkFormat;

  -- Integer_8
  procedure ToNetworkFormat(
      Item   : in Integer_8;
      Buffer : out Stream_Element_Array) is
  begin
    ToNetworkFormat( Cast( Item ), Buffer );
  end ToNetworkFormat;
  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Integer_8) is
    octets: Octets_1;
  begin
    FromNetworkFormat(Buffer, octets);
    Item := Cast( octets );
  end FromNetworkFormat;

end SMG_comms_types;

Something that irks me every time I look at the above: those for loops that convert octet by octet to Stream_Element. This is not only ugly but also rather inefficient, especially given that it's potentially the *second* time when those octets are read one by one (the first time being at the conversion from a protocol type - especially one of the more complex types - to an array of octets). However, I have no idea how to do that conversion from array of octets to array of Stream_Element in one single move! Do you know a better way to do this? A direct assign fails because the cast from one type to another would be on the array type rather than individual elements type. And I'm rather reluctant to work directly with Stream_Element as the basic type because this type is implementation dependent and outside my direct control - so I can't actually really know *what* it is.

The above being said regarding Stream_Element, it is important to note that the code above (and almost all the code for a network communication protocol) is anyway, strictly speaking, rather tightly linked to GNAT since it relies on Stream_Element being exactly 8 bits long. And to make this clear, here is the full trail I followed to make sure that I can indeed simply convert an octet (i.e. 8 bits) to a Stream_Element and the other way around while preserving exactly the number of bits specified in the protocol for each type: first, GNAT implements Stream_Element in a-stream.ads as follows:

type Stream_Element is mod 2 ** Standard'Storage_Unit;

Then, the definition of Standard'Storage_Unit for GNAT, which reads that Standard'Storage_Unit always has, according to the GNAT reference manual, the same value as System.Storage_Unit. In turn, System.Storage_Unit is indeed declared in GNAT's system.ads as a constant with value 8, so a Stream_Element in GNAT will indeed have exactly 8 bits:

Storage_Unit : constant := 8;

Moving on, the next step is to write the client-server part even if only a test version for now. Initially I took the easy way out here and simply wrote the separate server and client, each listening/sending one single packet. Running those on different machines worked perfectly fine. This is both fine and needed for a reasonable basic test of the whole thing, sure. However, at the moment as I'm just starting on this I'd rather *not* faff about with 2 machines each and every time I change or add something to this prototype protocol implementation. Moreover, this server/client part is perfect to experiment as well with threads4 in Ada - a topic that I'm still struggling to learn so perfect to practice! Therefore, in the basic test, server and client are implemented as different tasks (Ada's "threads") of the same main program - obviously, it follows that client and server will send data to one another on the same, local machine. This is of course not ideal nor sufficient as test in the long term but it'll do nicely for now and until the basic layer of the protocol at least is more fleshed out. The code in test_comms.adb:

 -- S.MG, 2018
 -- prototype implementation of S.MG communication protocol

with GNAT.Sockets; use GNAT.Sockets;
with Ada.Text_IO; use Ada.Text_IO;
with Ada.Streams; use Ada.Streams;
with Interfaces; use Interfaces;

with SMG_comms_types; use SMG_comms_types;

procedure test_comms is
  Port_No : constant := 2222;

  task type Client is
    entry Send;
  end Client;

  task type Server is
    entry Listen;
    entry Ready;
  end Server;

  task body Client is
    Sock: Socket_Type;
    Address: Sock_Addr_Type;
    Data: Ada.Streams.Stream_Element_Array(1..10) := (others => 42);
    Last: Ada.Streams.Stream_Element_Offset;
    N   : Integer_8 := -36;
  begin
    accept Send; -- task WILL block here until asked to send
    Address.Port := Port_No;
    Address.Addr := Inet_Addr("127.0.0.1");
    Create_Socket(Sock, Family_Inet, Socket_Datagram);

    ToNetworkFormat( N, Data(1..1));
    Send_Socket(Sock, Data, Last, Address);
    Put_Line("Client sent data " & "last: " & Last'Img);
  end Client;

  task body Server is
    Sock: Socket_Type;
    Address, From: Sock_Addr_Type;
    Data: Ada.Streams.Stream_Element_Array(1..512);
    Last: Ada.Streams.Stream_Element_Offset;
    N : Integer_8;
  begin
    accept Listen; -- wait to be started!
    Put_Line("Server started!");
    -- create UDP socket
    Create_Socket( Sock, Family_Inet, Socket_Datagram );

    -- set options on UDP socket
    Set_Socket_Option( Sock, Socket_Level, (Reuse_Address, True));
    Set_Socket_Option( Sock, Socket_Level, (Receive_Timeout, Timeout => 10.0));

    -- set address and bind
    Address.Addr := Any_Inet_Addr;
    Address.Port := Port_No;
    Bind_Socket( Sock, Address );

    accept Ready; -- server IS ready, when here
    -- receive on socket
    begin
      Receive_Socket( Sock, Data, Last, From );
      Put_Line("last: " & Last'Img);
      Put_Line("from: " & Image(From.Addr));
      Put_Line("data is:");
      for I in Data'First .. Last loop
        FromNetworkFormat(Data(I..I), N);
        Put_Line(N'Image);
      end loop;
    exception
      when Socket_Error =>
      Put_Line("Socket error! (timeout?)");
    end;  -- end of receive

  end Server;

  S: Server;
  C: Client;
begin
  S.Listen;
  S.Ready; -- WAIT for server to be ready!
  C.Send;  -- client is started only after server!
end test_comms;

Tasks in Ada can be defined as types as above. This is not mandatory - one can equally well simply define the tasks and they'll run in parallel with the main begin-end block. However, Task types are effectively needed if one wants to be able to explicitly and potentially dynamically create several workers of the same type. Since I'll need this later for sure, I might as well practice it at any occasion, so there they are, the Client Task type and Server Task type. To make sure that the server is ready *before* the client sends anything, I'm using the rendez-vous communication method that Ada provides: each "entry" of a task is basically a rendez-vous point between the task itself that "accepts" that entry at some specified point in its body and a caller task that "calls" that entry from outside. The main body of test_comms illustrates this: S and C start in parallel with the main body of test_comms but they both stop almost immediately as they wait on their first rendez-vous points (accept Listen for the Server type and accept Send for the Client type). The main body first calls S.Listen, effectively releasing the S (Server type) task from its wait. Immediately after that, the S.Ready entry is called so that now the main body is actually waiting for S to get to its "Ready" entry - basically to finish setting up the socket. Once S got to its Ready entry, the main body can go further and the next statement calls C.Send that releases the client for its wait. The client will send therefore its data and promptly finish, while the server (that was running in parallel all this time) will finally receive the data, process it and finish as well. Go ahead, give it ago and let me know if there's anything funny going on!

To compile all the above with one neat gprbuild call, there is of course a .gpr file:

-- S.MG, 2018
 -- prototype implementation of S.MG communication protocol
 -- http://trilema.com/2018/euloras-communication-protocol-restated/

project SMG_comms is
  for Languages use ("Ada");

  for Source_Dirs use ("src");
  for Ignore_Source_Sub_Dirs use (".svn", ".git", "@*");

  for Object_Dir use "obj";
  for Exec_Dir use ".";

  for Main use ("test_comms.adb");

  package Builder is
    for Executable ("test_comms.adb") use "test_comms";
  end Builder;

  package Compiler is
    for Default_Switches ("Ada") use ("-O2");
  end Compiler;

end SMG_comms;

An example of running the test_comms executable produced by gprbuild:

smg_comms$ ./test_comms
Server started!
Client sent data last:  10
last:  10
from: 127.0.0.1
data is:
-36
 42
 42
 42
 42
 42
 42
 42
 42
 42
smg_comms$

On a slightly different note, I had a bit of an internal debate on whether it's appropriate or not to still release this as a V tree given that it's work in progress and absolutely nowhere near a reference object of any sort or even a fully working item. I came to view V however as simply a versioning system rather than a "set in stone only the end products" sort of thing - for that matter what end products anyway, it's at most, in the happiest of situations, seed products rather than end products. So all the above code is the genesis of smg_comms and it's simply a tiny, far from perfect seed that will hopefully grow into something useful while preserving in its v-tree the whole history of that growth:


  1. Due to a large extent to ave1's work on forcing it into some useful shape so that it builds on a sane, musl-based system. 

  2. The Socket_Datagram mode to be set for UDP communications is at least mentioned in the examples although the rest of differences in effectively comunicating something are left to be discovered. 

  3. My Ada reference book is Barnes, John, "Programming in Ada 2012", Cambridge University Press, 2016, ISBN 978-1-107-42481-4. 

  4. On-demand, resilient threads are crucial for Eulora's server and I'm not yet confident at all that I really fully grasp Ada's mechanisms so I'm pushing it to the front, reading on it and otherwise banging my head on it at any and all occasions - how else to figure it out faster? 

« Newer PostsOlder Posts »

Theme and content by Diana Coman