Ossasepia

November 10, 2018

SMG Comms Chapter 7: Read/Write Serpent Keysets to/from Serpent Messages

Filed under: Coding, SMG_Comms — Diana Coman @ 4:06 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

Over this past week I've been going deeper into Ada looking again and again -among other things- at record representation clauses and unions and serialization1 and basically anything and everything that could perhaps provide an elegant solution to serializing data structures to arrays of octets and back. And by "looking" I don't mean just reading but rather the usual combination of reading, writing and coding or at least attempting to code all sorts of ideas that seemed promising. What I have as a result is a much better grasp of Ada's neat representation clauses and parametrized records. What I don't have as a result is the very elegant and short solution I was hoping to find. Instead, I have a first pass at a solution together with the whole chain of reasoning that led me to it. So I'm publishing this as it is - a small step - and I'll gladly listen to better suggestions that are really fit for the job at hand.

The issue is in principle simple enough: write game data to the raw arrays of octets that are SMG messages ensuring the exact format specified by the communication protocol; conversely, read from those same raw arrays of octets that are SMG messages the actual game data. Stated like this, the solution suggesting itself is to define records matching exactly the format specified by the protocol, use representation clauses to control the exact place of all components and then simply convert the whole thing in one go. There are however a few things that I really don't like about this approach of using records matching exactly the full message structure as defined in Eulora's communication protocol:

  • The actual game data is of course only a part of each message. The rest includes padding or communication-level information that really has no business leaking in any way outside of the communication part itself. So I don't quite see the point in defining records for the full message since the read/write (that are supposedly called from outside the communication level itself) should NOT require/produce such records! Instead, the output should be *game data* for read and *array of octets* for write. As a result, I really don't see any benefit in defining message-matching records just for read/write to use *internally* - a sort of filling those component by component and doing the conversion as a whole instead of just doing the conversion component by component directly.
  • The internal structure of most messages is effectively parametrized: there is usually at least one component that represents the number of other following components. Ada handles this sort of thing very gracefully through parametrized records but having parametrized records inside a record doesn't seem to me like a very elegant solution nor one leading to a very clear representation of the whole. I admit I am still not 100% sure of the actual, exact representation of such a record containing itself parametrized records since my understanding is that Ada will allocated maximum space (i.e. space to fit potentially the largest structure) but then simply make available only as much as needed according to the actual value of the parameter.

Considering the above issues, my current solution aims to keep the communication-level parts of messages as an internal concern of the communication level and provide/require at read/write only the actual game data that is needed for each type of message. The conversion to/from octets is then done indeed component by component, as they are written/read and endianness is handled simply in one single place, at conversion: if native endianness is Big Endian then octets are flipped for any component that is represented on more than one octet; otherwise, on Little Endian iron, octets are kept as they are found (i.e. the values in messages are in Little Endian representation, yes).

To illustrate the approach above in practice, I've implemented already the read/write for Serpent Keysets into Serpent Messages (section 4.1 of the procotol specification). The communication-level data in this type of message consists in the type ID, the CRC information *for each key* and the message padding (variable-sized since it simply fills the message as needed). The message counter is kept separate since the server for instance will likely need to keep different counters for each client, so I'll leave for now the decision on this for a higher /different level. The keyset itself is the actual data really and it consists in the number of keys, their actual octets and the flag that indicates whether they are for client or for server use. So I define a parametrized record for representing a keyset only rather than the whole message. This is in the new file data_structs.ads that I see at the moment as the place for defining structures for game data. Hence, it contains for instance also the "SMG_Object" record (fixed size) that illustrates some use of representation clauses:

 -- Data structures for SMG Communication Protocol
 -- S.MG, 2018

with Interfaces; -- for fixed-size types
with Raw_Types;  -- for protocol raw types
with System;     -- for Bit_Order
with Serpent;    -- for Serpent.Key type

package Data_Structs is
  Pragma Pure(Data_Structs);

  -- an object in the world, with record layout fully specified
  -- a storage element is 8-bit aka 1 octet
  -- "at" gives number of storage element
  -- "range" gives bits inside storage element

  type SMG_Object is
    record
      -- ID of the object
      ID         : Interfaces.Unsigned_32;

      -- Position of the object in the world.
      -- For a world with map coordinates (MC) between -500 and +500,
      -- the relationship with given figure (GF) is:
      -- MC = GF / 65.535 - 500
      -- where 65.535 comes from the mapping 65535 / (500 - - 500)
      X, Y, Z    : Interfaces.Integer_16;

      -- Rotation of the object (RO) linked to GF by:
      -- RO = GF / 128*pi
      RX, RY, RZ : Interfaces.Unsigned_8;
    end record;
  for SMG_Object'Size use 104; -- in bits!
  for SMG_Object'Bit_Order use System.Low_Order_First;
  for SMG_Object use
    record
      ID at 0  range 0 .. 31;
      X  at 4  range 0 .. 15;
      Y  at 6  range 0 .. 15;
      Z  at 8  range 0 .. 15;
      RX at 10 range 0 .. 7;
      RY at 11 range 0 .. 7;
      RZ at 12 range 0 .. 7;
    end record;

  -------------------------
  -- A set of Serpent Keys
  -- parametrized record for a Serpent Keyset
  -- parameters:
  --   N is number of Serpent Keys contained
  -- this can be the content of messages 4.1 or 5.2 in protocol spec.

  -- an array of Serpent Keys
  -- MAXIMUM 40 keys allowed in a message, hence subtype for key count
  subtype Keys_Count is Interfaces.Unsigned_8 range 1..40;
  type SKeys_Array is array( Keys_Count range <>) of Serpent.Key;

  type Serpent_Keyset( N : Keys_Count := Keys_Count'Last) is
    record
      -- actual Serpent Keys
      Keys      : SKeys_Array( 1..N );
      -- whether for talking to client (LSB set) or talking to server (MSB set)
      Flag      : Interfaces.Unsigned_8;
    end record;

  ------------------------------
  -- Serpent Keys Management
  type Keys_Mgm (N_Burnt: Interfaces.Unsigned_8) is
    record
      -- count of server keys requested
      N_Server: Interfaces.Unsigned_8;
      -- count of client keys requested
      N_Client: Interfaces.Unsigned_8;
      -- ID of Serpent key preferred for further inbound Serpent msgs.
      Key_ID  : Interfaces.Unsigned_8;
      -- IDs of Serpent keys burnt by this message
      Burnt   : SKeys_Array( 1..N_Burnt );
    end record;

end Data_Structs;

The communication-level information such as CRC, type ids and padding are all handled by messages.ads/adb and are effectively private matters, of no concern to anything outside this messages package:

 -- Message reader & writers for SMG Communication Protocol
 -- This part effectively serializes/deserializes game data structures
 --   for transmission between client and server.
 -- Note that messages themeselves are simply arrays of octets.
 -- (see also raw_types.ads/adb and packing.ads/adb for related parts)
 -- NB: message ids and padding as per protocol spec are handled HERE ONLY.
 -- Relies on:
 --   RNG (for random padding)
 --   CRC32 (for CRC calculations)
 --   Raw_Types
 --   Data_Structs
 -- S.MG, 2018

with Raw_Types;
with RNG;
with CRC32;
with Data_Structs; use Data_Structs;
with Interfaces;

package Messages is
  -- exception raised when given message to read fails sanity checks
  Invalid_Msg: exception;

  ------------------------------------------------
  -- Writes a Serpent Keyset to the given Serpent Message
  --
  -- Keyset  - the set of keys to write to message
  -- Counter - the message count
  procedure Write_SKeys_SMsg( Keyset  : in Serpent_Keyset;
                              Counter : in Interfaces.Unsigned_16;
                              Msg     : out Raw_Types.Serpent_Msg);

  -- Reads a Serpent Keyset from the given Serpent Message
  -- The opposite of Write_SKeys_SMsg above
  -- Raises Invalid_Message exception if given message fails sanity checks
  procedure Read_SKeys_SMsg( Msg     : in Raw_Types.Serpent_Msg;
                             Counter : out Interfaces.Unsigned_16;
                             Keyset  : out Serpent_Keyset);

private

  -- if native is little endian, does nothing;
  -- if native is big endian, it flips the input's octets.
  -- (strictly for arrays of octets)
  procedure Cast_LE( LE: in out Raw_Types.Octets );

  -- protocol message types IDs, fixed as per protocol specification
  -- Serpent messages end in "S_Type"
  -- RSA messages end in "R_Type"
  -- Character action types end in "A_Type"

  -- Serpent message types
  SKeys_S_Type         : constant Interfaces.Unsigned_8 := 1;
  Key_Mgm_S_Type       : constant Interfaces.Unsigned_8 := 2;
  File_Transfer_S_Type : constant Interfaces.Unsigned_8 := 3;
  File_Req_S_Type      : constant Interfaces.Unsigned_8 := 4;
  Client_Action_S_Type : constant Interfaces.Unsigned_8 := 5;
  World_Bulletin_S_Type: constant Interfaces.Unsigned_8 := 6;
  Obj_Request_S_Type   : constant Interfaces.Unsigned_8 := 7;
  Obj_Info_S_Type      : constant Interfaces.Unsigned_8 := 8;

  -- RSA message types
  RKeys_R_Type         : constant Interfaces.Unsigned_8 := 251;
  SKeys_R_Type         : constant Interfaces.Unsigned_8 := 157;
  Key_Mgm_R_Type       : constant Interfaces.Unsigned_8 := 233;

  -- Character action types
  Lock_A_Type          : constant Interfaces.Unsigned_8 := 0;
  Make_A_Type          : constant Interfaces.Unsigned_8 := 1;
  Explore_A_Type       : constant Interfaces.Unsigned_8 := 2;
  Exchange_A_Type      : constant Interfaces.Unsigned_8 := 3;
  Attack_A_Type        : constant Interfaces.Unsigned_8 := 4;
  Repair_A_Type        : constant Interfaces.Unsigned_8 := 5;
  Move_A_Type          : constant Interfaces.Unsigned_8 := 6;
  Train_A_Type         : constant Interfaces.Unsigned_8 := 7;

end Messages;

The implementation in messages.adb, including read/write for Serpent keysets to/from Serpent messages and handling endianness for any multi-octet entity:

 -- Message reader & writers for SMG Communication Protocol
 -- S.MG, 2018

with Interfaces; use Interfaces;
with Serpent;
with System; use System;

package body Messages is

  procedure Write_SKeys_SMsg( Keyset  : in Serpent_Keyset;
                              Counter : in Interfaces.Unsigned_16;
                              Msg     : out Raw_Types.Serpent_Msg) is
    Pos   : Integer := Msg'First;
    Check : CRC32.CRC32;
    PadLen: Integer;
    K     : Serpent.Key;
  begin
    -- write Type ID
    Msg(Pos) := SKeys_S_Type;
    Pos := Pos + 1;

    -- write count of keys (NB: this IS 8 bits by definition)
    Msg(Pos) := Keyset.Keys'Length;
    Pos := Pos + 1;

    -- write keys
    for I in Keyset.Keys'Range loop
      -- retrieve Key to write
      K := Keyset.Keys( I );

      -- write key itself
      Msg(Pos..Pos+K'Length-1) := K;
      -- ensure little endian order in message
      Cast_LE(Msg(Pos..Pos+K'Length-1));
      Pos := Pos + K'Length;

      -- write CRC of key
      Check := CRC32.CRC( K );
      Msg(Pos..Pos+3) := Raw_Types.Cast(Check);
      Cast_LE(Msg(Pos..Pos+3));
      Pos := Pos + 4;
    end loop;

    -- write flag
    Msg(Pos) := Keyset.Flag;
    Pos := Pos + 1;

    -- write message counter
    Msg(Pos..Pos+1) := Raw_Types.Cast(Counter);
    Cast_LE(Msg(Pos..Pos+1));
    Pos := Pos + 2;

    -- write padding as needed; endianness is irrelevant here
    PadLen := Msg'Last - Pos + 1;
    if PadLen > 0 then
      declare
        Pad : Raw_Types.Octets(1..PadLen);
      begin
        RNG.Get_Octets( Pad );
        Msg(Pos..Pos+PadLen-1) := Pad;
      end;
    end if;
  end Write_SKeys_SMsg;

  -- Reads a Serpent keyset from given Serpent Message
  procedure Read_SKeys_SMsg( Msg     : in Raw_Types.Serpent_Msg;
                             Counter : out Interfaces.Unsigned_16;
                             Keyset  : out Serpent_Keyset) is
    Pos: Integer := Msg'First;
  begin
    -- read type and check
    if Msg(Pos) = SKeys_S_Type then
      Pos := Pos + 1;
    else
      raise Invalid_Msg;
    end if;

    -- read count of keys and check
    if Msg(Pos) in Keys_Count'Range then
      declare
        N     : Keys_Count := Keys_Count(Msg(Pos));
        KS    : Serpent_Keyset(N);
        K     : Serpent.Key;
        Check : CRC32.CRC32;
        O4    : Raw_Types.Octets_4;
        O2    : Raw_Types.Octets_2;
      begin
        Pos := Pos + 1;
        --read keys and check crc for each
        for I in 1 .. N loop
          -- read key and advance pos
          K := Msg(Pos..Pos+K'Length-1);
          Cast_LE(K);
          Pos := Pos + K'Length;
          -- read crc and compare to crc32(key)
          O4 := Msg(Pos..Pos+3);
          Cast_LE(O4);
          Check   := Raw_Types.Cast(O4);
          Pos := Pos + 4;
          if Check /= CRC32.CRC(K) then
            raise Invalid_Msg;
          end if;
          -- if it got here, key is fine so add to set
          KS.Keys(KS.Keys'First + I -1) := K;
        end loop;
        -- read and set flag
        KS.Flag := Msg(Pos);
        Pos := Pos + 1;
        -- read and set message counter
        O2 := Msg(Pos..Pos+1);
        Cast_LE(O2);
        Counter := Raw_Types.Cast(O2);
        -- rest of message is padding so it's ignored
        -- copy keyset to output variable
        Keyset := KS;
      end;
    else
      raise Invalid_Msg;
    end if;
  end Read_SKeys_SMsg;

  -- private part
  procedure Cast_LE( LE: in out Raw_Types.Octets ) is
  begin
    -- flip octets ONLY if native is big endian.
    if System.Default_Bit_Order = System.High_Order_First then
      declare
        BE: constant Raw_Types.Octets := LE;
      begin
        for I in 1..LE'Length loop
          LE(LE'First+I-1) := BE(BE'Last-I+1);
        end loop;
      end;
    end if;
    -- NOTHING to do for native little endian
  end Cast_LE;

end Messages;

Note that Messages calls directly RNG to obtain padding as and when it needs it. Similarly, it calls directly CRC32 to calculate the checksums as and when needed. The CRC32 is the previously published table-lookup CRC32 implementation from EuCrypt, now integrated into SMG Comms and slightly adapted to use the Octets type in Raw_Types:


------------------------------------------------------------------------------
------------------------------------------------------------------------------
-- This file is part of 'CRC32'                                             --
--                                                                          --
-- You do not have, nor can you ever acquire the right to use, copy or      --
-- distribute this software ; Should you use this software for any purpose, --
-- or copy and distribute it to anyone or in any manner, you are breaking   --
-- the laws of whatever soi-disant jurisdiction, and you promise to         --
-- continue doing so for the indefinite future. In any case, please         --
-- always : read and understand any software ; verify any PGP signatures    --
-- that you use - for any purpose.                                          --
--                                                                          --
-- See also http://trilema.com/2015/a-new-software-licensing-paradigm .     --
------------------------------------------------------------------------------
------------------------------------------------------------------------------

  -- CRC32, lookup-based implementation
  -- S.MG, 2018
  --
  -- The CRC32 is a checksum calculated as the remainder of dividing
  -- the input data by the 0x04C11DB7 polynomial. Specifications:
  -- Name   : "CRC-32"
  -- Width  : 32        (number of bits)
  -- Poly   : 04C11DB7  (generator polynomial)
  -- Init   : FFFFFFFF
  -- RefIn  : True      (input is reflected)
  -- RefOut : True      (output is reflected)
  -- XorOut : FFFFFFFF
  -- Check  : CBF43926  (expected value for input "123456789")
  --
  -- This implementation is based on the CRC32 specification in:
  -- Sarwate, D.V. "Computation of Cyclic Redundancy Checks via Table Look-Up"
  -- in Communications of the ACM, Vol. 31 No. 8, pp.1008-1013, Aug. 1988

with Interfaces; use Interfaces;
with Raw_Types;

package CRC32 is
  -- local, shorthand version of Interfaces. types
  subtype CRC32 is Interfaces.Unsigned_32;
  subtype Octet is Interfaces.Unsigned_8;

  subtype Octet_Array is Raw_Types.Octets;

  -- interface for external callers
    -- calculate CRC32 for the given string
  function CRC( S: in String ) return CRC32;

    -- calculate CRC32 for the given array of octets
  function CRC( Data: in Octet_Array ) return CRC32;

  -- internal constants and helper methods
private
	function Shift_Right( Value  : CRC32;
                        Amount : Natural)
                        return CRC32;
  pragma Import(Intrinsic, Shift_Right); 

  Init_Value : constant CRC32 := 16#FFFF_FFFF#; -- Initial value
  Xor_Out    : constant CRC32 := 16#FFFF_FFFF#; -- For extracting result
  LSB_Mask   : constant CRC32 := 16#0000_00FF#; -- lsb mask for a CRC32 value

  -- lookup table with precomputed values for CRC32
  Lookup : constant array (CRC32 range 0 .. 255) of CRC32 :=
     (16#0000_0000#, 16#7707_3096#, 16#EE0E_612C#, 16#9909_51BA#,
      16#076D_C419#, 16#706A_F48F#, 16#E963_A535#, 16#9E64_95A3#,
      16#0EDB_8832#, 16#79DC_B8A4#, 16#E0D5_E91E#, 16#97D2_D988#,
      16#09B6_4C2B#, 16#7EB1_7CBD#, 16#E7B8_2D07#, 16#90BF_1D91#,
      16#1DB7_1064#, 16#6AB0_20F2#, 16#F3B9_7148#, 16#84BE_41DE#,
      16#1ADA_D47D#, 16#6DDD_E4EB#, 16#F4D4_B551#, 16#83D3_85C7#,
      16#136C_9856#, 16#646B_A8C0#, 16#FD62_F97A#, 16#8A65_C9EC#,
      16#1401_5C4F#, 16#6306_6CD9#, 16#FA0F_3D63#, 16#8D08_0DF5#,
      16#3B6E_20C8#, 16#4C69_105E#, 16#D560_41E4#, 16#A267_7172#,
      16#3C03_E4D1#, 16#4B04_D447#, 16#D20D_85FD#, 16#A50A_B56B#,
      16#35B5_A8FA#, 16#42B2_986C#, 16#DBBB_C9D6#, 16#ACBC_F940#,
      16#32D8_6CE3#, 16#45DF_5C75#, 16#DCD6_0DCF#, 16#ABD1_3D59#,
      16#26D9_30AC#, 16#51DE_003A#, 16#C8D7_5180#, 16#BFD0_6116#,
      16#21B4_F4B5#, 16#56B3_C423#, 16#CFBA_9599#, 16#B8BD_A50F#,
      16#2802_B89E#, 16#5F05_8808#, 16#C60C_D9B2#, 16#B10B_E924#,
      16#2F6F_7C87#, 16#5868_4C11#, 16#C161_1DAB#, 16#B666_2D3D#,
      16#76DC_4190#, 16#01DB_7106#, 16#98D2_20BC#, 16#EFD5_102A#,
      16#71B1_8589#, 16#06B6_B51F#, 16#9FBF_E4A5#, 16#E8B8_D433#,
      16#7807_C9A2#, 16#0F00_F934#, 16#9609_A88E#, 16#E10E_9818#,
      16#7F6A_0DBB#, 16#086D_3D2D#, 16#9164_6C97#, 16#E663_5C01#,
      16#6B6B_51F4#, 16#1C6C_6162#, 16#8565_30D8#, 16#F262_004E#,
      16#6C06_95ED#, 16#1B01_A57B#, 16#8208_F4C1#, 16#F50F_C457#,
      16#65B0_D9C6#, 16#12B7_E950#, 16#8BBE_B8EA#, 16#FCB9_887C#,
      16#62DD_1DDF#, 16#15DA_2D49#, 16#8CD3_7CF3#, 16#FBD4_4C65#,
      16#4DB2_6158#, 16#3AB5_51CE#, 16#A3BC_0074#, 16#D4BB_30E2#,
      16#4ADF_A541#, 16#3DD8_95D7#, 16#A4D1_C46D#, 16#D3D6_F4FB#,
      16#4369_E96A#, 16#346E_D9FC#, 16#AD67_8846#, 16#DA60_B8D0#,
      16#4404_2D73#, 16#3303_1DE5#, 16#AA0A_4C5F#, 16#DD0D_7CC9#,
      16#5005_713C#, 16#2702_41AA#, 16#BE0B_1010#, 16#C90C_2086#,
      16#5768_B525#, 16#206F_85B3#, 16#B966_D409#, 16#CE61_E49F#,
      16#5EDE_F90E#, 16#29D9_C998#, 16#B0D0_9822#, 16#C7D7_A8B4#,
      16#59B3_3D17#, 16#2EB4_0D81#, 16#B7BD_5C3B#, 16#C0BA_6CAD#,
      16#EDB8_8320#, 16#9ABF_B3B6#, 16#03B6_E20C#, 16#74B1_D29A#,
      16#EAD5_4739#, 16#9DD2_77AF#, 16#04DB_2615#, 16#73DC_1683#,
      16#E363_0B12#, 16#9464_3B84#, 16#0D6D_6A3E#, 16#7A6A_5AA8#,
      16#E40E_CF0B#, 16#9309_FF9D#, 16#0A00_AE27#, 16#7D07_9EB1#,
      16#F00F_9344#, 16#8708_A3D2#, 16#1E01_F268#, 16#6906_C2FE#,
      16#F762_575D#, 16#8065_67CB#, 16#196C_3671#, 16#6E6B_06E7#,
      16#FED4_1B76#, 16#89D3_2BE0#, 16#10DA_7A5A#, 16#67DD_4ACC#,
      16#F9B9_DF6F#, 16#8EBE_EFF9#, 16#17B7_BE43#, 16#60B0_8ED5#,
      16#D6D6_A3E8#, 16#A1D1_937E#, 16#38D8_C2C4#, 16#4FDF_F252#,
      16#D1BB_67F1#, 16#A6BC_5767#, 16#3FB5_06DD#, 16#48B2_364B#,
      16#D80D_2BDA#, 16#AF0A_1B4C#, 16#3603_4AF6#, 16#4104_7A60#,
      16#DF60_EFC3#, 16#A867_DF55#, 16#316E_8EEF#, 16#4669_BE79#,
      16#CB61_B38C#, 16#BC66_831A#, 16#256F_D2A0#, 16#5268_E236#,
      16#CC0C_7795#, 16#BB0B_4703#, 16#2202_16B9#, 16#5505_262F#,
      16#C5BA_3BBE#, 16#B2BD_0B28#, 16#2BB4_5A92#, 16#5CB3_6A04#,
      16#C2D7_FFA7#, 16#B5D0_CF31#, 16#2CD9_9E8B#, 16#5BDE_AE1D#,
      16#9B64_C2B0#, 16#EC63_F226#, 16#756A_A39C#, 16#026D_930A#,
      16#9C09_06A9#, 16#EB0E_363F#, 16#7207_6785#, 16#0500_5713#,
      16#95BF_4A82#, 16#E2B8_7A14#, 16#7BB1_2BAE#, 16#0CB6_1B38#,
      16#92D2_8E9B#, 16#E5D5_BE0D#, 16#7CDC_EFB7#, 16#0BDB_DF21#,
      16#86D3_D2D4#, 16#F1D4_E242#, 16#68DD_B3F8#, 16#1FDA_836E#,
      16#81BE_16CD#, 16#F6B9_265B#, 16#6FB0_77E1#, 16#18B7_4777#,
      16#8808_5AE6#, 16#FF0F_6A70#, 16#6606_3BCA#, 16#1101_0B5C#,
      16#8F65_9EFF#, 16#F862_AE69#, 16#616B_FFD3#, 16#166C_CF45#,
      16#A00A_E278#, 16#D70D_D2EE#, 16#4E04_8354#, 16#3903_B3C2#,
      16#A767_2661#, 16#D060_16F7#, 16#4969_474D#, 16#3E6E_77DB#,
      16#AED1_6A4A#, 16#D9D6_5ADC#, 16#40DF_0B66#, 16#37D8_3BF0#,
      16#A9BC_AE53#, 16#DEBB_9EC5#, 16#47B2_CF7F#, 16#30B5_FFE9#,
      16#BDBD_F21C#, 16#CABA_C28A#, 16#53B3_9330#, 16#24B4_A3A6#,
      16#BAD0_3605#, 16#CDD7_0693#, 16#54DE_5729#, 16#23D9_67BF#,
      16#B366_7A2E#, 16#C461_4AB8#, 16#5D68_1B02#, 16#2A6F_2B94#,
      16#B40B_BE37#, 16#C30C_8EA1#, 16#5A05_DF1B#, 16#2D02_EF8D#);

end CRC32;

The CRC32 implementation in crc32.adb:

------------------------------------------------------------------------------
------------------------------------------------------------------------------
-- This file is part of 'CRC32'                                             --
--                                                                          --
-- You do not have, nor can you ever acquire the right to use, copy or      --
-- distribute this software ; Should you use this software for any purpose, --
-- or copy and distribute it to anyone or in any manner, you are breaking   --
-- the laws of whatever soi-disant jurisdiction, and you promise to         --
-- continue doing so for the indefinite future. In any case, please         --
-- always : read and understand any software ; verify any PGP signatures    --
-- that you use - for any purpose.                                          --
--                                                                          --
-- See also http://trilema.com/2015/a-new-software-licensing-paradigm .     --
------------------------------------------------------------------------------
------------------------------------------------------------------------------

  -- CRC32 implementation
  -- S.MG, 2018

package body CRC32 is
  function CRC( S: in String ) return CRC32 is
    Result : CRC32 := Init_Value;
    Value  : CRC32;
  begin
    -- process input character by character
    for C of S loop
      Value  := CRC32( Character'Pos( C ) );
      Result := Shift_Right(Result, 8) xor
                  Lookup( Value xor (Result and LSB_MASK));
    end loop;
    -- reflect result
    Result := Result xor Xor_Out;

    return Result;
  end CRC;

  function CRC( Data: in Octet_Array ) return CRC32 is
    Result : CRC32 := Init_Value;
  begin
    -- process input octet by octet
    for C of Data loop
      Result := Shift_Right(Result, 8) xor
                  Lookup( CRC32(C) xor (Result and LSB_MASK));
    end loop;
    -- reflect result
    Result := Result xor Xor_Out;

    return Result;
  end CRC;

end CRC32;

Other than the above, the .vpatch for this chapter contains also some basic testing of the read/write methods for Serpent keysets:

  -- Tests for the serialization of data structures in SMG Protocol
 -- S.MG, 2018

with RNG;
with Data_Structs; use Data_Structs;
with Messages; use Messages;
with Interfaces; use Interfaces;
with System;
with System.Storage_Elements; use System.Storage_Elements;
with Ada.Text_IO; use Ada.Text_IO;

package body Test_Serializing is

  procedure Serialize_Keyset_SS is
    Msg  : Serpent_Msg;
    KSet : Serpent_Keyset(5);
    LSB : Interfaces.Unsigned_8 := 16#01#;
    MSB : Interfaces.Unsigned_8 := 16#80#;
    LMSB: Interfaces.Unsigned_8 := 16#81#;
    Counter : Interfaces.Unsigned_16 := 101;
    NewSet: Serpent_Keyset;
    NewCounter: Interfaces.Unsigned_16:=0;
  begin
    Put_Line("Generating the Serpent Keys...");
    -- fill a set of Serpent Keys
    for I in 1..KSet.N loop
      RNG.Get_Octets(KSet.Keys(Interfaces.Unsigned_8(I)));
    end loop;
    KSet.Flag := LSB;

    Put_Line("Writing the keys to message...");
    -- write keyset to serpent message
    Write_SKeys_SMsg( KSet, Counter, Msg );

    Put_Line("Reading keys back from message...");
    -- read keyset from message
    Read_SKeys_SMsg( Msg, Counter, NewSet );

    Put_Line("Comparing the keysets...");
    -- compare the two keysets
    if NewSet /= KSet then
      Put_Line("FAIL: keysets are different!");
    else
      Put_Line("PASS: keysets are the same!");
    end if;

    Put_Line("Attempting to read from mangled message");
    begin
      Msg(Msg'First) := Msg(Msg'First)+25;
      Read_SKeys_SMsg( Msg, Counter, NewSet);
      Put_Line("FAIL: read failed to raise invalid message exception!");
    exception
      when Invalid_Msg =>
        Put_Line("PASS: exception correctly raised for invalid message");
    end;
  end Serialize_Keyset_SS;

end Test_Serializing;

As I don't currently have any access to a Big Endian computer, I couldn't quite test the whole thing properly. If you have a Big Endian computer and get around to test this code, please let me know how it behaves.

The above method seems to me robust at least and reasonably easy to follow. Feel free however to loudly disagree and shoot it down in the comments below - I'll gladly read any better proposal. As mentioned in the very beginning of this post, this is not exactly what I hoped to obtain although it is the thing I have so far and therefore the thing I'm publishing - to be discussed and improved on, hopefully.

The .vpatch and my signature for it are as usual on my Reference Code Shelf and linked below for your convenience:


  1. The official solution for anything touching serialization seems to boil down to "Streams or gtfo" apparently. And for my specific case here I don't quite see the point of bringing in the whole Streams mammouth

November 4, 2018

SMG Comms Chapter 6: Packing and Unpacking RSA

Filed under: Coding, SMG_Comms — Diana Coman @ 1:38 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

Building on the new convenient way to simply call rsa+oaep encrypt/decrypt from Ada, this chapter provides the Pack/Unpack functions that convert between SMG Comms RSA packets (each of them 1470 octets long, effectively several RSA-encrypted chunks glued together) and SMG Comms messages aka the actual content of the packet from the game's point of view. To do this, the Pack/Unpack functions simply chop the input into fixed-size parts, apply rsa+oaep encrypt/decrypt as suitable and then glue back the results in order, providing the whole thing as final output. Note that from the point of view of Pack/Unpack, both packets and messages have fixed lengths, as specified! In other words: the padding that might be added to data structures to get the whole thing to the specified length of a RSA message is preserved - that is no concern at this layer at all1. In a nutshell: Unpack removes any padding that Pack added but preserves any other padding that was added *before* calling Pack. In turn, Pack expects as input a fixed-size array of octets aka a RSA message as defined in Raw_Types - and the size of this is simply a multiple of the maximum size that one single RSA-encrypted operation can handle.

While the above fixed-size helps considerably to keep the whole thing quite easy to follow and clear, things are still a bit more gnarly than with Serpent, there's no escaping that. In particular, the RSA Decrypt operation CAN fail visibly - when the decoded "length" of the actual content turns out to be larger than maximum/provided space. For this reason, the Unpack function has a "Success" flag that gets set to false when output is essentially undefined. Note that the Unpack will fail in this way as soon as one of the chunks fails the RSA+OAEP decrypt in any way - there is no point in even attempting to recover some "partial" thing. Even more importantly: a True value for the Success flag does NOT indicate in any way that the result obtained is valid - all it says is that the RSA+OAEP decrypt operations completed successfully and the result is the one returned but whether that's a valid data structure or not is not a concern at this point (nor could it be since Unpack can't check such a thing in any way).

The .vpatch for this chapter includes the following main changes/additions:

  • A new note is added next to the offending Keccak line, describing the proposed way2 to preserve the No_Implicit_Conditionals restriction if desired. As mentioned in the forum, I do not quite see the need for this change at this moment but perhaps it will be needed at a later time, so there it is, right where it'll come in handy.
  • There's a correction of the value for the constant RSA_MSG_OCTETS. The new, correct value matches the fact that this is meant to be the fixed-size length of a SMG Comms RSA message rather than (as it was before) the max length that can be encrypted with RSA in one operation. So the RSA_MSG_OCTETS is calculated based on other specification-defined constants and it is a multiple of the max length that can be encrypted with RSA in one operation.
  • As part of doing the above correction of RSA_MSG_OCTETS, I also moved the initialization of some of the OAEP constants into Raw_Types (from OAEP packet where they were before). More precisely, the OAEP packet still uses its own internal constants (e.g. the "reserved" octets spelling "TMSR-RSA") but it initialises those (at elaboration time) with the values that it reads from Raw_Types. This reflects the fact that the OAEP packet in this project is the one using the Raw_Types that the protocol defines, not the other way around (in other words: OAEP does not define/impose those values - it's SMG Comms that does!).
  • The Packing package lost its "Pure" status (i.e. the compiler can't rely on it having no internal state and can't therefore cache calls to its method). The reason for this is at root the need for random padding for RSA since yes, can't cache the call to a method that has to use different, random padding at every call. In any case, for anyone interested, the specific route to follow this is that the Pack method for RSA calls OAEP+RSA which in turn use the non-pure Sequential_IO (via the RNG package) to read random octets from a Fuckgoats.
  • The new Pack and Unpack methods for RSA messages <-> packets conversions are in packing.ads/adb.
  • Corresponding tests for the Pack/Unpack methods are added to test_packing.ads/adb and called from testall.adb.

The signatures of the new Pack/Unpack methods in packing.ads are the following:

  -- Packing a RSA message into RSA packet, using the given key
  function Pack( Msg : in Raw_Types.RSA_Msg;
                 K   : in RSA_OAEP.RSA_pkey)
                return Raw_Types.RSA_Pkt;

  -- Unpacking a RSA packet into contained message, using the given key
  function Unpack( Pkt     : in Raw_Types.RSA_Pkt;
                   K       : in RSA_OAEP.RSA_skey;
                   Success : out Boolean)
                  return Raw_Types.RSA_Msg;

And the corresponding code for the new Pack/Unpack in packing.adb:

  -- Packing a RSA message into RSA packet, using the given key
  function Pack( Msg : in Raw_Types.RSA_Msg;
                 K   : in RSA_OAEP.RSA_pkey)
                return Raw_Types.RSA_Pkt is

  -- a chunk that can be processed via rsa+oaep at any given time
    Chunk: Raw_Types.Octets(1..Raw_Types.OAEP_MAX_LEN) := (others => 0);

  -- number of chunks in the message to process
  -- NO incomplete chunks will be processed!
  -- NB: values are set so that there are no incomplete blocks here
    N   : constant Natural := Msg'Length / Chunk'Length;

  -- intermediate result, as obtained from rsa_oaep
    Encr: Raw_Types.RSA_len := (others => 0);

  -- final resulting RSA Packet
    Pkt : Raw_Types.RSA_Pkt := (others => 0);
  begin
    -- there should ALWAYS be precisely N chunks in Msg to feed to rsa_oaep
    -- process chunks of Msg one at a time
    for I in 1..N loop
      -- get current chunk
      Chunk := Msg(Msg'First + (I-1) * Chunk'Length ..
                   Msg'First + I * Chunk'Length - 1 );
      -- call rsa oaep encrypt on current chunk
      RSA_OAEP.Encrypt( Chunk, K, Encr );
      -- copy result to its place in final packet
      Pkt( Pkt'First + (I-1) * Encr'Length ..
           Pkt'First + I * Encr'Length - 1 ) := Encr;
    end loop;
    -- return final result
    return Pkt;
  end Pack;

  -- Unpacking a RSA packet into contained message, using the given key
  function Unpack( Pkt     : in Raw_Types.RSA_Pkt;
                   K       : in RSA_OAEP.RSA_skey;
                   Success : out Boolean)
                  return Raw_Types.RSA_Msg is
    -- a chunk - basically input for RSA_OAEP.Decrypt
    Chunk : Raw_Types.RSA_len := (others => 0);

    -- intermediate result of rsa_oaep decrypt
    Decr  : Raw_Types.Octets( 1..Raw_Types.OAEP_MAX_LEN ) := (others => 0);
    Len   : Natural;
    Flag  : Boolean;

    -- number of chunks in the packet
    -- NB: there should be only FULL chunks! otherwise -> fail
    N   : constant Natural := Pkt'Length / Chunk'Length;

    -- final resulting message content of the given RSA packet
    Msg : Raw_Types.RSA_Msg := (others => 0);
  begin
    -- initialize Success flag
    Success := True;

    -- process given packet, chunk by chunk
    for I in 1..N loop
      -- get current chunk
      Chunk := Pkt( Pkt'First + (I-1) * Chunk'Length ..
                    Pkt'First + I * Chunk'Length - 1 );
      -- decrypt it via rsa+oaep
      RSA_OAEP.Decrypt( Chunk, K, Decr, Len, Flag );
      -- check result and if ok then copy it to final result at its place
      -- NB: if returned length is EVER less than OAEP_MAX_LEN then -> fail!
      -- the reason for above: there will be undefined bits in the output!
      if Len /= Raw_Types.OAEP_MAX_LEN or (not Flag) then
        Success := False;
        return Msg;
      else
        Msg( Msg'First + (I-1) * Decr'Length ..
             Msg'First + I * Decr'Length - 1 ) := Decr;
      end if;
    end loop;

    -- return obtained message
    return Msg;
  end Unpack;

The new tests for the above, in test_packing.adb:

  procedure Test_Pack_Unpack_RSA is
    Msg       : RSA_Msg;
    Decr_Msg  : RSA_Msg;
    PKey      : RSA_pkey;
    SKey      : RSA_skey;
    Success   : Boolean;
    Pkt       : RSA_Pkt;
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
  begin
    -- initialize RSA key with values previously obtained from EuCrypt
    Hex2Octets( n, SKey.n );
    Hex2Octets( e, SKey.e );
    Hex2Octets( d, SKey.d );
    Hex2Octets( p, SKey.p );
    Hex2Octets( q, SKey.q );
    Hex2Octets( u, SKey.u );
    -- copy n and e for public key
    PKey.n := SKey.n;
    PKey.e := SKey.e;
    -- get random data for "message"
    RNG.Get_Octets(Msg);

    -- pack the message into corresponding packet
    Pkt := Packing.Pack( Msg, PKey );
    -- unpack and check the result
    Decr_Msg := Packing.Unpack( Pkt, SKey, Success );
    if (not Success) or (Decr_Msg /= Msg) then
      Put_Line("FAIL: pack/unpack with RSA.");
    else
      Put_Line("PASS: pack/unpack with RSA.");
    end if;

    -- try to unpack a mangled package
    Pkt(Pkt'First + 3) := Pkt(Pkt'First + 3) and 16#AB#;
    Decr_Msg := (others => 0);
    Decr_Msg := Packing.Unpack( Pkt, SKey, Success );
    if Success then
      Put_Line("FAIL: pack/unpack with RSA on mangled package.");
    else
      Put_Line("PASS: pack/unpack with RSA on mangled package.");
    end if;

  end Test_Pack_Unpack_RSA;

The .vpatch with all the above is on my Reference Code Shelf and linked below for your convenience:

UPDATE (5 November 2018): I've changed the tests so that they read the RSA keypair from a file rather than using a hard-coded key; as discussed in the logs, this is the only way I can see to keep all lines in code files under 80 columns while also keeping RSA components as they are i.e. well longer than 80 characters in any sane format. The added procedure simply reads key components in a rather blunt way, character by character and demanding *exactly* as many characters as needed to fill the length of key it expects. This is NOT a generic "key reading" but simply a quick switch of input for the basic tests:

  procedure ReadRSAKey( Filename: in String; Key: out RSA_OAEP.RSA_skey ) is
    package Char_IO is new Ada.Sequential_IO(Character);
    use Char_IO;
    Full : String(1..RSA_len'Length*2) := (others => '0');
    Half : String(1..RSA_half'Length*2) := (others => '0');
    F    : Char_IO.File_Type;
    C    : Character;
  begin
    Open( File => F, Mode => In_File, Name => Filename );

    -- read n
    for I in Full'Range loop
      Read(F, Full(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Full, Key.n);

    -- read e
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.e);

    -- read d
    for I in Full'Range loop
      Read(F, Full(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Full, Key.d);

    -- read p
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.p);

    -- read q
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.q);

    -- read u
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    Hex2Octets(Half, Key.u);

    -- Close file
    Close( F );

    exception
      when Char_IO.End_Error =>
        Put_Line("ReadRSAKey ERROR: Unexpected end of file in " & Filename);
      when others =>
        Put_Line("ReadRSAKey ERROR: can not open file " & Filename);

  end ReadRSAKey;

Since this change of inputs for tests has nothing to do with either protocol code per se or with advancing the protocol implementation, I'm considering it more of a fix of sorts to this chapter's code and so it'll be a .vpatch by itself, to be found both on the Reference Code Shelf and linked from here:

UPDATE (12 November 2018): I've refactored the tests to extract the reading of a RSA key in a package of its own and thus use it afterwards from everywhere it's needed (i.e. not only in testing oaep_rsa but also in testing the packing for instance). Since this happened after I published Chapter 7, it's a new .vpatch on top:


  1. It will be handled at a higher level, where data structures are populated. 

  2. Thank you, asciilifeform! 

October 31, 2018

SMG Comms Chapter 5: RSA with OAEP from Ada

Filed under: Coding, SMG_Comms — Diana Coman @ 5:35 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

This brings together RSA and OAEP as well as the raw representation as octets of all the things involved: true random padding, RSA key components, plain messages, encrypted outputs, OAEP blocks or Keccak hashes. The main goal was to have some clear and easy to use methods in Ada for doing the whole RSA+OAEP dance from start to finish and then back as needed. Due to the current situation re MPI and RSA, a significant part of this goal involves keeping the C swamp at bay: calling C methods only when there really is at the moment no alternative and in general as few times as possible, making sure that anything built on top of this part of smg_comms can effectively ignore that there is any C code involved at a lower level at all. The reason for insulating the C code away like this is of course that I hope to be able to get rid of it entirely at some point and have everything in Ada without having to touch anything higher up than this part right here. At some point, but not at this point here, not just yet.

At first glance, this chapter was meant to be quite easy and straightforward: glue the different parts together and that's it! As usual, the details were more gnarly and as a result there were a few things I had to change/re-implement in order to get this part into some sort of reasonable shape:

  • I've added a few new subtypes of Octets to Raw_Types: RSA_len and RSA_half to hold a component of precisely the length or, respectively, half of the length set for the modulus of a RSA key. Those are needed to store the RSA keys in raw format aka arrays of octets. While I could also use the generic Octets type, the clear specification of length reflects the expected lengths of each RSA component and brings clarity so it's well worth it in my books.
  • The random padding is now obtained directly from Ada code rather than via C_wrappers and truerandom.c. The new Ada code simply reads from a Fuckgoats that it expects to be available at a pre-set path, *already initialised*1 and ready to read from. This means that random octets do not require anymore a call to C and the corresponding conversions from array of octets to char * and back.
  • The comparison between an OAEP block and the RSA key's modulus is simplified: it's only the top octet of both that is compared. This was discussed previously in the logs but at any rate, the reason for this approach is that it greatly reduces the complexity of the code since it's really a one line compare of 2 8-bit values that are already available as such at the very spot where they are needed! So no more calling the C swamp, converting to char* and then to MPIs and then comparing the full MPIs and returning the result through to Ada. The potential additional OAEP encryption performed at times is considered a price well worth paying in order to avoid the whole Ada->C->MPI->ADA dance that needs to be performed otherwise *each and every time*.
  • The conversions from Ada's array of octets to C's char* are done in local code, octet by octet and strictly on arrays that are precisely the same size. I am aware that GNAT provides in principle methods for such conversions, namely To_Ada and To_C but they still fail to convince me that they provide a better service really. If you think they do though, feel free to state your case in the comments below.
  • The OAEP implementation is the one from EuCrypt but with different access/interface methods: instead of accepting and producing Strings like the EuCrypt version, the OAEP package in SMG COMMS accepts and produces arrays of octets (i.e. Raw_Types.Octets type) since this is effectively what is needed here. Not to mention that I'm not particularly sad to get rid of Strings even when they are fixed size. Obviously, this means that I also replaced the conversions String <-> Bitstream with Octets <-> Bitstream. The rest however stays the same (including the underlying Keccak for hashes).
  • Because of moving all the OAEP+RSA part to Ada and imposing only one-way calls from Ada to C, there was no need anymore for some of the wrappers that were provided in EuCrypt's OAEP and rsa_oaep code. Those were simply discarded so that the code you get by pressing this patch includes simply what is needed for SMG Comms and nothing more.
  • Whenever the Ada code (specifically the rsa+oaep code) retrieves an array of octets from C, it will basically reverse the "normalization" that the MPI lib performs so that one doesn't end up in the idiotic situation where the retrieved value is actually different than the one given to MPI just because of leading zeroes. This is part of the price paid for using the MPI lib and at any rate it should not be the concern of any of the calling code so it's kept well separated so that any callers of RSA_OAEP.Encrypt or RSA_OAEP.Decrypt don't have to deal with it directly: the encryption uses at most msg_max_len octets and produces a fixed-size array while the decrypt similarly requires the same fixed-size array as input and will produce as output the actual length of the plain message and the plain message itself.
  • The "No Implicit Conditionals" restriction for the whole smg comms had to be discarded because of the Keccak implementation - with the restriction applied, compilation of Keccak fails2. At the moment I don't consider this to be a big issue here but feel free to loudly disagree in the comments below - if you have a solid case, I can change my mind.
  • I've updated the relevant tests for RSA (including those from the C_Wrappers package previously published) to use RSA keys of the new size that SMG Comms uses (as opposed to the EuCrypt size). I've added also new tests for the new RSA+OAEP Ada code, of course.

The new Ada code in rng.ads for reading random octets directly from an FG that is already initialised:

 -- True Random octets generator reading from an FG and using Raw_Types.
 -- S.MG, 2018

with Ada.Sequential_IO;
with Interfaces;
with Raw_Types;

package RNG is

  -- for reading from the FG one Octet at a time
  package Octet_IO is new Ada.Sequential_IO(Element_Type =>
                                              Interfaces.Unsigned_8);

  -- path to the FG; NB: the FG has to be initialized prior to using SMG Comms!
  RNG_PATH: constant String := "/dev/ttyUSB0";

  -- exception raised when FG is not accessible / read fails
  FG_Failure : exception;

  -- reads O'Length octets from the FG at RNG_PATH
  procedure Get_Octets( O: out Raw_Types.Octets );
end RNG;

And the corresponding rng.adb, short and sweet:

 -- S.MG, 2018

with Ada.Exceptions; use Ada.Exceptions;

package body RNG is

  procedure Get_Octets( O: out Raw_Types.Octets ) is
    F : Octet_IO.File_Type;
  begin
    begin
      Octet_IO.Open( File => F, Mode => Octet_IO.In_File, Name => RNG_PATH );
      for I in O'Range loop
        Octet_IO.Read( F, O(I) );
      end loop;
      Octet_IO.Close(F);
    exception
      when others => Raise_Exception(FG_Failure'Identity,
                                     "Failed to access default RNG source!");
    end;
  end Get_Octets;

end RNG;

The code for doing oaep+rsa encrypt from Ada, in rsa_oaep.ads:

 -- Ada implementation of RSA with OAEP according to TMSR and Eulora spec
 -- Uses:
 --   - Eulora's raw types (Ada, raw_types.ads)
 --   - Keccak hashes (Ada, keccak.ads/adb)
 --   - OAEP schema (Ada, oaep.ads/adb)
 --   - RNG (Ada, rng.ads/adb) for true random padding
 --   - C wrappers lib (C, c_wrappers/) for:
 --       - MPI (C, mpi/)
 --       - RSA (C, rsa/rsa.c)
 --
 -- S.MG, 2018

with Raw_Types;
with Interfaces.C; use Interfaces.C;

package RSA_OAEP is
  -- exception for mismatched lengths when converting octets <-> char arrays
  Mismatched_Lengths_Error: exception;

  -- public RSA key with n,e stored as raw octets
  type RSA_pkey is
    record
      n : Raw_Types.RSA_len; --public modulus
      e : Raw_Types.RSA_half; --public exponent
    end record; --RSA_pkey

  -- private (secret) RSA key with components stored as raw octets
  type RSA_skey is
    record
      n : Raw_Types.RSA_len;  --public modulus
      e : Raw_Types.RSA_half; --public exponent
      d : Raw_Types.RSA_len;  --secret exponent e*d=1 mod phi; phi=(p-1)*(q-1)
      p : Raw_Types.RSA_half; --prime p
      q : Raw_Types.RSA_half; --prime q
      u : Raw_Types.RSA_half;  --inverse of p mod q; for faster calculations
    end record; --RSA_skey

  -- Encryption RSA+OAEP (i.e. using public key)
  -- NB: at most OAEP.MAX_LEN_MSG octets from Plain will be encrypted!
  -- Relies directly on:
  --   oaep.adb/ads
  --   c_wrappers.c ( RSA )
  --   rng.adb/ads ( random padding )
  procedure Encrypt( Plain: in Raw_Types.Octets;
                     Key  : in RSA_pkey;
                     Encr : out Raw_Types.RSA_len );

  -- Decryption RSA+OAEP (i.e. using private/secret key)
  -- The opposite of Encrypt above.
  -- NB: Plain has to have ENOUGH space for result!
  -- Result can be at most OAEP.MAX_LEN_MSG octets.
  -- Relies directly on:
  --   oaep.adb/ads
  --   c_wrappers.c (RSA)
  -- Plain_Len gives the length in OCTETS of the decrypted message.
  procedure Decrypt( Encr      : in Raw_Types.RSA_len;
                     Key       : in RSA_skey;
                     Plain     : out Raw_Types.Octets;
                     Plain_Len : out Natural;
                     Success   : out Boolean);

 --helper methods:
 -- mainly conversions to/from C's char* and imports from C_wrappers
  -- encrypt with public RSA key given as struct, Ada style
  -- NB: result is potentially 0-led (i.e. at end of Encr not at start!)
  procedure Public_RSA( Plain: in Raw_Types.Octets;
                        Key  : in RSA_pkey;
                        Encr : out Raw_Types.RSA_len);

  -- encrypt with public RSA key given as char arrays, via C_Wrappers
  -- this returns the length of result because C functions trim leading 0s!
  function Public_RSA_C( Encr      : out Interfaces.C.char_array;
                         Encr_Len  : in Integer;
                         Plain     : in Interfaces.C.char_array;
                         Plain_Len : in Integer;
                         RSA_N     : in Interfaces.C.char_array;
                         N_Len     : in Integer;
                         RSA_E     : in Interfaces.C.char_array;
                         E_Len     : in Integer)
                     return Integer;
   pragma Import(C, Public_RSA_C, "public_rsa_octets"); 

  -- decrypt with private RSA key given as struct, Ada style
  -- NB: Plain has to have ENOUGH space!
  -- NB: Result is potentially 0-led (i.e. at the end of Plain, not at start!)
  -- @return actual length of result
  procedure Private_RSA( Encr  : in Raw_Types.RSA_len;
                         Key   : in RSA_skey;
                         Plain : out Raw_Types.Octets);

  -- encrypt with private/secret RSA key given as char arrays (via C_wrappers)
  -- this returns length because C methods trim leading 0s
  function Private_RSA_C( Plain     : out Interfaces.C.char_array;
                          Plain_Len : in Integer;
                          Encr      : in Interfaces.C.char_array;
                          Encr_Len  : in Integer;
                          RSA_N     : in Interfaces.C.char_array;
                          N_Len     : in Integer;
                          RSA_E     : in Interfaces.C.char_array;
                          E_Len     : in Integer;
                          RSA_D     : in Interfaces.C.char_array;
                          D_Len     : in Integer;
                          RSA_P     : in Interfaces.C.char_array;
                          P_Len     : in Integer;
                          RSA_Q     : in Interfaces.C.char_array;
                          Q_Len     : in Integer;
                          RSA_U     : in Interfaces.C.char_array;
                          U_Len     : in Integer)
                      return Integer;
  pragma Import( C, Private_RSA_C, "private_rsa_octets" );

  -- convert from Ada's Octets (array of octets) to C's char* (char_array)

  -- This copies the octets from O to the beginning of A
  -- NB: there are NO checks or memory allocations here!
  -- Caller has to make sure that:
  --   A has allocated space for at least O'Length octets!
  procedure Octets_To_Char_Array( O   : in Raw_Types.Octets;
                                  A   : out Interfaces.C.char_array);

  -- This copies first O'Length characters from A to O
  -- NB: this does NOT allocate /check memory!
  -- Caller has to ensure that:
  --    A has space at least O'Length characters
  procedure Char_Array_To_Octets( A   : in Interfaces.C.char_array;
                                  O   : out Raw_Types.Octets);

end RSA_OAEP;

With the corresponding implementation in rsa_oaep.adb:

 -- S.MG, 2018

with Interfaces; use Interfaces;
with OAEP;
with RNG;

package body RSA_OAEP is

  -- OAEP + RSA with given public key.
  -- Steps:
  -- 1. repeat oaep.encrypt of Plain
  --      until the result has first octet < than first octet of Key's modulus.
  -- 2. RSA.encrypt with Key on the result from 1.
  procedure Encrypt( Plain: in Raw_Types.Octets;
                     Key  : in RSA_pkey;
                     Encr : out Raw_Types.RSA_len ) is
    -- block obtained through OAEP padding of given input Plain
    Blk     : OAEP.OAEP_Block := ( others => 0 );
    Entropy : OAEP.OAEP_Block;
    Len     : constant Natural := entropy'Length;
  begin
    -- loop cond: 1st octet of oaep block is < 1st octet of key's modulus
    loop
      -- get random bits via RNG (FG)
      RNG.Get_Octets( Entropy );

      -- oaep encrypt
      OAEP.OAEP_Encrypt( Plain, Entropy, Blk );

      -- check that the oaep block is suitable i.e. comparison of 1st octet
      if Blk(Blk'First) < Key.n(Key.n'First) then exit; end if;

    end loop; -- oaep block < modulus

    -- RSA encrypt
    Public_RSA( Blk, Key, Encr );
  end; --Encrypt (OAEP+RSA)

  -- RSA+OAEP Decrypt
  -- Steps:
  -- 1. RSA Decrypt (secret key)
  -- 2. OAEP Decrypt
  procedure Decrypt( Encr : in Raw_Types.RSA_len;
                     Key  : in RSA_skey;
                     Plain: out Raw_Types.Octets;
                     Plain_Len: out Natural;
                     Success: out Boolean) is
    Blk     : OAEP.OAEP_Block;
    Msg     : OAEP.OAEP_HALF;
    Msg_Len : Natural;
  begin
    -- RSA decrypt with provided secret key
    -- NB: result HAS TO BE AN OAEP BLOCK HERE! Even if - potentially - 0 led!
    Private_RSA( Encr, Key, Blk );

    -- OAEP decrypt
    OAEP.OAEP_Decrypt(Blk, Msg_Len, Msg, Success);
    -- switch to Length in OCTETS!
    Msg_Len := Msg_Len / 8;

    -- check that result FITS in given output array - otherwise, fail
    if Msg_Len > Plain'Length then
      Success := FALSE;
      Plain_Len := Msg_Len; --to give a clue to caller
    else
      Plain_Len := Msg_Len;
      Plain( Plain'First ..
             Plain'First + Plain_Len-1) := Msg( Msg'First ..
                                                Msg'First + Plain_Len -1 );
    end if;
  end Decrypt;

  -- helper methods
  -- encrypt with public RSA key given as struct, Ada style
  procedure Public_RSA( Plain: in Raw_Types.Octets;
                        Key  : in RSA_pkey;
                        Encr : out Raw_Types.RSA_len) is
    Encr_char  : char_array( size_t(Encr'First) .. size_t(Encr'Last));
    Plain_char : char_array( size_t(Plain'First) .. size_t(Plain'Last));
    N_char     : char_array( size_t(Key.n'First) .. size_t(Key.n'Last));
    E_char     : char_array( size_t(Key.e'First) .. size_t(Key.e'Last));
    out_len    : Integer;
  begin
    -- convert to char array
    Octets_To_Char_Array( Plain, Plain_char );
    Octets_To_Char_Array( Key.n, N_char );
    Octets_To_Char_Array( Key.e, E_char );
    -- call C imported function
    out_len := Public_RSA_C( Encr_char  , Encr'Length,
                             Plain_char , Plain'Length,
                             N_char     , Key.n'Length,
                             E_char     , Key.e'Length);
    -- convert back to octets
    Char_Array_To_Octets( Encr_char, Encr );
    -- C code trims leading 0s -> need to move octets if out_len0);
    end if;
    -- no need to return anything!
  end Public_RSA;

  procedure Private_RSA( Encr  : in Raw_Types.RSA_len;
                         Key   : in RSA_skey;
                         Plain : out Raw_Types.Octets) is
    Plain_Char : char_array( size_t(Plain'First) .. size_t(Plain'Last) );
    Plain_Len  : Integer;
    Encr_Char  : char_array( size_t(Encr'First)  .. size_t(Encr'Last) );
    N_Char     : char_array( size_t(Key.n'First) .. size_t(Key.n'Last) );
    E_Char     : char_array( size_t(Key.e'First) .. size_t(Key.e'Last) );
    D_Char     : char_array( size_t(Key.d'First) .. size_t(Key.d'Last) );
    P_Char     : char_array( size_t(Key.p'First) .. size_t(Key.p'Last) );
    Q_Char     : char_array( size_t(Key.q'First) .. size_t(Key.q'Last) );
    U_Char     : char_array( size_t(Key.u'First) .. size_t(Key.u'Last) );
  begin
    -- convert key and encrypted message to C char_arrays
    Octets_To_Char_Array( Encr, Encr_Char );
    Octets_To_Char_Array( Key.n, N_Char );
    Octets_To_Char_Array( Key.e, E_Char );
    Octets_To_Char_Array( Key.d, D_Char );
    Octets_To_Char_Array( Key.p, P_Char );
    Octets_To_Char_Array( Key.q, Q_Char );
    Octets_To_Char_Array( Key.u, U_Char );
    -- call RSA decrypt via C_Wrappers
    Plain_Len := Private_RSA_C( Plain_Char, Plain'Length,
                                Encr_Char , Encr'Length,
                                N_Char    , Key.n'Length,
                                E_Char    , Key.e'Length,
                                D_Char    , Key.d'Length,
                                P_Char    , Key.p'Length,
                                Q_Char    , Key.q'Length,
                                U_Char    , Key.u'Length);
    -- convert result back to Octets
    Char_Array_To_Octets( Plain_Char, Plain );
    -- if length < OAEP_Block'Length,it's 0-led and got trimmed, so move it
    if Plain_Len < Plain'Length then
      Plain(Plain'Last-Plain_Len+1..Plain'Last):= Plain(Plain'First ..
                                                   Plain'First + Plain_Len -1);
      Plain(Plain'First .. Plain'Last-Plain_Len) := (others => 0);
    end if;
    -- no need to return anything!
  end Private_RSA;

  procedure Octets_To_Char_Array( O   : in Raw_Types.Octets;
                                  A   : out Interfaces.C.char_array) is
  begin
    -- check that lengths ARE THE SAME!
    if A'Length /= O'Length then
      raise Mismatched_Lengths_Error;
    end if;
    -- copy values over octet by octet
    for I in 0 .. O'Length-1 loop
      A( A'First + Interfaces.C.size_t( I )) :=
                     Interfaces.C.Char( Character'Val(O(O'First + I)));
    end loop;
  end Octets_To_Char_Array;

  procedure Char_Array_To_Octets( A   : in Interfaces.C.char_array;
                                  O   : out Raw_Types.Octets) is
  begin
    -- check that lengths ARE THE SAME!
    if A'Length /= O'Length then
      raise Mismatched_Lengths_Error;
    end if;
    -- copy values over octet by octet
    for I in 0..O'Length -1 loop
      O( O'First + I ) := Character'Pos( Character(
                             A( A'First + Interfaces.C.size_t( I )) ));
    end loop;
  end Char_Array_To_Octets;

end RSA_OAEP;

The new oaep.ads using Raw_Types.Octets:

-- Implementation of TMSR's OAEP with Keccak as hash function
-- NB: this uses Eulora's protocol constants (esp. RSA key length) and types.
--
-- S.MG, 2018

with Keccak;    -- Keccak is used as hash function
with Raw_Types; -- Eulora's protocol raw types and constant values

with Interfaces; use Interfaces; -- for Unsigned_8 type and bit-level ops

package OAEP is
  pragma Pure( OAEP ); -- stateless, no side effects -> can cache calls

  -- constants for OAEP
  OAEP_LENGTH_OCTETS : constant := Raw_Types.RSA_KEY_OCTETS;
  OAEP_LENGTH_BITS   : constant := OAEP_LENGTH_OCTETS * 8;
  OAEP_HALF_OCTETS   : constant := OAEP_LENGTH_OCTETS / 2;
  TMSR_STR           : constant String := "TMSR-RSA";
    -- "TMSR-RSA" as unsigned_8 values:
  TMSR               : constant Raw_Types.Octets := (84,77,83,82,45,82,83,65);
  MAX_LEN_MSG        : constant Natural := OAEP_HALF_OCTETS -
                                           TMSR_STR'Length - 3;

  -- subtypes for OAEP encrypt/decrypt
  subtype OAEP_Block is Raw_Types.Octets( 1 .. OAEP_LENGTH_OCTETS );
  subtype OAEP_HALF is Raw_Types.Octets( 1 .. OAEP_HALF_OCTETS );

  -- padding & formatting of maximum MAX_LEN_MSG octets of the given input
  -- uses TMSR's OAEP schema:
  -- 1.format M00 as: [random octet][sz1][sz2]"TMSR-RSA"[random]*Message
  --    where sz1 and sz2 store the length of the message in bits
  --    the random octets before message are padding to make OAEP_LENGTH_OCTETS
  -- 2. R = OAEP_HALF_OCTETS random bits
  -- 3. X = M00 xor hash(R)
  -- 4. Y = R xor hash(X)
  -- 5. Result is X || Y
  -- NB: the Entropy parameter should be random octets from which this method
  -- will use as many as required for the OAEP encryption of given Msg
  -- NB: at MOST MAX_LEN_MSG octets of Msg! (Msg at most MAX_LEN_MSG*8 bits!)
  procedure OAEP_Encrypt( Msg      : in Raw_Types.Octets;
                          Entropy  : in OAEP_Block;
                          Output   : out OAEP_Block);

  -- This is the opposite of OAEP_Encrypt above.
  -- @param Encr - an OAEP block previously obtained from OAEP_Encrypt
  -- @param Len - this will hold the length of the obtained message (in bits!)
  -- @param Output - the first Len octets of this are the recovered message
  -- @param Success - set to TRUE if message was recovered, false otherwise
  -- NB: when Success is FALSE, both Len and Output have undefined values
  procedure OAEP_Decrypt( Encr    : in OAEP_Block;
                          Len     : out Natural;
                          Output  : out OAEP_HALF;
                          Success : out Boolean);

private
  -- gnat-specific methods for bit-level operations
	function Shift_Right( Value  : Unsigned_8;
                        Amount : Natural )
                        return Unsigned_8;
  pragma Import(Intrinsic, Shift_Right); 

	function Shift_Left( Value  : Unsigned_8;
                        Amount : Natural )
                        return Unsigned_8;
  pragma Import(Intrinsic, Shift_Left); 

  -- helper method: xor 2 half-oaep blocks
  function XOR_Octets(A : in OAEP_HALF;
                      B : in OAEP_HALF)
                     return OAEP_HALF;

  -- conversions between bitstream and string
  -- NB: caller has to ensure correct size of output parameter! no checks here.
  procedure ToOctets   ( B : in Keccak.Bitstream;
                         O : out Raw_Types.Octets );

  procedure ToBitstream( O : in Raw_Types.Octets;
                         B : out Keccak.Bitstream );

  -- wrapper for Sponge to use Octets for input/output
  procedure HashKeccak( Input     : in Raw_Types.Octets;
                        Output    : out Raw_Types.Octets;
                        Block_Len : in Keccak.Keccak_Rate :=
                                       Keccak.Default_Bitrate);

end OAEP;

And the implementation in oaep.adb:

-- S.MG, 2018

package body OAEP is

  -- padding & formatting of maximum MAX_LEN_MSG*8 bits of the given input
  -- uses TMSR's OAEP schema:
  -- 1.format M00 as: [random octet][sz1][sz2]"TMSR-RSA"[random]*Message
  --    where sz1 and sz2 store the length of the message in bits
  --    the random octets before message are padding to make OAEP_LENGTH_OCTETS
  -- 2. R = OAEP_HALF_OCTETS random bits
  -- 3. X = M00 xor hash(R)
  -- 4. Y = R xor hash(X)
  -- 5. Result is X || Y
  -- NB: the Entropy parameter should be random octets from which this method
  -- will use as many as required for the OAEP encryption of given Msg
  -- NB: at MOST MAX_LEN_MSG octets of Msg! (Msg at most 1960 bits)
  procedure OAEP_Encrypt( Msg     : in Raw_Types.Octets;
                          Entropy : in OAEP_Block;
                          Output  : out OAEP_Block) is
    M00    : OAEP_HALF;
    R      : OAEP_HALF;
    HashR  : OAEP_HALF;
    X      : OAEP_HALF;
    HashX  : OAEP_HALF;
    Y      : OAEP_HALF;
    MsgLen : Natural;
    PadLen : Natural;
  begin
    -- calculate maximum length of msg and needed amount of padding
    -- make sure also that only MAX_LEN_MSG octets at most are used from Msg
    MsgLen := Msg'Length;                          -- real msg length
    if MsgLen > MAX_LEN_MSG then
      MsgLen := MAX_LEN_MSG;  --only first MAX_LEN_MSG octets used
      PadLen := 0;            --no padding needed
    else
      PadLen := MAX_LEN_MSG - MsgLen; -- add padding as needed
    end if;

    -- step 1: header and format to obtain M00
      -- first octet is random bits
    M00( M00'First ) := Entropy( Entropy'First );

      -- next 2 octets hold the used length of Msg (number of octets)
    M00( M00'First + 2) := Unsigned_8( ( MsgLen * 8 ) mod 256 );
    M00( M00'First + 1) := Unsigned_8( ( (MsgLen * 8 ) / 256 ) mod 256 );

      -- next 8 octets are reserved for later use, currently "TMSR-RSA"
    M00( M00'First + 3 .. M00'First + 10 ) := TMSR;

      -- random bits for padding, if Msg is less than maximum length
    for I in 1 .. PadLen loop
      M00( M00'First + 10 + I ) := Entropy( Entropy'First + I );
    end loop;

      -- the message itself
    M00( M00'Last - MsgLen + 1 .. M00'Last ) :=
                               Msg( Msg'First .. Msg'First + MsgLen - 1 );

    -- step 2: R = Raw_Types.OAEP_HALF_OCTETS random octets
    -- can take LAST octets from given entropy as they are NOT used before
    -- (even if original message was empty, padding uses at most half - 10
    --   while entropy has full block length)
    R := Entropy( Entropy'Last - OAEP_HALF_OCTETS + 1 .. Entropy'Last );

    -- step 3: X = M00 xor hash(R)
    HashKeccak( R, HashR );
    X := XOR_Octets(M00, HashR);

    -- step 4: Y = R xor hash(X)
    HashKeccak( X, HashX );
    Y := XOR_Octets(R, HashX);

    -- step 5: Output is X || Y
    Output( Output'First .. Output'First + X'Length - 1 ) := X;
    Output( Output'Last - Y'Length + 1 .. Output'Last )   := Y;

  end OAEP_Encrypt;

  procedure OAEP_Decrypt( Encr    : in OAEP_Block;
                          Len     : out Natural;
                          Output  : out OAEP_HALF;
                          Success : out Boolean ) is
    X, Y, M, R   : OAEP_HALF;
    HashX, HashR : OAEP_HALF;
    LenOctets    : Natural;
  begin
    -- step 1: separate X and Y
    X := Encr( Encr'First .. Encr'First + X'Length - 1 );
    Y := Encr( Encr'Last - Y'Length + 1 .. Encr'Last );

    -- step 2: R = Y xor hash(X)
    HashKeccak( X, HashX );
    R := XOR_Octets(Y, HashX);

    -- step 3: M = X xor hash(R)
    HashKeccak( R, HashR );
    M := XOR_Octets(X, HashR);

    -- step 4: extract length and message
    Len := Natural(M( M'First + 1 )) * 256 +
           Natural(M( M'First + 2 ));
    LenOctets := Len / 8; 

    if LenOctets > MAX_LEN_MSG or LenOctets < 0 then
      Success := False;  -- error, failed to retrieve message
    else
      Success := True;
      Output( Output'First .. Output'First + LenOctets - 1 ) :=
        M( M'Last - LenOctets + 1 .. M'Last );
    end if;

  end OAEP_Decrypt;

-- private, helper methods
  procedure HashKeccak(Input     : in Raw_Types.Octets;
                       Output    : out Raw_Types.Octets;
                       Block_Len : in Keccak.Keccak_Rate :=
                                      Keccak.Default_Bitrate) is
    BIn  : Keccak.Bitstream( 0 .. Input'Length * 8 - 1 );
    BOut : Keccak.Bitstream( 0 .. Output'Length * 8 - 1 );
  begin
    ToBitstream( Input, BIn );
    Keccak.Sponge( BIn, BOut, Block_Len );
    ToOctets( BOut, Output );
  end HashKeccak;

  function XOR_Octets(A : in OAEP_HALF;
                      B : in OAEP_HALF)
                     return OAEP_HALF is
    R : OAEP_HALF;
  begin
    for I in R'Range loop
      R(I) := A(I) xor B(I);
    end loop;
    return R;
  end XOR_Octets;

  -- conversion between types
  procedure ToOctets(B: in Keccak.Bitstream; O: out Raw_Types.Octets ) is
    Pos : Natural;
  begin
    Pos := B'First;
    for I in O'Range loop
      O(I) := Unsigned_8( B( Pos     ) ) +
              Unsigned_8( B( Pos + 1 ) ) * 2 +
              Unsigned_8( B( Pos + 2 ) ) * 4 +
              Unsigned_8( B( Pos + 3 ) ) * 8 +
              Unsigned_8( B( Pos + 4 ) ) * 16 +
              Unsigned_8( B( Pos + 5 ) ) * 32 +
              Unsigned_8( B( Pos + 6 ) ) * 64 +
              Unsigned_8( B( Pos + 7 ) ) * 128;
      Pos := Pos + 8;
    end loop;
  end ToOctets;

  procedure ToBitstream(O: in Raw_Types.Octets; B: out Keccak.Bitstream ) is
    V   : Unsigned_8;
    Pos : Natural;
  begin
    Pos := B'First;
    for I in O'Range loop
      V := O( I );
      B( Pos     ) := Keccak.Bit( V and 1 );
      B( Pos + 1 ) := Keccak.Bit( Shift_Right( V, 1 ) and 1 );
      B( Pos + 2 ) := Keccak.Bit( Shift_Right( V, 2 ) and 1 );
      B( Pos + 3 ) := Keccak.Bit( Shift_Right( V, 3 ) and 1 );
      B( Pos + 4 ) := Keccak.Bit( Shift_Right( V, 4 ) and 1 );
      B( Pos + 5 ) := Keccak.Bit( Shift_Right( V, 5 ) and 1 );
      B( Pos + 6 ) := Keccak.Bit( Shift_Right( V, 6 ) and 1 );
      B( Pos + 7 ) := Keccak.Bit( Shift_Right( V, 7 ) and 1 );

      Pos := Pos + 8;
    end loop;
  end ToBitstream;

end OAEP;

The new tests added to tests/test_rsa_oaep.adb and called from tests/testall.adb, basically calling naked rsa from Ada, OAEP by itself and then rsa+oaep on a few different inputs:

 --S.MG, 2018

with Interfaces; use Interfaces;
with Interfaces.C; use Interfaces.C;
with RSA_OAEP; use RSA_OAEP;
with OAEP; use OAEP;
with Raw_Types; use Raw_Types;
with RNG; use RNG;
with Keccak; use Keccak;

with Ada.Text_IO; use Ada.Text_IO;

package body Test_RSA_OAEP is

  procedure test_char_array is
    S : String := OAEP.TMSR_STR;
    O : Octets := OAEP.TMSR;
    A : char_array(0..O'Length-1) := (others => '0');
    B : Octets(0..O'Length -1) := (others => 0);
    Fail : Boolean := FALSE;
  begin
    Octets_To_Char_Array(O, A);
    Char_Array_To_Octets(A, B);

    if B /= O then
      Put_Line("FAIL: char_array_to_octets");
    else
      Put_Line("PASS: char_array_to_octets");
    end if;

    for I in 0..S'Length-1 loop
      declare
        C : Character := Character(A(A'First + size_t(I)));
        E : Character := S(S'First + I);
      begin
        if C /= E then
          Fail := TRUE;
          Put("Error at pos " & Integer'Image(I) & ": ");
          Put(Integer'Image(Character'Pos(C)));
          Put_Line(" instead of " & Integer'Image(Character'Pos(E)));
        end if;
      end;
    end loop;
    if FAIL then
      Put_Line("FAIL: test octets_to_char_array");
    else
      Put_Line("PASS: test octets_to_char_array");
    end if;
  end test_char_array;

  -- test OAEP encrypt + decrypt
  procedure test_oaep is
    Plain: Octets(1..MAX_LEN_MSG);
    Short: Octets(0..10);
    Encr : OAEP_Block;
    Decr : OAEP_HALF;
    Len  : Natural;
    Entropy: OAEP_Block;
    Success : Boolean;
  begin
    RNG.Get_Octets(Plain);
    RNG.Get_Octets(Entropy);
    RNG.Get_Octets(Short);

    -- test full length message
    OAEP_Encrypt(Plain, Entropy, Encr);
    OAEP_Decrypt(Encr, Len, Decr, Success);

    if not Success or Len/8 /= Plain'Length then
      Put_Line("FAIL: oaep encrypt/decrypt on max len message.");
    else
      if Decr(Decr'First..Decr'First+Len/8-1) /=
              Plain(Plain'First..Plain'First+Len/8-1) then
        Put_Line("FAIL: oaep encrypt/decrypt on max len message - " &
                 "result different from expected.");
      else
        Put_Line("PASS: oaep encrypt/decrypt on max len message.");
      end if;
    end if;

    -- test short message
    OAEP_Encrypt(Short, Entropy, Encr);
    OAEP_Decrypt(Encr, Len, Decr, Success);
    if not Success or Len/8 /= Short'Length then
      Put_Line("FAIL: oaep encrypt/decrypt on short message.");
    else
      if Decr(Decr'First..Decr'First+Len/8-1) /=
             Short(Short'First..Short'First+Len/8-1) then
        Put_Line("FAIL: oaep encrypt/decrypt on short message - " &
                 "result different from expected.");
      else
        Put_Line("PASS: oaep encrypt/decrypt on short message.");
      end if;
    end if;

  end test_oaep;

  -- test JUST RSA (i.e. without oaep) with RSA key pair previously generated
  procedure test_rsa is
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
    Plain: OAEP_Block := (others => 0);
    Decr : OAEP_Block := (others => 0);
    Encr : RSA_len;
    pkey: RSA_pkey;
    skey: RSA_skey;
  begin
    -- initialize with RSA pair previously generated
    Hex2Octets( n, skey.n );
    Hex2Octets( e, skey.e );
    Hex2Octets( d, skey.d );
    Hex2Octets( p, skey.p );
    Hex2Octets( q, skey.q );
    Hex2Octets( u, skey.u );
    -- copy n and e for public key
    pkey.n := skey.n;
    pkey.e := skey.e;
    -- get random data
    RNG.Get_Octets(Plain);
    -- make first octet < RSA key's modulus first octet
    Plain(Plain'First) := 16#00#;
    -- naked rsa encrypt/decrypt
Put_Line("Encrypting with RSA public key...");
    Public_RSA( Plain, pkey, Encr );
Put_Line("Decrypting with RSA private key...");
    Private_RSA( Encr, skey, Decr );
Put_Line("Checking...");

    -- check result
    if Decr /= Plain then
      Put_Line("FAIL: RSA encrypt/decrypt result doesn't match plain.");
    else
      Put_Line("PASS: RSA encrypt/decrypt");
    end if;
  end test_rsa;

  -- test rsa+oaep with RSA key pair previously generated
  procedure test_rsa_oaep is
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
    Plain: Octets(1..MAX_LEN_MSG) := (others=>20);
    Short: Octets(1..10);
    Decr : RSA_len;
    Encr : RSA_len;
    pkey: RSA_pkey;
    skey: RSA_skey;
    Success: Boolean;
    Len : Natural;
  begin
    -- initialize with RSA pair previously generated
    Hex2Octets( n, skey.n );
    Hex2Octets( e, skey.e );
    Hex2Octets( d, skey.d );
    Hex2Octets( p, skey.p );
    Hex2Octets( q, skey.q );
    Hex2Octets( u, skey.u );
    -- copy n and e for public key
    pkey.n := skey.n;
    pkey.e := skey.e;

    -- test with 0 message of length Plain'Length
    RSA_OAEP.Encrypt(Plain, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Plain'Length
         or Plain /= Decr(Decr'First..Decr'First+Plain'Length-1) then
      Put_Line("FAIL: RSA_OAEP on max len message 20-filled.");
    else
      Put_Line("PASS: RSA_OAEP on max len message 20-filled.");
    end if;

    -- get random data for "plain" message
    RNG.Get_Octets(Plain);
    RSA_OAEP.Encrypt(Plain, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Plain'Length
         or Plain /= Decr(Decr'First..Decr'First+Plain'Length-1) then
      Put_Line("FAIL: RSA_OAEP on random data of max length.");
    else
      Put_Line("PASS: RSA_OAEP on random data of max length.");
    end if;

    -- get random data for "short" message
    RNG.Get_Octets(Short);
    RSA_OAEP.Encrypt(Short, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Short'Length
         or Short /= Decr(Decr'First..Decr'First+Short'Length-1) then
      Put_Line("FAIL: RSA_OAEP on random data of short length.");
    else
      Put_Line("PASS: RSA_OAEP on random data of short length.");
    end if; 

  end test_rsa_oaep;

-- helper methods
  procedure Hex2Octets( Hex: in String; O: out Raw_Types.Octets ) is
    S : String := "16#AA#";
    -- to make sure that input String has EVEN number of chars (ie full octets)
    H : String(1..Hex'Length+Hex'Length mod 2) := (others=>'0');
  begin
    -- first char is 0 if needed to cover full octet...
    H(H'Length-Hex'Length+1..H'Length) := Hex;
    O := (others => 0);
    for I in 0 .. H'Length/2-1 loop
      S := "16#" & H(H'First + I*2 .. H'First + I*2 + 1) & "#";
      O(O'Last - H'Length/2 + 1 + I) := Unsigned_8'Value(S);
    end loop;
  end Hex2Octets;

  procedure PrintOctets( O: in Raw_Types.Octets; Title: in String ) is
  begin
    Put_Line(Title);
    for V of O loop
      Put(Unsigned_8'Image(V) & " ");
    end loop;
    New_Line;
  end PrintOctets;

end Test_RSA_OAEP;

The full .vpatch for this chapter is quite long at 1714 lines but a significant part of that is essentially imported from EuCrypt (Keccak most notably but also the main part of OAEP). As usual, the .vpatch and my signature for it can be found on my Reference Code Shelf or directly from the links below:


  1. Yes, I adopted Stanislav's approach to reading from the FG. I also had a look at his code at www.loper-os.org/?p=2175  

  2. The exact line is this one: http://www.dianacoman.com/2018/01/18/eucrypt-chapter-6-keccak-transformations/#selection-159.735-159.812 and it's been pointed out already (in the logs, where else?) that there is an implicit check against div by 0 which could be avoided simply by rewriting the line as Output(X, Y) := Rotate_Left(Input(X,Y), ( (T+1)*(T+2)/2) and (Z_Length -1) );  

October 25, 2018

SMG Comms Chapter 4: C Wrappers for RSA and MPI

Filed under: Coding, SMG_Comms — Diana Coman @ 7:28 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

The C part of SMG Comms is - unsurprisingly - a pain in the ass and a three-pointed thorn in the backside:
1. All the RSA operations are coded in C because of the heavy dependency on the MPI lib. This is not something I can change at the moment since I don't yet have an Ada implementation of long integer arithmetics. Sigh.
2. The OAEP padding schema has to be called repeatedly until the result is indeed (when considered as a very large number) a value smaller than the RSA key's modulus. So although the OAEP padding itself (including underlying Keccak are implemented in Ada), the comparison between obtained value and the modulus of the key goes through the MPI swamp so it forces a call to C. Nevertheless, the "implemented in Ada" and "repeat until the result CAN be encrypted with given RSA key" are both absolutely right and proper and as they should be so there's nothing to change here. Hooray.
3. As a result of 1 and 2 above, the code flow can easily get ugly because one ends up with calls from Ada to C (to do RSA operations for instance) but *also* from C to Ada (to do Keccak, if one aims to provide oaep+rsa from C directly). This is currently the sad state of the wrappers provided as first attempt in EuCrypt for rsa-oaep. Nevertheless, it's horribly ugly, difficult to follow and overall unsanitary so it has to go.

My current solution to the above issue with rsa-oaep is to isolate properly the C code and stick to calls in only one direction, namely from Ada to C. To do this however, I first need to write some more ...C code. Eurgh. The new C code is a set of wrappers for RSA and MPI methods that I need to call from Ada (essentially from what will be Ada implementation of the rsa-with-oaep part): those wrappers provide a sane interface that matches SMG Comms' raw types layer - basically those methods can be called with arrays of octets as input/output without forcing onto the rest of the code MPI structures, unpredictable memory allocation and other abominations.

The above approach has two very important benefits: the code is much easier to follow since there is no ugly back and forth between Ada and C; the C part (MPI especially but also the RSA layer) is properly separated from the Ada part so that it should be relatively easy to simply replace it with a sane Ada implementation whenever such a wonder becomes available. The main downside to this is that it adds conversions from arrays of octets to MPIs and back as part and parcel of using the wrappers but I still think it's the better option available at this moment.

The C wrappers that I need for SMG Comms include the comparison of 2 MPI values (so mpi_cmp from MPI lib), the encryption and decryption with given RSA key as well as a procedure to generate a RSA key (from rsa lib). In addition, I wrote an mpi_to_octets function that does as the name suggests: reads the raw octets stored in the MPI and returns them to the caller. The reason for this to feature as a separate function is the sad fact that MPI's own "get buffer" allocates memory and then expects the caller to de-allocate it. Rather than import this headache and propagate it all through the new code, I wrapped it up in here and dealt with it: callers of mpi_to_octets should allocate their own memory and de-allocate it accordingly. The signatures of those wrapper methods are in all their enumerated-arrays-of-octets glory (RSA key components are passed as array of octets just as well as anything else) in c_wrappers/c_wrappers.c:

//Wrapper methods for C implementations of RSA and MPI.
//To be used / called from Ada so that the C part remains well separated and
//  can therefore be swapped easily at a later stage for something sane.
//S.MG, 2018

#include "mpi.h"

//Comparison of 2 arrays of octets interpreted as MPIs.
//This method creates 2 MPIs out of the given arrays of octes and then
//  calls the mpi_cmp method from mpi/mpi-cmp.c, returning its result.
// ************************************************************************
// ***NB: Make SURE that a and b have indeed allocated at least len_a and
// *****    len_b octets respectively! NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param a An array of octets representing the first MPI.
//@param len_a The length of the first array (number of octets).
//@param b An array of octets representing the second MPI.
//@param len_b The length of the second array (number of octets).
//@return 0 when a=b, -1 when ab
int mpi_cmp_octets(char *a, unsigned int len_a, char *b, unsigned int len_b);

//Encryption of given octets with public RSA key: n and e given as octets too.
// ************************************************************************
// ***Length of output is KEY_LENGTH_OCTETS.
// ***NB: Make SURE that out, input, n and e have enough space allocated!!
// ***NB: NO MEMORY ALLOCATED for its parameters by this method and
// *****    NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param out Pointer to ALREADY ALLOCATED space for the encrypted data.
//@param len_out Length of the allocated space for out (in octets).
//@param input Pointer to the data to be encrypted.
//@param len_input Length of the allocated space for input (in octets).
//@param n Pointer to the public RSA modulus to use for encryption.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent to use for encryption.
//@param len_e Length of e (in octets).
//@return The actual length of the output i.e. number of chars written to out.
int public_rsa_octets( char *out  , unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e);

//Encryption of given octets with *private* RSA key given as octets.
// ************************************************************************
// ***Length of output is KEY_LENGTH_OCTETS.
// ***NB: Make SURE that ALL pointers have enough space allocated!!
// ***NB: NO MEMORY ALLOCATED for its parameters by this method and
// *****    NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param out Pointer to ALREADY ALLOCATED space for the encrypted data.
//@param len_out Length of the allocated space for out (in octets).
//@param input Pointer to the data to be encrypted.
//@param len_input Length of the allocated space for input (in octets).
//@param n Pointer to the public RSA modulus of the given key.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent of the given key.
//@param len_e Length of e (in octets).
//@param d Pointer to the private RSA exponent of the given key.
//@param len_d Length of d (in octets).
//@param p Pointer to the prime p of the given key.
//@param len_p Length of p (in octets).
//@param q Pointer to the prime q of the given key.
//@param len_q Length of q (in octets).
//@param u Pointer to the inverse of p mod q for the given key.
//@param len_u Length of u (in octets).
//@return The actual length of the output i.e. number of chars written to out.
int private_rsa_octets( char *out, unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e,
                        char *d    , unsigned int len_d,
                        char *p    , unsigned int len_p,
                        char *q    , unsigned int len_q,
                        char *u    , unsigned int len_u);

//Generates a new RSA key and stores its components at the specified locations.
//@param n Pointer to the public RSA modulus of the given key.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent of the given key.
//@param len_e Length of e (in octets).
//@param d Pointer to the private RSA exponent of the given key.
//@param len_d Length of d (in octets).
//@param p Pointer to the prime p of the given key.
//@param len_p Length of p (in octets).
//@param q Pointer to the prime q of the given key.
//@param len_q Length of q (in octets).
//@param u Pointer to the inverse of p mod q for the given key.
//@param len_u Length of u (in octets).
void gen_rsa_octets( char *n, unsigned int *len_n,
                     char *e, unsigned int *len_e,
                     char *d, unsigned int *len_d,
                     char *p, unsigned int *len_p,
                     char *q, unsigned int *len_q,
                     char *u, unsigned int *len_u);

//Copies the buffer of m to the location to which out points.
//*****************************************************************
//*** This method does NOT allocate memory for out!
//*** NB: caller should allocate enough memory!
//*****************************************************************
//@param out pointer to allocated memory where to copy the MPI's octets
//@param len_out size of out; will be replaced by actual number of octets copied
//@param m The MPI whose octets are to be retrieved
void mpi_to_octets( char *out, unsigned int *len_out, MPI m);

Corresponding code for the wrapper methods:

//Wrapper methods for C implementations of RSA and MPI.
//To be used / called from Ada so that the C part remains well separated.
//S.MG, 2018

#include "c_wrappers.h"
#include "mpi.h"
#include "smg_rsa.h"
#include <assert.h>
#include <string.h> //for memmove...

//Wrapper for comparing the given arrays of octets as MPIs
//Steps:
//1. allocate space for 2 MPIs u and v
//2. set buffer of u to content of a, buffer of v to content of b
//3. call mpi_cmp(u,v) and store result
//4. de-allocate u and v
//5. return result
int mpi_cmp_octets(char *a, unsigned int len_a, char *b, unsigned int len_b) {
  //variable to hold the final result of comparing a to b as MPIs
  int result;

  //calculate how much space is needed for each MPI
	unsigned int nlimbs_a = mpi_nlimb_hint_from_nbytes( len_a );
	unsigned int nlimbs_b = mpi_nlimb_hint_from_nbytes( len_b );

  //allocate space for the 2 MPIs
	MPI u = mpi_alloc(nlimbs_a);
	MPI v = mpi_alloc(nlimbs_b);

  //set the given octets as the values of the 2 MPIs
  //the sign is set to 0 (last parameter).
  mpi_set_buffer(u, a, len_a, 0);
  mpi_set_buffer(v, b, len_b, 0);

  //compare the MPIs as numbers and store the result
  result = mpi_cmp(u, v);

  //tidy up: free the memory allocated for the 2 MPIs
  mpi_free(u);
  mpi_free(v);

  //return the result comparing a to b as MPIs
  return result;
}

//Encryption of given input with a public RSA key: n and e given as octets too.
//Steps:
//1. create (allocate memory for) MPIs for out and input;
//2. set the input as buffer for the corresponding MPI;
//3. create and set the public key structure with n,e as contents;
//4. call rsa/public_rsa and retrieve the result storing it in out;
//5. free allocated memory for all the MPIs.
//6. return the actual length of the encrypted result
int public_rsa_octets( char *out  , unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e) {

  // precondition: output has enough memory allocated
  assert( len_out >= KEY_LENGTH_OCTETS );

  //allocate memory for input and output MPIs
	unsigned int nlimbs_in  = mpi_nlimb_hint_from_nbytes( len_input );
	unsigned int nlimbs_out = mpi_nlimb_hint_from_nbytes( len_out   );
	MPI in_mpi  = mpi_alloc(nlimbs_in);
	MPI out_mpi = mpi_alloc(nlimbs_out);

  //set input as buffer for in_mpi
  mpi_set_buffer(in_mpi, input, len_input, 0);

  //create public key structure and set its contents to given n, e
  RSA_public_key pk;
	unsigned int nlimbs_n = mpi_nlimb_hint_from_nbytes( len_n );
	unsigned int nlimbs_e = mpi_nlimb_hint_from_nbytes( len_e );
  pk.n = mpi_alloc(nlimbs_n);
  pk.e = mpi_alloc(nlimbs_e);
  mpi_set_buffer(pk.n, n, len_n, 0);
  mpi_set_buffer(pk.e, e, len_e, 0);

  //call rsa public_key encryption and retrieve the result, storing it in out
  public_rsa( out_mpi, in_mpi, &pk);
  int len = len_out;
  mpi_to_octets( out, &len, out_mpi );

  //tidy up: free allocated memory for ALL MPIs.
  mpi_free(in_mpi);
  mpi_free(out_mpi);
  mpi_free(pk.n);
  mpi_free(pk.e);

  //return actual length
  return len;
}

//Decryption of given input with the private key given through its components.
//Steps:
//1. create (allocate memory for) MPIs for out and input;
//2. set the input as buffer for the corresponding MPI;
//3. create and set the private key structure with n,e,d,p,q,u as contents;
//4. call rsa/private_rsa and retrieve the result storing it in out;
//5. free allocated memory for all the MPIs.
//6. return the actual length of the result
int private_rsa_octets( char *out, unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e,
                        char *d    , unsigned int len_d,
                        char *p    , unsigned int len_p,
                        char *q    , unsigned int len_q,
                        char *u    , unsigned int len_u) {
  // precondition: output has enough memory allocated
  assert( len_out >= KEY_LENGTH_OCTETS );

  //allocate memory for input and output MPIs
	unsigned int nlimbs_in  = mpi_nlimb_hint_from_nbytes( len_input );
	unsigned int nlimbs_out = mpi_nlimb_hint_from_nbytes( len_out   );
	MPI in_mpi  = mpi_alloc(nlimbs_in);
	MPI out_mpi = mpi_alloc(nlimbs_out);

  //set input as buffer for in_mpi
  mpi_set_buffer(in_mpi, input, len_input, 0);

  //create private key structure and set its contents to given n,e,d,p,q,u
  RSA_secret_key sk;
	unsigned int nlimbs_n = mpi_nlimb_hint_from_nbytes( len_n );
	unsigned int nlimbs_e = mpi_nlimb_hint_from_nbytes( len_e );
	unsigned int nlimbs_d = mpi_nlimb_hint_from_nbytes( len_d );
	unsigned int nlimbs_p = mpi_nlimb_hint_from_nbytes( len_p );
	unsigned int nlimbs_q = mpi_nlimb_hint_from_nbytes( len_q );
	unsigned int nlimbs_u = mpi_nlimb_hint_from_nbytes( len_u );
  sk.n = mpi_alloc(nlimbs_n);
  sk.e = mpi_alloc(nlimbs_e);
  sk.d = mpi_alloc(nlimbs_d);
  sk.p = mpi_alloc(nlimbs_p);
  sk.q = mpi_alloc(nlimbs_q);
  sk.u = mpi_alloc(nlimbs_u);
  mpi_set_buffer(sk.n, n, len_n, 0);
  mpi_set_buffer(sk.e, e, len_e, 0);
  mpi_set_buffer(sk.d, d, len_d, 0);
  mpi_set_buffer(sk.p, p, len_p, 0);
  mpi_set_buffer(sk.q, q, len_q, 0);
  mpi_set_buffer(sk.u, u, len_u, 0);

  //call rsa secret_key encryption and retrieve the result, storing it in out
  secret_rsa( out_mpi, in_mpi, &sk );
  int len = len_out;
  mpi_to_octets( out, &len, out_mpi );

  //tidy up: free memory previously allocated for MPIs
  mpi_free(in_mpi);
  mpi_free(out_mpi);
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  //return number of octets copied in out - real length of result
  return len;
}

//Generates a new RSA key and stores its components at the specified locations.
void gen_rsa_octets( char *n, unsigned int *len_n,
                     char *e, unsigned int *len_e,
                     char *d, unsigned int *len_d,
                     char *p, unsigned int *len_p,
                     char *q, unsigned int *len_q,
                     char *u, unsigned int *len_u) {
  // precondition: all pointers have enough memory allocated
  assert( *len_n >= KEY_LENGTH_OCTETS );
  assert( *len_e >= KEY_LENGTH_OCTETS );
  assert( *len_d >= KEY_LENGTH_OCTETS );
  assert( *len_p >= KEY_LENGTH_OCTETS / 2);
  assert( *len_q >= KEY_LENGTH_OCTETS / 2);
  assert( *len_u >= KEY_LENGTH_OCTETS / 2);

  //the secret key structure that will hold generated key components
  RSA_secret_key sk;
	int nlimbs = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS );
	int nlimbs_pq = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS / 2 );

	sk.n = mpi_alloc(nlimbs);
	sk.e = mpi_alloc(nlimbs);
	sk.d = mpi_alloc(nlimbs);
	sk.p = mpi_alloc(nlimbs_pq);
	sk.q = mpi_alloc(nlimbs_pq);
	sk.u = mpi_alloc(nlimbs_pq);

  //generate the rsa key pair - this may take a while!
  gen_keypair(&sk);

  //copy components to their place
  mpi_to_octets( n, len_n, sk.n );
  mpi_to_octets( e, len_e, sk.e );
  mpi_to_octets( d, len_d, sk.d );
  mpi_to_octets( p, len_p, sk.p );
  mpi_to_octets( q, len_q, sk.q );
  mpi_to_octets( u, len_u, sk.u );

  //tidy up: free ALL MPIs
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);
}

void mpi_to_octets( char *out, unsigned int *len_out, MPI m) {
  //copy the components as raw octets to the given pointers
  int len = 0;
  int sign;
  unsigned char * buffer = mpi_get_buffer( m, &len, &sign );

  //check and don't copy MORE than there is allocated space in out!
  assert( len <= *len_out );
  memmove( out, buffer, len );
  *len_out = len;    //save actual length of the component

  xfree( buffer ); //free the buffer that was allocated by mpi_get_buffer
}

The basic tests for the new C wrappers are the following:

//S.MG, 2018

#include "mpi.h"
#include "smg_rsa.h"

#include "c_wrappers.h"

void test_mpi_cmp() {
  int result;
  int i;
  char a[KEY_LENGTH_OCTETS];
  char b[KEY_LENGTH_OCTETS];

  //initialize mpis
  for (i=0;i b
  a[240] = 241;
  result = mpi_cmp_octets(a, KEY_LENGTH_OCTETS, b, KEY_LENGTH_OCTETS);
  if (result == 1)
    printf("PASS: mpi_cmp_octets on a > b.\n");
  else {
    printf("FAIL: mpi_cmp_octets on a > b ");
    printf("returned %d instead of 1.\n", result);
  }

  //tidy up
}

void test_gen_rsa_octets() {
  RSA_secret_key sk;
  RSA_public_key pk;
	int nlimbs = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS );
	int nlimbs_pq = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS / 2 );

  //allocate memory
	sk.n = mpi_alloc(nlimbs);
	sk.e = mpi_alloc(nlimbs);
	sk.d = mpi_alloc(nlimbs);
	sk.p = mpi_alloc(nlimbs_pq);
	sk.q = mpi_alloc(nlimbs_pq);
	sk.u = mpi_alloc(nlimbs_pq);

	pk.n = mpi_alloc(nlimbs);
	pk.e = mpi_alloc(nlimbs);

  //generate key pair
  int len_n = KEY_LENGTH_OCTETS;
  int len_e = len_n;
  int len_d = len_n;
  int len_p = KEY_LENGTH_OCTETS / 2;
  int len_q = len_p;
  int len_u = len_p;
  char n[KEY_LENGTH_OCTETS];
  char e[KEY_LENGTH_OCTETS];
  char d[KEY_LENGTH_OCTETS];
  char p[KEY_LENGTH_OCTETS / 2];
  char q[KEY_LENGTH_OCTETS / 2];
  char u[KEY_LENGTH_OCTETS / 2];
  gen_rsa_octets(n, &len_n,
                 e, &len_e,
                 d, &len_d,
                 p, &len_p,
                 q, &len_q,
                 u, &len_u);

  //check encryption/decr works
  mpi_set_buffer(sk.n, n, len_n, 0);
  mpi_set_buffer(sk.e, e, len_e, 0);
  mpi_set_buffer(sk.d, d, len_d, 0);
  mpi_set_buffer(sk.p, p, len_p, 0);
  mpi_set_buffer(sk.q, q, len_q, 0);
  mpi_set_buffer(sk.u, u, len_u, 0);

  mpi_set_buffer(pk.n, n, len_n, 0);
  mpi_set_buffer(pk.e, e, len_e, 0);

  MPI encr = mpi_alloc(0);
	MPI plain = mpi_alloc(0);
  MPI out = mpi_alloc(0);
	mpi_fromstr(plain, "0x\
5B6A8A0ACF4F4DB3F82EAC2D20255E4DF3E4B7C799603210766F26EF87C8980E737579\
EC08E6505A51D19654C26D806BAF1B62F9C032E0B13D02AF99F7313BFCFD68DA46836E\
CA529D7360948550F982C6476C054A97FD01635AB44BFBDBE2A90BE06F7984AC8534C3\
28097EF92F6E78CAE0CB97");
  public_rsa(encr, plain, &pk);
  secret_rsa(out, encr, &sk);

  if (mpi_cmp(out, plain) != 0)
    printf("FAIL: test_gen_rsa encr/decr failed.\n");
  else
    printf("PASS: test_gen_rsa encr/decr passed.\n");

  //tidy up
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  mpi_free(pk.n);
  mpi_free(pk.e);

  mpi_free(plain);
  mpi_free(encr);
  mpi_free(out);
}

void test_rsa_octets() {
  int noctets = 512;
	RSA_public_key pk;
	pk.n = mpi_alloc(0);
	pk.e = mpi_alloc(0);

  RSA_secret_key sk;
  sk.n = mpi_alloc(0);
  sk.e = mpi_alloc(0);
  sk.d = mpi_alloc(0);
  sk.p = mpi_alloc(0);
  sk.q = mpi_alloc(0);
  sk.u = mpi_alloc(0);

//key pair previously generated with EuCrypt
  mpi_fromstr(sk.n, "0x\
CD2C025323BEA46FFF2FA8D7A9D39817EA713421F4AE03FA8120641193892A70BFECF5\
83101635A432110D3DDE6339E3CC7ECC0AD91C026FCACE832DD3888A6FCA7BCE56C390\
5A5AC8C7BC921DA675E4B62489B254EB34659D547D71165BC998983A81937BD251AEE1\
2D985EC387D5376F5DCC5EF7EC530FBD6FD2AA7285EE1AF3335EA73163F0954F30402E\
D7B374EE84A97B1849B0674B0DA0A2050BD79B71ABB1559F3A9CFDB8557DED7BC90CF2\
09E8A847E9C226140845B7D03842162E7DA5DD16326CB1F71A248D841FE9076A09911F\
2F4F5E3EA44EA8DE40332BF00406990BCCF61C322A03C456EF3A98B341E0BDBC1088CE\
683E78510E76B72C2BCC1EE9AEDD80FFF18ABFC5923B2F36B581C25114AB2DF9F6C2B1\
9481703FD19E313DCD7ACE15FA11B27D25BCE5388C180A7E21167FB87750599E1ED7C7\
50F4A844E1DC2270C62D19671CF8F4C25B81E366B09FC850AE642136D204A9160AEECE\
575B57378AA439E9DD46DC990288CD54BAA35EEE1C02456CD39458A6F1CBF012DCEDF4\
27CCF3F3F53645658FC49C9C9D7F2856DB571D92B967AB5845514E0054DDB49099F5DD\
04A6F6F5C5CE642276834B932881AEB648D1F25E9223971F56E249EF40CF7D80F22621\
CDD0260E9E7D23746960ADB52CF2987584FB1DE95A69A39E5CB12B76E0F5C1A0529C0C\
065D2E35720810F7C7983180B9A9EA0E00C11B79DC3D");

  mpi_fromstr(sk.e, "0x\
DD4856B4EE3D099A8604AE392D8EFEC094CDF01546A28BE87CB484F999E8E75CDFCD01\
D04D455A6A9254C60BD28C0B03611FC3E751CC27EF768C0B401C4FD2B27C092834A6F2\
49A145C4EDC47A3B3D363EC352462C945334D160AF9AA72202862912493AC6190AA3A6\
149D4D8B9996BA7927D3D0D2AD00D30FD630CF464E6CAF9CF49355B9A70E05DB7AE915\
F9F602772F8D11E5FCDFC7709210F248052615967090CC1F43D410C83724AA5912B2F0\
52E6B39449A89A97C79C92DC8CB8DEEFCF248C1E1D2FC5BFE85165ECA31839CAA9CEB3\
3A92EBDC0EB3BAC0F810938BB173C7DA21DCBB2220D44CBA0FD40A2C868FC93AC5243E\
C137C27B0A76D65634EBB3");

  mpi_fromstr(sk.d, "0x\
7C8A6FA1199D99DCA45E9BDF567CA49D02B237340D7E999150BC4883AE29DEC5158521\
B338F35DC883792356BDDBB3C8B3030A6DD4C6522599A3254E751F9BA1CB1061C5633C\
81BBFACF6FCD64502614102DFED3F3FA284066C342D5E00953B415915331E30812E5FB\
CD6680ADCCDEE40B8376A3A225F2E160EA59C7566804526D73BB660A648A3EF9802313\
B2F841E8458B2AAACE7AACF31083E8F3F630298138393BC88BBD7D4AA4334949651D25\
365B10DBF4A4A08E20A6CC74BFDD37C1C38E2ADC2A283DF06590DF06B46F67F6ACA67F\
AC464C795261659A2F9558802D0BBAA05FD1E1AF2CDC70654723DF7EFAEA148B8CDBEB\
C89EA2320AB9BBB1BC4311475DF3D91446F02EF192368DFEBAC598CCFD4407DEC58FDC\
1A94CCDD6E5FBA9C52164ACEA8AEE633E557BCCEACB7A1AF656C379482D784A120A725\
32F9B2B35173D505F21D5AD4CB9511BC836DC923730B70291B70290A216CA3B21CFF79\
E895C35F4F7AF80E1BD9ED2773BD26919A76E4298D169160593E0335BE2A2A2D2E8516\
948F657E1B1260E18808A9D463C108535FB60B3B28F711C81E5DE24F40214134A53CE5\
9A952C8970A1D771EBEFFA2F4359DCF157995B3F1950DE3C6EC41B7FF837148F55F323\
372AF3F20CE8B8038E750C23D8F5041FA951327859B0E47483F0A47103EF808C72C251\
006FA526245291C8C84C12D2EF63FB2301EA3EEDA42B");

  mpi_fromstr(sk.p, "0x\
E236732452039C14EC1D3B8095BDDCFB7625CE27B1EA5394CF4ED09D3CEECAA4FC0BF6\
2F7CE975E0C8929CE84B0259D773EA038396479BF15DA065BA70E549B248D77B4B23ED\
A267308510DBEE2FD44E35D880EE7CFB81E0646AA8630165BD8988C3A8776D9E704C20\
AA25CA0A3C32F27F592D5FD363B04DD57D8C61FFDCDFCCC59E2913DE0EE47769180340\
E1EA5A803AA2301A010FF553A380F002601F0853FCACDB82D76FE2FACBCD6E5F294439\
0799EA5AE9D7880D4E1D4AE146DC1D4E8495B9DD30E57E883923C5FC26682B7142D35C\
D8A0FC561FE725A6CF419B15341F40FE0C31132CBD81DD8E50697BD1EBFFA16B522E16\
F5B49A03B707218C7DA60B");

  mpi_fromstr(sk.q, "0x\
E830482A3C4F5C3A7E59C10FF8BA760DB1C6D55880B796FFDA4A82E0B60E974E81D04B\
2A4AD417823EBFB4E8EFB13782943562B19B6C4A680E3BA0C8E37B5023470F4F1AC1F8\
A0B10672EF75CD58BCD45E6B14503B8A6A70AFE79F6201AF56E7364A1C742BE1453FD2\
24FDC9D66522EAF4466A084BCB9E46D455A2946E94CBF028770F38D0B741C2CC59308F\
71D8C2B4B9C928E0AE8D68DEB48A3E9EFD84A10301EBD55F8221CA32FC567B306B2A8E\
116350AFB995859FDF4378C5CFD06901494E8CFA5D8FAC564D6531FA8A2E4761F5EFBA\
F78750B6F4662BE9EA4C2FAD67AF73EEB36B41FC15CB678810C19A51DF23555695C4C1\
546F3FACA39CAA7BB8DBD7");

  mpi_fromstr(sk.u, "0x\
846232322775C1CD7D5569DC59E2F3E61A885AE2E9C4A4F8CB3ACBE8C3A5441E5FE348\
A2A8AC9C2998FBF282222BF508AA1ECF66A76AEDD2D9C97028BFD3F6CA0542E38A5312\
603C70B95650CE73F80FDD729988FBDB5595A5BF8A007EA34E54994A697906CE56354C\
E00DF10EB711DEC274A62494E3D350D88736CF67A477FB600AC9F1D6580727585092BF\
5EBC092CC4D6CF75769051033A1197103BE269942F372168A53771746FBA18ED6972D5\
0B935A9B1D6B5B3DD50CD89A27FE93C10924E9103FACF7B4C5724A046C3D3B50CC1C78\
5F5C8E00DBE1D6561F120F5294C170914BC10F978ED4356EED67A9F3A60D70AFE540FC\
5373CBAE3D0A7FD1C87273");

// copy the public key components
  pk.n = mpi_copy( sk.n );
  pk.e = mpi_copy( sk.e );

// some plain text message
	MPI plain = mpi_alloc(0);
	mpi_fromstr(plain, "0x\
5B6A8A0ACF4F4DB3F82EAC2D20255E4DF3E4B7C799603210766F26EF87C8980E737579\
EC08E6505A51D19654C26D806BAF1B62F9C032E0B13D02AF99F7313BFCFD68DA46836E\
CA529D7360948550F982C6476C054A97FD01635AB44BFBDBE2A90BE06F7984AC8534C3\
28097EF92F6E78CAE0CB97");

// expected encrypted MPI (via rsa.c directly)
  MPI encr = mpi_alloc(0);
  public_rsa( encr, plain, &pk);
  MPI decr = mpi_alloc(0);
  secret_rsa( decr, encr, &sk);
  if (mpi_cmp(decr, plain) != 0)
    printf("FAIL: decrypted != plain in test_rsa_octets (MPI call)\n");

//allocate the char arrays for _octets rsa
  int len_n = noctets;
  int len_e = len_n;
  int len_d = len_n;
  int len_p = noctets / 2;
  int len_q = len_p;
  int len_u = len_p;
  char n[noctets];
  char e[noctets];
  char d[noctets];
  char p[noctets / 2];
  char q[noctets / 2];
  char u[noctets / 2];

//copy the key components into char arrays
  mpi_to_octets(n, &len_n, sk.n);
  mpi_to_octets(e, &len_e, sk.e);
  mpi_to_octets(d, &len_d, sk.d);
  mpi_to_octets(p, &len_p, sk.p);
  mpi_to_octets(q, &len_q, sk.q);
  mpi_to_octets(u, &len_u, sk.u);

//call _octets rsa and check results
  int len_encr = noctets;
  int len_decr = noctets;
  int len_plain = noctets;
  char plain_o[noctets];
  char encr_o[noctets];
  char expected_encr_o[noctets];
  char decr_o[noctets];
  char expected_decr_o[noctets];

  mpi_to_octets(plain_o, &len_plain, plain);
  mpi_to_octets(expected_encr_o, &len_encr, encr);
  mpi_to_octets(expected_decr_o, &len_decr, decr);
  len_decr = noctets;

  int len;
  len = public_rsa_octets( encr_o, len_encr, plain_o, len_plain,
                           n, len_n, e, len_e);
  if (len != len_encr)
    printf("FAIL: actual len of encr is %d; expected %d\n", len, len_encr);
  else
    printf("PASS: actual len of encr matches expected: %d\n", len);
  int errors= 0;
  int i;
  for (i=0;i0)
    printf("FAIL: found %d errors in public_rsa_octets output\n", errors);
  else
    printf("PASS: no errors found in public_rsa_octets output\n");

  len_encr = len;
  len = private_rsa_octets( decr_o, len_decr, encr_o, len_encr,
                            n, len_n, e, len_e,
                            d, len_d, p, len_p,
                            q, len_q, u, len_u);
  if (len != len_plain)
    printf("FAIL: actual len of decr is %d; expected %d\n", len, len_plain);
  else
    printf("PASS: actual len of decr matches expected: %d\n", len);

  errors = 0;
  for (i=0;i0)
    printf("FAIL: found %d errors in private_rsa_octets output\n", errors);
  else printf("PASS: no errors found in private_rsa_octets_output\n");

  //tidy up
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  mpi_free(pk.n);
  mpi_free(pk.e);

  mpi_free(plain);
  mpi_free(encr);
  mpi_free(decr);
}

int main(int ac, char **av) {
  if (ac < 2) {
    printf("Usage: %s testID\n", av[0]);
    return -1;
  }

  int id = atoi(av[1]);
  switch (id) {
    case 1:
      test_mpi_cmp();
      break;
    case 2:
      test_gen_rsa_octets();
      break;
    case 3:
      test_rsa_octets();
      break;
    default:
      printf("Current test ids:\n");
      printf("1 test of mpi_cmp_octets\n");
      printf("2 test of gen_rsa_octets (can be very SLOW!)\n");
      printf("3 test of rsa_octets (can take a few minutes)\n");
  }
  return 0;
}

The rest (and the longest part by far) of the .vpatch for this chapter includes the MPI lib from EuCrypt and the RSA lib from the same EuCrypt minus the oaep parts that are not going to be used by SMG Comms. Specifically, the changes to the RSA lib from EuCrypt are the following:

  • The constant KEY_LENGTH_OCTETS is changed to the value used by Eulora: 490. I've updated also all the references to the previously fixed value 4096 throughout the comments.
  • I discarded two methods from rsa.c/h: rsa_oaep_encrypt, rsa_oaep_decrypt, oaep_encrypt_c, oaep_decrypt_c.

In addition to the above, the .vpatch also makes a small change to the smg_comms.gpr file reflecting the fact that SMG Comms does not touch mpi and rsa directly but only through the new C wrappers so that the .gpr file uses "c_wrappers.gpr" and not rsa.gpr + mpi.gpr.

The .vpatch is as usual on my Reference Code Shelf and linked also below for your convenience:

October 18, 2018

SMG Comms Chapter 3: Packing Serpent

Filed under: Coding, SMG_Comms — Diana Coman @ 10:12 a.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

This chapter uses the raw types of the protocol as defined in the previous chapter and adds two methods that are still at layer 0 of the protocol: one method for packing Serpent messages into corresponding Serpent packets and one method for unpacking such Serpent packets to extract their contained Serpent message. It is a single small step forward but the corresponding .vpatch is still quite large as it contains the Serpent code in addition to the packing/unpacking methods. The packing is effectively an encryption with a given Serpent key while the unpacking is the corresponding decryption. Nevertheless, there are a few bits that are specific to this implementation as they reflect the requirements of Eulora's protocol:

  • Packing receives as input a Serpent_Msg and produced as output a Serpent_Pkt. Symmetrically, unpacking receives as input a Serpent_Pkt and produces a Serpent_Msg. Both Serpent_Pkt and Serpent_Msg are arrays of octets but of fixed, pre-defined size: 1472 octets and nothing else.
  • Both packing and unpacking will split their input into blocks of the size that Serpent can handle, encrypt/decrypt them and then glue together the results to produce the output. So there are effectively 921 encrypting / decrypting operations with the same, given Serpent key, for one single pack / unpack call.

I've adapted the Serpent implementation that I previously published as part of EuCrypt, effectively integrating it into SMG Comms and stripping away anything that isn't directly needed by SMG Comms:

  • Serpent is now simply a package like all the others rather than a stand-alone library. While it is true that any changes to the original will have to be manually ported to this one as well, that was always going to be the case anyway. So I don't really see much point in carrying about all the glue and additional files to make a library out of only 2 files. Hence, Serpent in SMG Comms has 2 files and nothing more: serpent.ads and serpent.adb. Short and clear.
  • Since this is production use already, testing parts such as the "Selftest" method don't really have any business in the code itself. I've moved this method where it belongs, namely with the tests for all the code, in the tests directory (test_serpent.ads/.adb)
  • Since Serpent here becomes part of SMG Comms it follows that it should also use the raw types of the protocol - it will anyway be called to use variables of those types rather than anything else. There is no point in forcing back and forth conversions between SMG Comms' "Octets" and Serpent's "Bytes" types that are both arrays of octets anyway. So I've changed the definition of the "Bytes" type in Serpent so that it is here simply a subtype of the "Octets" type. This has the advantage that it allows smooth calls to Serpent from SMG Comms while being a small, easily-reversible change that also maintains otherwise the clarity of Serpent's code as it is. Basically SMG Comms gets to call Serpent without having to do explicit conversions between types that are anyway the same thing and Serpent gets to keep calling arrays of octets Bytes internally as it does in its stand-alone lib version.

In addition to the above, the .vpatch for this chapter also adds tests for the packing/unpacking methods2. I've also made a small change to raw_types.ads so that there is now only one variable for Serpent length: SERPENT_OCTETS. This reflects better the fact that there really is only one length for Serpent and it still allows to keep code clear by having the two array types Serpent_Pkt and Serpent_Msg - they just use the same length. Clarity of code is a tricky choice, what more can I say. Here's the updated code in raw_types:

  -- constants from SMG.COMMS standard specification
    -- size of a serpent-encrypted packet and message, in octets
    -- note that this corresponds to 1472/16 = 92 Serpent blocks
    -- NB: lengths are the same!
  SERPENT_OCTETS : constant Positive := 1472;

    -- size of a RSA-encrypted packet and message in octets and bits
  RSA_PKT_OCTETS     : constant Positive := 1470;
  RSA_MSG_OCTETS     : constant Positive := 234;
  RSA_MSG_BITS       : constant Positive := RSA_MSG_OCTETS * 8; --1872

  -- raw, low-level types
  -- all messages and packets are simply arrays of octets at low level/raw
  type Octets is array( Natural range <> ) of Interfaces.Unsigned_8;

  -- raw representations of basic types (with fixed, well-defined sizes)
  subtype Octets_1 is Octets( 1 .. 1 );
  subtype Octets_2 is Octets( 1 .. 2 );
  subtype Octets_4 is Octets( 1 .. 4 );
  subtype Octets_8 is Octets( 1 .. 8 );

  -- RSA packets and contained raw messages
  subtype RSA_Pkt is Octets( 1 .. RSA_PKT_OCTETS );
  subtype RSA_Msg is Octets( 1 .. RSA_MSG_OCTETS );

  -- Serpent packets and contained raw messages
  -- NB: length is the same but the distinction makes the code clearer
  subtype Serpent_Pkt is Octets( 1 .. SERPENT_OCTETS );
  subtype Serpent_Msg is Octets( 1 .. SERPENT_OCTETS );

And the new code in packing.ads:

  -- Packing/unpacking for Eulora's communication protocol:
  -- Serpent Message to/from Serpent Packet
  -- RSA Message to/from RSA Packet
  -- S.MG, 2018

with Raw_Types;
with Serpent;

package Packing is
  -- no side effects or internal state
  Pragma Pure(Packing);

  -- Packing a Serpent message into Serpent package, using the given key
  function Pack( Msg : in Raw_Types.Serpent_Msg;
                    K   : in Serpent.Key )
                  return Raw_Types.Serpent_Pkt;

  -- Unpacking a Serpent packet into contained message, using the given key
  function Unpack( Pkt : in Raw_Types.Serpent_Pkt;
                    K   : in Serpent.Key)
                  return Raw_Types.Serpent_Msg;

  -- internals of this package, NOT for outside use
private
  -- length of 1 Serpent block
  Block_Len: constant Natural := Serpent.Block'Length;

  -- number of Serpent blocks in one single Serpent message/packet
  S_Blocks : constant Natural := Raw_Types.SERPENT_OCTETS / Block_Len;

end Packing;

The new code in packing.adb:

  -- Packing/unpacking for Eulora's communication protocol:
  -- Serpent Message to/from Serpent Packet
  -- RSA Message to/from RSA Packet
  -- S.MG, 2018

package body Packing is

  -- Packing a Serpent message into Serpent package, using the given key
  function Pack( Msg : in Raw_Types.Serpent_Msg;
                 K   : in Serpent.Key )
               return Raw_Types.Serpent_Pkt is

    -- single Serpent blocks containing plain / encrypted data
    Plain    : Serpent.Block;
    Encr     : Serpent.Block;

    -- Serpent Key Schedule - needed for direct encr/decr calls
    KS       : Serpent.Key_Schedule;

    -- final resulting Serpent package
    Pkt      : Raw_Types.Serpent_Pkt := (others => 0);
  begin
    -- prepare the Serpent key schedule based on given key
    Serpent.Prepare_Key( K, KS );

    -- encrypt message block by block and copy result in packet
    for I in 1 .. S_Blocks loop
      -- get current block to encrypt
      Plain := Msg( Msg'First + (I-1) * Block_Len ..
                    Msg'First +  I    * Block_Len - 1 );
      -- encrypt with Serpent
      Serpent.Encrypt( KS, Plain, Encr );
      -- copy result to output packet
      Pkt( Pkt'First + (I-1) * Block_Len ..
           Pkt'First +  I    * Block_Len - 1 )
         := Encr;
    end loop;

    -- return result
    return Pkt;
  end Pack;

  -- Unpacking a Serpent packet into contained message, using the given key
  function Unpack( Pkt : in Raw_Types.Serpent_Pkt;
                   K   : in Serpent.Key)
                 return Raw_Types.Serpent_Msg is
    -- single Serpent blocks containing plain / encrypted data
    Plain    : Serpent.Block;
    Encr     : Serpent.Block;

    -- Serpent Key Schedule - needed for direct encr/decr calls
    KS       : Serpent.Key_Schedule;

    -- the message extracted from the given packet
    Msg : Raw_Types.Serpent_Msg := (others => 0);
  begin
    -- prepare the Serpent key for use
    Serpent.Prepare_Key( K, KS );

    -- decrypt the Serpent packet block by block
    for I in 1 .. S_Blocks loop
      -- get current block from input and decrypt
      Encr := Pkt( Pkt'First + (I-1) * Block_Len ..
                   Pkt'First +  I    * Block_Len - 1 );
      Serpent.Decrypt( KS, Encr, Plain );

      -- copy result to its correct position in final output
      Msg( Msg'First + (I-1) * Block_Len ..
           Msg'First +  I    * Block_Len - 1 )
         := Plain;
    end loop;

    -- return the result - the message content of given package
    return Msg;
  end Unpack;

end Packing;

The .vpatch and my signature for it are as usual on my Reference Code Shelf as well as linked here for your convenience:


  1. 1472 / 16 = 92 

  2. And needed they were too for they actually caught an error that had survived somehow several re-readings of the code to the point that I was totally surprised when the tests first...failed. Never underestimate your own capacity of introducing idiotic errors in the simplest of things! 

October 16, 2018

SMG Comms Chapter 2: Raw Types

Filed under: Coding, SMG_Comms — Diana Coman @ 11:20 a.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

My previous Chapter 1 with its assorted warts and lamentations sparked a rather productive discussion in the forum that ended up with a significant revision of the protocol specification. The main change has to do with packet sizes and types: ALL packets will be either 1470 octets long and RSA encrypted OR 1472 octets long and Serpent encrypted. Those sizes are chosen to ensure that there is no fragmenting of UDP frames sent on the wire since such fragmenting increases significantly the chance of packet loss. A test with those UDP packet sizes sent back and forth between UK and UY did not uncover any trouble - if anything, the results were actually better than expected. In any case, increased chances of reliable communications aside, the added benefit of those fixed, unique sizes is a clearer protocol alltogether. So after a bit of digesting the new specification, I can finally say that I have some sort of plan for its implementation rather than grasping at it from any thread that I could perceive at all (as it was pretty much the case in Chapter 1). Specifically, my current implementation plan is made of 3 layers that build on one another and give me a structure to rely on:

  1. Layer 0 - Raw types
  2. Layer 1 - Sender/Receiver
  3. Layer 2 - Data structures

1. Layer 0 - Raw types

The raw types for Eulora's communication protocol are essentially two: octet (the basic unit, effectively 8 bits) and arrays of octets (various fixed sizes matching the basic types in the protocol's specification). However, for the arrays of octets, there is a useful distinction to be made between packets (i.e. the received array of octets of fixed length, either 1470 or 1472 octets) and messages (i.e. the content of a packet). Essentially messages are obtained from packets in Eulora's communication protocol through decryption using either Serpent (for 1472 octets long packets) or RSA (for 1470 octets long packet). Symmetrically, packets are obtained by encrypting messages with either Serpent (for 1472 octets long messages) or RSA (for 234 octets long messages). Consequently, this layer 0 offers the definitions of types that cover the full set of possible messages and packets together with the relevant conversion methods between packets and messages as well as between raw octets and int/unsigned/float data types. Note that a "message" here is simply the decrypted content of a packet, nothing more. The interpretation of the octets in a message and/or their use to fill some data structure that the protocol defines happens at a higher layer, not here.

2. Layer 1 - Sender/Receiver

Layer 1 is concerned with sending and receiving packets using the types and conversion methods provided by layer 0 together with the UDP lib for actual network communication. So far this seems to me at the moment a very thin layer really since it's little more than a basic receiver and sender: the receiver simply uses UDP lib to endlessly read all incoming packets and store them in a queue for later processing by consumer(s) external to this layer; the sender may have perhaps an input queue from which it keeps reading packets together with their intended destination and then sends them along using the UDP lib. This layer comes right after the raw types because it needs and uses those raw types but at the moment it's unclear to me whether it belongs as part of a protocol implementation since I can see the case for it being left to the user application of the protocol. At any rate, the protocol specification is not really concerned with this part and this description of the sender/receiver is not a given - it may change as required by each application.

4. Layer 2 - Data Structures

This layer is on top of Layer 0 - it provides the data structures for the different types of messages in Eulora's communication protocol. While any message in Eulora's protocol is just an array of octets at layer 0, the information contained in any message has to be structured according to the protocol's rules described in sections 4, 5 and 7 of the specification. This layer provides the definitions of those data structures as well as the conversion methods from messages to data structures and back.

Note that those 3 layers do not include protocol mechanics since those are really part of the server or client applications. Perhaps a demo / test application implementing the specified protocol mechanics would be useful to include as part of this implementation but I'd rather not specify it as such at this point. For now I'm fine with those 3 layers but if you see any holes in them or problems with them, let me know in the comments below!

Implementation

To start it all off, here's the first part of layer 0 - raw types, including the definitions of types and the most basic conversions via Ada.Unchecked_Conversion:

 -- raw types for the communication protocol
 -- these are used throughout at the lowest level of the protocol
 -- essentially they are the units of packets and of messages
 -- SMG.Comms has only 2 types of packets: RSA and Serpent
 -- a message is the decrypted content of a packet
 -- S.MG, 2018

with Interfaces; use Interfaces; -- Unsigned_n and Integer_n
with Ada.Unchecked_Conversion;

package Raw_Types is

  -- constants from SMG.COMMS standard specification
    -- size of a serpent-encrypted packet and message, in octets
    -- note that this corresponds to 1472/16 = 92 Serpent blocks
    -- NB: lengths are the same but the distinction makes the code clearer
  SERPENT_PKT_OCTETS : constant Positive := 1472;
  SERPENT_MSG_OCTETS : constant Positive := SERPENT_PKT_OCTETS;

    -- size of a RSA-encrypted packet and message in octets and bits
  RSA_PKT_OCTETS     : constant Positive := 1470;
  RSA_MSG_OCTETS     : constant Positive := 234;
  RSA_MSG_BITS       : constant Positive := RSA_MSG_OCTETS * 8; --1872

  -- raw, low-level types
  -- all messages and packets are simply arrays of octets at low level/raw
  type Octets is array( Natural range <> ) of Interfaces.Unsigned_8;

  -- raw representations of basic types (with fixed, well-defined sizes)
  subtype Octets_1 is Octets( 1 .. 1 );
  subtype Octets_2 is Octets( 1 .. 2 );
  subtype Octets_4 is Octets( 1 .. 4 );
  subtype Octets_8 is Octets( 1 .. 8 );

  -- RSA packets and contained raw messages
  subtype RSA_Pkt is Octets( 1 .. RSA_PKT_OCTETS );
  subtype RSA_Msg is Octets( 1 .. RSA_MSG_OCTETS );

  -- Serpent packets and contained raw messages
  -- NB: length is the same but the distinction makes the code clearer
  subtype Serpent_Pkt is Octets( 1 .. SERPENT_PKT_OCTETS );
  subtype Serpent_Msg is Octets( 1 .. SERPENT_MSG_OCTETS );

  -- blind, unchecked casts ( memcpy style )
  function Cast is new Ada.Unchecked_Conversion( Integer_8  , Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1   , Integer_8 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_8 , Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1   , Unsigned_8 );

  function Cast is new Ada.Unchecked_Conversion( Integer_16 , Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2   , Integer_16 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_16, Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2   , Unsigned_16 );

  function Cast is new Ada.Unchecked_Conversion( Integer_32 , Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4   , Integer_32 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_32, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4   , Unsigned_32 );

  -- Gnat's Float has 32 bits but this might be different with other compilers
  function Cast is new Ada.Unchecked_Conversion( Float, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4, Float );

  function Cast is new Ada.Unchecked_Conversion( Integer_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Integer_64 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Unsigned_64 );

end Raw_Types;

I packed the above code as a .vpatch on top of the previous genesis .vpatch mainly for keeping my original promise of showing this being built as it is with detours and corrections on the way. Otherwise this could equally well be a genesis in itself given how radically it changes pretty much everything in there. Nevertheless, at least for now - meaning until a final version is achieved and a regrind makes perhaps sense - I'll keep growing the smg_comms tree. I'll link all smg_comms .vpatches and my signatures for them from the Code Shelf page as usual, while linking the current ones here as well:

The next chapter will add to the raw types layer, providing conversions from packets to messages and back. That will bring in parts of EuCrypt of course, namely the Serpent and RSA implementations. For this reason, I aim to do this perhaps in two steps (first Serpent and only then RSA) in order to keep patches as clear and easy to follow as possible. And this is of course a very good opportunity to re-read the EuCrypt implementation, to re-write it on one level or another as I bring it over to SMG_Comms and to see first-hand how it is to use it!

September 14, 2018

SMG.Comms Implementation - Chapter 1

Filed under: Coding, SMG_Comms — Diana Coman @ 11:02 a.m.

~ This is a work in progress towards an Ada implementation of the communication protocol discussed on Trilema four months ago. ~

As far as I know, there isn't any more recent discussion of the above specification, nor any other attempt at all at any sort of implementation. Consequently, this is the first ever attempt - a prototype at this stage rather than a reference implementation. Moreover, it's also a sort of double first, since it clearly requires a deeper knowledge of Ada than I ever needed before. My approach to this pile of unknown here is to simply start working on it and expose the path travelled from this very first attempt to the final product, mistakes and detours and pitfalls to come included. You are welcome to follow along, to help if you can, to question if you don't understand, to extract perhaps in doing so some benefit for yourself.

The first decision I had to make before I could even really attempt any sort of prototype implementation at all concerned the library to use for the actual network communication. Since GNAT is the de-facto republican Ada compiler1, the logical decision is to simply use GNAT's own library, GNAT.Sockets and avoid otherwise as much as possible introducing additional, external dependencies - I really can't see any reason to add even more code even indirectly. So then GNAT.Sockets it is and hooray for that - except that there doesn't seem to be much documentation about it other than the comments in g-socket.ads and g-socket.adb! Still, the .ads file has a rather detailed introduction of the package with some commented examples in there so definitely worth reading as a starting point especially since... there isn't any other starting point really.

The g-socket files reveal essentially that any data to be sent or received through GNAT.Sockets will have to be stored at one point or another in an entity of Ada.Streams.Stream_Element_Array type. The examples in the file (and most everything else I could find on the topic) really focus almost exclusively2 on TCP connections - I suspect one "should prefer" TCP over UDP simply like that and we are even past the point of discussing therefore UDP in any scant docs or something. Nevertheless, I don't prefer it: Eulora's protocol is rather specifically designed to be stateless and to keep communications as simple and clear as possible - for as long as UDP is enough for the job, I'd rather use it!

Rummaging a bit in the g-sockets files reveals happily that UDP with GNAT.Sockets is in fact quite straightforward: one simply needs to specify "Socket_Datagram" as mode for the socket when calling Create_Socket and then use directly the Send_Socket( socket, data, last, to_address) and Receive_Socket(socket, data, last, from_address) methods. As one would expect, there is no reliable connection established, no "stream" of send and receive but only sockets that allow independent "send" and "receive" calls - each of those could in principle use a different address even. A quick example - keep reading for a a full, working example - of a server and client working with a UDP socket looks like this:

 -- on server:
     -- create UDP socket
    Create_Socket( Sock, Family_Inet, Socket_Datagram );

    -- set options on UDP socket
    Set_Socket_Option( Sock, Socket_Level, (Reuse_Address, True));

    -- set address and bind
    Address.Addr := Any_Inet_Addr;
    Address.Port := Port_No; -- the port number that server is listening on
    Bind_Socket( Sock, Address );
    Receive_Socket( Sock, Data, Last, From );

 -- on client:
    Address.Port := Port_No; -- server's port number
    Address.Addr := Inet_Addr("127.0.0.1"); -- this should be server's address here!
    Create_Socket(Sock, Family_Inet, Socket_Datagram);

    Send_Socket(Sock, Data, Last, Address);

With the very basics of the sockets part at least in place, the next part to decide on is how to send and receive over those sockets the actual data types defined by the protocol. After reading quite a bit on the Systems Programming (an annex of the Ada standard and therefore compiler-dependent) part of Ada (that is relevant for data representation) and on streams and on representation clauses and pragmas supported by GNAT and records and discriminants for records and everything else from the Ada reference book3 that I thought could help with this, I got to the conclusion that I'll keep it simple and clear especially for now, while I'm just starting to figure it all out! So I'll simply use GNAT's fixed width types defined in the Interfaces package and I'll copy otherwise the raw octets from and to those types using Ada.Unchecked_Conversion that works as far as I understand it precisely as a raw copy from memory. Once the data is simply obtained as a vector of raw octets, I can implement two simple functions to copy them into the Stream_Element_Array structure that can be sent directly through the UDP socket. Moreover, at this stage - and at this stage *only* - I'll worry about potentially different endianness of the local machine and the network: if local environment is little endian, the methods converting to and from network format will simply read the octets in reverse order. The relevant basic types and conversion methods are defined in the smg_comms_types.ads file:

 -- S.MG, 2018
 -- prototype implementation of S.MG communication protocol

with Ada.Streams; use Ada.Streams;
with Interfaces; use Interfaces; -- Integer_n and Unsigned_n
with Ada.Unchecked_Conversion; -- converting int/uint to array of octets

package SMG_comms_types is
  -- basic types with guaranteed lengths
  type Octet_Array is array(Natural range <>) of Unsigned_8;

  subtype Octets_1 is Octet_Array( 1 .. 1 );
  subtype Octets_2 is Octet_Array( 1 .. 2 );
  subtype Octets_4 is Octet_Array( 1 .. 4 );
  subtype Octets_8 is Octet_Array( 1 .. 8 );

  subtype Message is Octet_Array( 1 .. 512 );
  subtype RSAMessage is Octet_Array( 1 .. 245 );

  -- blind, unchecked casts ( memcpy style )
  function Cast is new Ada.Unchecked_Conversion( Integer_8, Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1, Integer_8 );
  function Cast is new Ada.Unchecked_Conversion( Integer_16, Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2, Integer_16 );

  function Cast is new Ada.Unchecked_Conversion( Integer_32, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4, Integer_32 );

  function Cast is new Ada.Unchecked_Conversion( Integer_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Integer_64 );

  -- to and from streams for network communications - general
  procedure ToNetworkFormat(
      Item   : in Octet_Array;
      Buffer : out Stream_Element_Array);

  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Octet_Array);

  -- specific, convenience methods for the basic types
    -- Integer_8
  procedure ToNetworkFormat(
      Item   : in Integer_8;
      Buffer : out Stream_Element_Array);

  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Integer_8);

end SMG_comms_types;

As you can easily notice, the above does not yet cover fully even the 3.0 "Basic types" part of the protocol specification. It's all right for now though - there is quite enough there for the first basic tests and once those are fine, I'll add gradually the rest of types too. There is little point in spending the time now to implement them all before I even got the chance to change my mind regarding *how* to implement them! So if you have a better implementation solution than the above, speak up in the comments section below and save me some time and a lot of headache! Note however that simplicity, clarity and hard guarantees are rather important here.

One small point on which I'm already rather undecided is whether to continue implementing the convenience methods for different types so that one can simply call ToNetworkFormat and FromNetworkFormat for anything or to leave only the generic methods at least for basic types. At the moment I incline towards providing all those methods (i.e. adding to the single pairof methods for the Integer_8 type defined above) because more complex types will likely need such methods anyway and moreover, this approach helps to keep all those casts (unchecked_conversion) in one place rather than scattered all through the rest of the code. However, it does add to the LOC count of this package. Anyway, moving further for now, the corresponding .adb file with the implementation of smg_comms_types:

  -- S.MG, 2018
  -- prototype implementation of S.MG communication protocol

with SMG_comms_types; use SMG_comms_types;
with System; use System; -- endianness
with Ada.Exceptions;
with Ada.Streams; use Ada.Streams;

package body SMG_comms_types is

  -- to and from network format (i.e. big endian, stream_element_array)
  procedure ToNetworkFormat(
      Item   : in Octet_Array;
      Buffer : out Stream_Element_Array) is
  begin
    if Item'Length /= Buffer'Length then
      raise Constraint_Error with "Item and Buffer lengths do NOT match!";
    end if;

    if Default_Bit_Order = Low_Order_First then
      for I in 0 .. Item'Length - 1 loop
        Buffer( Buffer'Last - Stream_Element_Offset(I) ) := Stream_Element(Item(Item'First + I));
      end loop;
    else
      for I in 0 .. Item'Length - 1 loop
        Buffer( Buffer'First + Stream_Element_Offset(I) ) := Stream_Element(Item(Item'First + I));
      end loop;
    end if;
  end ToNetworkFormat;

  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Octet_Array) is
  begin
    if Item'Length /= Buffer'Length then
      raise Constraint_Error with "Buffer and Item length do NOT match!";
    end if;

    if Default_Bit_Order = Low_Order_First then
      for I in 0 .. Buffer'Length - 1 loop
        Item( Item'Last - I ) :=
          Unsigned_8( Buffer( Buffer'First + Stream_Element_Offset( I ) ) );
      end loop;
    else
      for I in 0 .. Buffer'Length - 1 loop
        Item( Item'First + I ) :=
          Unsigned_8( Buffer( Buffer'First + Stream_Element_Offset( I ) ) );
      end loop;
    end if;
  end FromNetworkFormat;

  -- Integer_8
  procedure ToNetworkFormat(
      Item   : in Integer_8;
      Buffer : out Stream_Element_Array) is
  begin
    ToNetworkFormat( Cast( Item ), Buffer );
  end ToNetworkFormat;
  procedure FromNetworkFormat(
      Buffer : in Stream_Element_Array;
      Item   : out Integer_8) is
    octets: Octets_1;
  begin
    FromNetworkFormat(Buffer, octets);
    Item := Cast( octets );
  end FromNetworkFormat;

end SMG_comms_types;

Something that irks me every time I look at the above: those for loops that convert octet by octet to Stream_Element. This is not only ugly but also rather inefficient, especially given that it's potentially the *second* time when those octets are read one by one (the first time being at the conversion from a protocol type - especially one of the more complex types - to an array of octets). However, I have no idea how to do that conversion from array of octets to array of Stream_Element in one single move! Do you know a better way to do this? A direct assign fails because the cast from one type to another would be on the array type rather than individual elements type. And I'm rather reluctant to work directly with Stream_Element as the basic type because this type is implementation dependent and outside my direct control - so I can't actually really know *what* it is.

The above being said regarding Stream_Element, it is important to note that the code above (and almost all the code for a network communication protocol) is anyway, strictly speaking, rather tightly linked to GNAT since it relies on Stream_Element being exactly 8 bits long. And to make this clear, here is the full trail I followed to make sure that I can indeed simply convert an octet (i.e. 8 bits) to a Stream_Element and the other way around while preserving exactly the number of bits specified in the protocol for each type: first, GNAT implements Stream_Element in a-stream.ads as follows:

type Stream_Element is mod 2 ** Standard'Storage_Unit;

Then, the definition of Standard'Storage_Unit for GNAT, which reads that Standard'Storage_Unit always has, according to the GNAT reference manual, the same value as System.Storage_Unit. In turn, System.Storage_Unit is indeed declared in GNAT's system.ads as a constant with value 8, so a Stream_Element in GNAT will indeed have exactly 8 bits:

Storage_Unit : constant := 8;

Moving on, the next step is to write the client-server part even if only a test version for now. Initially I took the easy way out here and simply wrote the separate server and client, each listening/sending one single packet. Running those on different machines worked perfectly fine. This is both fine and needed for a reasonable basic test of the whole thing, sure. However, at the moment as I'm just starting on this I'd rather *not* faff about with 2 machines each and every time I change or add something to this prototype protocol implementation. Moreover, this server/client part is perfect to experiment as well with threads4 in Ada - a topic that I'm still struggling to learn so perfect to practice! Therefore, in the basic test, server and client are implemented as different tasks (Ada's "threads") of the same main program - obviously, it follows that client and server will send data to one another on the same, local machine. This is of course not ideal nor sufficient as test in the long term but it'll do nicely for now and until the basic layer of the protocol at least is more fleshed out. The code in test_comms.adb:

 -- S.MG, 2018
 -- prototype implementation of S.MG communication protocol

with GNAT.Sockets; use GNAT.Sockets;
with Ada.Text_IO; use Ada.Text_IO;
with Ada.Streams; use Ada.Streams;
with Interfaces; use Interfaces;

with SMG_comms_types; use SMG_comms_types;

procedure test_comms is
  Port_No : constant := 2222;

  task type Client is
    entry Send;
  end Client;

  task type Server is
    entry Listen;
    entry Ready;
  end Server;

  task body Client is
    Sock: Socket_Type;
    Address: Sock_Addr_Type;
    Data: Ada.Streams.Stream_Element_Array(1..10) := (others => 42);
    Last: Ada.Streams.Stream_Element_Offset;
    N   : Integer_8 := -36;
  begin
    accept Send; -- task WILL block here until asked to send
    Address.Port := Port_No;
    Address.Addr := Inet_Addr("127.0.0.1");
    Create_Socket(Sock, Family_Inet, Socket_Datagram);

    ToNetworkFormat( N, Data(1..1));
    Send_Socket(Sock, Data, Last, Address);
    Put_Line("Client sent data " & "last: " & Last'Img);
  end Client;

  task body Server is
    Sock: Socket_Type;
    Address, From: Sock_Addr_Type;
    Data: Ada.Streams.Stream_Element_Array(1..512);
    Last: Ada.Streams.Stream_Element_Offset;
    N : Integer_8;
  begin
    accept Listen; -- wait to be started!
    Put_Line("Server started!");
    -- create UDP socket
    Create_Socket( Sock, Family_Inet, Socket_Datagram );

    -- set options on UDP socket
    Set_Socket_Option( Sock, Socket_Level, (Reuse_Address, True));
    Set_Socket_Option( Sock, Socket_Level, (Receive_Timeout, Timeout => 10.0));

    -- set address and bind
    Address.Addr := Any_Inet_Addr;
    Address.Port := Port_No;
    Bind_Socket( Sock, Address );

    accept Ready; -- server IS ready, when here
    -- receive on socket
    begin
      Receive_Socket( Sock, Data, Last, From );
      Put_Line("last: " & Last'Img);
      Put_Line("from: " & Image(From.Addr));
      Put_Line("data is:");
      for I in Data'First .. Last loop
        FromNetworkFormat(Data(I..I), N);
        Put_Line(N'Image);
      end loop;
    exception
      when Socket_Error =>
      Put_Line("Socket error! (timeout?)");
    end;  -- end of receive

  end Server;

  S: Server;
  C: Client;
begin
  S.Listen;
  S.Ready; -- WAIT for server to be ready!
  C.Send;  -- client is started only after server!
end test_comms;

Tasks in Ada can be defined as types as above. This is not mandatory - one can equally well simply define the tasks and they'll run in parallel with the main begin-end block. However, Task types are effectively needed if one wants to be able to explicitly and potentially dynamically create several workers of the same type. Since I'll need this later for sure, I might as well practice it at any occasion, so there they are, the Client Task type and Server Task type. To make sure that the server is ready *before* the client sends anything, I'm using the rendez-vous communication method that Ada provides: each "entry" of a task is basically a rendez-vous point between the task itself that "accepts" that entry at some specified point in its body and a caller task that "calls" that entry from outside. The main body of test_comms illustrates this: S and C start in parallel with the main body of test_comms but they both stop almost immediately as they wait on their first rendez-vous points (accept Listen for the Server type and accept Send for the Client type). The main body first calls S.Listen, effectively releasing the S (Server type) task from its wait. Immediately after that, the S.Ready entry is called so that now the main body is actually waiting for S to get to its "Ready" entry - basically to finish setting up the socket. Once S got to its Ready entry, the main body can go further and the next statement calls C.Send that releases the client for its wait. The client will send therefore its data and promptly finish, while the server (that was running in parallel all this time) will finally receive the data, process it and finish as well. Go ahead, give it ago and let me know if there's anything funny going on!

To compile all the above with one neat gprbuild call, there is of course a .gpr file:

-- S.MG, 2018
 -- prototype implementation of S.MG communication protocol
 -- http://trilema.com/2018/euloras-communication-protocol-restated/

project SMG_comms is
  for Languages use ("Ada");

  for Source_Dirs use ("src");
  for Ignore_Source_Sub_Dirs use (".svn", ".git", "@*");

  for Object_Dir use "obj";
  for Exec_Dir use ".";

  for Main use ("test_comms.adb");

  package Builder is
    for Executable ("test_comms.adb") use "test_comms";
  end Builder;

  package Compiler is
    for Default_Switches ("Ada") use ("-O2");
  end Compiler;

end SMG_comms;

An example of running the test_comms executable produced by gprbuild:

smg_comms$ ./test_comms
Server started!
Client sent data last:  10
last:  10
from: 127.0.0.1
data is:
-36
 42
 42
 42
 42
 42
 42
 42
 42
 42
smg_comms$

On a slightly different note, I had a bit of an internal debate on whether it's appropriate or not to still release this as a V tree given that it's work in progress and absolutely nowhere near a reference object of any sort or even a fully working item. I came to view V however as simply a versioning system rather than a "set in stone only the end products" sort of thing - for that matter what end products anyway, it's at most, in the happiest of situations, seed products rather than end products. So all the above code is the genesis of smg_comms and it's simply a tiny, far from perfect seed that will hopefully grow into something useful while preserving in its v-tree the whole history of that growth:


  1. Due to a large extent to ave1's work on forcing it into some useful shape so that it builds on a sane, musl-based system. 

  2. The Socket_Datagram mode to be set for UDP communications is at least mentioned in the examples although the rest of differences in effectively comunicating something are left to be discovered. 

  3. My Ada reference book is Barnes, John, "Programming in Ada 2012", Cambridge University Press, 2016, ISBN 978-1-107-42481-4. 

  4. On-demand, resilient threads are crucial for Eulora's server and I'm not yet confident at all that I really fully grasp Ada's mechanisms so I'm pushing it to the front, reading on it and otherwise banging my head on it at any and all occasions - how else to figure it out faster? 

Theme and content by Diana Coman