Ossasepia

November 13, 2018

V with VTools, Keccak Hashes and Its Own Tree

Filed under: Coding, TMSR — Diana Coman @ 10:49 p.m.

The republican versioning control system, V for Victory, is very much used, very much needed but nevertheless not yet versioned itself. As this has caused already way more talk in the logs than it's worth it, I promised I'll do a write-up of my own V setup and publish it, with a proper versioning for V itself included. So I've dug up old versions as well as my current setup and packaged everything in 2 different ways:

  1. A V-tree (using Keccak hashes) that captures the changes to v.pl code1 from the first version that I ever used, namely 999942. To use this, simply download the .vpatches, the .sig files and my signature, check them and then press the tree with either a different V that you might have or otherwise semi-manually with phf's vtools (vpatch more precisely). Note that you WILL need vtools (or equivalent) at any rate! Once you pressed V itself successfuly, you should have a v.pl that you can run - it will check its dependencies and complain if it doesn't find something (most notably the vtools parts namely vpatch and ksum).
  2. A signed .zip meant as a starter package for someone who hears of V for the first time in their life. Download starter_v.zip and starter_v.zip.diana_coman.sig. Check the signature! If and ONLY IF the check passes, unzip and then read the scripts in there. The build.sh script will simply build3 the vtools that are included in this starter pack and it will copy to the starter_v directory all the executables that are needed (v.pl renamed as vk.pl to make it clear it uses Keccak hashes, vdiff, vpatch and ksum). The included vtools are the code obtained from pressing current vtools tree up to and including the ksum .vpatch.

My changes to Mod6's v.pl simply replace older sha-based dependencies and calls with the vtools-based ones. Note that you'll need to have ksum and vpatch in your PATH or otherwise ready and accessible as v.pl will simply try to call them when it needs them.

For potential reference, here's my usual workflow to make a .vpatch:
mkdir a
mkdir b
cp -r old_stuff a/stuff
cp -r new_sutff b/stuff
vdiff a b > newpatch.vpatch
gpg --armor --output newpatch.vpatch.diana_coman.sig --detach-sig newpatch.vpatch

To check / press a V tree:
mkdir patches
mkdir .wot
mkdir .seals
cp some_patches patches/
cp corresponding_sig_files .seals
cp corresponding_trusted_pubkeys .wot
vk4 f
vk l
vk p v testdir chosen_patch.vpatch
cd testdir
read, compile, run etc

The .vpatches and .sig files:

The .zip file and corresponding .sig file:

For something to test your new shiny V on, head over to my Reference Code Shelf and take your pick. For trouble and questions, use the comments box below.

  1. Note that you are warmly invited to implement your own V! This version here is mod6's V implementation that was much discussed and iterated upon in the early days. ↩

  2. Note that V's versions DECREASE rather than increase, as per the explanation. ↩

  3. It requires GNAT. If you have no idea what that is, dig around, read the logs, ask humbly. ↩

  4. As I have all sorts of V implementations living side by side, I tend to give them different names - this is the vk for V-Keccak! ↩

November 10, 2018

SMG Comms Chapter 7: Read/Write Serpent Keysets to/from Serpent Messages

Filed under: Coding, SMG_Comms — Diana Coman @ 4:06 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

Over this past week I've been going deeper into Ada looking again and again -among other things- at record representation clauses and unions and serialization1 and basically anything and everything that could perhaps provide an elegant solution to serializing data structures to arrays of octets and back. And by "looking" I don't mean just reading but rather the usual combination of reading, writing and coding or at least attempting to code all sorts of ideas that seemed promising. What I have as a result is a much better grasp of Ada's neat representation clauses and parametrized records. What I don't have as a result is the very elegant and short solution I was hoping to find. Instead, I have a first pass at a solution together with the whole chain of reasoning that led me to it. So I'm publishing this as it is - a small step - and I'll gladly listen to better suggestions that are really fit for the job at hand.

The issue is in principle simple enough: write game data to the raw arrays of octets that are SMG messages ensuring the exact format specified by the communication protocol; conversely, read from those same raw arrays of octets that are SMG messages the actual game data. Stated like this, the solution suggesting itself is to define records matching exactly the format specified by the protocol, use representation clauses to control the exact place of all components and then simply convert the whole thing in one go. There are however a few things that I really don't like about this approach of using records matching exactly the full message structure as defined in Eulora's communication protocol:

  • The actual game data is of course only a part of each message. The rest includes padding or communication-level information that really has no business leaking in any way outside of the communication part itself. So I don't quite see the point in defining records for the full message since the read/write (that are supposedly called from outside the communication level itself) should NOT require/produce such records! Instead, the output should be *game data* for read and *array of octets* for write. As a result, I really don't see any benefit in defining message-matching records just for read/write to use *internally* - a sort of filling those component by component and doing the conversion as a whole instead of just doing the conversion component by component directly.
  • The internal structure of most messages is effectively parametrized: there is usually at least one component that represents the number of other following components. Ada handles this sort of thing very gracefully through parametrized records but having parametrized records inside a record doesn't seem to me like a very elegant solution nor one leading to a very clear representation of the whole. I admit I am still not 100% sure of the actual, exact representation of such a record containing itself parametrized records since my understanding is that Ada will allocated maximum space (i.e. space to fit potentially the largest structure) but then simply make available only as much as needed according to the actual value of the parameter.

Considering the above issues, my current solution aims to keep the communication-level parts of messages as an internal concern of the communication level and provide/require at read/write only the actual game data that is needed for each type of message. The conversion to/from octets is then done indeed component by component, as they are written/read and endianness is handled simply in one single place, at conversion: if native endianness is Big Endian then octets are flipped for any component that is represented on more than one octet; otherwise, on Little Endian iron, octets are kept as they are found (i.e. the values in messages are in Little Endian representation, yes).

To illustrate the approach above in practice, I've implemented already the read/write for Serpent Keysets into Serpent Messages (section 4.1 of the procotol specification). The communication-level data in this type of message consists in the type ID, the CRC information *for each key* and the message padding (variable-sized since it simply fills the message as needed). The message counter is kept separate since the server for instance will likely need to keep different counters for each client, so I'll leave for now the decision on this for a higher /different level. The keyset itself is the actual data really and it consists in the number of keys, their actual octets and the flag that indicates whether they are for client or for server use. So I define a parametrized record for representing a keyset only rather than the whole message. This is in the new file data_structs.ads that I see at the moment as the place for defining structures for game data. Hence, it contains for instance also the "SMG_Object" record (fixed size) that illustrates some use of representation clauses:

 -- Data structures for SMG Communication Protocol
 -- S.MG, 2018

with Interfaces; -- for fixed-size types
with Raw_Types;  -- for protocol raw types
with System;     -- for Bit_Order
with Serpent;    -- for Serpent.Key type

package Data_Structs is
  Pragma Pure(Data_Structs);

  -- an object in the world, with record layout fully specified
  -- a storage element is 8-bit aka 1 octet
  -- "at" gives number of storage element
  -- "range" gives bits inside storage element

  type SMG_Object is
    record
      -- ID of the object
      ID         : Interfaces.Unsigned_32;

      -- Position of the object in the world.
      -- For a world with map coordinates (MC) between -500 and +500,
      -- the relationship with given figure (GF) is:
      -- MC = GF / 65.535 - 500
      -- where 65.535 comes from the mapping 65535 / (500 - - 500)
      X, Y, Z    : Interfaces.Integer_16;

      -- Rotation of the object (RO) linked to GF by:
      -- RO = GF / 128*pi
      RX, RY, RZ : Interfaces.Unsigned_8;
    end record;
  for SMG_Object'Size use 104; -- in bits!
  for SMG_Object'Bit_Order use System.Low_Order_First;
  for SMG_Object use
    record
      ID at 0  range 0 .. 31;
      X  at 4  range 0 .. 15;
      Y  at 6  range 0 .. 15;
      Z  at 8  range 0 .. 15;
      RX at 10 range 0 .. 7;
      RY at 11 range 0 .. 7;
      RZ at 12 range 0 .. 7;
    end record;

  -------------------------
  -- A set of Serpent Keys
  -- parametrized record for a Serpent Keyset
  -- parameters:
  --   N is number of Serpent Keys contained
  -- this can be the content of messages 4.1 or 5.2 in protocol spec.

  -- an array of Serpent Keys
  -- MAXIMUM 40 keys allowed in a message, hence subtype for key count
  subtype Keys_Count is Interfaces.Unsigned_8 range 1..40;
  type SKeys_Array is array( Keys_Count range <>) of Serpent.Key;

  type Serpent_Keyset( N : Keys_Count := Keys_Count'Last) is
    record
      -- actual Serpent Keys
      Keys      : SKeys_Array( 1..N );
      -- whether for talking to client (LSB set) or talking to server (MSB set)
      Flag      : Interfaces.Unsigned_8;
    end record;

  ------------------------------
  -- Serpent Keys Management
  type Keys_Mgm (N_Burnt: Interfaces.Unsigned_8) is
    record
      -- count of server keys requested
      N_Server: Interfaces.Unsigned_8;
      -- count of client keys requested
      N_Client: Interfaces.Unsigned_8;
      -- ID of Serpent key preferred for further inbound Serpent msgs.
      Key_ID  : Interfaces.Unsigned_8;
      -- IDs of Serpent keys burnt by this message
      Burnt   : SKeys_Array( 1..N_Burnt );
    end record;

end Data_Structs;

The communication-level information such as CRC, type ids and padding are all handled by messages.ads/adb and are effectively private matters, of no concern to anything outside this messages package:

 -- Message reader & writers for SMG Communication Protocol
 -- This part effectively serializes/deserializes game data structures
 --   for transmission between client and server.
 -- Note that messages themeselves are simply arrays of octets.
 -- (see also raw_types.ads/adb and packing.ads/adb for related parts)
 -- NB: message ids and padding as per protocol spec are handled HERE ONLY.
 -- Relies on:
 --   RNG (for random padding)
 --   CRC32 (for CRC calculations)
 --   Raw_Types
 --   Data_Structs
 -- S.MG, 2018

with Raw_Types;
with RNG;
with CRC32;
with Data_Structs; use Data_Structs;
with Interfaces;

package Messages is
  -- exception raised when given message to read fails sanity checks
  Invalid_Msg: exception;

  ------------------------------------------------
  -- Writes a Serpent Keyset to the given Serpent Message
  --
  -- Keyset  - the set of keys to write to message
  -- Counter - the message count
  procedure Write_SKeys_SMsg( Keyset  : in Serpent_Keyset;
                              Counter : in Interfaces.Unsigned_16;
                              Msg     : out Raw_Types.Serpent_Msg);

  -- Reads a Serpent Keyset from the given Serpent Message
  -- The opposite of Write_SKeys_SMsg above
  -- Raises Invalid_Message exception if given message fails sanity checks
  procedure Read_SKeys_SMsg( Msg     : in Raw_Types.Serpent_Msg;
                             Counter : out Interfaces.Unsigned_16;
                             Keyset  : out Serpent_Keyset);

private

  -- if native is little endian, does nothing;
  -- if native is big endian, it flips the input's octets.
  -- (strictly for arrays of octets)
  procedure Cast_LE( LE: in out Raw_Types.Octets );

  -- protocol message types IDs, fixed as per protocol specification
  -- Serpent messages end in "S_Type"
  -- RSA messages end in "R_Type"
  -- Character action types end in "A_Type"

  -- Serpent message types
  SKeys_S_Type         : constant Interfaces.Unsigned_8 := 1;
  Key_Mgm_S_Type       : constant Interfaces.Unsigned_8 := 2;
  File_Transfer_S_Type : constant Interfaces.Unsigned_8 := 3;
  File_Req_S_Type      : constant Interfaces.Unsigned_8 := 4;
  Client_Action_S_Type : constant Interfaces.Unsigned_8 := 5;
  World_Bulletin_S_Type: constant Interfaces.Unsigned_8 := 6;
  Obj_Request_S_Type   : constant Interfaces.Unsigned_8 := 7;
  Obj_Info_S_Type      : constant Interfaces.Unsigned_8 := 8;

  -- RSA message types
  RKeys_R_Type         : constant Interfaces.Unsigned_8 := 251;
  SKeys_R_Type         : constant Interfaces.Unsigned_8 := 157;
  Key_Mgm_R_Type       : constant Interfaces.Unsigned_8 := 233;

  -- Character action types
  Lock_A_Type          : constant Interfaces.Unsigned_8 := 0;
  Make_A_Type          : constant Interfaces.Unsigned_8 := 1;
  Explore_A_Type       : constant Interfaces.Unsigned_8 := 2;
  Exchange_A_Type      : constant Interfaces.Unsigned_8 := 3;
  Attack_A_Type        : constant Interfaces.Unsigned_8 := 4;
  Repair_A_Type        : constant Interfaces.Unsigned_8 := 5;
  Move_A_Type          : constant Interfaces.Unsigned_8 := 6;
  Train_A_Type         : constant Interfaces.Unsigned_8 := 7;

end Messages;

The implementation in messages.adb, including read/write for Serpent keysets to/from Serpent messages and handling endianness for any multi-octet entity:

 -- Message reader & writers for SMG Communication Protocol
 -- S.MG, 2018

with Interfaces; use Interfaces;
with Serpent;
with System; use System;

package body Messages is

  procedure Write_SKeys_SMsg( Keyset  : in Serpent_Keyset;
                              Counter : in Interfaces.Unsigned_16;
                              Msg     : out Raw_Types.Serpent_Msg) is
    Pos   : Integer := Msg'First;
    Check : CRC32.CRC32;
    PadLen: Integer;
    K     : Serpent.Key;
  begin
    -- write Type ID
    Msg(Pos) := SKeys_S_Type;
    Pos := Pos + 1;

    -- write count of keys (NB: this IS 8 bits by definition)
    Msg(Pos) := Keyset.Keys'Length;
    Pos := Pos + 1;

    -- write keys
    for I in Keyset.Keys'Range loop
      -- retrieve Key to write
      K := Keyset.Keys( I );

      -- write key itself
      Msg(Pos..Pos+K'Length-1) := K;
      -- ensure little endian order in message
      Cast_LE(Msg(Pos..Pos+K'Length-1));
      Pos := Pos + K'Length;

      -- write CRC of key
      Check := CRC32.CRC( K );
      Msg(Pos..Pos+3) := Raw_Types.Cast(Check);
      Cast_LE(Msg(Pos..Pos+3));
      Pos := Pos + 4;
    end loop;

    -- write flag
    Msg(Pos) := Keyset.Flag;
    Pos := Pos + 1;

    -- write message counter
    Msg(Pos..Pos+1) := Raw_Types.Cast(Counter);
    Cast_LE(Msg(Pos..Pos+1));
    Pos := Pos + 2;

    -- write padding as needed; endianness is irrelevant here
    PadLen := Msg'Last - Pos + 1;
    if PadLen > 0 then
      declare
        Pad : Raw_Types.Octets(1..PadLen);
      begin
        RNG.Get_Octets( Pad );
        Msg(Pos..Pos+PadLen-1) := Pad;
      end;
    end if;
  end Write_SKeys_SMsg;

  -- Reads a Serpent keyset from given Serpent Message
  procedure Read_SKeys_SMsg( Msg     : in Raw_Types.Serpent_Msg;
                             Counter : out Interfaces.Unsigned_16;
                             Keyset  : out Serpent_Keyset) is
    Pos: Integer := Msg'First;
  begin
    -- read type and check
    if Msg(Pos) = SKeys_S_Type then
      Pos := Pos + 1;
    else
      raise Invalid_Msg;
    end if;

    -- read count of keys and check
    if Msg(Pos) in Keys_Count'Range then
      declare
        N     : Keys_Count := Keys_Count(Msg(Pos));
        KS    : Serpent_Keyset(N);
        K     : Serpent.Key;
        Check : CRC32.CRC32;
        O4    : Raw_Types.Octets_4;
        O2    : Raw_Types.Octets_2;
      begin
        Pos := Pos + 1;
        --read keys and check crc for each
        for I in 1 .. N loop
          -- read key and advance pos
          K := Msg(Pos..Pos+K'Length-1);
          Cast_LE(K);
          Pos := Pos + K'Length;
          -- read crc and compare to crc32(key)
          O4 := Msg(Pos..Pos+3);
          Cast_LE(O4);
          Check   := Raw_Types.Cast(O4);
          Pos := Pos + 4;
          if Check /= CRC32.CRC(K) then
            raise Invalid_Msg;
          end if;
          -- if it got here, key is fine so add to set
          KS.Keys(KS.Keys'First + I -1) := K;
        end loop;
        -- read and set flag
        KS.Flag := Msg(Pos);
        Pos := Pos + 1;
        -- read and set message counter
        O2 := Msg(Pos..Pos+1);
        Cast_LE(O2);
        Counter := Raw_Types.Cast(O2);
        -- rest of message is padding so it's ignored
        -- copy keyset to output variable
        Keyset := KS;
      end;
    else
      raise Invalid_Msg;
    end if;
  end Read_SKeys_SMsg;

  -- private part
  procedure Cast_LE( LE: in out Raw_Types.Octets ) is
  begin
    -- flip octets ONLY if native is big endian.
    if System.Default_Bit_Order = System.High_Order_First then
      declare
        BE: constant Raw_Types.Octets := LE;
      begin
        for I in 1..LE'Length loop
          LE(LE'First+I-1) := BE(BE'Last-I+1);
        end loop;
      end;
    end if;
    -- NOTHING to do for native little endian
  end Cast_LE;

end Messages;

Note that Messages calls directly RNG to obtain padding as and when it needs it. Similarly, it calls directly CRC32 to calculate the checksums as and when needed. The CRC32 is the previously published table-lookup CRC32 implementation from EuCrypt, now integrated into SMG Comms and slightly adapted to use the Octets type in Raw_Types:


------------------------------------------------------------------------------
------------------------------------------------------------------------------
-- This file is part of 'CRC32'                                             --
--                                                                          --
-- You do not have, nor can you ever acquire the right to use, copy or      --
-- distribute this software ; Should you use this software for any purpose, --
-- or copy and distribute it to anyone or in any manner, you are breaking   --
-- the laws of whatever soi-disant jurisdiction, and you promise to         --
-- continue doing so for the indefinite future. In any case, please         --
-- always : read and understand any software ; verify any PGP signatures    --
-- that you use - for any purpose.                                          --
--                                                                          --
-- See also http://trilema.com/2015/a-new-software-licensing-paradigm .     --
------------------------------------------------------------------------------
------------------------------------------------------------------------------

  -- CRC32, lookup-based implementation
  -- S.MG, 2018
  --
  -- The CRC32 is a checksum calculated as the remainder of dividing
  -- the input data by the 0x04C11DB7 polynomial. Specifications:
  -- Name   : "CRC-32"
  -- Width  : 32        (number of bits)
  -- Poly   : 04C11DB7  (generator polynomial)
  -- Init   : FFFFFFFF
  -- RefIn  : True      (input is reflected)
  -- RefOut : True      (output is reflected)
  -- XorOut : FFFFFFFF
  -- Check  : CBF43926  (expected value for input "123456789")
  --
  -- This implementation is based on the CRC32 specification in:
  -- Sarwate, D.V. "Computation of Cyclic Redundancy Checks via Table Look-Up"
  -- in Communications of the ACM, Vol. 31 No. 8, pp.1008-1013, Aug. 1988

with Interfaces; use Interfaces;
with Raw_Types;

package CRC32 is
  -- local, shorthand version of Interfaces. types
  subtype CRC32 is Interfaces.Unsigned_32;
  subtype Octet is Interfaces.Unsigned_8;

  subtype Octet_Array is Raw_Types.Octets;

  -- interface for external callers
    -- calculate CRC32 for the given string
  function CRC( S: in String ) return CRC32;

    -- calculate CRC32 for the given array of octets
  function CRC( Data: in Octet_Array ) return CRC32;

  -- internal constants and helper methods
private
	function Shift_Right( Value  : CRC32;
                        Amount : Natural)
                        return CRC32;
  pragma Import(Intrinsic, Shift_Right); 

  Init_Value : constant CRC32 := 16#FFFF_FFFF#; -- Initial value
  Xor_Out    : constant CRC32 := 16#FFFF_FFFF#; -- For extracting result
  LSB_Mask   : constant CRC32 := 16#0000_00FF#; -- lsb mask for a CRC32 value

  -- lookup table with precomputed values for CRC32
  Lookup : constant array (CRC32 range 0 .. 255) of CRC32 :=
     (16#0000_0000#, 16#7707_3096#, 16#EE0E_612C#, 16#9909_51BA#,
      16#076D_C419#, 16#706A_F48F#, 16#E963_A535#, 16#9E64_95A3#,
      16#0EDB_8832#, 16#79DC_B8A4#, 16#E0D5_E91E#, 16#97D2_D988#,
      16#09B6_4C2B#, 16#7EB1_7CBD#, 16#E7B8_2D07#, 16#90BF_1D91#,
      16#1DB7_1064#, 16#6AB0_20F2#, 16#F3B9_7148#, 16#84BE_41DE#,
      16#1ADA_D47D#, 16#6DDD_E4EB#, 16#F4D4_B551#, 16#83D3_85C7#,
      16#136C_9856#, 16#646B_A8C0#, 16#FD62_F97A#, 16#8A65_C9EC#,
      16#1401_5C4F#, 16#6306_6CD9#, 16#FA0F_3D63#, 16#8D08_0DF5#,
      16#3B6E_20C8#, 16#4C69_105E#, 16#D560_41E4#, 16#A267_7172#,
      16#3C03_E4D1#, 16#4B04_D447#, 16#D20D_85FD#, 16#A50A_B56B#,
      16#35B5_A8FA#, 16#42B2_986C#, 16#DBBB_C9D6#, 16#ACBC_F940#,
      16#32D8_6CE3#, 16#45DF_5C75#, 16#DCD6_0DCF#, 16#ABD1_3D59#,
      16#26D9_30AC#, 16#51DE_003A#, 16#C8D7_5180#, 16#BFD0_6116#,
      16#21B4_F4B5#, 16#56B3_C423#, 16#CFBA_9599#, 16#B8BD_A50F#,
      16#2802_B89E#, 16#5F05_8808#, 16#C60C_D9B2#, 16#B10B_E924#,
      16#2F6F_7C87#, 16#5868_4C11#, 16#C161_1DAB#, 16#B666_2D3D#,
      16#76DC_4190#, 16#01DB_7106#, 16#98D2_20BC#, 16#EFD5_102A#,
      16#71B1_8589#, 16#06B6_B51F#, 16#9FBF_E4A5#, 16#E8B8_D433#,
      16#7807_C9A2#, 16#0F00_F934#, 16#9609_A88E#, 16#E10E_9818#,
      16#7F6A_0DBB#, 16#086D_3D2D#, 16#9164_6C97#, 16#E663_5C01#,
      16#6B6B_51F4#, 16#1C6C_6162#, 16#8565_30D8#, 16#F262_004E#,
      16#6C06_95ED#, 16#1B01_A57B#, 16#8208_F4C1#, 16#F50F_C457#,
      16#65B0_D9C6#, 16#12B7_E950#, 16#8BBE_B8EA#, 16#FCB9_887C#,
      16#62DD_1DDF#, 16#15DA_2D49#, 16#8CD3_7CF3#, 16#FBD4_4C65#,
      16#4DB2_6158#, 16#3AB5_51CE#, 16#A3BC_0074#, 16#D4BB_30E2#,
      16#4ADF_A541#, 16#3DD8_95D7#, 16#A4D1_C46D#, 16#D3D6_F4FB#,
      16#4369_E96A#, 16#346E_D9FC#, 16#AD67_8846#, 16#DA60_B8D0#,
      16#4404_2D73#, 16#3303_1DE5#, 16#AA0A_4C5F#, 16#DD0D_7CC9#,
      16#5005_713C#, 16#2702_41AA#, 16#BE0B_1010#, 16#C90C_2086#,
      16#5768_B525#, 16#206F_85B3#, 16#B966_D409#, 16#CE61_E49F#,
      16#5EDE_F90E#, 16#29D9_C998#, 16#B0D0_9822#, 16#C7D7_A8B4#,
      16#59B3_3D17#, 16#2EB4_0D81#, 16#B7BD_5C3B#, 16#C0BA_6CAD#,
      16#EDB8_8320#, 16#9ABF_B3B6#, 16#03B6_E20C#, 16#74B1_D29A#,
      16#EAD5_4739#, 16#9DD2_77AF#, 16#04DB_2615#, 16#73DC_1683#,
      16#E363_0B12#, 16#9464_3B84#, 16#0D6D_6A3E#, 16#7A6A_5AA8#,
      16#E40E_CF0B#, 16#9309_FF9D#, 16#0A00_AE27#, 16#7D07_9EB1#,
      16#F00F_9344#, 16#8708_A3D2#, 16#1E01_F268#, 16#6906_C2FE#,
      16#F762_575D#, 16#8065_67CB#, 16#196C_3671#, 16#6E6B_06E7#,
      16#FED4_1B76#, 16#89D3_2BE0#, 16#10DA_7A5A#, 16#67DD_4ACC#,
      16#F9B9_DF6F#, 16#8EBE_EFF9#, 16#17B7_BE43#, 16#60B0_8ED5#,
      16#D6D6_A3E8#, 16#A1D1_937E#, 16#38D8_C2C4#, 16#4FDF_F252#,
      16#D1BB_67F1#, 16#A6BC_5767#, 16#3FB5_06DD#, 16#48B2_364B#,
      16#D80D_2BDA#, 16#AF0A_1B4C#, 16#3603_4AF6#, 16#4104_7A60#,
      16#DF60_EFC3#, 16#A867_DF55#, 16#316E_8EEF#, 16#4669_BE79#,
      16#CB61_B38C#, 16#BC66_831A#, 16#256F_D2A0#, 16#5268_E236#,
      16#CC0C_7795#, 16#BB0B_4703#, 16#2202_16B9#, 16#5505_262F#,
      16#C5BA_3BBE#, 16#B2BD_0B28#, 16#2BB4_5A92#, 16#5CB3_6A04#,
      16#C2D7_FFA7#, 16#B5D0_CF31#, 16#2CD9_9E8B#, 16#5BDE_AE1D#,
      16#9B64_C2B0#, 16#EC63_F226#, 16#756A_A39C#, 16#026D_930A#,
      16#9C09_06A9#, 16#EB0E_363F#, 16#7207_6785#, 16#0500_5713#,
      16#95BF_4A82#, 16#E2B8_7A14#, 16#7BB1_2BAE#, 16#0CB6_1B38#,
      16#92D2_8E9B#, 16#E5D5_BE0D#, 16#7CDC_EFB7#, 16#0BDB_DF21#,
      16#86D3_D2D4#, 16#F1D4_E242#, 16#68DD_B3F8#, 16#1FDA_836E#,
      16#81BE_16CD#, 16#F6B9_265B#, 16#6FB0_77E1#, 16#18B7_4777#,
      16#8808_5AE6#, 16#FF0F_6A70#, 16#6606_3BCA#, 16#1101_0B5C#,
      16#8F65_9EFF#, 16#F862_AE69#, 16#616B_FFD3#, 16#166C_CF45#,
      16#A00A_E278#, 16#D70D_D2EE#, 16#4E04_8354#, 16#3903_B3C2#,
      16#A767_2661#, 16#D060_16F7#, 16#4969_474D#, 16#3E6E_77DB#,
      16#AED1_6A4A#, 16#D9D6_5ADC#, 16#40DF_0B66#, 16#37D8_3BF0#,
      16#A9BC_AE53#, 16#DEBB_9EC5#, 16#47B2_CF7F#, 16#30B5_FFE9#,
      16#BDBD_F21C#, 16#CABA_C28A#, 16#53B3_9330#, 16#24B4_A3A6#,
      16#BAD0_3605#, 16#CDD7_0693#, 16#54DE_5729#, 16#23D9_67BF#,
      16#B366_7A2E#, 16#C461_4AB8#, 16#5D68_1B02#, 16#2A6F_2B94#,
      16#B40B_BE37#, 16#C30C_8EA1#, 16#5A05_DF1B#, 16#2D02_EF8D#);

end CRC32;

The CRC32 implementation in crc32.adb:

------------------------------------------------------------------------------
------------------------------------------------------------------------------
-- This file is part of 'CRC32'                                             --
--                                                                          --
-- You do not have, nor can you ever acquire the right to use, copy or      --
-- distribute this software ; Should you use this software for any purpose, --
-- or copy and distribute it to anyone or in any manner, you are breaking   --
-- the laws of whatever soi-disant jurisdiction, and you promise to         --
-- continue doing so for the indefinite future. In any case, please         --
-- always : read and understand any software ; verify any PGP signatures    --
-- that you use - for any purpose.                                          --
--                                                                          --
-- See also http://trilema.com/2015/a-new-software-licensing-paradigm .     --
------------------------------------------------------------------------------
------------------------------------------------------------------------------

  -- CRC32 implementation
  -- S.MG, 2018

package body CRC32 is
  function CRC( S: in String ) return CRC32 is
    Result : CRC32 := Init_Value;
    Value  : CRC32;
  begin
    -- process input character by character
    for C of S loop
      Value  := CRC32( Character'Pos( C ) );
      Result := Shift_Right(Result, 8) xor
                  Lookup( Value xor (Result and LSB_MASK));
    end loop;
    -- reflect result
    Result := Result xor Xor_Out;

    return Result;
  end CRC;

  function CRC( Data: in Octet_Array ) return CRC32 is
    Result : CRC32 := Init_Value;
  begin
    -- process input octet by octet
    for C of Data loop
      Result := Shift_Right(Result, 8) xor
                  Lookup( CRC32(C) xor (Result and LSB_MASK));
    end loop;
    -- reflect result
    Result := Result xor Xor_Out;

    return Result;
  end CRC;

end CRC32;

Other than the above, the .vpatch for this chapter contains also some basic testing of the read/write methods for Serpent keysets:

  -- Tests for the serialization of data structures in SMG Protocol
 -- S.MG, 2018

with RNG;
with Data_Structs; use Data_Structs;
with Messages; use Messages;
with Interfaces; use Interfaces;
with System;
with System.Storage_Elements; use System.Storage_Elements;
with Ada.Text_IO; use Ada.Text_IO;

package body Test_Serializing is

  procedure Serialize_Keyset_SS is
    Msg  : Serpent_Msg;
    KSet : Serpent_Keyset(5);
    LSB : Interfaces.Unsigned_8 := 16#01#;
    MSB : Interfaces.Unsigned_8 := 16#80#;
    LMSB: Interfaces.Unsigned_8 := 16#81#;
    Counter : Interfaces.Unsigned_16 := 101;
    NewSet: Serpent_Keyset;
    NewCounter: Interfaces.Unsigned_16:=0;
  begin
    Put_Line("Generating the Serpent Keys...");
    -- fill a set of Serpent Keys
    for I in 1..KSet.N loop
      RNG.Get_Octets(KSet.Keys(Interfaces.Unsigned_8(I)));
    end loop;
    KSet.Flag := LSB;

    Put_Line("Writing the keys to message...");
    -- write keyset to serpent message
    Write_SKeys_SMsg( KSet, Counter, Msg );

    Put_Line("Reading keys back from message...");
    -- read keyset from message
    Read_SKeys_SMsg( Msg, Counter, NewSet );

    Put_Line("Comparing the keysets...");
    -- compare the two keysets
    if NewSet /= KSet then
      Put_Line("FAIL: keysets are different!");
    else
      Put_Line("PASS: keysets are the same!");
    end if;

    Put_Line("Attempting to read from mangled message");
    begin
      Msg(Msg'First) := Msg(Msg'First)+25;
      Read_SKeys_SMsg( Msg, Counter, NewSet);
      Put_Line("FAIL: read failed to raise invalid message exception!");
    exception
      when Invalid_Msg =>
        Put_Line("PASS: exception correctly raised for invalid message");
    end;
  end Serialize_Keyset_SS;

end Test_Serializing;

As I don't currently have any access to a Big Endian computer, I couldn't quite test the whole thing properly. If you have a Big Endian computer and get around to test this code, please let me know how it behaves.

The above method seems to me robust at least and reasonably easy to follow. Feel free however to loudly disagree and shoot it down in the comments below - I'll gladly read any better proposal. As mentioned in the very beginning of this post, this is not exactly what I hoped to obtain although it is the thing I have so far and therefore the thing I'm publishing - to be discussed and improved on, hopefully.

The .vpatch and my signature for it are as usual on my Reference Code Shelf and linked below for your convenience:

  1. The official solution for anything touching serialization seems to boil down to "Streams or gtfo" apparently. And for my specific case here I don't quite see the point of bringing in the whole Streams mammouth. ↩

November 4, 2018

SMG Comms Chapter 6: Packing and Unpacking RSA

Filed under: Coding, SMG_Comms — Diana Coman @ 1:38 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

Building on the new convenient way to simply call rsa+oaep encrypt/decrypt from Ada, this chapter provides the Pack/Unpack functions that convert between SMG Comms RSA packets (each of them 1470 octets long, effectively several RSA-encrypted chunks glued together) and SMG Comms messages aka the actual content of the packet from the game's point of view. To do this, the Pack/Unpack functions simply chop the input into fixed-size parts, apply rsa+oaep encrypt/decrypt as suitable and then glue back the results in order, providing the whole thing as final output. Note that from the point of view of Pack/Unpack, both packets and messages have fixed lengths, as specified! In other words: the padding that might be added to data structures to get the whole thing to the specified length of a RSA message is preserved - that is no concern at this layer at all1. In a nutshell: Unpack removes any padding that Pack added but preserves any other padding that was added *before* calling Pack. In turn, Pack expects as input a fixed-size array of octets aka a RSA message as defined in Raw_Types - and the size of this is simply a multiple of the maximum size that one single RSA-encrypted operation can handle.

While the above fixed-size helps considerably to keep the whole thing quite easy to follow and clear, things are still a bit more gnarly than with Serpent, there's no escaping that. In particular, the RSA Decrypt operation CAN fail visibly - when the decoded "length" of the actual content turns out to be larger than maximum/provided space. For this reason, the Unpack function has a "Success" flag that gets set to false when output is essentially undefined. Note that the Unpack will fail in this way as soon as one of the chunks fails the RSA+OAEP decrypt in any way - there is no point in even attempting to recover some "partial" thing. Even more importantly: a True value for the Success flag does NOT indicate in any way that the result obtained is valid - all it says is that the RSA+OAEP decrypt operations completed successfully and the result is the one returned but whether that's a valid data structure or not is not a concern at this point (nor could it be since Unpack can't check such a thing in any way).

The .vpatch for this chapter includes the following main changes/additions:

  • A new note is added next to the offending Keccak line, describing the proposed way2 to preserve the No_Implicit_Conditionals restriction if desired. As mentioned in the forum, I do not quite see the need for this change at this moment but perhaps it will be needed at a later time, so there it is, right where it'll come in handy.
  • There's a correction of the value for the constant RSA_MSG_OCTETS. The new, correct value matches the fact that this is meant to be the fixed-size length of a SMG Comms RSA message rather than (as it was before) the max length that can be encrypted with RSA in one operation. So the RSA_MSG_OCTETS is calculated based on other specification-defined constants and it is a multiple of the max length that can be encrypted with RSA in one operation.
  • As part of doing the above correction of RSA_MSG_OCTETS, I also moved the initialization of some of the OAEP constants into Raw_Types (from OAEP packet where they were before). More precisely, the OAEP packet still uses its own internal constants (e.g. the "reserved" octets spelling "TMSR-RSA") but it initialises those (at elaboration time) with the values that it reads from Raw_Types. This reflects the fact that the OAEP packet in this project is the one using the Raw_Types that the protocol defines, not the other way around (in other words: OAEP does not define/impose those values - it's SMG Comms that does!).
  • The Packing package lost its "Pure" status (i.e. the compiler can't rely on it having no internal state and can't therefore cache calls to its method). The reason for this is at root the need for random padding for RSA since yes, can't cache the call to a method that has to use different, random padding at every call. In any case, for anyone interested, the specific route to follow this is that the Pack method for RSA calls OAEP+RSA which in turn use the non-pure Sequential_IO (via the RNG package) to read random octets from a Fuckgoats.
  • The new Pack and Unpack methods for RSA messages <-> packets conversions are in packing.ads/adb.
  • Corresponding tests for the Pack/Unpack methods are added to test_packing.ads/adb and called from testall.adb.

The signatures of the new Pack/Unpack methods in packing.ads are the following:

  -- Packing a RSA message into RSA packet, using the given key
  function Pack( Msg : in Raw_Types.RSA_Msg;
                 K   : in RSA_OAEP.RSA_pkey)
                return Raw_Types.RSA_Pkt;

  -- Unpacking a RSA packet into contained message, using the given key
  function Unpack( Pkt     : in Raw_Types.RSA_Pkt;
                   K       : in RSA_OAEP.RSA_skey;
                   Success : out Boolean)
                  return Raw_Types.RSA_Msg;

And the corresponding code for the new Pack/Unpack in packing.adb:

  -- Packing a RSA message into RSA packet, using the given key
  function Pack( Msg : in Raw_Types.RSA_Msg;
                 K   : in RSA_OAEP.RSA_pkey)
                return Raw_Types.RSA_Pkt is

  -- a chunk that can be processed via rsa+oaep at any given time
    Chunk: Raw_Types.Octets(1..Raw_Types.OAEP_MAX_LEN) := (others => 0);

  -- number of chunks in the message to process
  -- NO incomplete chunks will be processed!
  -- NB: values are set so that there are no incomplete blocks here
    N   : constant Natural := Msg'Length / Chunk'Length;

  -- intermediate result, as obtained from rsa_oaep
    Encr: Raw_Types.RSA_len := (others => 0);

  -- final resulting RSA Packet
    Pkt : Raw_Types.RSA_Pkt := (others => 0);
  begin
    -- there should ALWAYS be precisely N chunks in Msg to feed to rsa_oaep
    -- process chunks of Msg one at a time
    for I in 1..N loop
      -- get current chunk
      Chunk := Msg(Msg'First + (I-1) * Chunk'Length ..
                   Msg'First + I * Chunk'Length - 1 );
      -- call rsa oaep encrypt on current chunk
      RSA_OAEP.Encrypt( Chunk, K, Encr );
      -- copy result to its place in final packet
      Pkt( Pkt'First + (I-1) * Encr'Length ..
           Pkt'First + I * Encr'Length - 1 ) := Encr;
    end loop;
    -- return final result
    return Pkt;
  end Pack;

  -- Unpacking a RSA packet into contained message, using the given key
  function Unpack( Pkt     : in Raw_Types.RSA_Pkt;
                   K       : in RSA_OAEP.RSA_skey;
                   Success : out Boolean)
                  return Raw_Types.RSA_Msg is
    -- a chunk - basically input for RSA_OAEP.Decrypt
    Chunk : Raw_Types.RSA_len := (others => 0);

    -- intermediate result of rsa_oaep decrypt
    Decr  : Raw_Types.Octets( 1..Raw_Types.OAEP_MAX_LEN ) := (others => 0);
    Len   : Natural;
    Flag  : Boolean;

    -- number of chunks in the packet
    -- NB: there should be only FULL chunks! otherwise -> fail
    N   : constant Natural := Pkt'Length / Chunk'Length;

    -- final resulting message content of the given RSA packet
    Msg : Raw_Types.RSA_Msg := (others => 0);
  begin
    -- initialize Success flag
    Success := True;

    -- process given packet, chunk by chunk
    for I in 1..N loop
      -- get current chunk
      Chunk := Pkt( Pkt'First + (I-1) * Chunk'Length ..
                    Pkt'First + I * Chunk'Length - 1 );
      -- decrypt it via rsa+oaep
      RSA_OAEP.Decrypt( Chunk, K, Decr, Len, Flag );
      -- check result and if ok then copy it to final result at its place
      -- NB: if returned length is EVER less than OAEP_MAX_LEN then -> fail!
      -- the reason for above: there will be undefined bits in the output!
      if Len /= Raw_Types.OAEP_MAX_LEN or (not Flag) then
        Success := False;
        return Msg;
      else
        Msg( Msg'First + (I-1) * Decr'Length ..
             Msg'First + I * Decr'Length - 1 ) := Decr;
      end if;
    end loop;

    -- return obtained message
    return Msg;
  end Unpack;

The new tests for the above, in test_packing.adb:

  procedure Test_Pack_Unpack_RSA is
    Msg       : RSA_Msg;
    Decr_Msg  : RSA_Msg;
    PKey      : RSA_pkey;
    SKey      : RSA_skey;
    Success   : Boolean;
    Pkt       : RSA_Pkt;
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
  begin
    -- initialize RSA key with values previously obtained from EuCrypt
    Hex2Octets( n, SKey.n );
    Hex2Octets( e, SKey.e );
    Hex2Octets( d, SKey.d );
    Hex2Octets( p, SKey.p );
    Hex2Octets( q, SKey.q );
    Hex2Octets( u, SKey.u );
    -- copy n and e for public key
    PKey.n := SKey.n;
    PKey.e := SKey.e;
    -- get random data for "message"
    RNG.Get_Octets(Msg);

    -- pack the message into corresponding packet
    Pkt := Packing.Pack( Msg, PKey );
    -- unpack and check the result
    Decr_Msg := Packing.Unpack( Pkt, SKey, Success );
    if (not Success) or (Decr_Msg /= Msg) then
      Put_Line("FAIL: pack/unpack with RSA.");
    else
      Put_Line("PASS: pack/unpack with RSA.");
    end if;

    -- try to unpack a mangled package
    Pkt(Pkt'First + 3) := Pkt(Pkt'First + 3) and 16#AB#;
    Decr_Msg := (others => 0);
    Decr_Msg := Packing.Unpack( Pkt, SKey, Success );
    if Success then
      Put_Line("FAIL: pack/unpack with RSA on mangled package.");
    else
      Put_Line("PASS: pack/unpack with RSA on mangled package.");
    end if;

  end Test_Pack_Unpack_RSA;

The .vpatch with all the above is on my Reference Code Shelf and linked below for your convenience:

UPDATE (5 November 2018): I've changed the tests so that they read the RSA keypair from a file rather than using a hard-coded key; as discussed in the logs, this is the only way I can see to keep all lines in code files under 80 columns while also keeping RSA components as they are i.e. well longer than 80 characters in any sane format. The added procedure simply reads key components in a rather blunt way, character by character and demanding *exactly* as many characters as needed to fill the length of key it expects. This is NOT a generic "key reading" but simply a quick switch of input for the basic tests:

  procedure ReadRSAKey( Filename: in String; Key: out RSA_OAEP.RSA_skey ) is
    package Char_IO is new Ada.Sequential_IO(Character);
    use Char_IO;
    Full : String(1..RSA_len'Length*2) := (others => '0');
    Half : String(1..RSA_half'Length*2) := (others => '0');
    F    : Char_IO.File_Type;
    C    : Character;
  begin
    Open( File => F, Mode => In_File, Name => Filename );

    -- read n
    for I in Full'Range loop
      Read(F, Full(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Full, Key.n);

    -- read e
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.e);

    -- read d
    for I in Full'Range loop
      Read(F, Full(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Full, Key.d);

    -- read p
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.p);

    -- read q
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    -- read new line character and convert to hex
    Read(F, C);
    Hex2Octets(Half, Key.q);

    -- read u
    for I in Half'Range loop
      Read(F, Half(I));
    end loop;
    Hex2Octets(Half, Key.u);

    -- Close file
    Close( F );

    exception
      when Char_IO.End_Error =>
        Put_Line("ReadRSAKey ERROR: Unexpected end of file in " & Filename);
      when others =>
        Put_Line("ReadRSAKey ERROR: can not open file " & Filename);

  end ReadRSAKey;

Since this change of inputs for tests has nothing to do with either protocol code per se or with advancing the protocol implementation, I'm considering it more of a fix of sorts to this chapter's code and so it'll be a .vpatch by itself, to be found both on the Reference Code Shelf and linked from here:

UPDATE (12 November 2018): I've refactored the tests to extract the reading of a RSA key in a package of its own and thus use it afterwards from everywhere it's needed (i.e. not only in testing oaep_rsa but also in testing the packing for instance). Since this happened after I published Chapter 7, it's a new .vpatch on top:

  1. It will be handled at a higher level, where data structures are populated. ↩

  2. Thank you, asciilifeform! ↩

October 31, 2018

SMG Comms Chapter 5: RSA with OAEP from Ada

Filed under: Coding, SMG_Comms — Diana Coman @ 5:35 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

This brings together RSA and OAEP as well as the raw representation as octets of all the things involved: true random padding, RSA key components, plain messages, encrypted outputs, OAEP blocks or Keccak hashes. The main goal was to have some clear and easy to use methods in Ada for doing the whole RSA+OAEP dance from start to finish and then back as needed. Due to the current situation re MPI and RSA, a significant part of this goal involves keeping the C swamp at bay: calling C methods only when there really is at the moment no alternative and in general as few times as possible, making sure that anything built on top of this part of smg_comms can effectively ignore that there is any C code involved at a lower level at all. The reason for insulating the C code away like this is of course that I hope to be able to get rid of it entirely at some point and have everything in Ada without having to touch anything higher up than this part right here. At some point, but not at this point here, not just yet.

At first glance, this chapter was meant to be quite easy and straightforward: glue the different parts together and that's it! As usual, the details were more gnarly and as a result there were a few things I had to change/re-implement in order to get this part into some sort of reasonable shape:

  • I've added a few new subtypes of Octets to Raw_Types: RSA_len and RSA_half to hold a component of precisely the length or, respectively, half of the length set for the modulus of a RSA key. Those are needed to store the RSA keys in raw format aka arrays of octets. While I could also use the generic Octets type, the clear specification of length reflects the expected lengths of each RSA component and brings clarity so it's well worth it in my books.
  • The random padding is now obtained directly from Ada code rather than via C_wrappers and truerandom.c. The new Ada code simply reads from a Fuckgoats that it expects to be available at a pre-set path, *already initialised*1 and ready to read from. This means that random octets do not require anymore a call to C and the corresponding conversions from array of octets to char * and back.
  • The comparison between an OAEP block and the RSA key's modulus is simplified: it's only the top octet of both that is compared. This was discussed previously in the logs but at any rate, the reason for this approach is that it greatly reduces the complexity of the code since it's really a one line compare of 2 8-bit values that are already available as such at the very spot where they are needed! So no more calling the C swamp, converting to char* and then to MPIs and then comparing the full MPIs and returning the result through to Ada. The potential additional OAEP encryption performed at times is considered a price well worth paying in order to avoid the whole Ada->C->MPI->ADA dance that needs to be performed otherwise *each and every time*.
  • The conversions from Ada's array of octets to C's char* are done in local code, octet by octet and strictly on arrays that are precisely the same size. I am aware that GNAT provides in principle methods for such conversions, namely To_Ada and To_C but they still fail to convince me that they provide a better service really. If you think they do though, feel free to state your case in the comments below.
  • The OAEP implementation is the one from EuCrypt but with different access/interface methods: instead of accepting and producing Strings like the EuCrypt version, the OAEP package in SMG COMMS accepts and produces arrays of octets (i.e. Raw_Types.Octets type) since this is effectively what is needed here. Not to mention that I'm not particularly sad to get rid of Strings even when they are fixed size. Obviously, this means that I also replaced the conversions String <-> Bitstream with Octets <-> Bitstream. The rest however stays the same (including the underlying Keccak for hashes).
  • Because of moving all the OAEP+RSA part to Ada and imposing only one-way calls from Ada to C, there was no need anymore for some of the wrappers that were provided in EuCrypt's OAEP and rsa_oaep code. Those were simply discarded so that the code you get by pressing this patch includes simply what is needed for SMG Comms and nothing more.
  • Whenever the Ada code (specifically the rsa+oaep code) retrieves an array of octets from C, it will basically reverse the "normalization" that the MPI lib performs so that one doesn't end up in the idiotic situation where the retrieved value is actually different than the one given to MPI just because of leading zeroes. This is part of the price paid for using the MPI lib and at any rate it should not be the concern of any of the calling code so it's kept well separated so that any callers of RSA_OAEP.Encrypt or RSA_OAEP.Decrypt don't have to deal with it directly: the encryption uses at most msg_max_len octets and produces a fixed-size array while the decrypt similarly requires the same fixed-size array as input and will produce as output the actual length of the plain message and the plain message itself.
  • The "No Implicit Conditionals" restriction for the whole smg comms had to be discarded because of the Keccak implementation - with the restriction applied, compilation of Keccak fails2. At the moment I don't consider this to be a big issue here but feel free to loudly disagree in the comments below - if you have a solid case, I can change my mind.
  • I've updated the relevant tests for RSA (including those from the C_Wrappers package previously published) to use RSA keys of the new size that SMG Comms uses (as opposed to the EuCrypt size). I've added also new tests for the new RSA+OAEP Ada code, of course.

The new Ada code in rng.ads for reading random octets directly from an FG that is already initialised:

 -- True Random octets generator reading from an FG and using Raw_Types.
 -- S.MG, 2018

with Ada.Sequential_IO;
with Interfaces;
with Raw_Types;

package RNG is

  -- for reading from the FG one Octet at a time
  package Octet_IO is new Ada.Sequential_IO(Element_Type =>
                                              Interfaces.Unsigned_8);

  -- path to the FG; NB: the FG has to be initialized prior to using SMG Comms!
  RNG_PATH: constant String := "/dev/ttyUSB0";

  -- exception raised when FG is not accessible / read fails
  FG_Failure : exception;

  -- reads O'Length octets from the FG at RNG_PATH
  procedure Get_Octets( O: out Raw_Types.Octets );
end RNG;

And the corresponding rng.adb, short and sweet:

 -- S.MG, 2018

with Ada.Exceptions; use Ada.Exceptions;

package body RNG is

  procedure Get_Octets( O: out Raw_Types.Octets ) is
    F : Octet_IO.File_Type;
  begin
    begin
      Octet_IO.Open( File => F, Mode => Octet_IO.In_File, Name => RNG_PATH );
      for I in O'Range loop
        Octet_IO.Read( F, O(I) );
      end loop;
      Octet_IO.Close(F);
    exception
      when others => Raise_Exception(FG_Failure'Identity,
                                     "Failed to access default RNG source!");
    end;
  end Get_Octets;

end RNG;

The code for doing oaep+rsa encrypt from Ada, in rsa_oaep.ads:

 -- Ada implementation of RSA with OAEP according to TMSR and Eulora spec
 -- Uses:
 --   - Eulora's raw types (Ada, raw_types.ads)
 --   - Keccak hashes (Ada, keccak.ads/adb)
 --   - OAEP schema (Ada, oaep.ads/adb)
 --   - RNG (Ada, rng.ads/adb) for true random padding
 --   - C wrappers lib (C, c_wrappers/) for:
 --       - MPI (C, mpi/)
 --       - RSA (C, rsa/rsa.c)
 --
 -- S.MG, 2018

with Raw_Types;
with Interfaces.C; use Interfaces.C;

package RSA_OAEP is
  -- exception for mismatched lengths when converting octets <-> char arrays
  Mismatched_Lengths_Error: exception;

  -- public RSA key with n,e stored as raw octets
  type RSA_pkey is
    record
      n : Raw_Types.RSA_len; --public modulus
      e : Raw_Types.RSA_half; --public exponent
    end record; --RSA_pkey

  -- private (secret) RSA key with components stored as raw octets
  type RSA_skey is
    record
      n : Raw_Types.RSA_len;  --public modulus
      e : Raw_Types.RSA_half; --public exponent
      d : Raw_Types.RSA_len;  --secret exponent e*d=1 mod phi; phi=(p-1)*(q-1)
      p : Raw_Types.RSA_half; --prime p
      q : Raw_Types.RSA_half; --prime q
      u : Raw_Types.RSA_half;  --inverse of p mod q; for faster calculations
    end record; --RSA_skey

  -- Encryption RSA+OAEP (i.e. using public key)
  -- NB: at most OAEP.MAX_LEN_MSG octets from Plain will be encrypted!
  -- Relies directly on:
  --   oaep.adb/ads
  --   c_wrappers.c ( RSA )
  --   rng.adb/ads ( random padding )
  procedure Encrypt( Plain: in Raw_Types.Octets;
                     Key  : in RSA_pkey;
                     Encr : out Raw_Types.RSA_len );

  -- Decryption RSA+OAEP (i.e. using private/secret key)
  -- The opposite of Encrypt above.
  -- NB: Plain has to have ENOUGH space for result!
  -- Result can be at most OAEP.MAX_LEN_MSG octets.
  -- Relies directly on:
  --   oaep.adb/ads
  --   c_wrappers.c (RSA)
  -- Plain_Len gives the length in OCTETS of the decrypted message.
  procedure Decrypt( Encr      : in Raw_Types.RSA_len;
                     Key       : in RSA_skey;
                     Plain     : out Raw_Types.Octets;
                     Plain_Len : out Natural;
                     Success   : out Boolean);

 --helper methods:
 -- mainly conversions to/from C's char* and imports from C_wrappers
  -- encrypt with public RSA key given as struct, Ada style
  -- NB: result is potentially 0-led (i.e. at end of Encr not at start!)
  procedure Public_RSA( Plain: in Raw_Types.Octets;
                        Key  : in RSA_pkey;
                        Encr : out Raw_Types.RSA_len);

  -- encrypt with public RSA key given as char arrays, via C_Wrappers
  -- this returns the length of result because C functions trim leading 0s!
  function Public_RSA_C( Encr      : out Interfaces.C.char_array;
                         Encr_Len  : in Integer;
                         Plain     : in Interfaces.C.char_array;
                         Plain_Len : in Integer;
                         RSA_N     : in Interfaces.C.char_array;
                         N_Len     : in Integer;
                         RSA_E     : in Interfaces.C.char_array;
                         E_Len     : in Integer)
                     return Integer;
   pragma Import(C, Public_RSA_C, "public_rsa_octets"); 

  -- decrypt with private RSA key given as struct, Ada style
  -- NB: Plain has to have ENOUGH space!
  -- NB: Result is potentially 0-led (i.e. at the end of Plain, not at start!)
  -- @return actual length of result
  procedure Private_RSA( Encr  : in Raw_Types.RSA_len;
                         Key   : in RSA_skey;
                         Plain : out Raw_Types.Octets);

  -- encrypt with private/secret RSA key given as char arrays (via C_wrappers)
  -- this returns length because C methods trim leading 0s
  function Private_RSA_C( Plain     : out Interfaces.C.char_array;
                          Plain_Len : in Integer;
                          Encr      : in Interfaces.C.char_array;
                          Encr_Len  : in Integer;
                          RSA_N     : in Interfaces.C.char_array;
                          N_Len     : in Integer;
                          RSA_E     : in Interfaces.C.char_array;
                          E_Len     : in Integer;
                          RSA_D     : in Interfaces.C.char_array;
                          D_Len     : in Integer;
                          RSA_P     : in Interfaces.C.char_array;
                          P_Len     : in Integer;
                          RSA_Q     : in Interfaces.C.char_array;
                          Q_Len     : in Integer;
                          RSA_U     : in Interfaces.C.char_array;
                          U_Len     : in Integer)
                      return Integer;
  pragma Import( C, Private_RSA_C, "private_rsa_octets" );

  -- convert from Ada's Octets (array of octets) to C's char* (char_array)

  -- This copies the octets from O to the beginning of A
  -- NB: there are NO checks or memory allocations here!
  -- Caller has to make sure that:
  --   A has allocated space for at least O'Length octets!
  procedure Octets_To_Char_Array( O   : in Raw_Types.Octets;
                                  A   : out Interfaces.C.char_array);

  -- This copies first O'Length characters from A to O
  -- NB: this does NOT allocate /check memory!
  -- Caller has to ensure that:
  --    A has space at least O'Length characters
  procedure Char_Array_To_Octets( A   : in Interfaces.C.char_array;
                                  O   : out Raw_Types.Octets);

end RSA_OAEP;

With the corresponding implementation in rsa_oaep.adb:

 -- S.MG, 2018

with Interfaces; use Interfaces;
with OAEP;
with RNG;

package body RSA_OAEP is

  -- OAEP + RSA with given public key.
  -- Steps:
  -- 1. repeat oaep.encrypt of Plain
  --      until the result has first octet < than first octet of Key's modulus.
  -- 2. RSA.encrypt with Key on the result from 1.
  procedure Encrypt( Plain: in Raw_Types.Octets;
                     Key  : in RSA_pkey;
                     Encr : out Raw_Types.RSA_len ) is
    -- block obtained through OAEP padding of given input Plain
    Blk     : OAEP.OAEP_Block := ( others => 0 );
    Entropy : OAEP.OAEP_Block;
    Len     : constant Natural := entropy'Length;
  begin
    -- loop cond: 1st octet of oaep block is < 1st octet of key's modulus
    loop
      -- get random bits via RNG (FG)
      RNG.Get_Octets( Entropy );

      -- oaep encrypt
      OAEP.OAEP_Encrypt( Plain, Entropy, Blk );

      -- check that the oaep block is suitable i.e. comparison of 1st octet
      if Blk(Blk'First) < Key.n(Key.n'First) then exit; end if;

    end loop; -- oaep block < modulus

    -- RSA encrypt
    Public_RSA( Blk, Key, Encr );
  end; --Encrypt (OAEP+RSA)

  -- RSA+OAEP Decrypt
  -- Steps:
  -- 1. RSA Decrypt (secret key)
  -- 2. OAEP Decrypt
  procedure Decrypt( Encr : in Raw_Types.RSA_len;
                     Key  : in RSA_skey;
                     Plain: out Raw_Types.Octets;
                     Plain_Len: out Natural;
                     Success: out Boolean) is
    Blk     : OAEP.OAEP_Block;
    Msg     : OAEP.OAEP_HALF;
    Msg_Len : Natural;
  begin
    -- RSA decrypt with provided secret key
    -- NB: result HAS TO BE AN OAEP BLOCK HERE! Even if - potentially - 0 led!
    Private_RSA( Encr, Key, Blk );

    -- OAEP decrypt
    OAEP.OAEP_Decrypt(Blk, Msg_Len, Msg, Success);
    -- switch to Length in OCTETS!
    Msg_Len := Msg_Len / 8;

    -- check that result FITS in given output array - otherwise, fail
    if Msg_Len > Plain'Length then
      Success := FALSE;
      Plain_Len := Msg_Len; --to give a clue to caller
    else
      Plain_Len := Msg_Len;
      Plain( Plain'First ..
             Plain'First + Plain_Len-1) := Msg( Msg'First ..
                                                Msg'First + Plain_Len -1 );
    end if;
  end Decrypt;

  -- helper methods
  -- encrypt with public RSA key given as struct, Ada style
  procedure Public_RSA( Plain: in Raw_Types.Octets;
                        Key  : in RSA_pkey;
                        Encr : out Raw_Types.RSA_len) is
    Encr_char  : char_array( size_t(Encr'First) .. size_t(Encr'Last));
    Plain_char : char_array( size_t(Plain'First) .. size_t(Plain'Last));
    N_char     : char_array( size_t(Key.n'First) .. size_t(Key.n'Last));
    E_char     : char_array( size_t(Key.e'First) .. size_t(Key.e'Last));
    out_len    : Integer;
  begin
    -- convert to char array
    Octets_To_Char_Array( Plain, Plain_char );
    Octets_To_Char_Array( Key.n, N_char );
    Octets_To_Char_Array( Key.e, E_char );
    -- call C imported function
    out_len := Public_RSA_C( Encr_char  , Encr'Length,
                             Plain_char , Plain'Length,
                             N_char     , Key.n'Length,
                             E_char     , Key.e'Length);
    -- convert back to octets
    Char_Array_To_Octets( Encr_char, Encr );
    -- C code trims leading 0s -> need to move octets if out_len0);
    end if;
    -- no need to return anything!
  end Public_RSA;

  procedure Private_RSA( Encr  : in Raw_Types.RSA_len;
                         Key   : in RSA_skey;
                         Plain : out Raw_Types.Octets) is
    Plain_Char : char_array( size_t(Plain'First) .. size_t(Plain'Last) );
    Plain_Len  : Integer;
    Encr_Char  : char_array( size_t(Encr'First)  .. size_t(Encr'Last) );
    N_Char     : char_array( size_t(Key.n'First) .. size_t(Key.n'Last) );
    E_Char     : char_array( size_t(Key.e'First) .. size_t(Key.e'Last) );
    D_Char     : char_array( size_t(Key.d'First) .. size_t(Key.d'Last) );
    P_Char     : char_array( size_t(Key.p'First) .. size_t(Key.p'Last) );
    Q_Char     : char_array( size_t(Key.q'First) .. size_t(Key.q'Last) );
    U_Char     : char_array( size_t(Key.u'First) .. size_t(Key.u'Last) );
  begin
    -- convert key and encrypted message to C char_arrays
    Octets_To_Char_Array( Encr, Encr_Char );
    Octets_To_Char_Array( Key.n, N_Char );
    Octets_To_Char_Array( Key.e, E_Char );
    Octets_To_Char_Array( Key.d, D_Char );
    Octets_To_Char_Array( Key.p, P_Char );
    Octets_To_Char_Array( Key.q, Q_Char );
    Octets_To_Char_Array( Key.u, U_Char );
    -- call RSA decrypt via C_Wrappers
    Plain_Len := Private_RSA_C( Plain_Char, Plain'Length,
                                Encr_Char , Encr'Length,
                                N_Char    , Key.n'Length,
                                E_Char    , Key.e'Length,
                                D_Char    , Key.d'Length,
                                P_Char    , Key.p'Length,
                                Q_Char    , Key.q'Length,
                                U_Char    , Key.u'Length);
    -- convert result back to Octets
    Char_Array_To_Octets( Plain_Char, Plain );
    -- if length < OAEP_Block'Length,it's 0-led and got trimmed, so move it
    if Plain_Len < Plain'Length then
      Plain(Plain'Last-Plain_Len+1..Plain'Last):= Plain(Plain'First ..
                                                   Plain'First + Plain_Len -1);
      Plain(Plain'First .. Plain'Last-Plain_Len) := (others => 0);
    end if;
    -- no need to return anything!
  end Private_RSA;

  procedure Octets_To_Char_Array( O   : in Raw_Types.Octets;
                                  A   : out Interfaces.C.char_array) is
  begin
    -- check that lengths ARE THE SAME!
    if A'Length /= O'Length then
      raise Mismatched_Lengths_Error;
    end if;
    -- copy values over octet by octet
    for I in 0 .. O'Length-1 loop
      A( A'First + Interfaces.C.size_t( I )) :=
                     Interfaces.C.Char( Character'Val(O(O'First + I)));
    end loop;
  end Octets_To_Char_Array;

  procedure Char_Array_To_Octets( A   : in Interfaces.C.char_array;
                                  O   : out Raw_Types.Octets) is
  begin
    -- check that lengths ARE THE SAME!
    if A'Length /= O'Length then
      raise Mismatched_Lengths_Error;
    end if;
    -- copy values over octet by octet
    for I in 0..O'Length -1 loop
      O( O'First + I ) := Character'Pos( Character(
                             A( A'First + Interfaces.C.size_t( I )) ));
    end loop;
  end Char_Array_To_Octets;

end RSA_OAEP;

The new oaep.ads using Raw_Types.Octets:

-- Implementation of TMSR's OAEP with Keccak as hash function
-- NB: this uses Eulora's protocol constants (esp. RSA key length) and types.
--
-- S.MG, 2018

with Keccak;    -- Keccak is used as hash function
with Raw_Types; -- Eulora's protocol raw types and constant values

with Interfaces; use Interfaces; -- for Unsigned_8 type and bit-level ops

package OAEP is
  pragma Pure( OAEP ); -- stateless, no side effects -> can cache calls

  -- constants for OAEP
  OAEP_LENGTH_OCTETS : constant := Raw_Types.RSA_KEY_OCTETS;
  OAEP_LENGTH_BITS   : constant := OAEP_LENGTH_OCTETS * 8;
  OAEP_HALF_OCTETS   : constant := OAEP_LENGTH_OCTETS / 2;
  TMSR_STR           : constant String := "TMSR-RSA";
    -- "TMSR-RSA" as unsigned_8 values:
  TMSR               : constant Raw_Types.Octets := (84,77,83,82,45,82,83,65);
  MAX_LEN_MSG        : constant Natural := OAEP_HALF_OCTETS -
                                           TMSR_STR'Length - 3;

  -- subtypes for OAEP encrypt/decrypt
  subtype OAEP_Block is Raw_Types.Octets( 1 .. OAEP_LENGTH_OCTETS );
  subtype OAEP_HALF is Raw_Types.Octets( 1 .. OAEP_HALF_OCTETS );

  -- padding & formatting of maximum MAX_LEN_MSG octets of the given input
  -- uses TMSR's OAEP schema:
  -- 1.format M00 as: [random octet][sz1][sz2]"TMSR-RSA"[random]*Message
  --    where sz1 and sz2 store the length of the message in bits
  --    the random octets before message are padding to make OAEP_LENGTH_OCTETS
  -- 2. R = OAEP_HALF_OCTETS random bits
  -- 3. X = M00 xor hash(R)
  -- 4. Y = R xor hash(X)
  -- 5. Result is X || Y
  -- NB: the Entropy parameter should be random octets from which this method
  -- will use as many as required for the OAEP encryption of given Msg
  -- NB: at MOST MAX_LEN_MSG octets of Msg! (Msg at most MAX_LEN_MSG*8 bits!)
  procedure OAEP_Encrypt( Msg      : in Raw_Types.Octets;
                          Entropy  : in OAEP_Block;
                          Output   : out OAEP_Block);

  -- This is the opposite of OAEP_Encrypt above.
  -- @param Encr - an OAEP block previously obtained from OAEP_Encrypt
  -- @param Len - this will hold the length of the obtained message (in bits!)
  -- @param Output - the first Len octets of this are the recovered message
  -- @param Success - set to TRUE if message was recovered, false otherwise
  -- NB: when Success is FALSE, both Len and Output have undefined values
  procedure OAEP_Decrypt( Encr    : in OAEP_Block;
                          Len     : out Natural;
                          Output  : out OAEP_HALF;
                          Success : out Boolean);

private
  -- gnat-specific methods for bit-level operations
	function Shift_Right( Value  : Unsigned_8;
                        Amount : Natural )
                        return Unsigned_8;
  pragma Import(Intrinsic, Shift_Right); 

	function Shift_Left( Value  : Unsigned_8;
                        Amount : Natural )
                        return Unsigned_8;
  pragma Import(Intrinsic, Shift_Left); 

  -- helper method: xor 2 half-oaep blocks
  function XOR_Octets(A : in OAEP_HALF;
                      B : in OAEP_HALF)
                     return OAEP_HALF;

  -- conversions between bitstream and string
  -- NB: caller has to ensure correct size of output parameter! no checks here.
  procedure ToOctets   ( B : in Keccak.Bitstream;
                         O : out Raw_Types.Octets );

  procedure ToBitstream( O : in Raw_Types.Octets;
                         B : out Keccak.Bitstream );

  -- wrapper for Sponge to use Octets for input/output
  procedure HashKeccak( Input     : in Raw_Types.Octets;
                        Output    : out Raw_Types.Octets;
                        Block_Len : in Keccak.Keccak_Rate :=
                                       Keccak.Default_Bitrate);

end OAEP;

And the implementation in oaep.adb:

-- S.MG, 2018

package body OAEP is

  -- padding & formatting of maximum MAX_LEN_MSG*8 bits of the given input
  -- uses TMSR's OAEP schema:
  -- 1.format M00 as: [random octet][sz1][sz2]"TMSR-RSA"[random]*Message
  --    where sz1 and sz2 store the length of the message in bits
  --    the random octets before message are padding to make OAEP_LENGTH_OCTETS
  -- 2. R = OAEP_HALF_OCTETS random bits
  -- 3. X = M00 xor hash(R)
  -- 4. Y = R xor hash(X)
  -- 5. Result is X || Y
  -- NB: the Entropy parameter should be random octets from which this method
  -- will use as many as required for the OAEP encryption of given Msg
  -- NB: at MOST MAX_LEN_MSG octets of Msg! (Msg at most 1960 bits)
  procedure OAEP_Encrypt( Msg     : in Raw_Types.Octets;
                          Entropy : in OAEP_Block;
                          Output  : out OAEP_Block) is
    M00    : OAEP_HALF;
    R      : OAEP_HALF;
    HashR  : OAEP_HALF;
    X      : OAEP_HALF;
    HashX  : OAEP_HALF;
    Y      : OAEP_HALF;
    MsgLen : Natural;
    PadLen : Natural;
  begin
    -- calculate maximum length of msg and needed amount of padding
    -- make sure also that only MAX_LEN_MSG octets at most are used from Msg
    MsgLen := Msg'Length;                          -- real msg length
    if MsgLen > MAX_LEN_MSG then
      MsgLen := MAX_LEN_MSG;  --only first MAX_LEN_MSG octets used
      PadLen := 0;            --no padding needed
    else
      PadLen := MAX_LEN_MSG - MsgLen; -- add padding as needed
    end if;

    -- step 1: header and format to obtain M00
      -- first octet is random bits
    M00( M00'First ) := Entropy( Entropy'First );

      -- next 2 octets hold the used length of Msg (number of octets)
    M00( M00'First + 2) := Unsigned_8( ( MsgLen * 8 ) mod 256 );
    M00( M00'First + 1) := Unsigned_8( ( (MsgLen * 8 ) / 256 ) mod 256 );

      -- next 8 octets are reserved for later use, currently "TMSR-RSA"
    M00( M00'First + 3 .. M00'First + 10 ) := TMSR;

      -- random bits for padding, if Msg is less than maximum length
    for I in 1 .. PadLen loop
      M00( M00'First + 10 + I ) := Entropy( Entropy'First + I );
    end loop;

      -- the message itself
    M00( M00'Last - MsgLen + 1 .. M00'Last ) :=
                               Msg( Msg'First .. Msg'First + MsgLen - 1 );

    -- step 2: R = Raw_Types.OAEP_HALF_OCTETS random octets
    -- can take LAST octets from given entropy as they are NOT used before
    -- (even if original message was empty, padding uses at most half - 10
    --   while entropy has full block length)
    R := Entropy( Entropy'Last - OAEP_HALF_OCTETS + 1 .. Entropy'Last );

    -- step 3: X = M00 xor hash(R)
    HashKeccak( R, HashR );
    X := XOR_Octets(M00, HashR);

    -- step 4: Y = R xor hash(X)
    HashKeccak( X, HashX );
    Y := XOR_Octets(R, HashX);

    -- step 5: Output is X || Y
    Output( Output'First .. Output'First + X'Length - 1 ) := X;
    Output( Output'Last - Y'Length + 1 .. Output'Last )   := Y;

  end OAEP_Encrypt;

  procedure OAEP_Decrypt( Encr    : in OAEP_Block;
                          Len     : out Natural;
                          Output  : out OAEP_HALF;
                          Success : out Boolean ) is
    X, Y, M, R   : OAEP_HALF;
    HashX, HashR : OAEP_HALF;
    LenOctets    : Natural;
  begin
    -- step 1: separate X and Y
    X := Encr( Encr'First .. Encr'First + X'Length - 1 );
    Y := Encr( Encr'Last - Y'Length + 1 .. Encr'Last );

    -- step 2: R = Y xor hash(X)
    HashKeccak( X, HashX );
    R := XOR_Octets(Y, HashX);

    -- step 3: M = X xor hash(R)
    HashKeccak( R, HashR );
    M := XOR_Octets(X, HashR);

    -- step 4: extract length and message
    Len := Natural(M( M'First + 1 )) * 256 +
           Natural(M( M'First + 2 ));
    LenOctets := Len / 8; 

    if LenOctets > MAX_LEN_MSG or LenOctets < 0 then
      Success := False;  -- error, failed to retrieve message
    else
      Success := True;
      Output( Output'First .. Output'First + LenOctets - 1 ) :=
        M( M'Last - LenOctets + 1 .. M'Last );
    end if;

  end OAEP_Decrypt;

-- private, helper methods
  procedure HashKeccak(Input     : in Raw_Types.Octets;
                       Output    : out Raw_Types.Octets;
                       Block_Len : in Keccak.Keccak_Rate :=
                                      Keccak.Default_Bitrate) is
    BIn  : Keccak.Bitstream( 0 .. Input'Length * 8 - 1 );
    BOut : Keccak.Bitstream( 0 .. Output'Length * 8 - 1 );
  begin
    ToBitstream( Input, BIn );
    Keccak.Sponge( BIn, BOut, Block_Len );
    ToOctets( BOut, Output );
  end HashKeccak;

  function XOR_Octets(A : in OAEP_HALF;
                      B : in OAEP_HALF)
                     return OAEP_HALF is
    R : OAEP_HALF;
  begin
    for I in R'Range loop
      R(I) := A(I) xor B(I);
    end loop;
    return R;
  end XOR_Octets;

  -- conversion between types
  procedure ToOctets(B: in Keccak.Bitstream; O: out Raw_Types.Octets ) is
    Pos : Natural;
  begin
    Pos := B'First;
    for I in O'Range loop
      O(I) := Unsigned_8( B( Pos     ) ) +
              Unsigned_8( B( Pos + 1 ) ) * 2 +
              Unsigned_8( B( Pos + 2 ) ) * 4 +
              Unsigned_8( B( Pos + 3 ) ) * 8 +
              Unsigned_8( B( Pos + 4 ) ) * 16 +
              Unsigned_8( B( Pos + 5 ) ) * 32 +
              Unsigned_8( B( Pos + 6 ) ) * 64 +
              Unsigned_8( B( Pos + 7 ) ) * 128;
      Pos := Pos + 8;
    end loop;
  end ToOctets;

  procedure ToBitstream(O: in Raw_Types.Octets; B: out Keccak.Bitstream ) is
    V   : Unsigned_8;
    Pos : Natural;
  begin
    Pos := B'First;
    for I in O'Range loop
      V := O( I );
      B( Pos     ) := Keccak.Bit( V and 1 );
      B( Pos + 1 ) := Keccak.Bit( Shift_Right( V, 1 ) and 1 );
      B( Pos + 2 ) := Keccak.Bit( Shift_Right( V, 2 ) and 1 );
      B( Pos + 3 ) := Keccak.Bit( Shift_Right( V, 3 ) and 1 );
      B( Pos + 4 ) := Keccak.Bit( Shift_Right( V, 4 ) and 1 );
      B( Pos + 5 ) := Keccak.Bit( Shift_Right( V, 5 ) and 1 );
      B( Pos + 6 ) := Keccak.Bit( Shift_Right( V, 6 ) and 1 );
      B( Pos + 7 ) := Keccak.Bit( Shift_Right( V, 7 ) and 1 );

      Pos := Pos + 8;
    end loop;
  end ToBitstream;

end OAEP;

The new tests added to tests/test_rsa_oaep.adb and called from tests/testall.adb, basically calling naked rsa from Ada, OAEP by itself and then rsa+oaep on a few different inputs:

 --S.MG, 2018

with Interfaces; use Interfaces;
with Interfaces.C; use Interfaces.C;
with RSA_OAEP; use RSA_OAEP;
with OAEP; use OAEP;
with Raw_Types; use Raw_Types;
with RNG; use RNG;
with Keccak; use Keccak;

with Ada.Text_IO; use Ada.Text_IO;

package body Test_RSA_OAEP is

  procedure test_char_array is
    S : String := OAEP.TMSR_STR;
    O : Octets := OAEP.TMSR;
    A : char_array(0..O'Length-1) := (others => '0');
    B : Octets(0..O'Length -1) := (others => 0);
    Fail : Boolean := FALSE;
  begin
    Octets_To_Char_Array(O, A);
    Char_Array_To_Octets(A, B);

    if B /= O then
      Put_Line("FAIL: char_array_to_octets");
    else
      Put_Line("PASS: char_array_to_octets");
    end if;

    for I in 0..S'Length-1 loop
      declare
        C : Character := Character(A(A'First + size_t(I)));
        E : Character := S(S'First + I);
      begin
        if C /= E then
          Fail := TRUE;
          Put("Error at pos " & Integer'Image(I) & ": ");
          Put(Integer'Image(Character'Pos(C)));
          Put_Line(" instead of " & Integer'Image(Character'Pos(E)));
        end if;
      end;
    end loop;
    if FAIL then
      Put_Line("FAIL: test octets_to_char_array");
    else
      Put_Line("PASS: test octets_to_char_array");
    end if;
  end test_char_array;

  -- test OAEP encrypt + decrypt
  procedure test_oaep is
    Plain: Octets(1..MAX_LEN_MSG);
    Short: Octets(0..10);
    Encr : OAEP_Block;
    Decr : OAEP_HALF;
    Len  : Natural;
    Entropy: OAEP_Block;
    Success : Boolean;
  begin
    RNG.Get_Octets(Plain);
    RNG.Get_Octets(Entropy);
    RNG.Get_Octets(Short);

    -- test full length message
    OAEP_Encrypt(Plain, Entropy, Encr);
    OAEP_Decrypt(Encr, Len, Decr, Success);

    if not Success or Len/8 /= Plain'Length then
      Put_Line("FAIL: oaep encrypt/decrypt on max len message.");
    else
      if Decr(Decr'First..Decr'First+Len/8-1) /=
              Plain(Plain'First..Plain'First+Len/8-1) then
        Put_Line("FAIL: oaep encrypt/decrypt on max len message - " &
                 "result different from expected.");
      else
        Put_Line("PASS: oaep encrypt/decrypt on max len message.");
      end if;
    end if;

    -- test short message
    OAEP_Encrypt(Short, Entropy, Encr);
    OAEP_Decrypt(Encr, Len, Decr, Success);
    if not Success or Len/8 /= Short'Length then
      Put_Line("FAIL: oaep encrypt/decrypt on short message.");
    else
      if Decr(Decr'First..Decr'First+Len/8-1) /=
             Short(Short'First..Short'First+Len/8-1) then
        Put_Line("FAIL: oaep encrypt/decrypt on short message - " &
                 "result different from expected.");
      else
        Put_Line("PASS: oaep encrypt/decrypt on short message.");
      end if;
    end if;

  end test_oaep;

  -- test JUST RSA (i.e. without oaep) with RSA key pair previously generated
  procedure test_rsa is
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
    Plain: OAEP_Block := (others => 0);
    Decr : OAEP_Block := (others => 0);
    Encr : RSA_len;
    pkey: RSA_pkey;
    skey: RSA_skey;
  begin
    -- initialize with RSA pair previously generated
    Hex2Octets( n, skey.n );
    Hex2Octets( e, skey.e );
    Hex2Octets( d, skey.d );
    Hex2Octets( p, skey.p );
    Hex2Octets( q, skey.q );
    Hex2Octets( u, skey.u );
    -- copy n and e for public key
    pkey.n := skey.n;
    pkey.e := skey.e;
    -- get random data
    RNG.Get_Octets(Plain);
    -- make first octet < RSA key's modulus first octet
    Plain(Plain'First) := 16#00#;
    -- naked rsa encrypt/decrypt
Put_Line("Encrypting with RSA public key...");
    Public_RSA( Plain, pkey, Encr );
Put_Line("Decrypting with RSA private key...");
    Private_RSA( Encr, skey, Decr );
Put_Line("Checking...");

    -- check result
    if Decr /= Plain then
      Put_Line("FAIL: RSA encrypt/decrypt result doesn't match plain.");
    else
      Put_Line("PASS: RSA encrypt/decrypt");
    end if;
  end test_rsa;

  -- test rsa+oaep with RSA key pair previously generated
  procedure test_rsa_oaep is
    n: String := "C6579F8646180EED0DC1F02E0DDD2B43EABB3F702D79D9928E2CDA5E1D42DF5D9ED7773F80B1F8D9B0DB7D4D00F55647640D70768F63D3CED56A39C681D08D6191F318BB79DC969B470A7364D53335C8318EF35E39D5DF706AB6F2393C6DD2128C142DBAB1806EB35E26C908F0A48419313D2D0F33DD430655DBFEC722899EC21C238E8DB7003430BBC39BAD990F9887F6B03E1344F537EC97389B78DBC656718ACD7B0FDC13DD24534F417BC7A18F077A0C4227354CEA19670331B6CAA3DFC17BBA7E70C14510D9EB3B63F3014994EC87BD23E868C0AE6E9EC55027577F62C0280B2D7DD1135001844923E5455C4566E066B3FDE968C6BC4DC672F229FCE366440403D7A4F4A8BFBA5679B7D0844BA1231277D13A77C9E2B5A1CB138C1B7AB5B4D4832448723A3DE70ED2E86D5FC5174F949A02DE8E404304BEB95F9BF40F3AA3CA15622D2776294BE7E19233406FF563CB8C25A1CB5AADBC1899DA3F2AE38533931FE032EE3232C2CD4F219FADF95B91635C0762A476A4DE5013F4384093F0FB715028D97F93B2E6F057B99EE344D83ADF2686FD5C9C793928BEF3182E568C4339C36C744C8E9CA7D4B9A16AA039CBF6F38CC97B12D87644E94C9DBD6BC93A93A03ED61ECC5874586E3A310E958F858735E30019D345C62E5127B80652C8A970A14B31F03B3A157CD5";
    e: String := "F74D78E382FC19B064411C6C20E0FDB2985F843007A54C7D8400BB459468624126E7D175F397E55C57AF25858EAE2D2952FB7998C119A6103606733EB5E1D27FCA1FACF14ADE94101D383D1B25DA511805569BC344EAD384EDBF3F3A541B34887FE199D99D7F62E6E9D516F88D6F5AD3E020DF04D402A02CC628A0064362FE8516CF7CD6040E9521407AB90EE6B5AFFF9EA9EBB16A7D3407CE81FD3844F519880556AB94AB349C1F3BBB6FDB4C4B377FE4C091EBDC2C3A1BD3AA56382D8D80E7742B5C751008FD6ECDD2EC3B2E3B6C566F698ED672000B403766DD63C3ACBDE16A14FB02E83A2EB6AA018BFC0020401E790DEE24E9";
    d: String := "698DA05DA25B230211EEF0CBA12083A1457B749A11937AC9993859F69A3BF38D575E5166AF2EC88D77F1DF04E68AEA358EACF7659FD4722A4F5A1C8BA7676DA97A9FBA75451152F8F68887D3451A9CCFFFE9EB80979786E37495B17687A6212F77FA616E4C0CD8A8EB7AEB88EA6CCABB7F3E854FB94B35394A09F95F0D6F997947E865CC0606F437C30FE8C48D96FBF5E2F52807BC9E9ED7BBEB23D5C45EDDCD16FE2BF410A9A1E5EF879E71C0D41FAE270C0C5D442860103F8C3944E802F33DB38432F11F763A7AF593656108E4A98A44A8549913CE5DCEC1A6500F280E3190991B2B938561CFACD8BC5183AAC9A4914BFE52C3BE39BB83688E1DE52479107EF8E087DCDB409432FC954C6349407E81DDFB11AE92BABB32A31868597958C9C76E0B4156F380955F0E09C1F3B98BB4CDD59E1B5C7D8CC2AA7491B0D319D219CF459A527CE1AA2729DEC53269653BF0ED3E0253F4451168437E3B069E48350CA4C3EC82134E87135624C768D1330B0D70C6E447FD9945BF06FCB91AA334C0FD8EEF1ADBC15928B3DB62077B537F7E9F468CC95CD5AAFEAE1F760A863B48D07B163F670E2E5B550BB3E960230BA9FDAED9903AE2E669A7F3C4D1F1E25B8E8EDB8CC6E6FD2164E66F4E64ED77BEF1EC9E6CEA5624FD84C0680248746DC1C8187145F3CD2411659DAEAD11D";
    p: String := "CDD6F7673A501FB24C44D56CA1D434F6CB3334E193E02F8E906241906BCB7412DD2159825B24C22002F373E647C2DA62A854F3841C00FD5985D03227CA9B54A69380BA9D63BE738BDF9E65C247E43E1220EEDD9281DCA78B32A4E1B786B7697ED0C3195D5AF2990881B11D6FC9EC9F940067B2DEA2A516FAA5F269C98F0B67628A6D2708515A4A58041AA17A93E4C4DD95C85BC38351DDA1DCF3DFD91C505B22383132649CF9F9233852C7207075BCF43C71038F043F1EC53E9787FB051B7927D020903233C16897B993C8089D8464451F086E756CF20E46CE6ED4A6AC5C327A0AAFBECBAAFD177969E7C952C76A4F4E7C85BF7F63";
    q: String := "F6ACF0790A250802C8D45DAC549CDBEF7806D5877A5DF0069136A458FAC4F0B0858060A873DA6355A965A064A0BC1BBB874872CD7ED89674AD407533041E74BCA317EC73597D335115523F61A05071E5ED81EE2A05331F65D4DC7A25AD7938B124CF03F49154B6693FB0B598B33ABDEF85C599A57A9B7347EAFF82638E1CBC28FCDFFF1FF04A18C2DBF3938395C2F8D1782B43D3A25EF7633B5DDAC89EFD3BAA64D976425A0891E00B876E9DE9FE4B6492B0EA8DFC7C8DEEC61721356EC816295B1BD9CD9DA3E30D2D90DC9CB3987F4BE042104900E036F3044A016749EF910CCFB9F377A90849B4CCCF4471A74E67EF6C814C9467";
    u: String := "854B89ED10F52258D00D6B3FA7F1FD22752804668F51FF7806DB82E22CB8B3AA8448D9B8E9DB14D31A36AEC2BCFA89E341B7334D494E97ED8051244136192233332C4612D963E7B6AF2535FDB7FE97E28DDFEBDFB3E1AFC29D05DBDF37106A817D3AB1864C7F7F247982897EDA6A92BED47D9C68305CD170C7301ACEB05F8A6382E73CC7614B2D8D758669B3A99AB64114809254B0BE21F40341A5B48B9B032603B14875B87EB5E16603FD16552E146A0FC6964958DFC25AA9FFCCD1ED1F4DEAF9FBAA0D7357F5FF0803FEB9BA78E74AC6B3070F417CEC6CFC7A3CF1E305FC7B76B7ED71893999AF797B2EBDE41FE90F076CCEDBFB";
    Plain: Octets(1..MAX_LEN_MSG) := (others=>20);
    Short: Octets(1..10);
    Decr : RSA_len;
    Encr : RSA_len;
    pkey: RSA_pkey;
    skey: RSA_skey;
    Success: Boolean;
    Len : Natural;
  begin
    -- initialize with RSA pair previously generated
    Hex2Octets( n, skey.n );
    Hex2Octets( e, skey.e );
    Hex2Octets( d, skey.d );
    Hex2Octets( p, skey.p );
    Hex2Octets( q, skey.q );
    Hex2Octets( u, skey.u );
    -- copy n and e for public key
    pkey.n := skey.n;
    pkey.e := skey.e;

    -- test with 0 message of length Plain'Length
    RSA_OAEP.Encrypt(Plain, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Plain'Length
         or Plain /= Decr(Decr'First..Decr'First+Plain'Length-1) then
      Put_Line("FAIL: RSA_OAEP on max len message 20-filled.");
    else
      Put_Line("PASS: RSA_OAEP on max len message 20-filled.");
    end if;

    -- get random data for "plain" message
    RNG.Get_Octets(Plain);
    RSA_OAEP.Encrypt(Plain, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Plain'Length
         or Plain /= Decr(Decr'First..Decr'First+Plain'Length-1) then
      Put_Line("FAIL: RSA_OAEP on random data of max length.");
    else
      Put_Line("PASS: RSA_OAEP on random data of max length.");
    end if;

    -- get random data for "short" message
    RNG.Get_Octets(Short);
    RSA_OAEP.Encrypt(Short, pkey, Encr);
    RSA_OAEP.Decrypt(Encr, skey, Decr, Len, Success);
    if (not Success) or Len /= Short'Length
         or Short /= Decr(Decr'First..Decr'First+Short'Length-1) then
      Put_Line("FAIL: RSA_OAEP on random data of short length.");
    else
      Put_Line("PASS: RSA_OAEP on random data of short length.");
    end if; 

  end test_rsa_oaep;

-- helper methods
  procedure Hex2Octets( Hex: in String; O: out Raw_Types.Octets ) is
    S : String := "16#AA#";
    -- to make sure that input String has EVEN number of chars (ie full octets)
    H : String(1..Hex'Length+Hex'Length mod 2) := (others=>'0');
  begin
    -- first char is 0 if needed to cover full octet...
    H(H'Length-Hex'Length+1..H'Length) := Hex;
    O := (others => 0);
    for I in 0 .. H'Length/2-1 loop
      S := "16#" & H(H'First + I*2 .. H'First + I*2 + 1) & "#";
      O(O'Last - H'Length/2 + 1 + I) := Unsigned_8'Value(S);
    end loop;
  end Hex2Octets;

  procedure PrintOctets( O: in Raw_Types.Octets; Title: in String ) is
  begin
    Put_Line(Title);
    for V of O loop
      Put(Unsigned_8'Image(V) & " ");
    end loop;
    New_Line;
  end PrintOctets;

end Test_RSA_OAEP;

The full .vpatch for this chapter is quite long at 1714 lines but a significant part of that is essentially imported from EuCrypt (Keccak most notably but also the main part of OAEP). As usual, the .vpatch and my signature for it can be found on my Reference Code Shelf or directly from the links below:

  1. Yes, I adopted Stanislav's approach to reading from the FG. I also had a look at his code at www.loper-os.org/?p=2175  ↩

  2. The exact line is this one: http://www.dianacoman.com/2018/01/18/eucrypt-chapter-6-keccak-transformations/#selection-159.735-159.812 and it's been pointed out already (in the logs, where else?) that there is an implicit check against div by 0 which could be avoided simply by rewriting the line as Output(X, Y) := Rotate_Left(Input(X,Y), ( (T+1)*(T+2)/2) and (Z_Length -1) );  ↩

October 25, 2018

SMG Comms Chapter 4: C Wrappers for RSA and MPI

Filed under: Coding, SMG_Comms — Diana Coman @ 7:28 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

The C part of SMG Comms is - unsurprisingly - a pain in the ass and a three-pointed thorn in the backside:
1. All the RSA operations are coded in C because of the heavy dependency on the MPI lib. This is not something I can change at the moment since I don't yet have an Ada implementation of long integer arithmetics. Sigh.
2. The OAEP padding schema has to be called repeatedly until the result is indeed (when considered as a very large number) a value smaller than the RSA key's modulus. So although the OAEP padding itself (including underlying Keccak are implemented in Ada), the comparison between obtained value and the modulus of the key goes through the MPI swamp so it forces a call to C. Nevertheless, the "implemented in Ada" and "repeat until the result CAN be encrypted with given RSA key" are both absolutely right and proper and as they should be so there's nothing to change here. Hooray.
3. As a result of 1 and 2 above, the code flow can easily get ugly because one ends up with calls from Ada to C (to do RSA operations for instance) but *also* from C to Ada (to do Keccak, if one aims to provide oaep+rsa from C directly). This is currently the sad state of the wrappers provided as first attempt in EuCrypt for rsa-oaep. Nevertheless, it's horribly ugly, difficult to follow and overall unsanitary so it has to go.

My current solution to the above issue with rsa-oaep is to isolate properly the C code and stick to calls in only one direction, namely from Ada to C. To do this however, I first need to write some more ...C code. Eurgh. The new C code is a set of wrappers for RSA and MPI methods that I need to call from Ada (essentially from what will be Ada implementation of the rsa-with-oaep part): those wrappers provide a sane interface that matches SMG Comms' raw types layer - basically those methods can be called with arrays of octets as input/output without forcing onto the rest of the code MPI structures, unpredictable memory allocation and other abominations.

The above approach has two very important benefits: the code is much easier to follow since there is no ugly back and forth between Ada and C; the C part (MPI especially but also the RSA layer) is properly separated from the Ada part so that it should be relatively easy to simply replace it with a sane Ada implementation whenever such a wonder becomes available. The main downside to this is that it adds conversions from arrays of octets to MPIs and back as part and parcel of using the wrappers but I still think it's the better option available at this moment.

The C wrappers that I need for SMG Comms include the comparison of 2 MPI values (so mpi_cmp from MPI lib), the encryption and decryption with given RSA key as well as a procedure to generate a RSA key (from rsa lib). In addition, I wrote an mpi_to_octets function that does as the name suggests: reads the raw octets stored in the MPI and returns them to the caller. The reason for this to feature as a separate function is the sad fact that MPI's own "get buffer" allocates memory and then expects the caller to de-allocate it. Rather than import this headache and propagate it all through the new code, I wrapped it up in here and dealt with it: callers of mpi_to_octets should allocate their own memory and de-allocate it accordingly. The signatures of those wrapper methods are in all their enumerated-arrays-of-octets glory (RSA key components are passed as array of octets just as well as anything else) in c_wrappers/c_wrappers.c:

//Wrapper methods for C implementations of RSA and MPI.
//To be used / called from Ada so that the C part remains well separated and
//  can therefore be swapped easily at a later stage for something sane.
//S.MG, 2018

#include "mpi.h"

//Comparison of 2 arrays of octets interpreted as MPIs.
//This method creates 2 MPIs out of the given arrays of octes and then
//  calls the mpi_cmp method from mpi/mpi-cmp.c, returning its result.
// ************************************************************************
// ***NB: Make SURE that a and b have indeed allocated at least len_a and
// *****    len_b octets respectively! NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param a An array of octets representing the first MPI.
//@param len_a The length of the first array (number of octets).
//@param b An array of octets representing the second MPI.
//@param len_b The length of the second array (number of octets).
//@return 0 when a=b, -1 when ab
int mpi_cmp_octets(char *a, unsigned int len_a, char *b, unsigned int len_b);

//Encryption of given octets with public RSA key: n and e given as octets too.
// ************************************************************************
// ***Length of output is KEY_LENGTH_OCTETS.
// ***NB: Make SURE that out, input, n and e have enough space allocated!!
// ***NB: NO MEMORY ALLOCATED for its parameters by this method and
// *****    NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param out Pointer to ALREADY ALLOCATED space for the encrypted data.
//@param len_out Length of the allocated space for out (in octets).
//@param input Pointer to the data to be encrypted.
//@param len_input Length of the allocated space for input (in octets).
//@param n Pointer to the public RSA modulus to use for encryption.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent to use for encryption.
//@param len_e Length of e (in octets).
//@return The actual length of the output i.e. number of chars written to out.
int public_rsa_octets( char *out  , unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e);

//Encryption of given octets with *private* RSA key given as octets.
// ************************************************************************
// ***Length of output is KEY_LENGTH_OCTETS.
// ***NB: Make SURE that ALL pointers have enough space allocated!!
// ***NB: NO MEMORY ALLOCATED for its parameters by this method and
// *****    NO CHECKS PERFORMED BY THIS METHOD!
// ************************************************************************
//@param out Pointer to ALREADY ALLOCATED space for the encrypted data.
//@param len_out Length of the allocated space for out (in octets).
//@param input Pointer to the data to be encrypted.
//@param len_input Length of the allocated space for input (in octets).
//@param n Pointer to the public RSA modulus of the given key.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent of the given key.
//@param len_e Length of e (in octets).
//@param d Pointer to the private RSA exponent of the given key.
//@param len_d Length of d (in octets).
//@param p Pointer to the prime p of the given key.
//@param len_p Length of p (in octets).
//@param q Pointer to the prime q of the given key.
//@param len_q Length of q (in octets).
//@param u Pointer to the inverse of p mod q for the given key.
//@param len_u Length of u (in octets).
//@return The actual length of the output i.e. number of chars written to out.
int private_rsa_octets( char *out, unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e,
                        char *d    , unsigned int len_d,
                        char *p    , unsigned int len_p,
                        char *q    , unsigned int len_q,
                        char *u    , unsigned int len_u);

//Generates a new RSA key and stores its components at the specified locations.
//@param n Pointer to the public RSA modulus of the given key.
//@param len_n Length of n (in octets).
//@param e Pointer to the public RSA exponent of the given key.
//@param len_e Length of e (in octets).
//@param d Pointer to the private RSA exponent of the given key.
//@param len_d Length of d (in octets).
//@param p Pointer to the prime p of the given key.
//@param len_p Length of p (in octets).
//@param q Pointer to the prime q of the given key.
//@param len_q Length of q (in octets).
//@param u Pointer to the inverse of p mod q for the given key.
//@param len_u Length of u (in octets).
void gen_rsa_octets( char *n, unsigned int *len_n,
                     char *e, unsigned int *len_e,
                     char *d, unsigned int *len_d,
                     char *p, unsigned int *len_p,
                     char *q, unsigned int *len_q,
                     char *u, unsigned int *len_u);

//Copies the buffer of m to the location to which out points.
//*****************************************************************
//*** This method does NOT allocate memory for out!
//*** NB: caller should allocate enough memory!
//*****************************************************************
//@param out pointer to allocated memory where to copy the MPI's octets
//@param len_out size of out; will be replaced by actual number of octets copied
//@param m The MPI whose octets are to be retrieved
void mpi_to_octets( char *out, unsigned int *len_out, MPI m);

Corresponding code for the wrapper methods:

//Wrapper methods for C implementations of RSA and MPI.
//To be used / called from Ada so that the C part remains well separated.
//S.MG, 2018

#include "c_wrappers.h"
#include "mpi.h"
#include "smg_rsa.h"
#include <assert.h>
#include <string.h> //for memmove...

//Wrapper for comparing the given arrays of octets as MPIs
//Steps:
//1. allocate space for 2 MPIs u and v
//2. set buffer of u to content of a, buffer of v to content of b
//3. call mpi_cmp(u,v) and store result
//4. de-allocate u and v
//5. return result
int mpi_cmp_octets(char *a, unsigned int len_a, char *b, unsigned int len_b) {
  //variable to hold the final result of comparing a to b as MPIs
  int result;

  //calculate how much space is needed for each MPI
	unsigned int nlimbs_a = mpi_nlimb_hint_from_nbytes( len_a );
	unsigned int nlimbs_b = mpi_nlimb_hint_from_nbytes( len_b );

  //allocate space for the 2 MPIs
	MPI u = mpi_alloc(nlimbs_a);
	MPI v = mpi_alloc(nlimbs_b);

  //set the given octets as the values of the 2 MPIs
  //the sign is set to 0 (last parameter).
  mpi_set_buffer(u, a, len_a, 0);
  mpi_set_buffer(v, b, len_b, 0);

  //compare the MPIs as numbers and store the result
  result = mpi_cmp(u, v);

  //tidy up: free the memory allocated for the 2 MPIs
  mpi_free(u);
  mpi_free(v);

  //return the result comparing a to b as MPIs
  return result;
}

//Encryption of given input with a public RSA key: n and e given as octets too.
//Steps:
//1. create (allocate memory for) MPIs for out and input;
//2. set the input as buffer for the corresponding MPI;
//3. create and set the public key structure with n,e as contents;
//4. call rsa/public_rsa and retrieve the result storing it in out;
//5. free allocated memory for all the MPIs.
//6. return the actual length of the encrypted result
int public_rsa_octets( char *out  , unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e) {

  // precondition: output has enough memory allocated
  assert( len_out >= KEY_LENGTH_OCTETS );

  //allocate memory for input and output MPIs
	unsigned int nlimbs_in  = mpi_nlimb_hint_from_nbytes( len_input );
	unsigned int nlimbs_out = mpi_nlimb_hint_from_nbytes( len_out   );
	MPI in_mpi  = mpi_alloc(nlimbs_in);
	MPI out_mpi = mpi_alloc(nlimbs_out);

  //set input as buffer for in_mpi
  mpi_set_buffer(in_mpi, input, len_input, 0);

  //create public key structure and set its contents to given n, e
  RSA_public_key pk;
	unsigned int nlimbs_n = mpi_nlimb_hint_from_nbytes( len_n );
	unsigned int nlimbs_e = mpi_nlimb_hint_from_nbytes( len_e );
  pk.n = mpi_alloc(nlimbs_n);
  pk.e = mpi_alloc(nlimbs_e);
  mpi_set_buffer(pk.n, n, len_n, 0);
  mpi_set_buffer(pk.e, e, len_e, 0);

  //call rsa public_key encryption and retrieve the result, storing it in out
  public_rsa( out_mpi, in_mpi, &pk);
  int len = len_out;
  mpi_to_octets( out, &len, out_mpi );

  //tidy up: free allocated memory for ALL MPIs.
  mpi_free(in_mpi);
  mpi_free(out_mpi);
  mpi_free(pk.n);
  mpi_free(pk.e);

  //return actual length
  return len;
}

//Decryption of given input with the private key given through its components.
//Steps:
//1. create (allocate memory for) MPIs for out and input;
//2. set the input as buffer for the corresponding MPI;
//3. create and set the private key structure with n,e,d,p,q,u as contents;
//4. call rsa/private_rsa and retrieve the result storing it in out;
//5. free allocated memory for all the MPIs.
//6. return the actual length of the result
int private_rsa_octets( char *out, unsigned int len_out,
                        char *input, unsigned int len_input,
                        char *n    , unsigned int len_n,
                        char *e    , unsigned int len_e,
                        char *d    , unsigned int len_d,
                        char *p    , unsigned int len_p,
                        char *q    , unsigned int len_q,
                        char *u    , unsigned int len_u) {
  // precondition: output has enough memory allocated
  assert( len_out >= KEY_LENGTH_OCTETS );

  //allocate memory for input and output MPIs
	unsigned int nlimbs_in  = mpi_nlimb_hint_from_nbytes( len_input );
	unsigned int nlimbs_out = mpi_nlimb_hint_from_nbytes( len_out   );
	MPI in_mpi  = mpi_alloc(nlimbs_in);
	MPI out_mpi = mpi_alloc(nlimbs_out);

  //set input as buffer for in_mpi
  mpi_set_buffer(in_mpi, input, len_input, 0);

  //create private key structure and set its contents to given n,e,d,p,q,u
  RSA_secret_key sk;
	unsigned int nlimbs_n = mpi_nlimb_hint_from_nbytes( len_n );
	unsigned int nlimbs_e = mpi_nlimb_hint_from_nbytes( len_e );
	unsigned int nlimbs_d = mpi_nlimb_hint_from_nbytes( len_d );
	unsigned int nlimbs_p = mpi_nlimb_hint_from_nbytes( len_p );
	unsigned int nlimbs_q = mpi_nlimb_hint_from_nbytes( len_q );
	unsigned int nlimbs_u = mpi_nlimb_hint_from_nbytes( len_u );
  sk.n = mpi_alloc(nlimbs_n);
  sk.e = mpi_alloc(nlimbs_e);
  sk.d = mpi_alloc(nlimbs_d);
  sk.p = mpi_alloc(nlimbs_p);
  sk.q = mpi_alloc(nlimbs_q);
  sk.u = mpi_alloc(nlimbs_u);
  mpi_set_buffer(sk.n, n, len_n, 0);
  mpi_set_buffer(sk.e, e, len_e, 0);
  mpi_set_buffer(sk.d, d, len_d, 0);
  mpi_set_buffer(sk.p, p, len_p, 0);
  mpi_set_buffer(sk.q, q, len_q, 0);
  mpi_set_buffer(sk.u, u, len_u, 0);

  //call rsa secret_key encryption and retrieve the result, storing it in out
  secret_rsa( out_mpi, in_mpi, &sk );
  int len = len_out;
  mpi_to_octets( out, &len, out_mpi );

  //tidy up: free memory previously allocated for MPIs
  mpi_free(in_mpi);
  mpi_free(out_mpi);
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  //return number of octets copied in out - real length of result
  return len;
}

//Generates a new RSA key and stores its components at the specified locations.
void gen_rsa_octets( char *n, unsigned int *len_n,
                     char *e, unsigned int *len_e,
                     char *d, unsigned int *len_d,
                     char *p, unsigned int *len_p,
                     char *q, unsigned int *len_q,
                     char *u, unsigned int *len_u) {
  // precondition: all pointers have enough memory allocated
  assert( *len_n >= KEY_LENGTH_OCTETS );
  assert( *len_e >= KEY_LENGTH_OCTETS );
  assert( *len_d >= KEY_LENGTH_OCTETS );
  assert( *len_p >= KEY_LENGTH_OCTETS / 2);
  assert( *len_q >= KEY_LENGTH_OCTETS / 2);
  assert( *len_u >= KEY_LENGTH_OCTETS / 2);

  //the secret key structure that will hold generated key components
  RSA_secret_key sk;
	int nlimbs = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS );
	int nlimbs_pq = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS / 2 );

	sk.n = mpi_alloc(nlimbs);
	sk.e = mpi_alloc(nlimbs);
	sk.d = mpi_alloc(nlimbs);
	sk.p = mpi_alloc(nlimbs_pq);
	sk.q = mpi_alloc(nlimbs_pq);
	sk.u = mpi_alloc(nlimbs_pq);

  //generate the rsa key pair - this may take a while!
  gen_keypair(&sk);

  //copy components to their place
  mpi_to_octets( n, len_n, sk.n );
  mpi_to_octets( e, len_e, sk.e );
  mpi_to_octets( d, len_d, sk.d );
  mpi_to_octets( p, len_p, sk.p );
  mpi_to_octets( q, len_q, sk.q );
  mpi_to_octets( u, len_u, sk.u );

  //tidy up: free ALL MPIs
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);
}

void mpi_to_octets( char *out, unsigned int *len_out, MPI m) {
  //copy the components as raw octets to the given pointers
  int len = 0;
  int sign;
  unsigned char * buffer = mpi_get_buffer( m, &len, &sign );

  //check and don't copy MORE than there is allocated space in out!
  assert( len <= *len_out );
  memmove( out, buffer, len );
  *len_out = len;    //save actual length of the component

  xfree( buffer ); //free the buffer that was allocated by mpi_get_buffer
}

The basic tests for the new C wrappers are the following:

//S.MG, 2018

#include "mpi.h"
#include "smg_rsa.h"

#include "c_wrappers.h"

void test_mpi_cmp() {
  int result;
  int i;
  char a[KEY_LENGTH_OCTETS];
  char b[KEY_LENGTH_OCTETS];

  //initialize mpis
  for (i=0;i b
  a[240] = 241;
  result = mpi_cmp_octets(a, KEY_LENGTH_OCTETS, b, KEY_LENGTH_OCTETS);
  if (result == 1)
    printf("PASS: mpi_cmp_octets on a > b.\n");
  else {
    printf("FAIL: mpi_cmp_octets on a > b ");
    printf("returned %d instead of 1.\n", result);
  }

  //tidy up
}

void test_gen_rsa_octets() {
  RSA_secret_key sk;
  RSA_public_key pk;
	int nlimbs = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS );
	int nlimbs_pq = mpi_nlimb_hint_from_nbytes( KEY_LENGTH_OCTETS / 2 );

  //allocate memory
	sk.n = mpi_alloc(nlimbs);
	sk.e = mpi_alloc(nlimbs);
	sk.d = mpi_alloc(nlimbs);
	sk.p = mpi_alloc(nlimbs_pq);
	sk.q = mpi_alloc(nlimbs_pq);
	sk.u = mpi_alloc(nlimbs_pq);

	pk.n = mpi_alloc(nlimbs);
	pk.e = mpi_alloc(nlimbs);

  //generate key pair
  int len_n = KEY_LENGTH_OCTETS;
  int len_e = len_n;
  int len_d = len_n;
  int len_p = KEY_LENGTH_OCTETS / 2;
  int len_q = len_p;
  int len_u = len_p;
  char n[KEY_LENGTH_OCTETS];
  char e[KEY_LENGTH_OCTETS];
  char d[KEY_LENGTH_OCTETS];
  char p[KEY_LENGTH_OCTETS / 2];
  char q[KEY_LENGTH_OCTETS / 2];
  char u[KEY_LENGTH_OCTETS / 2];
  gen_rsa_octets(n, &len_n,
                 e, &len_e,
                 d, &len_d,
                 p, &len_p,
                 q, &len_q,
                 u, &len_u);

  //check encryption/decr works
  mpi_set_buffer(sk.n, n, len_n, 0);
  mpi_set_buffer(sk.e, e, len_e, 0);
  mpi_set_buffer(sk.d, d, len_d, 0);
  mpi_set_buffer(sk.p, p, len_p, 0);
  mpi_set_buffer(sk.q, q, len_q, 0);
  mpi_set_buffer(sk.u, u, len_u, 0);

  mpi_set_buffer(pk.n, n, len_n, 0);
  mpi_set_buffer(pk.e, e, len_e, 0);

  MPI encr = mpi_alloc(0);
	MPI plain = mpi_alloc(0);
  MPI out = mpi_alloc(0);
	mpi_fromstr(plain, "0x\
5B6A8A0ACF4F4DB3F82EAC2D20255E4DF3E4B7C799603210766F26EF87C8980E737579\
EC08E6505A51D19654C26D806BAF1B62F9C032E0B13D02AF99F7313BFCFD68DA46836E\
CA529D7360948550F982C6476C054A97FD01635AB44BFBDBE2A90BE06F7984AC8534C3\
28097EF92F6E78CAE0CB97");
  public_rsa(encr, plain, &pk);
  secret_rsa(out, encr, &sk);

  if (mpi_cmp(out, plain) != 0)
    printf("FAIL: test_gen_rsa encr/decr failed.\n");
  else
    printf("PASS: test_gen_rsa encr/decr passed.\n");

  //tidy up
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  mpi_free(pk.n);
  mpi_free(pk.e);

  mpi_free(plain);
  mpi_free(encr);
  mpi_free(out);
}

void test_rsa_octets() {
  int noctets = 512;
	RSA_public_key pk;
	pk.n = mpi_alloc(0);
	pk.e = mpi_alloc(0);

  RSA_secret_key sk;
  sk.n = mpi_alloc(0);
  sk.e = mpi_alloc(0);
  sk.d = mpi_alloc(0);
  sk.p = mpi_alloc(0);
  sk.q = mpi_alloc(0);
  sk.u = mpi_alloc(0);

//key pair previously generated with EuCrypt
  mpi_fromstr(sk.n, "0x\
CD2C025323BEA46FFF2FA8D7A9D39817EA713421F4AE03FA8120641193892A70BFECF5\
83101635A432110D3DDE6339E3CC7ECC0AD91C026FCACE832DD3888A6FCA7BCE56C390\
5A5AC8C7BC921DA675E4B62489B254EB34659D547D71165BC998983A81937BD251AEE1\
2D985EC387D5376F5DCC5EF7EC530FBD6FD2AA7285EE1AF3335EA73163F0954F30402E\
D7B374EE84A97B1849B0674B0DA0A2050BD79B71ABB1559F3A9CFDB8557DED7BC90CF2\
09E8A847E9C226140845B7D03842162E7DA5DD16326CB1F71A248D841FE9076A09911F\
2F4F5E3EA44EA8DE40332BF00406990BCCF61C322A03C456EF3A98B341E0BDBC1088CE\
683E78510E76B72C2BCC1EE9AEDD80FFF18ABFC5923B2F36B581C25114AB2DF9F6C2B1\
9481703FD19E313DCD7ACE15FA11B27D25BCE5388C180A7E21167FB87750599E1ED7C7\
50F4A844E1DC2270C62D19671CF8F4C25B81E366B09FC850AE642136D204A9160AEECE\
575B57378AA439E9DD46DC990288CD54BAA35EEE1C02456CD39458A6F1CBF012DCEDF4\
27CCF3F3F53645658FC49C9C9D7F2856DB571D92B967AB5845514E0054DDB49099F5DD\
04A6F6F5C5CE642276834B932881AEB648D1F25E9223971F56E249EF40CF7D80F22621\
CDD0260E9E7D23746960ADB52CF2987584FB1DE95A69A39E5CB12B76E0F5C1A0529C0C\
065D2E35720810F7C7983180B9A9EA0E00C11B79DC3D");

  mpi_fromstr(sk.e, "0x\
DD4856B4EE3D099A8604AE392D8EFEC094CDF01546A28BE87CB484F999E8E75CDFCD01\
D04D455A6A9254C60BD28C0B03611FC3E751CC27EF768C0B401C4FD2B27C092834A6F2\
49A145C4EDC47A3B3D363EC352462C945334D160AF9AA72202862912493AC6190AA3A6\
149D4D8B9996BA7927D3D0D2AD00D30FD630CF464E6CAF9CF49355B9A70E05DB7AE915\
F9F602772F8D11E5FCDFC7709210F248052615967090CC1F43D410C83724AA5912B2F0\
52E6B39449A89A97C79C92DC8CB8DEEFCF248C1E1D2FC5BFE85165ECA31839CAA9CEB3\
3A92EBDC0EB3BAC0F810938BB173C7DA21DCBB2220D44CBA0FD40A2C868FC93AC5243E\
C137C27B0A76D65634EBB3");

  mpi_fromstr(sk.d, "0x\
7C8A6FA1199D99DCA45E9BDF567CA49D02B237340D7E999150BC4883AE29DEC5158521\
B338F35DC883792356BDDBB3C8B3030A6DD4C6522599A3254E751F9BA1CB1061C5633C\
81BBFACF6FCD64502614102DFED3F3FA284066C342D5E00953B415915331E30812E5FB\
CD6680ADCCDEE40B8376A3A225F2E160EA59C7566804526D73BB660A648A3EF9802313\
B2F841E8458B2AAACE7AACF31083E8F3F630298138393BC88BBD7D4AA4334949651D25\
365B10DBF4A4A08E20A6CC74BFDD37C1C38E2ADC2A283DF06590DF06B46F67F6ACA67F\
AC464C795261659A2F9558802D0BBAA05FD1E1AF2CDC70654723DF7EFAEA148B8CDBEB\
C89EA2320AB9BBB1BC4311475DF3D91446F02EF192368DFEBAC598CCFD4407DEC58FDC\
1A94CCDD6E5FBA9C52164ACEA8AEE633E557BCCEACB7A1AF656C379482D784A120A725\
32F9B2B35173D505F21D5AD4CB9511BC836DC923730B70291B70290A216CA3B21CFF79\
E895C35F4F7AF80E1BD9ED2773BD26919A76E4298D169160593E0335BE2A2A2D2E8516\
948F657E1B1260E18808A9D463C108535FB60B3B28F711C81E5DE24F40214134A53CE5\
9A952C8970A1D771EBEFFA2F4359DCF157995B3F1950DE3C6EC41B7FF837148F55F323\
372AF3F20CE8B8038E750C23D8F5041FA951327859B0E47483F0A47103EF808C72C251\
006FA526245291C8C84C12D2EF63FB2301EA3EEDA42B");

  mpi_fromstr(sk.p, "0x\
E236732452039C14EC1D3B8095BDDCFB7625CE27B1EA5394CF4ED09D3CEECAA4FC0BF6\
2F7CE975E0C8929CE84B0259D773EA038396479BF15DA065BA70E549B248D77B4B23ED\
A267308510DBEE2FD44E35D880EE7CFB81E0646AA8630165BD8988C3A8776D9E704C20\
AA25CA0A3C32F27F592D5FD363B04DD57D8C61FFDCDFCCC59E2913DE0EE47769180340\
E1EA5A803AA2301A010FF553A380F002601F0853FCACDB82D76FE2FACBCD6E5F294439\
0799EA5AE9D7880D4E1D4AE146DC1D4E8495B9DD30E57E883923C5FC26682B7142D35C\
D8A0FC561FE725A6CF419B15341F40FE0C31132CBD81DD8E50697BD1EBFFA16B522E16\
F5B49A03B707218C7DA60B");

  mpi_fromstr(sk.q, "0x\
E830482A3C4F5C3A7E59C10FF8BA760DB1C6D55880B796FFDA4A82E0B60E974E81D04B\
2A4AD417823EBFB4E8EFB13782943562B19B6C4A680E3BA0C8E37B5023470F4F1AC1F8\
A0B10672EF75CD58BCD45E6B14503B8A6A70AFE79F6201AF56E7364A1C742BE1453FD2\
24FDC9D66522EAF4466A084BCB9E46D455A2946E94CBF028770F38D0B741C2CC59308F\
71D8C2B4B9C928E0AE8D68DEB48A3E9EFD84A10301EBD55F8221CA32FC567B306B2A8E\
116350AFB995859FDF4378C5CFD06901494E8CFA5D8FAC564D6531FA8A2E4761F5EFBA\
F78750B6F4662BE9EA4C2FAD67AF73EEB36B41FC15CB678810C19A51DF23555695C4C1\
546F3FACA39CAA7BB8DBD7");

  mpi_fromstr(sk.u, "0x\
846232322775C1CD7D5569DC59E2F3E61A885AE2E9C4A4F8CB3ACBE8C3A5441E5FE348\
A2A8AC9C2998FBF282222BF508AA1ECF66A76AEDD2D9C97028BFD3F6CA0542E38A5312\
603C70B95650CE73F80FDD729988FBDB5595A5BF8A007EA34E54994A697906CE56354C\
E00DF10EB711DEC274A62494E3D350D88736CF67A477FB600AC9F1D6580727585092BF\
5EBC092CC4D6CF75769051033A1197103BE269942F372168A53771746FBA18ED6972D5\
0B935A9B1D6B5B3DD50CD89A27FE93C10924E9103FACF7B4C5724A046C3D3B50CC1C78\
5F5C8E00DBE1D6561F120F5294C170914BC10F978ED4356EED67A9F3A60D70AFE540FC\
5373CBAE3D0A7FD1C87273");

// copy the public key components
  pk.n = mpi_copy( sk.n );
  pk.e = mpi_copy( sk.e );

// some plain text message
	MPI plain = mpi_alloc(0);
	mpi_fromstr(plain, "0x\
5B6A8A0ACF4F4DB3F82EAC2D20255E4DF3E4B7C799603210766F26EF87C8980E737579\
EC08E6505A51D19654C26D806BAF1B62F9C032E0B13D02AF99F7313BFCFD68DA46836E\
CA529D7360948550F982C6476C054A97FD01635AB44BFBDBE2A90BE06F7984AC8534C3\
28097EF92F6E78CAE0CB97");

// expected encrypted MPI (via rsa.c directly)
  MPI encr = mpi_alloc(0);
  public_rsa( encr, plain, &pk);
  MPI decr = mpi_alloc(0);
  secret_rsa( decr, encr, &sk);
  if (mpi_cmp(decr, plain) != 0)
    printf("FAIL: decrypted != plain in test_rsa_octets (MPI call)\n");

//allocate the char arrays for _octets rsa
  int len_n = noctets;
  int len_e = len_n;
  int len_d = len_n;
  int len_p = noctets / 2;
  int len_q = len_p;
  int len_u = len_p;
  char n[noctets];
  char e[noctets];
  char d[noctets];
  char p[noctets / 2];
  char q[noctets / 2];
  char u[noctets / 2];

//copy the key components into char arrays
  mpi_to_octets(n, &len_n, sk.n);
  mpi_to_octets(e, &len_e, sk.e);
  mpi_to_octets(d, &len_d, sk.d);
  mpi_to_octets(p, &len_p, sk.p);
  mpi_to_octets(q, &len_q, sk.q);
  mpi_to_octets(u, &len_u, sk.u);

//call _octets rsa and check results
  int len_encr = noctets;
  int len_decr = noctets;
  int len_plain = noctets;
  char plain_o[noctets];
  char encr_o[noctets];
  char expected_encr_o[noctets];
  char decr_o[noctets];
  char expected_decr_o[noctets];

  mpi_to_octets(plain_o, &len_plain, plain);
  mpi_to_octets(expected_encr_o, &len_encr, encr);
  mpi_to_octets(expected_decr_o, &len_decr, decr);
  len_decr = noctets;

  int len;
  len = public_rsa_octets( encr_o, len_encr, plain_o, len_plain,
                           n, len_n, e, len_e);
  if (len != len_encr)
    printf("FAIL: actual len of encr is %d; expected %d\n", len, len_encr);
  else
    printf("PASS: actual len of encr matches expected: %d\n", len);
  int errors= 0;
  int i;
  for (i=0;i0)
    printf("FAIL: found %d errors in public_rsa_octets output\n", errors);
  else
    printf("PASS: no errors found in public_rsa_octets output\n");

  len_encr = len;
  len = private_rsa_octets( decr_o, len_decr, encr_o, len_encr,
                            n, len_n, e, len_e,
                            d, len_d, p, len_p,
                            q, len_q, u, len_u);
  if (len != len_plain)
    printf("FAIL: actual len of decr is %d; expected %d\n", len, len_plain);
  else
    printf("PASS: actual len of decr matches expected: %d\n", len);

  errors = 0;
  for (i=0;i0)
    printf("FAIL: found %d errors in private_rsa_octets output\n", errors);
  else printf("PASS: no errors found in private_rsa_octets_output\n");

  //tidy up
  mpi_free(sk.n);
  mpi_free(sk.e);
  mpi_free(sk.d);
  mpi_free(sk.p);
  mpi_free(sk.q);
  mpi_free(sk.u);

  mpi_free(pk.n);
  mpi_free(pk.e);

  mpi_free(plain);
  mpi_free(encr);
  mpi_free(decr);
}

int main(int ac, char **av) {
  if (ac < 2) {
    printf("Usage: %s testID\n", av[0]);
    return -1;
  }

  int id = atoi(av[1]);
  switch (id) {
    case 1:
      test_mpi_cmp();
      break;
    case 2:
      test_gen_rsa_octets();
      break;
    case 3:
      test_rsa_octets();
      break;
    default:
      printf("Current test ids:\n");
      printf("1 test of mpi_cmp_octets\n");
      printf("2 test of gen_rsa_octets (can be very SLOW!)\n");
      printf("3 test of rsa_octets (can take a few minutes)\n");
  }
  return 0;
}

The rest (and the longest part by far) of the .vpatch for this chapter includes the MPI lib from EuCrypt and the RSA lib from the same EuCrypt minus the oaep parts that are not going to be used by SMG Comms. Specifically, the changes to the RSA lib from EuCrypt are the following:

  • The constant KEY_LENGTH_OCTETS is changed to the value used by Eulora: 490. I've updated also all the references to the previously fixed value 4096 throughout the comments.
  • I discarded two methods from rsa.c/h: rsa_oaep_encrypt, rsa_oaep_decrypt, oaep_encrypt_c, oaep_decrypt_c.

In addition to the above, the .vpatch also makes a small change to the smg_comms.gpr file reflecting the fact that SMG Comms does not touch mpi and rsa directly but only through the new C wrappers so that the .gpr file uses "c_wrappers.gpr" and not rsa.gpr + mpi.gpr.

The .vpatch is as usual on my Reference Code Shelf and linked also below for your convenience:

October 18, 2018

SMG Comms Chapter 3: Packing Serpent

Filed under: Coding, SMG_Comms — Diana Coman @ 10:12 a.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

This chapter uses the raw types of the protocol as defined in the previous chapter and adds two methods that are still at layer 0 of the protocol: one method for packing Serpent messages into corresponding Serpent packets and one method for unpacking such Serpent packets to extract their contained Serpent message. It is a single small step forward but the corresponding .vpatch is still quite large as it contains the Serpent code in addition to the packing/unpacking methods. The packing is effectively an encryption with a given Serpent key while the unpacking is the corresponding decryption. Nevertheless, there are a few bits that are specific to this implementation as they reflect the requirements of Eulora's protocol:

  • Packing receives as input a Serpent_Msg and produced as output a Serpent_Pkt. Symmetrically, unpacking receives as input a Serpent_Pkt and produces a Serpent_Msg. Both Serpent_Pkt and Serpent_Msg are arrays of octets but of fixed, pre-defined size: 1472 octets and nothing else.
  • Both packing and unpacking will split their input into blocks of the size that Serpent can handle, encrypt/decrypt them and then glue together the results to produce the output. So there are effectively 921 encrypting / decrypting operations with the same, given Serpent key, for one single pack / unpack call.

I've adapted the Serpent implementation that I previously published as part of EuCrypt, effectively integrating it into SMG Comms and stripping away anything that isn't directly needed by SMG Comms:

  • Serpent is now simply a package like all the others rather than a stand-alone library. While it is true that any changes to the original will have to be manually ported to this one as well, that was always going to be the case anyway. So I don't really see much point in carrying about all the glue and additional files to make a library out of only 2 files. Hence, Serpent in SMG Comms has 2 files and nothing more: serpent.ads and serpent.adb. Short and clear.
  • Since this is production use already, testing parts such as the "Selftest" method don't really have any business in the code itself. I've moved this method where it belongs, namely with the tests for all the code, in the tests directory (test_serpent.ads/.adb)
  • Since Serpent here becomes part of SMG Comms it follows that it should also use the raw types of the protocol - it will anyway be called to use variables of those types rather than anything else. There is no point in forcing back and forth conversions between SMG Comms' "Octets" and Serpent's "Bytes" types that are both arrays of octets anyway. So I've changed the definition of the "Bytes" type in Serpent so that it is here simply a subtype of the "Octets" type. This has the advantage that it allows smooth calls to Serpent from SMG Comms while being a small, easily-reversible change that also maintains otherwise the clarity of Serpent's code as it is. Basically SMG Comms gets to call Serpent without having to do explicit conversions between types that are anyway the same thing and Serpent gets to keep calling arrays of octets Bytes internally as it does in its stand-alone lib version.

In addition to the above, the .vpatch for this chapter also adds tests for the packing/unpacking methods2. I've also made a small change to raw_types.ads so that there is now only one variable for Serpent length: SERPENT_OCTETS. This reflects better the fact that there really is only one length for Serpent and it still allows to keep code clear by having the two array types Serpent_Pkt and Serpent_Msg - they just use the same length. Clarity of code is a tricky choice, what more can I say. Here's the updated code in raw_types:

  -- constants from SMG.COMMS standard specification
    -- size of a serpent-encrypted packet and message, in octets
    -- note that this corresponds to 1472/16 = 92 Serpent blocks
    -- NB: lengths are the same!
  SERPENT_OCTETS : constant Positive := 1472;

    -- size of a RSA-encrypted packet and message in octets and bits
  RSA_PKT_OCTETS     : constant Positive := 1470;
  RSA_MSG_OCTETS     : constant Positive := 234;
  RSA_MSG_BITS       : constant Positive := RSA_MSG_OCTETS * 8; --1872

  -- raw, low-level types
  -- all messages and packets are simply arrays of octets at low level/raw
  type Octets is array( Natural range <> ) of Interfaces.Unsigned_8;

  -- raw representations of basic types (with fixed, well-defined sizes)
  subtype Octets_1 is Octets( 1 .. 1 );
  subtype Octets_2 is Octets( 1 .. 2 );
  subtype Octets_4 is Octets( 1 .. 4 );
  subtype Octets_8 is Octets( 1 .. 8 );

  -- RSA packets and contained raw messages
  subtype RSA_Pkt is Octets( 1 .. RSA_PKT_OCTETS );
  subtype RSA_Msg is Octets( 1 .. RSA_MSG_OCTETS );

  -- Serpent packets and contained raw messages
  -- NB: length is the same but the distinction makes the code clearer
  subtype Serpent_Pkt is Octets( 1 .. SERPENT_OCTETS );
  subtype Serpent_Msg is Octets( 1 .. SERPENT_OCTETS );

And the new code in packing.ads:

  -- Packing/unpacking for Eulora's communication protocol:
  -- Serpent Message to/from Serpent Packet
  -- RSA Message to/from RSA Packet
  -- S.MG, 2018

with Raw_Types;
with Serpent;

package Packing is
  -- no side effects or internal state
  Pragma Pure(Packing);

  -- Packing a Serpent message into Serpent package, using the given key
  function Pack( Msg : in Raw_Types.Serpent_Msg;
                    K   : in Serpent.Key )
                  return Raw_Types.Serpent_Pkt;

  -- Unpacking a Serpent packet into contained message, using the given key
  function Unpack( Pkt : in Raw_Types.Serpent_Pkt;
                    K   : in Serpent.Key)
                  return Raw_Types.Serpent_Msg;

  -- internals of this package, NOT for outside use
private
  -- length of 1 Serpent block
  Block_Len: constant Natural := Serpent.Block'Length;

  -- number of Serpent blocks in one single Serpent message/packet
  S_Blocks : constant Natural := Raw_Types.SERPENT_OCTETS / Block_Len;

end Packing;

The new code in packing.adb:

  -- Packing/unpacking for Eulora's communication protocol:
  -- Serpent Message to/from Serpent Packet
  -- RSA Message to/from RSA Packet
  -- S.MG, 2018

package body Packing is

  -- Packing a Serpent message into Serpent package, using the given key
  function Pack( Msg : in Raw_Types.Serpent_Msg;
                 K   : in Serpent.Key )
               return Raw_Types.Serpent_Pkt is

    -- single Serpent blocks containing plain / encrypted data
    Plain    : Serpent.Block;
    Encr     : Serpent.Block;

    -- Serpent Key Schedule - needed for direct encr/decr calls
    KS       : Serpent.Key_Schedule;

    -- final resulting Serpent package
    Pkt      : Raw_Types.Serpent_Pkt := (others => 0);
  begin
    -- prepare the Serpent key schedule based on given key
    Serpent.Prepare_Key( K, KS );

    -- encrypt message block by block and copy result in packet
    for I in 1 .. S_Blocks loop
      -- get current block to encrypt
      Plain := Msg( Msg'First + (I-1) * Block_Len ..
                    Msg'First +  I    * Block_Len - 1 );
      -- encrypt with Serpent
      Serpent.Encrypt( KS, Plain, Encr );
      -- copy result to output packet
      Pkt( Pkt'First + (I-1) * Block_Len ..
           Pkt'First +  I    * Block_Len - 1 )
         := Encr;
    end loop;

    -- return result
    return Pkt;
  end Pack;

  -- Unpacking a Serpent packet into contained message, using the given key
  function Unpack( Pkt : in Raw_Types.Serpent_Pkt;
                   K   : in Serpent.Key)
                 return Raw_Types.Serpent_Msg is
    -- single Serpent blocks containing plain / encrypted data
    Plain    : Serpent.Block;
    Encr     : Serpent.Block;

    -- Serpent Key Schedule - needed for direct encr/decr calls
    KS       : Serpent.Key_Schedule;

    -- the message extracted from the given packet
    Msg : Raw_Types.Serpent_Msg := (others => 0);
  begin
    -- prepare the Serpent key for use
    Serpent.Prepare_Key( K, KS );

    -- decrypt the Serpent packet block by block
    for I in 1 .. S_Blocks loop
      -- get current block from input and decrypt
      Encr := Pkt( Pkt'First + (I-1) * Block_Len ..
                   Pkt'First +  I    * Block_Len - 1 );
      Serpent.Decrypt( KS, Encr, Plain );

      -- copy result to its correct position in final output
      Msg( Msg'First + (I-1) * Block_Len ..
           Msg'First +  I    * Block_Len - 1 )
         := Plain;
    end loop;

    -- return the result - the message content of given package
    return Msg;
  end Unpack;

end Packing;

The .vpatch and my signature for it are as usual on my Reference Code Shelf as well as linked here for your convenience:

  1. 1472 / 16 = 92 ↩

  2. And needed they were too for they actually caught an error that had survived somehow several re-readings of the code to the point that I was totally surprised when the tests first...failed. Never underestimate your own capacity of introducing idiotic errors in the simplest of things! ↩

October 16, 2018

SMG Comms Chapter 2: Raw Types

Filed under: Coding, SMG_Comms — Diana Coman @ 11:20 a.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

My previous Chapter 1 with its assorted warts and lamentations sparked a rather productive discussion in the forum that ended up with a significant revision of the protocol specification. The main change has to do with packet sizes and types: ALL packets will be either 1470 octets long and RSA encrypted OR 1472 octets long and Serpent encrypted. Those sizes are chosen to ensure that there is no fragmenting of UDP frames sent on the wire since such fragmenting increases significantly the chance of packet loss. A test with those UDP packet sizes sent back and forth between UK and UY did not uncover any trouble - if anything, the results were actually better than expected. In any case, increased chances of reliable communications aside, the added benefit of those fixed, unique sizes is a clearer protocol alltogether. So after a bit of digesting the new specification, I can finally say that I have some sort of plan for its implementation rather than grasping at it from any thread that I could perceive at all (as it was pretty much the case in Chapter 1). Specifically, my current implementation plan is made of 3 layers that build on one another and give me a structure to rely on:

  1. Layer 0 - Raw types
  2. Layer 1 - Sender/Receiver
  3. Layer 2 - Data structures

1. Layer 0 - Raw types

The raw types for Eulora's communication protocol are essentially two: octet (the basic unit, effectively 8 bits) and arrays of octets (various fixed sizes matching the basic types in the protocol's specification). However, for the arrays of octets, there is a useful distinction to be made between packets (i.e. the received array of octets of fixed length, either 1470 or 1472 octets) and messages (i.e. the content of a packet). Essentially messages are obtained from packets in Eulora's communication protocol through decryption using either Serpent (for 1472 octets long packets) or RSA (for 1470 octets long packet). Symmetrically, packets are obtained by encrypting messages with either Serpent (for 1472 octets long messages) or RSA (for 234 octets long messages). Consequently, this layer 0 offers the definitions of types that cover the full set of possible messages and packets together with the relevant conversion methods between packets and messages as well as between raw octets and int/unsigned/float data types. Note that a "message" here is simply the decrypted content of a packet, nothing more. The interpretation of the octets in a message and/or their use to fill some data structure that the protocol defines happens at a higher layer, not here.

2. Layer 1 - Sender/Receiver

Layer 1 is concerned with sending and receiving packets using the types and conversion methods provided by layer 0 together with the UDP lib for actual network communication. So far this seems to me at the moment a very thin layer really since it's little more than a basic receiver and sender: the receiver simply uses UDP lib to endlessly read all incoming packets and store them in a queue for later processing by consumer(s) external to this layer; the sender may have perhaps an input queue from which it keeps reading packets together with their intended destination and then sends them along using the UDP lib. This layer comes right after the raw types because it needs and uses those raw types but at the moment it's unclear to me whether it belongs as part of a protocol implementation since I can see the case for it being left to the user application of the protocol. At any rate, the protocol specification is not really concerned with this part and this description of the sender/receiver is not a given - it may change as required by each application.

4. Layer 2 - Data Structures

This layer is on top of Layer 0 - it provides the data structures for the different types of messages in Eulora's communication protocol. While any message in Eulora's protocol is just an array of octets at layer 0, the information contained in any message has to be structured according to the protocol's rules described in sections 4, 5 and 7 of the specification. This layer provides the definitions of those data structures as well as the conversion methods from messages to data structures and back.

Note that those 3 layers do not include protocol mechanics since those are really part of the server or client applications. Perhaps a demo / test application implementing the specified protocol mechanics would be useful to include as part of this implementation but I'd rather not specify it as such at this point. For now I'm fine with those 3 layers but if you see any holes in them or problems with them, let me know in the comments below!

Implementation

To start it all off, here's the first part of layer 0 - raw types, including the definitions of types and the most basic conversions via Ada.Unchecked_Conversion:

 -- raw types for the communication protocol
 -- these are used throughout at the lowest level of the protocol
 -- essentially they are the units of packets and of messages
 -- SMG.Comms has only 2 types of packets: RSA and Serpent
 -- a message is the decrypted content of a packet
 -- S.MG, 2018

with Interfaces; use Interfaces; -- Unsigned_n and Integer_n
with Ada.Unchecked_Conversion;

package Raw_Types is

  -- constants from SMG.COMMS standard specification
    -- size of a serpent-encrypted packet and message, in octets
    -- note that this corresponds to 1472/16 = 92 Serpent blocks
    -- NB: lengths are the same but the distinction makes the code clearer
  SERPENT_PKT_OCTETS : constant Positive := 1472;
  SERPENT_MSG_OCTETS : constant Positive := SERPENT_PKT_OCTETS;

    -- size of a RSA-encrypted packet and message in octets and bits
  RSA_PKT_OCTETS     : constant Positive := 1470;
  RSA_MSG_OCTETS     : constant Positive := 234;
  RSA_MSG_BITS       : constant Positive := RSA_MSG_OCTETS * 8; --1872

  -- raw, low-level types
  -- all messages and packets are simply arrays of octets at low level/raw
  type Octets is array( Natural range <> ) of Interfaces.Unsigned_8;

  -- raw representations of basic types (with fixed, well-defined sizes)
  subtype Octets_1 is Octets( 1 .. 1 );
  subtype Octets_2 is Octets( 1 .. 2 );
  subtype Octets_4 is Octets( 1 .. 4 );
  subtype Octets_8 is Octets( 1 .. 8 );

  -- RSA packets and contained raw messages
  subtype RSA_Pkt is Octets( 1 .. RSA_PKT_OCTETS );
  subtype RSA_Msg is Octets( 1 .. RSA_MSG_OCTETS );

  -- Serpent packets and contained raw messages
  -- NB: length is the same but the distinction makes the code clearer
  subtype Serpent_Pkt is Octets( 1 .. SERPENT_PKT_OCTETS );
  subtype Serpent_Msg is Octets( 1 .. SERPENT_MSG_OCTETS );

  -- blind, unchecked casts ( memcpy style )
  function Cast is new Ada.Unchecked_Conversion( Integer_8  , Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1   , Integer_8 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_8 , Octets_1 );
  function Cast is new Ada.Unchecked_Conversion( Octets_1   , Unsigned_8 );

  function Cast is new Ada.Unchecked_Conversion( Integer_16 , Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2   , Integer_16 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_16, Octets_2 );
  function Cast is new Ada.Unchecked_Conversion( Octets_2   , Unsigned_16 );

  function Cast is new Ada.Unchecked_Conversion( Integer_32 , Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4   , Integer_32 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_32, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4   , Unsigned_32 );

  -- Gnat's Float has 32 bits but this might be different with other compilers
  function Cast is new Ada.Unchecked_Conversion( Float, Octets_4 );
  function Cast is new Ada.Unchecked_Conversion( Octets_4, Float );

  function Cast is new Ada.Unchecked_Conversion( Integer_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Integer_64 );
  function Cast is new Ada.Unchecked_Conversion( Unsigned_64, Octets_8 );
  function Cast is new Ada.Unchecked_Conversion( Octets_8, Unsigned_64 );

end Raw_Types;

I packed the above code as a .vpatch on top of the previous genesis .vpatch mainly for keeping my original promise of showing this being built as it is with detours and corrections on the way. Otherwise this could equally well be a genesis in itself given how radically it changes pretty much everything in there. Nevertheless, at least for now - meaning until a final version is achieved and a regrind makes perhaps sense - I'll keep growing the smg_comms tree. I'll link all smg_comms .vpatches and my signatures for them from the Code Shelf page as usual, while linking the current ones here as well:

The next chapter will add to the raw types layer, providing conversions from packets to messages and back. That will bring in parts of EuCrypt of course, namely the Serpent and RSA implementations. For this reason, I aim to do this perhaps in two steps (first Serpent and only then RSA) in order to keep patches as clear and easy to follow as possible. And this is of course a very good opportunity to re-read the EuCrypt implementation, to re-write it on one level or another as I bring it over to SMG_Comms and to see first-hand how it is to use it!

October 13, 2018

Results of Testing UDP - Take 2

Filed under: Coding, UDP — Diana Coman @ 8:32 p.m.

As the first test revealed that most UDP packets make it safely at least when sent 1 second apart, I decided that the next week-long test should run with a substantially smaller delay: 0.01 seconds between each 2 packets. Moreover, I've changed the sender to simply send 10000 packets at each run, alternating between 1472 and 1470 octets in size. The focus on those 2 sizes reflects the fact that ALL packets are either 14721 or 14702 octets long in the newest version of Eulora's communication protocol. Here's the summary of the data I collected:

UY UK
Sent: 21400003 2140000
Received: 2134569 (99.74%) 2139788 (99.99%)
Errors received: 0 0
1470-size sent: 1070000 1070000
1470-size received: 1067322 (99.75%) 1069901 (99.99%)
1472-size sent: 1070000 1070000
1472-size received: 1067247 (99.74%) 1069887 (99.99%)

From the above, it would seem that even 0.01 delay between any two packets being sent is not causing any problem at all - at least not for the sizes considered4. If anything, the % of received packets increased slightly compared to the previous test. Of the packets that made it to the other side, none were mangled, hence no errors received5. Unsurprising perhaps, the 2 octets difference in size between the 2 types of packets does not seem to have the slightest impact on % of packets that make it to the other side - the route itself is likely to be more important in this respect.

The timings remain iffier to investigate as I did not change the tester specifically for this. Applying the same correction as in the previous test (i.e. calculating first the difference between the logged local times and then simply adjusting so that the minimum difference is 0) yields again the same stats on both sides (if slightly different values than during the last test):

Data Min 1st Q Median Mean 3rd Q Max
UK -> UY 0 7 13 13.09 20 27
UY -> UK 0 7 13 13.19 20 27

As before, I would not suggest taking the above timings for anything really. The only thing that can perhaps be said is that packets don't seem to take all that much to arrive at their destination at least on those routes. This is of course no guarantee at all, just a limited observation of this particular data set.

  1. Serpent encrypted packets that are meant for most communications in Eulora. ↩

  2. RSA encrypted packets that are meant only for new accounts in Eulora. ↩

  3. I've ran this for almost 9 days so more than the previous 1 week for the first test. ↩

  4. Arguably larger sizes that would get fragmented on the way might have higher chances of becoming lost. ↩

  5. There was however one spam packet received by the node in UY from an entirely unrelated IP but I removed that before making the summary above since it has nothing to do with the test itself. Also, the fact that one will receive spam is not exactly news otherwise so there isn't much to say about it at this moment. ↩

October 10, 2018

EuCrypt Chapter 14: CRC32 Implementation with Lookup Table

Filed under: Coding, EuCrypt — Diana Coman @ 1:51 p.m.

~ This is part of the EuCrypt series. Start with Introducing EuCrypt. ~

The communication protocol for Eulora uses CRC32 as checksum for its packages and so I found myself looking in disbelief at the fact that GNAT imposes the use of Streams for something as simple as calculating the checksum of anything at all, no matter how small no matter what use, no matter what you might need or want or indeed find useful, no matter what. No matter! As usual, the forum quickly pointed me in the right direction - thank you ave1! - namely looking under the hood of course, in this case GNAT's own hood, the Systems.CRC32 package. Still, this package makes a whole dance of eating one single character at a time since it is written precisely to support the stream monstrosity on top rather than to support the user with what they might indeed need. Happily though, CRC32 is a very simple thing and absolutely easy to lift and package into 52 lines of code in the .adb file + 130 in the .ads file so 182 all in total1, comments and two types of input (string or raw array of octets) included. And since a CRC32 implementation is anyway likely to be useful outside of Eulora's communication protocol too, I'm adding it as a .vpatch on the EuCrypt tree where it seems to fit best at the moment. It's a lib on its own as "CRC32" when compiled via its own crc32.gpr or otherwise part of the aggregate EuCrypt project if you'd rather compile all of EuCrypt via its eucrypt.gpr.

My CRC32 lib uses the 0x04C11DB7 generator polynomial and a lookup table with pre-calculated values for faster execution. As Stanislav points out, implementing the actual division and living with the slow-down incurred is not a huge problem but at least for now I did not bother with it. The CRC32 lib provides as output 32 bits of checksum for either a string or an array of octets. At the moment at least I really don't see the need for anything more complicated than this - even the string-input method is more for the convenience of other/later uses than anything else. For my own current work on Eulora's protocol I expect to use CRC32 on arrays of octets directly.

The .vpatch adding CRC32 to EuCrypt and my signature for it are as usual on my Reference Code Shelf with additional links here for your convenience:

  1. Specifically: 52 lines is the count of crc32.adb that does the work. The .ads file brings in another 130 lines that are mostly the lookup table with pre-calculated values. The .gpr file has another 61 lines and the restrict.adc another 80 lines. ↩

October 4, 2018

Results of Testing UDP - Take 1

Filed under: Coding, UDP — Diana Coman @ 6:09 p.m.

For one full week, between 26 September 2018 and 3 October 2018, my UDP Tester ran on 2 computers, one in the UK and one in Uruguay (UY), sending and receiving UDP messages in both directions. On each side, the receiver ran continuously, logging all UDP messages that it received during the whole interval. By contrast, the sender ran on both sides hourly but at different times so that the communications did not overlap. I don’t expect it would have been any trouble even if they did overlap but this was meant to be a test of UDP under best conditions and for this reason I set the times so that the sender on one end always finished a full run before the one on the other end started. For the same reason, the messages were sent at a rate of at most 1 message per second1.

At each run, the sender sent exactly 2043 UDP messages with lengths between 6 and 2048, each message having a different length. The order of messages was pseudo-random, relying on the Mersenne Twister prng using as seed the local time at the start of the run (in unix format). The sender kept a log of all messages it sent, including destination IP and port, seed used for MT, local time when message was sent and size of message. The receiver also logged basic information about each message: source IP and port, local time and message size as observed by receiver as well as those contained in the message’s own header, number of observed incorrect bits in the message’s payload as well as the expected and actual values of incorrect octets.

A first look at the week-long data yielded a bit of a surprise in that the UY receiver had actually received *more* messages than were sent from the UK! At a closer look, it turned out that 4933 UDP messages arriving at the UY node were actually sent by its own local switch! And moreover, they were all, without exception, recorded as corrupted since neither size nor payload matched the expected values2. At the moment those switch-generated messages are a bit of a mystery - it’s unclear what they are exactly or why and how they appeared. Working hypotheses would be that they are either local dhcp messages (although the port number would be a weird choice for those) or stray frags of bigger UDP messages. My one single attempt to replicate this behaviour while simultaneously capturing everything with tcpdump has so far failed - there were no such unexpected messages at all over several hours of UK sender at work. I might perhaps try again at a later date after I’m done with the more pressing tests that I need for SMG comms or simply process the existing error log and reconstitute the already observed weird messages from there. Anyway, for now I put those anomalous messages to the side and focus instead on the rest of the messages (which were sent as expected either by the node in the UK or by the one in UY). Here’s a summary of the data thus cleaned:

UK node UY node
Total sent: 3459283 3452674
Total received: 3447175 3451836
% received: 99.84%7 99.78%8
Errors received9: 0 0

Arguably the lost messages are of most interest in all the above: can one say perhaps that the largest messages10 get lost more often? Not really or at least not based on this little set of data. Compare the summary stats for three groups of messages: all messages sent from the UK (reflecting as expected the sizes sent and the fact that the same number of messages of each size are sent), all messages lost at UY (i.e. did not make it on the way from UK to UY) and all messages lost at UK (i.e. did not make it on the way from UY to UK):

Data Min 1st Q Median Mean 3rd Q Max
All sent from UK 6 516 1027 1027 1538 2048
Lost at UY (UK->UY) 13 513 1049 1051 1602 2045
Lost at UK (UY->UK) 16 553.5 1072 1061 1576 2047

While the data set of lost messages is quite small (550 messages lost at UK and 745 at UY), note that this is mainly due to the fact that there are relatively few losses overall: less than 0.4% of messages sent got lost on the way. So it would seem that at least under the conditions and on the routes considered11, UDP is not all that unreliable anyway. In any case, those summaries above seem to me remarkably close to one another - meaning that there isn’t any visible evidence that some sizes would get lost more than others, at least not for the set of sizes considered. Arguably sizes of up to 2048 octets of message are quite fine for communications over UDP - or at any rate, just as fine as smaller sizes.

In terms of order of received messages, the UY node received ALL messages precisely in the order in which they were sent but the UK node reported 66 messages in total that arrived out of order. Although this is a tiny number, it is perhaps reasonable to assume that it might increase in worse conditions (e.g. significantly less than 1 second between sending messages).

The actual timings are a bit iffier to investigate since the precision of UDP Tester turns out to be less than what would be needed for such task. Moreover, there is something weird going on with the way I recorded the time because the difference between the two nodes should be of ~34 seconds (UY node local time = UK node local time + 34) but this doesn't quite square with all the data especially at the UY receiver end12. On the more positive side though, at least the measurement bias there is constant for all the data and it doesn't introduce any weird effects so I can still attempt to infer something considering also that observed behaviour suggests that most UDP messages really make it to the other end within 1 second. Consequently, I calculated the delta on both sides as TR - TS at first and then I added (on UK side) respectively subtracted (on UY side) the quantity needed to make the lowest delta 0. So at the UY receiver, delta = TR - TS - 11 while at the UK receiver, delta = TR - TS + 32. With this correction, the summary stats for the delta on both sides turn out to be remarkably similar:

Data Min 1st Q Median Mean 3rd Q Max
Deltas at UK node: 0 5 11 10.60 16 21
Deltas at UY node: 0 5 11 10.62 16 21

Note that I do *not* recommend taking the above delta values for anything really, as the tester's precision in recording time is just not enough for this.

You are of course warmly invited to run your own tests and to play with this dataset in any way you find fit. So here's the data from both nodes, including the additional 4933 messages that the UY node received from its own switch:

  • udp_test_take1_data.zip (~10MB)
  • SHA512SUM: 963b8a1467630eea35532122ab7c2d25cb8741001808841f7cf02b34abb6ad5300adcb1d667dd902b4278dd2b373dc46427b0b0bbc918ee52f326456535a4114 udp_test_take1_data.zip

Have fun!

  1. Specifically: the sender had a delay of 1 second between any two consecutive messages. ↩

  2. The UDP tester simply fills the message up to any length with values calculated as Pos mod 256 where Pos is the position of the respective octet in the full message. ↩

  3. This is precisely 345928/2043=169 runs. ↩

  4. This includes a partial 170th run since I stopped the whole test while the UY sender was running already its 170th run. ↩

  5. 550 messages lost in total. ↩

  6. 745 messages lost in total. ↩

  7. 344717 / 345267 * 100 ↩

  8. 345183 / 345928 * 100 ↩

  9. This refers to messages received but with payloads that don’t match the expected values. ↩

  10. Note that this test capped the messages at 2048 so “largest” here means strictly < 2049 octets. ↩

  11. The UK node is a “consumer” node i.e. behind a router and on a residential connection; the UY node is S.MG’s test server with Pizarro. ↩

  12. Considering TR as TimeReceived and TS as TimeSent, at UY receiver the delta should be calculated as TR - (TS + 34) = TR - TS - 34; however, there are entries with TR-TS as low as 11 so basically it would seem that messages arrived before they were even sent. ↩

Older Posts »

Theme and content by Diana Coman