Ossasepia

February 7, 2019

Seppuku Job Market: Minimal Dynamic Tasking in Ada

Filed under: Coding — Diana Coman @ 10:44 p.m.

Eulora's server needs a reliable and robust way of performing - preferably in parallel whenever possible - various jobs for all the players that might connect to the game at any given time. Given the parallel requirement, there isn't really any way around the fact that multi-threading is needed. Nevertheless, since multi-threading is by its nature complex enough to give subtle errors and heavy headaches at any time, I'd really much rather make sure any implementation that deals with multiple threads of execution is as small, clear, plain and easy to follow as possible. In other words, if it has to be multi-threaded then it should better be minimal, self-healing, self-adjusting and ruthlessly functional with all and any bells and whistles chucked as far away from it as possible. To drive this point home and keep it in mind at all times1, I'll call this self-reliant unit of the server the Seppuku Job Market or SJM for short.

The list of requirements for the SJM is this:
1. Accept Jobs from all and sundry in a thread-safe manner and execute them in order of their priorities.
2. Generate and kill Worker tasks2 *dynamically* and on an *as-needed basis* to perform jobs as soon as possible but remaining at all times within a pre-set maximum number of Workers3.
3. Creation and destruction of Workers should be reliable and robust: in particular, SJM should run for ever unless explicitly stopped and it should re-spawn Workers as needed, even if they get killed from outside the code (cosmic-ray event or not).
4. Aim to perform jobs in order of their specified priority but taking into account that at most ONE job per player is actually executed at any given time. In other words: do NOT allow a player to hog the whole thing and run as many jobs as they want; this is parallelism aimed to increase the number of players served not merely the number of jobs performed!

Point 1 of requirements hints at the nature of SJM: as it needs to accept jobs from many, unknown sources, it is effectively a "server" of sorts and moreover it is essentially a resource from the point of view of job producers. The best Ada construct that readily fits this description is a protected unit (aka a passive entity that guarantees thread-safe access to the data it encapsulates - in this case to the queue of jobs waiting to be performed). One significant benefit of an Ada protected entity is the fact that it is specifically not a task itself nor is there a task associated with it. Instead, the mutually exclusive access to services provided by a protected unit is ensured by the run-time system and therefore the whole thing has at least one less headache to think of: while Worker tasks may get killed, the SJM itself at least cannot get killed unless the whole program (i.e the main thread of execution of the server itself) gets killed.

Point 2 of requirements (dynamic, self-adjusting number of tasks) means that I'll need to actually create and dispose of tasks programmatically - there is no way to have only statically allocated tasks. In turn, this means that a few restrictions have to go away: No_Allocators, No_Finalization, No_Task_Allocators, No_Tasking, No_Unchecked_Deallocation. The need to drop the No_Finalization and No_Unchecked_Deallocation restrictions comes from the way in which Ada handles memory allocated dynamically even when on the stack. Essentially, dynamically allocated tasks receive memory from a "pool". Once allocated, memory from a pool is reclaimed ONLY when the whole pool goes out of scope or in other words when it can be guaranteed that there is no piece of code left that can actually attempt to access that bit of memory. This is very robust and quite useful of course but in the case of dynamically allocated tasks it means that tasks that finish will STILL effectively occupy memory unless specifically deallocated (with unchecked_deallocation as that's the only way to do it as far as I can tell). In turn, this creates the undesirable but very real and quite horrible possibility that the code will run just fine *until* the pool in which tasks are created runs out of memory because of all previous tasks that finished long time ago but whose space was never reclaimed. To avoid this, the code has to keep track of terminated tasks and explicitly deallocate the memory they occupy before chucking away their pointer and/or re-spawning a replacement Worker (as there is no way to "restart" a task).

Point 3 means that Workers need to be effectively managed based on the evolution of the number of available jobs and the number of Workers themselves. One approach would be of course to have a Supervisor task but the problem then is twofold: first, the Supervisor needs to be aware of changes to the jobs queue as they happen; second, having a Supervisor task creates the potential problem of who supervises the supervisor (esp. with respect to recovery from killed thread since in this case the Supervisor itself might die unexpectedly). Given however that the SJM protected unit effectively guards precisely the jobs queue, it's also in the best position to react promptly to an increase or decrease in jobs and so it follows that it should in fact manage the Workers too. After all, it can do a bit more on receiving a job than merely chucking it into the queue: ideally it would in fact pass it on to a Worker immediately.

While at first sight "take job, spawn Worker, pass it on and let him do it" sounds precisely fine, in practice it's really not fine at all and not least because of the requirement at Point 4: passing a job on to a Worker requires some ordering of jobs (by priority) and even a sort of guarded access to a player since a new job cannot be accepted (and especially cannot be passed on to a Worker for execution) while an existing Worker may still be toiling away on a previous job for the same player. So the SJM needs to find out when a job is finished in order to accept again jobs for that specific player. As always, there are only a few ways to know when something finished: either look for it4 as one rather has to do when Workers are just passive executors of jobs or otherwise expect a signal to be sent back by a more active type of Worker task when it finished the job it had.

This distinction between active and passive Workers (or tasks in general) is quite significant. As passive entities, Workers can at most simply wait to be handed a job or any other signal. Typically, a Worker would be created and handed a job, they would do it and then they would quietly die keeping out of the way of everyone else. This can be a great fit in various cases but I can see several problems with this for Eulora's server: first, Workers cannot be reused even when jobs are available so there is a rather inefficient kill/create overhead5 precisely at busy time when one wants it even less than at any other time; second, the only way for the SJM to find out when a job finished is by a sort of polling i.e. going through the whole set of workers and checking which one is in a terminated state - note that it is not at all clear just *when* should this be done or how would it be triggered (sure, one can use a sort of scheduled event e.g. check it every 3 seconds or some such but it's more of a workaround than addressing the problem); third, the SJM needs to do both Worker creation and Job allocation (i.e. priority ordering + only one job per player at any given time) at the same time and while keeping a job creator waiting.

The first of the above issues (no reuse of Workers) is easily addressed by making Workers active rather than passive: they get created and then they actively ask for a job; once they got a job, they do it and then they go back and report it done, after which they queue again to get another job or perhaps the boot if there are no jobs to be had. And since such active Workers do not finish by default when a task is finished, they need to have rather suicidal tendencies and ask not merely for a job but rather for either a job or permission to seppuku (hopefully in a clean manner though!).

Making Workers active (if suicidal) neatly separates Worker creation from Job allocation: when jobs start pouring in, the SJM can simply create a bunch of Workers and release the job creators before it makes time to actually hand the jobs out to queuing workers. When the jobs keep pouring in, Workers keep working and there's no need to kill them now to only create them a few milliseconds later. Moreover, finished jobs are simply reported and marked as such without any need to poll. In the (hopefully rare) case when a Worker dies unexpectedly before sending the signal that it finished its job, they will be anyway observed sooner or later when the state of Workers is assessed to decide if more or fewer Workers are needed. Essentially the only trouble this approach brings is the added responsibility on the SJM: it controls access to the Job queue for job creators AND for Workers while ALSO effectively managing and keeping track of all Worker-related aspects. But then it's not a Seppuku Job Market for no reason: if it needs to do it, it will have to do it and do it well.

As a proof of concept of the above, I have implemented the SJM precisely as described: as a protected unit that encapsulates a Job queue and manages active Worker tasks, creating and destroying them as needed while also de-allocating memory of any terminated Workers, ensuring that only one Job per player is accepted at any given time and allowing a graceful stop that does not block any job producers that may come at a later time and does not leave dangling Worker tasks either. Jobs are simply record types with a discriminant that specifies their type and therefore the exact form a variable part of the record takes (since each Job type is likely to have specific data structures it requires). Note that I specifically avoided the Object-Oriented option (i.e. tagged type in Ada) with a hierarchy of Job types and reliance on polymorphism for "Complete" to do the right thing depending on the exact type of Job. The reason for this avoidance is mainly that there really isn't much to gain from it as far as I can see at the moment. Similarly, I prefer to not rely on generic containers (for the Job Queue for instance) unless they become clearly and absolutely needed. Finally, I am quite aware of Ada's relevant annexes such as Real-Time Systems and I know that it provides a whole infrastructure of worker pools and jobs with futures even (i.e. a way to provide results at a later time) but they are quite at odds with the significant aim of keeping it all as short6 and clear and easy to follow as possible (not to mention potential issues with the way in which some parts might be implemented using a secondary stack for instance which I specifically do not want to have).

The public part of the EuJobs package is this:

with Interfaces; use Interfaces;
with Data_Structs;
with Ada.Finalization;
with Ada.Unchecked_Deallocation; -- to clean up worker tasks if needed.

package EuJobs is

  pragma Elaborate_Body;

  -- knobs and constants
  Max_Workers : constant Natural := 64;
  Max_Idle_W  : constant Natural := Max_Workers;
  -- max jobs
  Max_Jobs    : constant Natural := Max_Workers * Max_Workers;

  -----------------------WORKERS--------------------
  -- Generic Eulora Workers type: simply perform given Jobs
  subtype Worker_Index is Natural range 1..Max_Workers;
  -- Those are to be FULLY managed (including created/ended) by the Job Market
  -- ACTIVE but suicidal elements:
  --   a worker will keep requesting jobs/permission to seppuku
  --     until allowed to terminate
  -- Pos is a token identifying Worker with the Job Market
  -- NB: ALL workers WILL use this Job Market
  -- NB: do NOT create workers from outside the Job Market!
  task type Worker( Pos: Worker_Index );

  -- needed to dynamically generate Workers
  type Worker_Address is access Worker;
  procedure Free is new Ada.Unchecked_Deallocation(Worker, Worker_Address);

  -- ALL the info that the Job Market holds on workers to manage them
  type Worker_Rec is
    record
      Assigned  : Boolean := False;
      Player_Id : Interfaces.Unsigned_64;
      WA        : Worker_Address;  -- actual pointer to worker
    end record;  

  -- for storing pointers to generated workers including if assigned and id
  type Worker_Array is array( Worker_Index'First ..
                              Worker_Index'Last) of Worker_Rec;

  -- limited controlled type that ensures no dangling workers at Finalize time
  type Controlled_Workers is new Ada.Finalization.Limited_Controlled with
    record
      Workers: Worker_Array;
    end record;

  overriding
  procedure Finalize( S: in out Controlled_Workers );
  overriding
  procedure Initialize( S: in out Controlled_Workers );

  -------------------------------JOBS-------------------
  -- Job types; NB: do NOT map (nor have to) directly on message types!
  type Job_Types is ( Do_Nothing,
                      Print_Job,
                      Create_Acct );

  -- Data structure with relevant information for each type of job
  type Job_Data ( T: Job_Types := Do_Nothing ) is
    record
      -- common information relating to the one requesting this job
      Player_ID: Interfaces.Unsigned_64 := 0;
      Source_IP: Interfaces.Unsigned_32 := 0;
      --NB: this is SOURCE port - reply WILL be sent here, whether RSA or S!
      Source_P : Interfaces.Unsigned_16 := 0;
      -- Message counter, as received
      Counter  : Interfaces.Unsigned_16 := 0;
      Priority : Natural := 0; --lowest possible priority
      case T is
        when Create_Acct =>
          Acct_Info: Data_Structs.Player_RSA;
        when Do_Nothing =>
          null;
        when Print_Job =>
          null;
      end case;
    end record;

  procedure Complete( JD   : in Job_Data );

  subtype Job_Count is Natural range 0..Max_Jobs;
  type Job_Array is array( 1..Max_Jobs ) of Job_Data;
  type Jobs_List is
    record
      Len  : Job_Count := 0;
      JA   : Job_Array;
    end record;

  ---------------------------Job_Market--------------------

  -- FULLY self-managed Job Market for euloran jobs:
  --   -- accepts jobs to do
  --   -- spawns, kills and managed workers that complete the jobs
  -- NB: Job_Market will DISCARD a new job when:
  --    -- it is FULL (i.e. can't handle anymore)
  --    -- it is stopping
  --    -- it already has a job for the same player
  -- Jobs are performed according to specific criteria (not strictly fifo):
  --   - FIFO but ensuring no more than 1 job per player served at any time
  --   - ALSO: there might be other priorities (e.g. type of job)
  protected Job_Market is
    -- adding a new job that needs to be done
    -- this can be ANY derivated type of Job_Data
    -- NB: Added will be true if J was indeed accepted and False otherwise
    entry Add_Job( J     : in Job_Data;
                   Added : out Boolean );

    -- workers request jobs when they are out of work
    -- workers need to provide their token (Pos)
    -- they can get to do: either a job OR seppuku signal.
    procedure Get_Job( Pos    : in Worker_Index;
                       J      : out Job_Data;
                       Seppuku: out Boolean );

    -- workers have to report back when a job is done
    -- (or they get sweeped up eventually if/when they abort).
    procedure Done_Job( Pos: in Worker_Index );

    -- sets in motion the process to stop gracefully:
    --   -- no more jobs received, existing discarded
    --   -- all workers will be given Seppuku signal
    -- NB: NO reverse for this.
    procedure Stop;

    -- for any external supervisors
    -- returns TRUE if it is NOT stopping
    -- returns False if it is stopping
    function Operating(  Waiting_Jobs: out Natural;
                         Idle_Workers: out Natural;
                         Active_Workers: out Natural;
                         Terminated_Workers: out Natural;
                         Is_Full: out Boolean)
      return Boolean;

  private

    -- internal storage of jobs and mgm of workers
    Board  : Jobs_List;

    -- NB: Workers are in the BODY of the package
    --   because they HAVE to be after the body of Finalize

    -- when stopping:
    -- discard new jobs; give out stop on get/done; empty jobs map
    Stopping : Boolean := False;
    Fullboard: Boolean := False;

    -- Retrieves next available job from the Board and returns it in JD
    -- Sets Found to True if an available job was found (i.e. JD is valid)
    -- Sets Found to False (and JD is undefined) if NO available job was found.
    -- NB: this DOES remove the element from the board!
    procedure Get_Available( JD    : out Job_Data;
                             Found : out Boolean );

    -- checks if the given player_id IS currently served by any worker
    function Is_Assigned( Player_ID: in Interfaces.Unsigned_64 )
             return Boolean;

    -- Checks in Board list ONLY if there is a job for this player
    -- Returns True if a job was found (i.e. a job waiting for a worker)
    -- Returns False otherwise.
    -- NB: Player might STILL have a job in progress (assigned to a worker)
    function Has_Waiting_Job( Player_ID: in Interfaces.Unsigned_64 )
             return Boolean;

    -- releases any player_id that might be stuck with aborted workers
    -- *creates* new workers if needed (specific conditions met)
    procedure Manage_Workers;

  end Job_Market;

private

  -- create new Worker with identification token (position) P
  function Create_Worker(P: in Worker_Index)
             return Worker_Address;

end EuJobs;

Workers are very simple tasks with an ID received at creation time to identify them within the Job_Market (very simply by position in the array of Worker addresses). They run a loop in which they request tasks or permission to Seppuku and when they receive either of them they proceed to do as instructed. Perhaps you noticed above that the array of Worker pointers is wrapped inside Controlled_Workers, which is a controlled type. A controlled type in Ada guarantees that the provided Initialize and Finalize routines are run precisely at the stages that their names suggest to enable the type to start off cleanly and to end up cleaning after itself. In the case of Controlled_Workers, the Initialize simply makes sure that the array has all pointers marked as null and moreover as not assigned any tasks while the Finalize goes one more time through the array and finishes off (with abort) any workers that are not null already. Note that the scope of Worker tasks is in fact the package level since the Worker_Address type is declared at this level (and that's how the scope is defined for such types in Ada). You might have noticed also that there is no concrete array of Workers defined anyhere so far: indeed, the array of workers is defined inside the package body for two main reasons: first, it should NOT be accessed by anyone from outside (not even potential children packages at a later time); second, it has to be defined after the bodies of Initialize and Finalize since otherwise it can't be created.

Jobs are barely sketched for now as Job_Data structures with a discriminant to distinguish different types and a variable part for specific data that each type of job needs. The Complete procedure then simply does different things for each type of job in a straightforward manner (at the moment it does something for the print job only for basic testing purposes).

The Job_Market itself is a protected object that offers a handfull of services (aka public entries, procedures or functions): entry Add_Job for job producers to provide their new jobs; procedure Get_Job for Workers who are looking for something to do; procedure Done_Job for Workers who report they finished their previously allocated job; procedure Stop for any higher-level caller who is in a position to turn off the whole Job_Market; function Operating that simply provides information on the current state (i.e. operating or stopping) and status (e.g. number of jobs and workers) of the Job_Market. Note that there are important differences between functions, procedures and entries: functions can only *read* protected data so they are effectively banned from modifying anything, hence Operating being exactly a function as it provides a snapshot of current state and metrics for the Job_Market; procedures can modify data but a call to them is unconditional meaning it gets accepted as soon as the protected object is available and the caller is first in queue for it, without any further restrictions - hence Stop, Done_Job and Get_Job are procedures since there is no constraint on them being called at any time; finally, entries can also modify data but they have entry barriers meaning they accept a call only when certain conditions are met - in this case Get_Job has the simple but necessary condition that either the Job_Market is stopping (in which case callers should not be blocked since it's pointless to wait anyway) or the Job queue is not full since it makes little sense to allow a job producer in just to discard their job for lack of space anyway. Note however that this is merely for completeness here since in practice there will be several other levels of measures taken so that the job queue does NOT become full since that is clearly not a sane way to have the server running.

In addition to the above public services, the Job_Market also has a private part where it keeps the job queue (as a basic array for now - this can easily change at a later time if there is a good reason for the change), a flag to know if it's stopping and one to register if/when the board is full as well as a few helper procedures and functions for its own use. The Get_Available procedure effectively implements the strategy of picking next Job to execute: it's here that priorities are considered really and it's here that there is another check to make sure that no two jobs of the same player are ever executed at the same time. The Is_Assigned procedure checks the set of Workers to see if any of them is performing a job for the specified player. The Has_Waiting_Job on the other hand checks the job queue to see if there is any job from the specified player waiting in the queue. Arguably the most important of those is "Manage_Workers" that does precisely what the name says: it does a headcount of Workers in various states, cleans up any aborted/unexpectedly dead ones, reclaims memory for terminated ones and then, if required, creates new Workers to match the current Job provision. Note that there really are only 64 workers in total (and at any rate this is unlikely to become a huge number) so this headcount of workers is not really terribly costly.

The overall package further has a private function that dynamically creates a new Worker task with the given ID, returning its address. This is more for convenience than anything else since one could easily call new directly so perhaps it will even go away at the next round of trimming the code.

The implementation in eujobs.adb starts with the Initialize and Finalize procedures, declares the Controlled_Workers object and then proceed with the internals of the Job_Market itself:

with Ada.Text_IO; use Ada.Text_IO;

package body EuJobs is

  procedure Finalize( S: in out Controlled_Workers ) is
    -- ALL this needs to do is to make SURE no worker is still running!
  begin
    for I in S.Workers'First .. S.Workers'Last loop
      if S.Workers(I).WA /= null then
        abort S.Workers(I).WA.all;
        S.Workers(I).WA := null;
        S.Workers(I).Assigned := False;
      end if;
    end loop;
  end Finalize;

  procedure Initialize( S: in out Controlled_Workers ) is
  begin
    for I in S.Workers'First .. S.Workers'Last loop
      S.Workers(I).WA := null;
      S.Workers(I).Assigned := False;
    end loop;
  end Initialize;

  -- actual workers slots; workers are managed internally here
  -- this type is needed though, to Finalize properly
  CW: Controlled_Workers;

  protected body Job_Market is
    -- adding a new job that needs to be done
    -- this can be ANY derivated type of Job_Data
    entry Add_Job( J     : in Job_Data;
                   Added : out Boolean )
      when Stopping or    --to unblock producers
           (not Fullboard) is
    begin
      -- if stopping, discard job -- allows callers to finish too...
      -- check Player_ID and add job ONLY if none exist for this player
      if (not Stopping) and
         (not Is_Assigned(J.Player_ID)) and
         (not Has_Waiting_Job(J.Player_ID)) then
        -- board is known to have space, so add to it
        Board.JA(Board.JA'First + Board.Len) := J;
        Board.Len := Board.Len + 1;

        -- job added may mean full board
        FullBoard := Board.Len >= Board.JA'Last;

        -- Quick worker management to adjust if needed
        Manage_Workers;
        -- Let caller know that job was indeed added
        Added := True;
      else
        Added := False; --not added, aka discarded
      end if;
    end Add_Job;

    -- workers request jobs or seppuku when they are out of work
    procedure Get_Job( Pos    : in Worker_Index;
                   J      : out Job_Data;
                   Seppuku: out Boolean ) is
      Found : Boolean;
    begin
      if Stopping then
        -- when stopping: all seppuku
        Seppuku := True;
      else
        -- try first to get some job that should be done
        Get_Available(J, Found);
        if (not Found) then
          Seppuku := True; --since no job is available..
        else
          -- have a job so no seppuku for now
          Seppuku := False;
          -- update Worker record to mark player as being served etc.
          CW.Workers(Pos).Assigned := True;
          CW.Workers(Pos).Player_ID := J.Player_ID;
          -- this SURELY means board is NOT full!
          Fullboard := False;
        end if;
      end if;
      -- LAST: manage workers in ANY CASE!
      Manage_Workers;
    end Get_Job;

    -- workers have to report back when a job is done
    procedure Done_Job( Pos: in Worker_Index ) is
    begin
      -- update record for this worker and let him go
      CW.Workers(Pos).Assigned := False;
    end Done_Job;

    -- aim to stop gracefully:
    --   -- no new jobs stored, existing discarded, workers killed.
    -- NB: NO reverse for this.
    procedure Stop is
    begin
      Stopping := True; -- NO need for anything else, really
    end Stop;

    function Operating(  Waiting_Jobs: out Natural;
                         Idle_Workers: out Natural;
                         Active_Workers: out Natural;
                         Terminated_Workers: out Natural;
                         Is_Full: out Boolean)
      return Boolean is
    begin
      Waiting_Jobs := Natural( Board.Len );
      Is_Full := Fullboard;
      Idle_Workers := 0;
      Active_Workers := 0;
      Terminated_Workers := 0;

      for I in CW.Workers'Range loop
        if CW.Workers(I).WA /= null then
          if CW.Workers(I).WA'Terminated then
            Terminated_Workers := Terminated_Workers+1;
          elsif CW.Workers(I).Assigned then
            Active_Workers := Active_Workers + 1;
          else
            Idle_Workers := Idle_Workers + 1;
          end if;
        end if;
      end loop;
      return (not Stopping);
    end Operating;

    -- anything needed for external load checking (?)

--private stuff

    procedure Get_Available( JD    : out Job_Data;
                             Found : out Boolean ) is
      Pos   : Job_Count;
      P     : Natural := 0; --priority of job found so far
    begin
      Found := False;
      -- ALWAYS walk the FULL set: higher priority might have come in later
      for I in 1 .. Board.Len loop
        if ( (not Found) or (Board.JA(I).Priority > P) ) and
           (not Is_Assigned(Board.JA(I).Player_ID) ) then
          Found := True;
          Pos   := I;
          P     := Board.JA(I).Priority;
          -- but don't copy just yet, as there might be higher priority further
        end if;
      end loop;
      -- retrieve the found job data but ONLY if found!
      if Found then
        JD := Board.JA(Pos);
        -- if not last job, shift to avoid gaps in the array
        if Pos < Board.Len then
          Board.JA(Pos..Board.Len-1) :=
              Board.JA(Pos + 1 .. Board.Len);
        end if;
        -- update count of jobs in the array
        Board.Len := Board.Len -1;
      end if;
    end Get_Available;

    function Is_Assigned( Player_ID: in Interfaces.Unsigned_64 )
             return Boolean is
      Found: Boolean := False;
    begin
      -- walk the array of workers and check
      for I in CW.Workers'Range loop
        if CW.Workers(I).WA /= null and
           CW.Workers(I).Assigned and
-- Will have to rely on .assigned being SET properly by the manager!
--  (not CW.Workers(I).WA'Terminated) and
           CW.Workers(I).Player_ID = Player_ID then
          -- found it!
          Found := True;
          exit;
        end if;
      end loop;
      return Found;
    end Is_Assigned;

    function Has_Waiting_Job( Player_ID: in Interfaces.Unsigned_64 )
             return Boolean is
      Found: Boolean := False;
    begin
      for I in Board.JA'First .. Board.JA'First + Board.Len loop
        if Board.JA(I).Player_ID = Player_ID then
          Found := True;
          exit;
        end if;
      end loop;
      return Found;
    end Has_Waiting_Job;

    procedure Manage_Workers is
      Active_W: Natural := 0;
      Idle_W  : Natural := 0;
      Total_W : Natural := 0;
      To_Create: Natural:= 0;
    begin
      -- release player ids if workers terminated
      -- count also precisely how many are active
      for I in CW.Workers'Range loop
        if CW.Workers(I).WA /= null then
          if CW.Workers(I).WA'Terminated then
            -- this terminated abnormally -> LOG?
            CW.Workers(I).Assigned := False;
            -- claim this space to restart a worker here if needed
            --CW.Workers(I).WA := null;
            -- deallocate it too as otherwise memory space slowly gets lost
            -- NB: Free proc sets it to null anyway
            Free(CW.Workers(I).WA);

          --if NOT null and NOT terminated-> idle or active
          elsif CW.Workers(I).Assigned then
              -- this is an active worker, count it
              Active_W := Active_W + 1;
          else
            -- this is an idle worker, count it
            Idle_W := Idle_W + 1;
          end if;
          -- null workers are simply empty spaces, no need to count them
        end if;
      end loop;
      -- calculate total workers
      Total_W := Active_W + Idle_W;

      if (not Stopping) and
         (Board.Len > Total_W) and
         (Total_W < Max_Workers ) and
         (Idle_W = 0) then
        -- need (perhaps) to create workers: how many?
        To_Create := Board.Len - Total_W;

        -- create them for as long as there is ANY space..
        -- NB: MORE workers MIGHT have terminated meanwhile,
        -- but they won't be null!
        for I in CW.Workers'Range loop
          if CW.Workers(I).WA = null then
            -- found a place, so create a worker
            CW.Workers(I).Assigned := False;
            CW.Workers(I).WA := Create_Worker(I);
            To_Create := To_Create - 1;
            Total_W := Total_W + 1;

            if To_Create <= 0 or Total_W >= Max_Workers then
              exit;
            end if;
          end if;
        end loop;
      end if;
     end Manage_Workers;

  end Job_Market;

  -- Worker body
  task body Worker is
    JD      : Job_Data;
    Seppuku : Boolean := False;
  begin
    -- main Loop: get a job or die, work and repeat.
    Work_Loop:
    loop
      -- ask the Job Market for a job or permission to seppuku
      Job_Market.Get_Job( Pos, JD, Seppuku );

      if Seppuku then
        exit Work_Loop;
      else
        -- do the job
        EuJobs.Complete( JD );
        -- report job done
        Job_Market.Done_Job( Pos );
      end if;
    end loop Work_Loop;
    -- worker is done and will die gracefully!
  end Worker;

  -- Jobs themselves
  procedure Complete( JD   : in Job_Data ) is
    Stop: Boolean;
  begin
     -- do different things for different types of jobs...
    case JD.T is
        when Create_Acct =>
          --Acct_Info: Data_Structs.Player_RSA;
          Stop := False;
        when Set_SKeys =>
          -- SKes: Data_Structs.Serpent_Keyset;
          Stop := False;
        when Mgm_SKeys =>
          --SMgm: Data_Structs.Keys_Mgm;
          Stop := False;
        when Print_Job =>
          Put_Line("Completing: job counter " &
                   Interfaces.Unsigned_16'Image(JD.Counter) &
                   " priority " & Natural'Image(JD.Priority) &
                   " for player " &
                   Interfaces.Unsigned_64'Image(JD.Player_ID) &
                   " from IP:P " & Interfaces.Unsigned_32'Image(JD.Source_IP) &
                   ":" & Interfaces.Unsigned_16'Image(JD.Source_P));
        when others =>
          -- no job or dubious at best, better stop.
          Stop := True;
    end case;
  end Complete;

  function Create_Worker(P: in Worker_Index)
             return Worker_Address is
  begin
    return new Worker(P);
  end;

end EuJobs;

Your thoughts, observations and critiques on the above are welcome below in the comments section. If there is a problem with the above approach or with the code itself I really want to hear of it sooner rather than later since it's of course easier to do something about it now - this is after all the whole reason why I'm publishing this proof of concept so go ahead and point out any faults you see.

  1. Also to reflect some suicidal tendencies of my Workers but that becomes clearer later. ↩

  2. "Threads" if you prefer non-Ada terminology. ↩

  3. There isn't much point in having more Workers than your underlying iron can actually support ↩

  4. blocking until it's done or checking at some intervals ↩

  5. Ada's documentation claims that dynamic creation of a task has a big overhead anyway so it's best avoided whenever possible but I can't say I have any idea just what "big overhead" means here. ↩

  6. The full .ads + .adb code+comments shown below is 500 lines, it uses no secondary stack, no heap and no containers or other similar external packages. Even the "use Ada.Text_IO" will go away as it's in there now strictly to allow the Print job to be seen as it completes for testing purposes. ↩

January 12, 2019

Compiling Ada Library for Use with Non-Ada Main

Filed under: Coding — Diana Coman @ 5:40 p.m.

Following a rather bumpy road of compilation troubles trying to link an Ada lib into a CPP main program, I found first a working solution to the task at hand and then a headache trying to disentangle the confusion of what is exactly a "standalone encapsulated dynamic" library, why is it needed and how exactly does it differ on the initialization front from a boring static library. Fortunately it turns out that my headache was due mainly to all that bumping into walls combined with the rather confusing terms used in .gpr files - there was at least nothing that a good dose of hands-on experimentation and several re-reading of the GNAT docs couldn't cure! Still, as I'd rather not repeat the whole process next time I need to mix Ada with others, I'll summarise here my notes on the options I found for compiling an Ada library1 so that it can be safely used from a non-Ada main program.

To use Ada code from a non-Ada main program, one needs to find a way to actually start the Ada run-time environment *before* any calls to Ada code. The Ada run-time does the crucial task of elaboration of Ada code (i.e. getting everything ready for executing code, so broadly speaking it takes care of initializing variables and constants as well as running any code found in the main body of packages that are used). Since elaboration is a concept entirely specific to Ada, there is no way to rely on the non-Ada main code (C, C++ or whatever it might be) to take care of this. Instead, the solution is to make sure that the Ada library itself contains and exposes an initialization procedure that does exactly this: starts the Ada run-time and performs the required elaboration for the library code. Once this exists, the non-Ada code simply has to make sure it calls this initialization procedure *before* calling *any* Ada code from that library and that's all2. This much was clear from the beginning - it's from here on that the headache and confusion started since not ALL Ada libraries actually contain/expose such an initialization routine. Essentially, in addition to the usual classification of libraries into static or dynamic, Ada has another parallel classification: standalone or not! And asking gprbuild to produce a standalone library is NOT done by using the "Library_Standlone" option but by defining an... interface for the library via the "Library_Interface" option in the .gpr file. Specifically, from the beginning:

  • To use an Ada library from a non-Ada main program, one needs to compile the library as "standalone". The standalone type of Ada library is the only one that contains and exposes for outside use an initialization routine that will start the Ada run-time and perform all elaboration tasks required for the library itself. NB: the initialization routine will be called libnameinit so if the library is called "eunet" then the routine will be "eunetinit".
  • To create a standalone Ada library with gprbuild, the corresponding .gpr file has to include the option "Library_Interface" that lists the packages that are actually exposed for use from outside the library. This option is enough by itself to obtain a standlone library and therefore to have the initialization routine! NB: you can build a standalone library as static or dynamic, as you want, simply specifying the kind, via "Library_Kind" - in both cases, the resulting .a or .so file will contain the initialization routine. For example: 
      for Object_Dir use "obj";
  for Library_Dir use "lib";
  for Library_Name use "eunet";
  for Library_Kind use "static";
  for Library_Interface use ("Eunet", "Raw_Types");
  • Standalone libraries have subtypes too and it is actually the subtype that is specified via the option "Library_Standalone" in a .gpr file! According to GNAT's user guide, the Library_Standalone can take 3 values: standard (default), no, encapsulated.
    • The "standard" is the option used if your .gpr file does not even mention "Library_Standalone" (but DOES mention "Library_Interface"!) and it means that the initialization routine is contained and exposed.
    • The "encapsulated" option means in addition that the library will depend only on static libraries except for system libraries - so this option will effectively pull in everything the library needs, including the GNAT run-time. This makes for a significantly *easier* use and linkage further downstream BUT it forces the Library_Kind to... "dynamic". I could NOT find out any clear explanation as to WHY this is so but if I'm to guess I'd say it's probably a way of "protecting" users so that they don't encapsulate the Ada run-time in 10 separate libraries and then use all of them in the same program or something.
    • Finally, the "no" option means - surprisingly! - what you'd expect: the library is NOT to be a standalone library after all (and "Library_Interface" be damned)!
  • Summarizing the messy interplay between Library_Interface and Library_Standalone above: you can have a static or dynamic standalone library as long as you leave "Library_Standalone" option alone; you can have only a dynamic standalone library if you actually want to include the GNAT run-time. This item is called "encapsulated standalone library" and means that "Library_Standalone" is set to "encapsulated". You can - unclear why/when is it useful - specify explicitly that you do NOT want a standalone library by setting "Library_Standalone" to "no". Essentially the Library_Standlone chooses between "types" of standalone that include standard, encapsulated or... not standalone at all. I still get slightly nauseaous.
  • Assumming you did go for one sort or another of standalone library, there is a further option to ask the library to "automatically" run its initialization. This is done via "Library_Auto_Init" option being set to "true" (the default value). However, this is the sort of gun that can easily explode in your face since the actual behaviour is platform dependent so you can't rely on it for anything. As a result, I'd say this is best set to "false" clearly and explicitly so that one is not lulled into the idea that someone else will do the initialization auto-magically.
  • If you build a static standalone library, note that its linking into the main program requires also the linking of GNAT runtime as a minimum. The exact things you need depend on what your library really uses but things can get quite gnarly. For instance3 the line for a basic main.cpp test that does ~nothing but it does it with the whole smg_comms + some glue for handling net stuff of Eulora's client:

    gcc main.cpp -o main.o lib/libeunet.a
-Wl,-z,origin,-rpath,/home/eu-test/eulora/eunet/c_wrappers/bin/:/home/eu-test/eulora/eunet
/rsa/bin/:/home/eu-test/eulora/eunet/mpi/bin/
-ldl -lpthread -lrt -L/home/eu-test/eulora/eunet/c_wrappers/bin/ -lC_Wrappers
-L/home/eu-test/eulora/eunet/rsa/bin/ -lRSA -L/home/eu-test/eulora/eunet/mpi/bin/
-lMPI
/home/eu-test/x86_64-linux-musl-native/lib/gcc/x86_64-linux-musl/4.9.4/adalib/libgnarl.a
/home/eu-test/x86_64-linux-musl-native/lib/gcc/x86_64-linux-musl/4.9.4/adalib/libgnat.a

On the bright side, the investigation that resulted in the above notes means that I'm now satisfied that I can in fact link an eunet Ada library both with Eulora's client (that links mainly with dynamic libs so possibly easier as encapsulated standalone) and with code that runs on GNAT with static libs only. In addition, I certainly got also a much better understanding of Ada's elaboration and elaboration order and how it should be handled for safe use of tasks from within a library. But a set of notes for that might be the topic for another time!

  1. even a rather complex one with tasks that start at elaboration time among other things ↩

  2. In principle there is also a symmetric finalization procedure that is to be called at the very end by the non-Ada code: this one shuts down the Ada run-time but in practice it doesn't seem to be required all that often. ↩

  3. Isn't "libgnarl.a" such a great name? ↩

January 4, 2019

Tandretea inceputului de an

Filed under: Lyf, Young, old and oldest — Diana Coman @ 11:10 a.m.

Daca inceputul a fost cu argumentari iar urmarea a fost cu mirari cat fiinta si purtari de tot felul, 2019 incepe aproape pisiceste asa, tandru si zambitor si cat pe ce sa nu mai am loc de el in fotoliu:

Victor_2019

Creste el, mai cresc si eu, va doresc sa tot cresteti si voi ca stat pe loc nu se poate niciodata pe lumea asta. La multi ani!

December 24, 2018

A Week in TMSR: 10 - 16 December 2018

Filed under: A Week, TMSR — Diana Coman @ 3:55 p.m.

On Monday, 10th of December 2018:

Mircea Popescu notices that the MP-WP1 installation on Pizarro's shared server seems broken as it fails to correctly process footnotes. In response, Asciilifeform2 asks BingoBoingo3 to look into it, noting also that other accounts on the server (Hanbot's) don't exhibit the same problem and therefore the issue has to be linked to Nicoleci's account on the shared server.

Mircea Popescu expresses his surprise at Nicoleci's apparent inability to express herself in writing anyway nearly as well as she is able to express herself orally. The difference is significant enough to be rather hard to believe if not directly witnessed. Further discussion with Trinque and Asciilifeform of Nicoleci's public writings on her blog - mainly summaries of TMSR logs - and of the sad state of what passes as "writing" in the US nowadays leads to the conclusion that the core issue with her writing is that it lacks any narrative structure: instead of telling a story of any kind, she seems to attempt to just give the gist of her thoughts at one moment or another.

Asciilifeform expresses his pleasant surprise at having recently tried a 3D device. He suggests it for Eulora but Mircea Popescu notes that Eulora is significantly more intellectual than visceral or graphical at the moment and the current struggle in this direction is anyway simply getting even basic art done for the game rather than improving public's access to it.

Danielpbarron publishes on his blog another snippet of his conversations with some dudes on religious matters. Trinque struggles to make any sense of the published snippet and points to Danielpbarron the solipsistic nature of his current activities as they can be perceived based on his publications. Danielpbarron fails to see Trinque's point and enquires whether there is anyway any significant difference between talking publicly of religion as he does and talking publicly of sex as Mircea Popescu does. This enquiry is promptly answered by Mircea Popescu who points out some significant differences: while every human being is interested in sex seeing how sex is fundamental to humans, not every human being is actually interested in religion seeing how religion is fundamentally gossip; moreover, while other types of gossip are at least interesting as they touch on interesting people, religion fails to captivate as it concerns nobodies. Danielpbarron disagrees with this view of religion and affirms that the "truth of the Bible is universally known", offering as unique support to this assertion a few citations from his Bible.

Nicoleci publishes her 101th post on her blog detailing some interactions with people from her past who failed to impress her as much as they told themselves they did even when she was younger while positively making her laugh currently with their unsolicited emails.

Ben Vulpes publishes an accounting statement for Pizarro for November, relying on a semi-automated process (numbers are produced automatically but the final format requires manual work to put everything together).

Diana Coman realises that her previously mentioned problem of an empty genesis .vpatch as a result of Cuntoo's bootstrap script is caused by an issue with the vdiff tool on the machine running the script (so nothing to do with Cuntoo's bootstrap script after all). After fixing the vdiff tool she reports that the Cuntoo script runs successfully and produces a .vpatch but the signature for it fails to verify. Bvt chimes in to report that he has a similar problem on his computer as the .vpatch he obtained from Cuntoo's bootstrap script fails to verify against Trinque's provided signature. Later during the day, Diana Coman publishes the .vpatch she obtained and Trinque is able to compare it with his own noting that there are several differences that he will need to fix, including his use of sha-based vdiff rather than the keccak-based vdiff. Diana Coman also notes that the Cuntoo bootstrap script fails on a different machine configuration (different operating system mainly), stopping with an error. She provides a paste of the error and Trinque is able at a later time to point her to the potential issue - an un-met requirement (having /dev/shm mounted) for compiling Python.

Diana Coman gives a talk on Bitcoin to students at Reading University in the UK. Later during the day she publishes a write-up of it including a detailed account of her Bitcoin talk and the supporting slides that she used.

Diana Coman offers to Asciilifeform the results of a tcpdump running on SMG's test server with Pizarro for several months during the year. The dump provides the content of some unexpected UDP packages that were observed during a previous test of UDP communications in October 2018. The dump includes some VoIP apparent scam that seems to originate from Iceland. As Asciilifeform is interested to investigate more into this, Diana Coman points out to him that it's all on Pizarro-owned infrastructure and so he asks BingoBoingo to reroute to one of his own computers with Pizarro all packets with unassigned IP destination.

On Tuesday, 11th of December 2018:

Commenting on Diana Coman's write-up of her talk at Reading Uni on the previous day, Mircea Popescu notes that the lack of a recording of the talk is rather unfortunate especially given how simple it is to obtain normally. Diana Coman and Mircea Popescu then discuss a bit the practical aspects of recording a talk and the rather shockingly basic conditions offered by Reading University on this occasion. Mircea Popescu notes in conclusion that the write-up of the talk looks good and the missing recording is more a matter of "missing out on a possible fanbase!" than anything else.

BingoBoingo reports that his Peruvian girlfriend finds Argentina very beautiful especially compared to what she knows of Uruguay. This prompts Mircea Popescu's "eh" and Asciilifeform's observation that Argentina hasn't quite managed yet to fully burry/destroy/run down the beautiful buildings it inherited from back when it mattered. The conversation then moves onto the significant differences in quality of buildings in different parts of the world and at different times, with Asciilifeform revealing that he can actually distinguish what he considers well-built structures by their smell that might be - or might not be - due to a combination of aging plaster, actual wood and perhaps old books in significant quantities.

Nicoleci publishes on her blog her summary of TMSR logs of 19th November 2018.

BingoBoingo publishes on Qntra an update on Macron's adventures in France and another update on Ebola's adventures in Congo.

Mircea Popescu draws on his extensive knowledge of world history and his extremely numerous interactions with a wide range of people to discuss his emergent view that multiculturalism fails first and foremost for lack of multiple actual cultures rather than for lack of potential merit in the idea of culture-mixing itself. Asciilifeform points out to the merits of China (at the time of Confucius) as an example of different actual culture that existed but Mircea Popescu notes that merits are irrelevant for the issue at hand: in practice, there is only a very narrow and unique way to culture and so everything that counts as such inevitably finds itself on this same path without much diversity possible. China is given again as an example since its current relevancy in the world is, in Mircea Popescu's view, fully due to and limited by the extent to which it copied white man culture. Addressing Asciilifeform's point, Mircea Popescu also notes that previously to this copying, China was simply a large bureaucratic state in a similar way in which the Inca state had also been one but still failing to actually develop as a culture since working organisation by itself is not enough. To support his point, Mircea Popescu remarks also that an actual alternative culture in China would be directly identifiable simply by its results. Given the obvious lack of such results - as there is no equivalent Chinese #trilema at all, let alone one bigger in size as it should logically be given China's size and more efficient organisation - it means therefore that there can't possibly be a culture there in any sense either. Both Mircea Popescu and Asciilifeform acknowledge that this might still be proven incorrect at a later time although the chances for such proof seem to them rather low. The more likely explanation for the current situation is in Mircea Popescu's opinion the simple fact that China can't seem to be able to advance past its remarkable efficiency at copying - currently copy successes that stop short of developing anything new including for instance mining Bitcoin but also owning the full fab stack and still failing at the same time to produce its own CPU architecture.

Asciilifeform rages at html's failure to provide a reliable way to format even basic equations so that they look the same across different displays and browsers (in particular without using javascript and/or images). Trinque suggests using SVG might be a good approach for the task but Asciilifeform rejects it because it won't be of any use for text-based browsers. Mircea Popescu provides a solution based on the use of html tables and top/bottom floating alignments, publishing it on Trilema as well, for future reference. At first, Asciilifeform balks at the proposed approach as he says it doesn't work with the Lynx text-based browser but Mircea Popescu points out that there is no way that works exactly the same in both text-based and graphical mode.

BingoBoingo announces that Pizarro's price for BTC is set at $4000 per 1 BTC for the month of December 2018. This price is based on an auction of $2000 that concluded on the 7th of December with the sale of the $2000 to Mats for 499.99ECu. Using this exchange rate, BingoBoingo produces Pizarro's invoices for provided services to bvt, jurov and trinque. Further invoices are likely due for Mocky's and Nicoleci's shared hosting with Pizarro and for SMG's test server.

Mircea Popescu states that he considered for some time Diana Coman's innovation/subversion distinction and he finds it to be well founded. He further notes that this distinction makes it clear that there is very little difference between subversion and "inclusion." Diana Coman agrees with this observation and notes that those finding change (hence, innovation by another name) difficult will simply push for subversion instead for as long and in as many ways as they can. Mircea Popescu adds to this the funny fact that Spanish uses the same word for expressing that something is expensive ("cuesta mucho") and that one finds something difficult ("me cuesta"), driving home the inescapable conclusion that indeed, the sort of person who finds it difficult to think (and therefore to change) has indeed no busines in #trilema or with Bitcoin for that matter. Diana Coman further links this "cost" of personal difficulty to the oft-heard complaint of "it's not fair" but Mircea Popescu considers the matter to be a much more intricate ball of nonsense than that. Nevertheless, he notes that a preocuppation with "fairness" (as opposed to correctness) is indeed a good heuristic for lack of useful intellect since it betrays significant inner voids that make it all together doubtful the subject is really a person at all.

A side note by Mircea Popescu on the provenience of the "arena" word in English from the Spanish word for sand turns into a short discussion with Diana Coman on the Arenal volcano in Costa Rica and subsequently with Asciilifeform on the properties of volcanic sand and the importance of semiconductors.

Asciilifeform announces that he will bid on a Symbolics MacIvory model and he will have it xray tomographied if he obtains it.

Mircea Popescu rages at Mozilla Firefox's idea of "releases" of the browser that include executables of all sorts and assorted signatures without any clear apparent meaning. Asciilifeform is rather amused at the idea that there is anything other than ceremonial in latest Mozilla offerings but notes also that he is not aware of any version of Firefox that did not suck to start with. Trinque chimes in to say that he has a version of Firefox that he built on Musl so that there is at least that as a potential de-facto graphical browser for Cuntoo. Mircea Popescu notes that at some point the republic will likely have to write its own sane browser anyway, getting rid in the process of all sorts of useless junk that currently come stuck with any graphical browser.

On Wednesday, 12th of December 2018:

Nicoleci publishes on her blog her summary of TMSR logs of 20 November 2018. She also notes that fetlife has deleted a post of Mircea Popescu from 2 days before since it was apparently more liberal than their liberalism can take.

Phf brings back the discussion on fairness/correctness from the previous day noting that he naturally considered fairness to mean exactly that: a recognition of correctness even when it's not to one's own advantage. In response, Mircea Popescu points out that this meaning of fairness as unpleasant-but-correct has always been a purely eastern one while the western definition always focused on a sort of weighing and comparing of outcomes. He links this to Hajnal's line in the sense that fewer and later marriages give more idle time to be spent on the contemplated sort of "fairness" considerations.

Mircea Popescu redirects Nicoleci away from attempting to summarize TMSR logs and on to transcribing old proceedings of the Royal Society of London that are rather interesting to read but are hardly readable in their existing format since they've been mangled by the automated OCR process.

One of Trilema's readers suggests to use a Wordpress Latex plugin to properly format equations. Mircea Popescu passes on the suggestion but Asciilifeform says he already investigated the plugin and it fails to solve his problem as it still relies on images and therefore it produces output that is not entirely suited for text-only browsers. Mircea Popescu points out that Mathematical notations are simply not fully alphabetic and as such they can't ever be pure text and therefore it's up to terminals to work correctly by being able to handle text + adnotations rather than text only. The discussion further advances on to what sort of text preprocessing should be actually done by a browser with Mircea Popescu noting that this question doesn't yet have a clear answer and Asciilifeform noting that at any rate, existing answers such as tags totally fail to actually answer anything. The mention of tags touches a nerve with Mircea Popescu and he notes that they are a very good example of the fundamentally broken approach that created the significant current technological debt: "simplification" implemented without regards to actual secondary costs incurred and by removing the barriers to entry that kept out precisely the sort of people that had no place to enter in the first place. While Asciilifeform heartily agrees with this view, he considers it old news and summarises it as "mechanization + idiocy == mechanized idiocy." He adds however that this sort of simplification "works" anyway simply by subversion of the very object called computer since actual computers are even more difficult to obtain than they were before while the objects that are now indeed very easily obtained are computers in name only.

BingoBoingo publishes on Qntra four articles: on Britain's no confidence vote in its Prime Minister, on the death of a Physics professor at Stanford University, on the secret conviction of George Pell in Australia and on one of the FBI's terrorism charges.

Mircea Popescu discusses with Asciilifeform the dilemma of Free Software raised by Naggum: while useful code has indeed value as Naggum clearly argues, its valuation cannot be approached in the way that Naggum seems to suggest, namely by attaching some value to the lines of code itself and/or closing code so that its source is not freely available anymore. In Mircea Popescu's view, free access to useful code source does not take away value of the code but instead adds a very useful entry point that works also as a passive but effective filter so that valuable contributors can be discerned from time wasters. Asciilifeform does not disagree with this view but expresses some reserve: he says he did not learn from reading ugly code despite reading loads of it but rather from reading non-code text; he also notes that Naggum seems to have been aware of the fact that lines of code added do not translate into value added but rather the opposite (the best code is no code); he adds also that Naggum's statement regarding the loss of value of software through free publishing are likely the result of his own personal history of trying to make a living by solving complex problems and seeing the tools he needed gradually vanishing as their producers failed to be valued enough to be able to continue their work. Mircea Popescu acknowledges that this is very possibly true and even proposes the neat packing of this pattern into a foundational myth under the title of "avik killed naggum"4 but notes that nevertheless the view that publication destroys value is not only misplaced but dubious in that it actively serves only those authors that attempt to extract more than their work is worth on closer examination. And since the abstract work of computer programming is much more similar to other abstract work such as that performed by doctors, the correct valuation should also follow similar patterns rather than attempting to follow patterns (such as copyright) that are derived from valuation of non-abstract work. As a result, Mircea Popescu notes that on one hand the requirement to publish code does not have to apply without discrimination and on the other hand the only correct way to pay for abstract computer work is through the crown allocation process: authors of abstract work may receive their payment as a recognition by a higher authority (the crown) of their valuable contribution but not as some quantifiable, formula-calculated amount that most users can decide on since most users are utterly unqualified to evaluate this type of abstract work in the first place.

As a continuation of the previous discussion on evaluating abstract work in general and code in particular, Mircea Popescu further stresses the important fact that valuable abstract work is by its very nature and fundamentally a surplus phenomena - meaning that there has to be first some surplus in order for one to be capable of performing abstract work of any value. In practical terms, this means that the authors do it without strictly needing the payment for it and as their own personal choice of doing it in preference to doing other things - some of them with clear payment even - that they are perfectly able to do. Asciilifeform also links this to operating from causes rather than for purposes (i.e. for obtaining some specific payment in this case).

A further continuation of the same discussion explores also to some extent the further difficulty in assigning rewards for valuable abstract work even through the crown allocation process. The process does not make the evaluation of abstract work any easier and it also doesn't provide a clear way to ensure optimal labour allocation at times of need. Essentially, Mircea Popescu notes that existing tools (money as a signal of value and market forces as regulators) although a good fit for concrete work and objects are nevertheless a disastrous fit where abstract work is involved and their failure is so significant that it likely drives intelligent people towards some form of socialism (as the only sort of alternative perceived) in their attempt to find a solution to the problems caused. The conclusion overall is, in Mircea Popescu's own words: "labour allocation is broken and nobody has any better".

BingoBoingo issues Pizarro invoices to Mircea Popescu for Nicoleci's shared hosting and for SMG's test server. He also updates Pizarro's public page to reflect the 10% discount offered on shared hosting for annual subscriptions over monthly subscriptions. Later, following Mocky's request, he also invoices Mocky for an annual shared hosting subscription.

Mocky asks BingoBoingo to bill him for his shared hosting with Pizarro on an annual bassis. He reports that his search for a job is still ongoing although slowed to some extent by holidays of his interviewers. Mircea Popescu suggests perhaps pooling resources through running a TMSR version of bourbaki: specialist appliers to remote jobs dumping tasks in a file that gets passed around for TMSR people to choose from as and when they want to do some non-TMSR work. Mocky chimes in to say that he previously considered outsourcing some of his own work, while Trinque notes that Oracle for instance is known to actually do precisely this. Asciilifeform says he'd be delighted to work in this way but expresses his doubts at the scheme, mainly due to the difficulty he perceives with task level/definition/discussion and the potentially problematic case of tasks that nobody wants to pick up within the allocated timeframe. Mocky says that his concern with this model is the fact that it can take him a year to become capable of actually solving specific problems within a reasonable time.

Asciilifeform and Mircea Popescu discuss the actual relationship between employer and employees with specific focus on Asciilifeform's apparent inability of escaping employee status. Mircea Popescu notes that the core issue seems to be the mismatch between the favourite "select first and then talk to selected" approach of most republicans and the opposite "filter the ocean" approach5 that is actually required for any search outside of TMSR (a search for employer included). Relatedly, he asks whether Pizarro has managed to do anything of the sort in order to find the clients it needs for survival. At a later time, BingoBoingo replies, revealing that the short answer is no, Pizarro has not yet managed to do anything of the sort but it might perhaps still manage to do it if only an "awk ninja" materializes to write the needed scripts.

On Thursday, 13th of December 2018:

Nicoleci publishes on her blog her summary of TMSR logs for 21 November 2018.

Diana Coman negrates douchebag for obstinately wasting her time. Probably in retaliation, douchebag carves in his own nick on freenode the message that nobody is interested in listening to, doing the irc equivalent of sandwich man. This prompts some laughter and merriment all around and Phf notes that douchebag's vulnerability finding is all about form rather than substance. Diana Coman says that the proposed view fits indeed the observed behaviour and moreover makes the whole activity very similar to a form of political correctness applied to code. Mircea Popescu takes this further and says that in this case the whole thing is also the precise equivalent of period politruks that attempt to police the code as the currently relevant form of speech.

Diana Coman publishes Chapter 12 of SMG Comms containing a thread-safe Ada implementation of simple queues that are specific to the needs of Eulora (as opposed to the generic thread-safe Ada queues of the standard).

Diana Coman informs Trinque that she experienced some problems obtaining an answer from deedbot to the !!ledger command. Trinque notes that the command currently works for him and he suspects the issue was most likely due to a lost connection between the irc bot that receives the command and the back service that actually handles all wallet functionality.

Diana Coman rates juliankunkel, the lecturer at Reading University that invited her to give a Bitcoin talk to students. Asciilifeform and BingoBoingo welcome him but he doesn't have much to say.

Asciilifeform reports he acquired the bolix on which he previously bid and he says he will therefore xray it. A bit later, Mircea Popescu contributes to this with a warning for Asciilifeform to check the rated power of the equipment he intends to use since not all equipment is powerful enough for such a task. Asciilifeform however reveals that the task is not likely to require high power anyway as there is no middle metal layer.

BingoBoingo publishes on Qntra: on the use of facial recognition at a pop concert.

Mircea Popescu, Asciilifeform and Diana Coman discuss the best approach to take for implementing (or not!) a sender/receiver layer as part of SMG Comms. The conclusion is that there will be such a layer as part of SMG Comms but a very thin one that simply moves UDP messages from/to outbound/inbound queue and the UDP socket. The reason for this layer to exist is the need to move UDP messages quickly from the relatively small queue on the IP stack to the larger in-memory queue. The reason for it to be part of SMG Comms is that it's not specific in any way to any given application since it's so thin as to focus exclusively on moving messages from/to socket and queues.

On Friday, 14th of December 2018:

Asciilifeform provides a paste of his talk to adlai in #asciilifeform as proof that quitting drinking has at least *some* effects. Mircea Popescu suggests a recuperative scholarly series on the SNS server and later notes that this is simply for documentation value rather than some silver bullet (or indeed any sort of working bullet at all). Asciilifeform indulges his love of old/interesting hardware mentioning items rarer than the bolix: xerox lispm and tandem. Phf reveals he has worked on a tandem (known as HP NonStop) at some point and he appreciated the architecture but noted that the software was entirely written in Cobol. This is interesting to Asciilifeform but it makes him poke Phf about some promised Bolix documentation that he previously said he might have. Following on from this, Asciilifeform reveals that he will likely perform the xray of his Bolix machine with his own hands but that he'd like to have Phf's papers (if there are any) to check against. The next day, Mircea Popescu adds to this discussion noting that there is proper xray-hygiene to follow when performing such a task.

Danielpbarron publishes on his blog another talk with unknown people on religious matters. A reference to reddit in there prompts Trinque to enquire if Danielpbarron is meanwhile militantly anti-republican. Danielpbarron flatly answers "no".

Nicoleci publishes on her blog her summary of TMSR logs of 22 November 2018.

BingoBoingo publishes on Qntra: on Germany's three choices of sex on paper.

On Saturday, 15th of December 2018:

Danielpbarron publishes on his blog another talk with random people on religious matters. Asciilifeform gets from it the impression that Danielbarron's approach is essentially calvinistic but Danielpbarron rejects this assessment on the grounds that "calvinism leads to hell". Some further talk reveals that Asciilifeform hasn't followed the religious life of Danielpbarron all that closely.

Asciilifeform further discusses with Phf his current plan for xray-ing his Bolix machine and then using the gained knowledge to build probes for further knowledge gain.

Asciilifeform notes that Ben Vulpes' logging bot is not working and Mircea Popescu notes that anyone can start another logging bot and simply aim it at the chans of interest as the bot code is published already.

BingoBoingo publishes on Qntra: on the latest adventures of Macron in France.

Mircea Popescu provides another sample from the responses he gets to one of his ocean-filtering actions. Asciilifeform is curious on the percentage of responses that manage to at least read the full initial message that usually gets cut off on various mobile phones and the like. The response is that there are some that pass this basic test but the percentage is very small.

Nicoleci publishes on her blog her summary of TMSR logs of 23 November 2018.

On Sunday, 16th of December 2018:

Asciilifeform announces he received his Bolix in perfect packing, with all accessories and able to run. He notes that he was in the end the only one to bid on this machine but he still did not want to miss the opportunity to buy it since the price apparently increases by $1000 every year. Phf suspects that there aren't that many bidders anyway and it's all more of a show with other owners of older hardware simply hoarding it as they notice they can't replace it anymore at any cost. The emerging picture seems to be that 2009 is the cutoff point for hardware that one can trust. A bit later, Phf provides Asciilifeform with DKS patches for Bolix and a port of 'zork' to Bolix.

Mircea Popescu laughs heartily at the Unity Web Player being now reportedly no longer working on Chrome, Firefox and Edge browsers. His faint interest in the matter focuses on the fact that it's totally unclear why and how did Unity exactly achieve its previous popularity. His hypothesis on this is that Unity got "chosen" simply for lack of any alternative. Asciilifeform offers as similar puzzle the success of qt but Mircea Popescu notes that they are in fact not comparable since Unity never actually worked nor did it ever have serious resources to speak of while qt both works and is not in fact going anywhere. Asciilifeform then links this to Bolix noting that the 3d engine for it still exists and is called Mirai. Its forum however is not working as it was overrun by spam. Amberglint joins in the discussion to correct Asciilifeform's assertion that Mirai was ported to CPP - he says it was in fact ported to Allegro Common Lisp. He also mentions that the most well-known work done in Mirai is the Gollum character for the Lord of the Rings film.

Asciilifeform publishes on his blog Chapter 14A of the FFA series covering the first half of Barretts Modular Reduction.

Mircea Popescu publishes on his blog the result of his wtf is a "Post Malone".

Diana Coman publishes on her blog a summary of TMSR logs from 3 to 9 December 2018. Mircea Popescu provides feedback and some corrections to it. The summary also prompts Mircea Popescu to add to one of the main topics previously touched namely the existing conflict between different versions of database management systems (mysql and postgres) being needed for different sort of tasks. This spills into the next day and the next week.

  1. A customized version of the Wordpress blogging platform produced by Mircea Popescu, packaged in V format by Hanbot and currently used by most republican blogs and by the Pizarro ISP for its clients that share space on a server. ↩

  2. Main tech for Pizzaro ISP. ↩

  3. Founder of Pizarro ISP and the only current local pair of hands. ↩

  4. Avik is the self-styled "master" that reportedly feeds his slaves a cocktail of pills, works at precisely the sort of software that crowded out of the market the tools needed by Naggum for his work and otherwise keeps pestering Nicoleci with unsolicited emails. ↩

  5. This is a term of art meaning literally talking to EVERYONE and then filtering out 1 / 1mn or in similar ratios those contacts that are in fact of any interest. ↩

December 19, 2018

A Week in TMSR: 26 November - 2 December 2018

Filed under: A Week, TMSR — Diana Coman @ 4:45 p.m.

On Monday:

Asciilifeform publishes on his blog Chapter 13 of the FFA series.

Trinque announces that he has a working Cuntoo bootstrapper that runs entirely offline and reliably produces the same genesis .vpatch that can then be verified against his signature. His write-up on the topic is due for next day. As part of his work on this, Trinque wonders whether vdiff could or should perhaps be able to produce a genesis .vpatch without requiring an empty directory. Asciilifeform points out in response that the current approach is both standard and without fault to his eyes. Moreover, later during the day, Mircea Popescu also notes that the empty directory is perfectly fine from a philosophical point of view since it represents the perfect code. He also states that an alternative, specific solution (such as diff against /dev/null) is perfectly valid as well for as long as it doesn't turn into a different mode of operation: essentially as long as vdiff simply diffs whether for genesis or not, it's fine; as soon as one wants to define some diff as diff and then something-else as genesis, insanity creeps in. This turns quickly quite vicious as Asciilifeform points out that such insanity is usually the "being smart" of code and as such the bane of any programmer who wants to do something as opposed to just write more code; in turn, Mircea Popescu strikes back with the even pointier point that the "being smart" of people is the even greater bane of any man who wants to do something as opposed to just breathe in and out for yet another day.

As further result of his work on the Cuntoo script, Trinque notes that he doesn't find sane the default behaviour of vdiff to exit with what is normally an error code (i.e. returning something other than 0) just because the given parameters are not as expected. Nevertheless, he makes do with it for now and uses it as it is in his Cuntoo script. Mircea Popescu chimes in to note that Trinque's point on this is valid - there shouldn't be a need to keep adding the check for this case. Phf takes note of the issue raised and says that he is adding it to his backlog of issues to fix and curiosities to look at (with naked eye, powerful microscope or a bigger hammer, as required).

Nicoleci publishes her summaries of #trilema logs of 9 and 10 November 2018. Asciilifeform finds at least one sentence in the latter hilarious.

Spyked comes back from a longish absence due to unexpected health issues of his father. He rates and introduces his new bot, feedbot, points republicans to its help page, promises to publish its V tree as well in the following days and tells Trinque that he can disable the rss part of his deedbot since feedbot is taking over that job. Mircea Popescu would rather have a smoother approach to this take over and points out to Spyked that a planned and gradual take over is likely to be less bumpy (for everyone involved) than the original para-dropping of feedbot into TMSR territory.

Asciilifeform enquires whether there is some automated or semi-automated way to submit new .vpatch and/or .sig files for inclusion into the main repository at http://btcbase.org/patches. His enquiry fails to get an answer and so far he doesn't seem bothered enough to press the issue further.

The current bot count in channel further increases by one as Asciilifeform resurrects his own FFA-bot - his name is PehBot. Asciilifeform gives PehBot a spin, illustrating the newly added capabilities of the bot that matches now the FFA content up to and including Chapter 13. Initially, Asciilifeform plans to move PehBot to #asciilifeform but Mircea Popescu points out that bot or not, one can stay in #trilema for as long as one doesn't become a nuisance.

BingoBoingo publishes on Qntra an article on the seizing of some Ukrainian ships in the Kerch strait. Both deedbot and feedbot jump on the new feed and announce it, prompting Mircea Popescu to poke spyked who hasn't yet noticed the call for a smoother take over. Smoothness might need to be forced upon with a bigger gun.

Asciilifeform fixes the commenting form on his blog after more than a year since they last worked and more like a few days since the latest round of pointed complaints about it from people who wanted to provide feedback on his FFA series. As part of the fix, he replaces one set of tripping wires for commenters with a single tripping thread hoping that the decreased complexity means that at least *some* commenters survive it now.

As part of his www winter-cleaning, Asciilifeform also installs the selection script that enables one to link to specific parts of text in a post. He goes on to have as a result an oft-repeated argument with Mircea Popescu regarding the proper way to keep - since you have to! - and especially keep in check your www stable that includes dirty beasts such as php and wordpress.

The ups and downs of reported fiat-value of Bitcoin continue to entertain as the "valuation" goes down to 3000 green papers out of infinity per one coin out of 21 million total. Asciilifeform saves his laughter for when the valuation goes to 3 green papers for the same coin but as part of the discussion, Mircea Popescu digs out and links several older posts on trilema.com that are a good read at any time.

From valuation of Bitcoin, the discussion moves seamlessly to valuation of people. Mircea Popescu notes that the bar to being a "wise man" in TMSR keeps increasing but there is hope that this increase is actually capped since the steep increases recently witnessed are likely due to catching up with neglected work rather than anything else. Illustrations range from dirty socks to recent (douchebag) and a bit less recent (kakobrekla) failures in #trilema and from Arthur Blair to C.S. Lewis or cardinal Newman in the world at large. As a side point, there is also a definition of "fractional girlfriend" and the observation that not everybody seemingly asking a question is actually looking for its or indeed for any actual answer.

On Tuesday:

Mircea Popescu provides a concrete example of the need to filter an ocean to find a crumb of usefulness. Empirical results seem to suggest that it's about 1 person in 1000 that seem to interact at all with what they read.

Mircea Popescu asks for feedback on his recently published (last week) first draft towards defining a republican replacement for DNS aka the GNS. Trinque says he hasn't yet got around to read the published piece but he promises he will read, digest and come back with a response.

As promised on the previous day, Trinque publishes on his blog his new script for bootstrapping Cuntoo. He asks people to let him know if they try it and with what results. Asciilifeform quickly looks and promises to try it at a later time but for the time being he questions the way in which the script steps through 2 GCC versions to get to the desired 4.9.4 version and the fact that it lacks ave1's gcc. In response, Trinque says that having ave1's GCC is the plan in the longer term but the point of the current item (the script) is to provide a stable starting point made of what-is so including mainstream 4.9.4 gcc. Mircea Popescu chimes in at a later point to say that nevertheless the ebuild with ave1's gcc should be made and preferably quite quickly. He also sketches out the roadmap for the longer term, including the full removal of SSL (all flavours and from all places) that is to be replaced with straight RSA. This sort of replacement is meant indeed for all republican items including for instance TRB although Mircea Popescu notes that the replacement might be FFA or a different republican RSA implementation depending on the practical requirements and constraints of each application.

Joining in the conversation sparked by Trinque's Cuntoo script, phf reveals that he has various POC1 bits and pieces that explore some potential ways of installing packages on a system in a V-reliable way. Trinque would like to see those and Phf promises to pull them out from their respective hiding places but only after he finishes his current move from the US to Russia. At a later point, Diana Coman reads the script and asks for clarification from Trinque on whether the published script can also be used to upgrade an existing gentoo installation to Cuntoo. Trinque replies that such a feat is in principle possible but it's currently undocumented and as such unexplored territory to be yet tried at explorer's own risk.

Nicoleci publishes her summaries of #trilema logs of 11 and 12 November 2018.

Spyked notices the request for smoother transition to his feedbot and acts accordingly to synchronize with Trinque.

Phf "snarfs" the latest FFA vpatches from Asciilifeform. The "snarf" is a term of art and it means that the new .vpatches and their signatures are now mirrored in the main TMSR repository at http://btcbase.org/patches. Asciilifeform profers his fondness of Phf's repository infrastructure that is "unspeakably helpful".

Asciilifeform, Diana Coman, Mircea Popescu and Phf discuss briefly the way in which Ada transitioned from initial ugly and gnarly find to republican standard language for programming. As part of the discussion, Asciilifeform links to Mocky's useful summary of the arguments for using Ada.

Davout was last seen in TMSR in April 2018, more than 5 months ago. Meaningful work from Davout was last seen several years ago. Mircea Popescu gives a quick summary of Davout's known involvement so far: being a tech in the early days and doing the receivership for BitBet at a later date.

A "grubles" from 2014 joins #trilema in 2017 and doesn't last long. By contrast, the negative rating he acquired in 2014 continues to last.

Asciilifeform corrects his own oversight and negrates "Hasimir".

BingoBoingo publishes on Qntra a short announcement of the release of Cuntoo bootstrapper and a brief note on the adventures of some people who run any code they happen to find.

On Wednesday:

Danielpbarron's blog sends delayed pingbacks from 2016 as he is finally uploading old articles on the new hosting with Pizarro. Following the delayed pingbacks, Mircea Popescu reads the content in which they are embedded and expresses puzzlement over the mismatch between the interests and worldview they reflect and Danielpbarron's desire to continue being a lord of TMSR. When asked directly, Danielpbarron states that Bitcoin is still an interest of his and it's simply a matter of enjoying his position and being materially invested in Bitcoin via running a node and having items and ECu in Eulora. He also says he may be lazy but not "morally opposed" to doing meaningful work in TMSR.

Amberglint pops by to offer Asciilifeform a pointer to someone who wants to decap an Ivory processor board. Asciilifeform points out that the operation can't be trusted to anybody walking in through the door as the Ivory is a very scarce resource so the wannabe de-capper is cordially invited to get in the WoT and convince people that he can be trusted with such a task. Phf also chimes in to ask Asciilifeform to postpone any attempt at decapping an Ivory until he gets the chance to provide the docs that he has somewhere burried among other stuff and might shed some light without the dangers of decapping anyone. Asciilifeform publishes a high resolution photo of the Mac Ivory Model 2 processor board together with a shouty "Do Not Touch!" warning neatly guarding it.

BingoBoingo publishes on Qntra 2 articles, on the jailing of a 64 year old woman over holding cotton candy and on sanctions issued against Bitcoin addresses, respectively. He also publishes on his own blog the Peso exchange rate of the day and his practical cooking lesson involving birds and stuffings.

Nicoleci publishes her summary of #trilema logs of 13 November 2018.

Asciilifeform wants to store his valuable Ivories with Mircea Popescu or even just pass them on to him but the latter is neither interested in adding to his pile of rarities nor offering vault space.

Mircea Popescu briefly toys with the idea of sending some bitcoin to the 2 addresses that were "sanctioned" but he notes that they haven't really been used anyway. In the process, he reveals that he can't quite tell apart the orlov of today from the orwell of yesteryear - they are all an orlol to him. Nevertheless, with Asciilifeform's help, he unearths the desired reference that turns out to be from the more recent writer - the topic being that a strongly held "no" is deeply disturbing to those who never encountered such a thing before.

Mocky briefly stops by to let BingoBoingo know about the fate of most recent wires that he sent. He also notes that most employers in his area now require "secret government clearance" for would-be employees.

On Thursday:

Mocky wonders at the latest job descriptions that include "blockchain engineer". He also states that searching for employment sucks but his previous solution for it - to stay with the same company for 14 years - also ended up sucking. Mocky then points out that he got so far precisely what he wanted, namely a lot of kids and a lot of code but he feels like an idjit for not having saved anything during all this time. Mircea Popescu validates Mocky's feelings on this matter.

Nicoleci publishes her summary of #trilema logs of 14 November 2018.

Spyked coordinates with Trinque to take over feeds in the switch from deedbot to feedbot. He also announces he bought himself a c101pa thinking it is a good place to test Trinque's Cuntoo bootstrapper on. Asciilifeform quickly points out to him that no, the c101pa is no such thing.

Mircea Popescu reveals he is burnt to peeling as a result of slut wrestling in the Costa Rican sun. Based on this information, Asciilifeform estimates that he'd last about 10 minutes under similar circumstances and reveals that he can also become crisplifeform under the sun on the 39th parallel.

Asciilifeform compares the LOC2 count for his FFA series that is in the low thousands with that of TRB which is in the tens of thousands not including additional unknown ball of dependencies pulled in. Diana Coman points out that the quality of any lines of code also matters - so it gets even better than the naked numbers show since she'd gladly read Asciilifeform's 1000 LOC at any time in preference of reading even 100 lines of Koch's.

Diana Coman states that she read and will sign Chapter 3 of the FFA series. She has however a question for Asciilifeform on it, regarding the exact meaning of overflow for a shift operation. As she points Asciilifeform to the exact code in question, he is able to confirm that she is right in observing that the code can produce garbage if called with arbitrary arguments but the procedure in question is strictly for internal use of the lib and as such strictly called with correct arguments that don't result in garbage.

Asciilifeform points out node 69.197.146.42 that is stuck behind the top of the chain and he suggests that the aggression patch of trb should be deployed to help it catch up. Lobbes acknowledges that the node is his but notes that it is already running with the aggresion patch so that further investigation is needed to find out the reason for its sad situation.

On Friday:

Mircea Popescu notes the remarkable similarities between apparently different things such as the fate of competent versus incompetent engineers in the current environment or the ability to remain synchronised with the network of Bitcoin nodes hosted with a reliable service as opposed to those hosted with a less reliable one. The similarity comes from the overwhelmingly socialist streak of the environment that is essentially described as "hindering the worthy to prop up the unworthy."

Nicoleci publishes her summaries of #trilema logs of 15 and 16 November 2018.

A certain "zx2c4" revisits #trilema providing Asciilifeform with ample opportunities for restating various basic points including the fact that the technical can never be separated from the political, the fact that a "proof" that requires faith (be it in unread, supporting code) is at most a proof of the proponent's idiocy and otherwise no proof at all and the competent opinion that "Rust" is a "leprous pile of shit" no matter how one looks at it. Upon coming online to the whole zx2c4 display, Mircea Popescu swiftly negrates zx2c4. The action prompts Asciilifeform to cite from "Левый марш": ваше слово, товарищ маузер! (tm). Upon coming online to this last line in Russian, Ave1 embarks upon making sense of it and as a result swiftly publishes his attempt at translating the "Левый марш" to English. Asciilifeform contributes in the comments to Ave1's post with his own quick translation of the whole thing.

Diana Coman publishes Chapter 10 of her SMG Comms series including an implementation of Action and RSA keys types of messages for Eulora's needs. She also signs chapter 3 of Asciilifeform's FFA series.

BingoBoingo publishes on Qntra 2 brief notes on some US new fines and Argentina's new deal to buy soy from the US. He also updates his previous Qntra post on the 2 Bitcoin addresses that were sanctioned: the update includes the clear evidence that the sanctions are worthless since there are newly confirmed transactions to those addresses.

The potential decapper of Ivories turns up under the nickname of SeanRiddle. Mircea Popescu confusingly thinks he's the author of pbfcomics.com and rates him accordingly. Further discussion regarding SeanRiddle's procedure for decapping reveals that he does decapping only as a weekend hobby, uses rust remover to remove top layers, leaves stuff with multiple layers for someone with better equipment, doesn't do any comic stuff and works as a programmer for most of the time. He also provides on his blog only low resolution photos because of limitations of his initial blog setup that involved blogspot and wikimedia. Mircea Popescu points out that he is better served with an actual blog of his own where he can also simply upload files as big as they might be to have the high resolution that is needed for the task. Asciilifeform notes that the number of layers to remove from the Ivory is not known and Mircea Popescu re-rates SeanRiddle with the more apt reference to his existing decapping hobby as opposed to his inexisting comic work. Upon further discussion, Asciilifeform passes on the offer of using SeanRiddle as a decapper for the precious and rare Ivory and SeanRiddle himself agrees that he'd rather leave this job for someone else. Amberglint joins in and provides a few .pdf files of potential interest, prompting Mircea Popescu to suggest he starts his own blog already, possibly hosted with Pizarro.

Mircea Popescu complains to Mocky about an issue with Mocky's bot in Eulora but Mocky says he won't be able to look at it until he is done finding his new "daily bread overlord". Mircea Popescu acknowledges the answer and points out to anyone able to read that there is this opportunity waiting for them to contribute by fixing the bot issue.

On Saturday:

Danielpbarron publishes on his blog his conversation on religious issues - it turns out he took out parts of it anyway so it's only some of the conversation, with some dudes in some chat room called #LRH.

BingoBoingo publishes on Qntra the November 2018 Report on Qntra activity noting 3117 words published during the month, all of them by BingoBoingo himself. He also publishes on Qntra an article on an FBI raid.

BingoBoingo and Asciilifeform discuss the unreal state of the real estate in Uruguay and Argentina noting mostly the inflated prices of flats and the similarly inflated expecations of owners.

Amberglint pops by to point to Asciilifeform the Soviet Refal machine and other hints of Soviet Lisp machines but Asciilifeform is already familiar with those and points to Amberglint the place they all went to: /dev/null.

Danielpbarron will consider visiting Uruguay but doesn't actually plan a visit. He also considers running a poker bot but his consideration is stopped dead by the realisation that he can't legally operate a gambling anything from the country he is in. Mircea Popescu points out that this aspect is a problem of the country itself to the point that one cannot legally operate almost anything there but Danielpbarron doesn't consider that to be the worst of things anyway.

Mircea Popescu illustrates the issue of naive extension of notions to domains or situations where they don't apply. As concrete example, he notes that amortization does not make sense as a concept to be considered by a country when deciding what to do with its current generation since each generation is a set of resources that will be spent anyway and always without any possibility of saving any of it.

Mircea Popescu and Asciilifeform discuss the nature of what distinguishes individuals out of a mass. Initially Asciilifeform seems to consider it is a matter of having more or less of some characteristic such as courage but Mircea Popescu points out that the only practical way to distinguish is the answer or lack of answer to some specific situation - essentially whether one gets the "calling" or not.

Asciilifeform and Mircea Popescu disagree in their view and interpretation of Lavrentii Beria. The discussion spills onto the next day.

On Sunday:

Asciilifeform and Mircea Popescu continue at length their discussion of Beria, Stalin and the whole entourage. Asciilifeform provides a curated fragment by Bukovsky in support of his own point. Mircea Popescu attempts to read it but quickly runs into abundant examples of stupidity and therefore stops before getting to full blown rage.

Ben Vulpes asks BingoBoingo to send him over the Pizarro transactions so that he can produce the full statements and move on to making and filling with data a customer equity tracker for Pizarro. BingoBoingo provides the required data.

BingoBoingo publishes on Qntra a short post on the protests in France. He also publishes on Pizarro's blog a summary of his activities for the business.

  1. Pics and pocs might be anything but here I read it as proof of concept! ↩

  2. Lines of Code ↩

December 17, 2018

SMG Comms Chapter 13: Sender and Receiver

Filed under: Coding, SMG_Comms — Diana Coman @ 1:39 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

This chapter adds to SMG Comms a thin wrapper package that effectively rescues the queue of UDP messages from the IP stack (where it's relatively small) into memory (where it can be, by comparison, large). Once the decision has been clearly made as to what the sender/receiver should do and moreover I finally seem to have gotten my head around using Ada's threads1, the implementation is deliciously straightforward. Who could have predicted this ?! Let's see directly the new Snd_Rcv package as it's very easy to read indeed:

 --Sender and Receiver task types for Eulora's Communication Protocol
 --This is a THIN layer on top of UDP lib, mainly to move messages out
 -- of the small queue of the IP stack onto a bigger, in-memory queue.
 --There is NO processing of messages here: just read/write from/to UDP.
 --S.MG, 2018

with Interfaces;
with Msg_Queue;
with UDP;

generic
  -- exact length of payload aka whether RSA or Serpent
  Len: in Positive;

package Snd_Rcv is
  -- queue package with specified payload length
  package M_Q is new Msg_Queue( Payload_Len => Len);

  -- outbound and inbound messages queues
  -- those are meant to be accessed from outside the package too!
  out_q : M_Q.Queue;
  in_q  : M_Q.Queue;

  -- sender type of task: takes msgs out of out_q and sends them via UDP
  task type Sender( Port: Interfaces.Unsigned_16);

  -- receiver type of tasks: reads incoming msgs from UDP and puts them in in_q
  task type Receiver( Port: Interfaces.Unsigned_16);

private
  -- udp lib package with specified payload length
  package M_UDP is new UDP( Payload_Size => Len);

end Snd_Rcv;

As it can be seen above, the package simply packs in one place an outbound message queue (out_q), an inbound message queue (in_q) and the definitions of two types of tasks: the Sender and the Receiver. The two queues act effectively as mailboxes: all and any tasks from anywhere else are invited to just drop their outbound packages into out_q and /or get incoming packages from in_q. Note that both those queues are thread-safe so there is no concern here over who tries to read/write and when - at most, a task may end up blocked waiting on an empty queue (when trying to read a message) or on a full queue (when trying to write a message).

If the two out_q and in_q are mailboxes, then the two types of tasks, Sender and Receiver, are postmen. They share the same underlying UDP package that is private here (only postmen are allowed to use the UDP post van!) and has a fixed size of messages. Note that this fixed size is given as a parameter to the Snd_Rcv package itself and is then used both for the queues and for the UDP package. Essentially the snd_rcv package is a postal service that handles just one pre-defined length of messages. An application may use of course as many different lengths of message it requires - all it needs to do is to create a different snd_rcv package (i.e. "postal service") for each of those lengths. Note also that the actual ports used by the Sender and Receiver are given as parameters - an application can create as many Sender/Receiver tasks as it wants and even bind them to different ports or to the same port, as desired. This gives maximum flexibility: an application can listen for messages on one port and send them out on a different port, while still having everything in one single queue; or it can listen and send through the same port via any number of Sender/Receiver tasks. Each Sender and Receiver task will simply bind its own local socket and then enter an endless loop in which the Sender picks up messages from the out_q and sends them through its socket via UDP lib, while the Receiver picks up messages from its socket via the UDP lib and writes them into in_q. The corresponding code is short (and it's made slightly longer by my choice of having each Sender/Receiver use its own local socket):

 -- S.MG, 2018

package body snd_rcv is
  -- sender
  task body Sender is
    E       : M_UDP.Endpoint;
    S       : M_UDP.Socket;
    Payload : M_Q.Payload_Type;
    Dest    : M_UDP.Endpoint;
  begin
    -- open the socket on local interface, specified port
    E.Address := M_UDP.INADDR_ANY;
    E.Port := Port;
    M_UDP.Open_Socket( S, E );

    -- infinite loop reading from out queue and sending via udp
    -- caller will have to call abort to stop this!
    loop
      out_q.Get( Payload, Dest.Address, Dest.Port);
      M_UDP.Transmit( S, Dest, Payload);
    end loop;
  end Sender;

  -- receiver
  task body Receiver is
    E      : M_UDP.Endpoint;
    Source : M_UDP.Endpoint;
    S      : M_UDP.Socket;
    Payload: M_Q.Payload_Type;
    Valid  : Boolean;
  begin
    -- open the socket on local interface, specified port
    E.Address := M_UDP.INADDR_ANY;
    E.Port := Port;
    M_UDP.Open_Socket( S, E );

    -- infinite loop reading from out udp and writing to inbound queue
    -- caller will have to call abort to stop this!
    loop
      M_UDP.Receive( S, Source, Payload, Valid);
      -- store ONLY if valid, otherwise discard
      if Valid then
        in_q.Put( Payload, Source.Address, Source.Port);
      end if;
    end loop;

  end Receiver;

end snd_rcv;

An alternative approach to the above (and one that I have implemented at first) was to have a single task Snd_Rcv that bound one single socket and then started on it its own sub-tasks for the actual sender and receiver. However, I find such an approach needlessly complicated and inflexible: it creates an additional layer in the hierarchy of tasks for no clear benefit (perhaps it would make sense if one added some sort of additional management of the sender/receiver tasks in there but at the moment it's unclear that any such thing is actually needed or needed here of all places); it is harder to read with the single and so far unconvincing benefit of a shared socket (so no repeated binding code); it forces some choices on any application using this package: the sender/receiver are forced as a package so there is no more option of just listening on a port and/or just sending on it; there is also no option of listening on one port and sending on another or indeed of creating - if needed - more senders than receivers or the other way around. Sure, it can be argued that several senders and receivers are anyway not likely to be required or that binding too many is likely to just increase packet loss or any other trouble. This is however up to higher levels of the application rather than the concern of this thin sender/receiver and since this implementation offers both highest flexibility AND highest clarity, I think it's the best option so far. As usual, feel free to let me know in the comments your reasons for disagreeing with this and your better solution for implementing a sender/receiver layer.

The above tiny amount of code would be all for this chapter if it weren't for 3 things: the need to relax yet another few restrictions; an example/test of using the above sender/receiver package; my decision to include the UDP lib as just another package of SMG comms rather than keeping it as a separate lib. This last part concerning the UDP lib accounts for most lines in the .vpatch and is essentially some noise at this level (since vdiff is not yet bright enough to figure out a move of files). The reason for it is mainly the fact that the UDP code is really meant to be used from this snd_rcv package and from nowhere else so I can't quite see the justification in keeping it entirely separate, with a .gpr file and everything else of its own and moreover - perhaps more importantly from a practical point of view - unable to directly use the basic types of smg_comms in raw_types. Note that this move does *not* make it in any significant way more difficult to replace this UDP code with another at a later time if that becomes available - it's still one package and those pesky C files, nothing else.

Going back to the need to relax a few restrictions - those are mainly restrictions related to the use of tasks. As both Sender and Receiver work in infinite loops2, the caller has to ruthlessly abort them when it needs them to finish (in Ada a task has to wait for all its sub-tasks to finish before it can finish itself). So the "No_Abort_Statements" restriction needs to go. The use of Abort is illustrated in the test/example code I wrote aka test_client and test_server. Similarly, because of the queues that use protected objects, the "No_Local_Protected_Objects" restriction had to go too. Here I must say that I am not sure I fully grasp why would it be better to have protected objects only as global rather than local? They are of course meant to be accessed from many places and therefore in "global" but this doesn't mean that they don't still belong somewhere and/ or that "access from several places" has to mean "access from ALL places". Finally, the restriction "No_Nested_Finalization" also had to go to allow the testing code to create the snd_rcv packages with different length of messages.

The testing code itself provides more of an example of using the snd_rcv package rather than a test as such since UDP communications are unreliable and therefore one can't really say in advance what one should get on the other side of the connection. At any rate, the test_server package provides an example of a basic "echo server" end of the connection: there are 2 Sender and 2 Receiver tasks working with Serpent-length and RSA-length packages on 2 different ports, respectively; there is also a "consumer" task for each type of package, simply taking it out of the inbound queue, printing it at the console and then echoing it back to the source aka writing it into the outbound queue for the Sender to send. The example awaits for a pre-defined total number of packages so it may remain waiting if the other end sends fewer packages or fewer packages make it all the way. At any rate, once all the expected messages are received, the whole application (aka the main task) simply aborts all the tasks it created and then finishes itself:

 -- S.MG, 2018
with Ada.Text_IO; use Ada.Text_IO;
with Interfaces;
with Snd_Rcv;
with Raw_Types;

procedure Test_Server is
  PortRSA: Interfaces.Unsigned_16 := 44340;
  PortS  : Interfaces.Unsigned_16 := 44341;
  N_S    : Interfaces.Unsigned_8 := 105;
  N_RSA  : Interfaces.Unsigned_8 := 82;
  package Snd_Rcv_RSA is new Snd_Rcv(Raw_Types.RSA_Pkt'Length);
  package Snd_Rcv_S is new Snd_Rcv(Raw_Types.Serpent_Pkt'Length);

  -- sender/receiver tasks --
  -- sender RSA and Serpent
  Sender_RSA: Snd_Rcv_RSA.Sender( PortRSA );
  Sender_S  : Snd_Rcv_S.Sender( PortS );
  -- receiver RSA and Serpent
  Receiver_RSA: Snd_Rcv_RSA.Receiver( PortRSA );
  Receiver_S: Snd_Rcv_S.Receiver( PortS );

  -- Serpent Consumer
  task s_cons is
    Entry Finish;
  end s_cons;
  task body s_cons is
    Payload: Raw_Types.Serpent_Pkt;
    A: Interfaces.Unsigned_32;
    P: Interfaces.Unsigned_16;
  begin
    for I in 1..N_S loop
      -- consume one message and echo it back
      Snd_Rcv_S.in_q.Get(Payload, A, P);
      Put_Line("S msg " &
               Interfaces.Unsigned_8'Image(Payload(Payload'First)) &
               " from " & Interfaces.Unsigned_32'Image(A) &
               ":" & Interfaces.Unsigned_16'Image(P));
      -- echo it back
      Snd_Rcv_S.out_q.Put(Payload, A, P);
    end loop;

    accept Finish;
    Put_Line("S Cons got the finish.");
  end s_cons;

  -- RSA Consumer
  task rsa_cons is
    Entry Finish;
  end rsa_cons;
  task body rsa_cons is
    Payload: Raw_Types.RSA_Pkt;
    A: Interfaces.Unsigned_32;
    P: Interfaces.Unsigned_16;
  begin
    for I in 1..N_RSA loop
      -- consume one message and echo it back
      Snd_Rcv_RSA.in_q.Get(Payload, A, P);
      Put_Line("RSA msg " &
               Interfaces.Unsigned_8'Image(Payload(Payload'First)) &
               " from " & Interfaces.Unsigned_32'Image(A) &
               ":" & Interfaces.Unsigned_16'Image(P));
      -- echo it back
      Snd_Rcv_RSA.out_q.Put(Payload, A, P);
    end loop;

    accept Finish;
    Put_Line("RSA Cons got the finish.");
  end rsa_cons;

begin
  Put_Line("Test server");
  -- wait for consumers to finish
  rsa_cons.Finish;
  s_cons.Finish;

  -- abort the sender & receiver to be able to finish
  abort Sender_S, Receiver_S, Sender_RSA, Receiver_RSA;
end Test_Server;

Similarly to the server example code above, an example client sends both RSA and Serpent packages and has consumer and producer tasks for both:

 -- S.MG, 2018
with Snd_Rcv;
with Interfaces;
with Ada.Text_IO; use Ada.Text_IO;
with Raw_Types;
with UDP;

procedure Test_Client is
  PortRSA   : Interfaces.Unsigned_16 := 34340;
  PortS     : Interfaces.Unsigned_16 := 34341;
  N_S       : Interfaces.Unsigned_8 := 105;
  N_RSA     : Interfaces.Unsigned_8 := 82;

  Server    : String := "127.0.0.1";
  package test_udp is new UDP(10);
  ServerA   : Interfaces.Unsigned_32 :=
                test_udp.IP_From_String(Server);
  ServerRSA : Interfaces.Unsigned_16 := 44340;
  ServerS   : Interfaces.Unsigned_16 := 44341;
  package Snd_Rcv_RSA is new Snd_Rcv(Raw_Types.RSA_Pkt'Length);
  package Snd_Rcv_S is new Snd_Rcv(Raw_Types.Serpent_Pkt'Length);
  -- sender RSA and Serpent
  Sender_RSA: Snd_Rcv_RSA.Sender( PortRSA );
  Sender_S  : Snd_Rcv_S.Sender( PortS );
  -- receiver RSA and Serpent
  Receiver_RSA: Snd_Rcv_RSA.Receiver( PortRSA );
  Receiver_S: Snd_Rcv_S.Receiver( PortS );

  -- producer of serpent messages
  task s_prod is
    entry Finish;
  end s_prod;
  task body s_prod is
    Payload : Raw_Types.Serpent_Pkt := (others => 10);
  begin
    Put_Line("S Producer with " &
             Interfaces.Unsigned_8'Image(N_S) & "messages.");
    -- send the messages with first octet the number
    for I in 1..N_S loop
      Payload(Payload'First) := I;
      Snd_Rcv_S.out_q.Put( Payload, ServerA, ServerS);
      Put_Line("Sent S message " &
                Interfaces.Unsigned_8'Image(I));
    end loop;

    -- signal it's done
    accept Finish;
    Put_Line("S prod got the finish.");

  end s_prod;

  -- producer of RSA messages
  task rsa_prod is
    Entry Finish;
  end rsa_prod;
  task body rsa_prod is
    Payload : Raw_Types.RSA_Pkt := (others => 20);
  begin
    Put_Line("RSA Producer with " &
             Interfaces.Unsigned_8'Image(N_RSA) & "messages.");

    -- send the messages with first octet the number
    for I in 1..N_RSA loop
      Payload(Payload'First) := I;
      Snd_Rcv_RSA.out_q.Put( Payload, ServerA, ServerRSA);
      Put_Line("Sent RSA message " &
                Interfaces.Unsigned_8'Image(I));
    end loop;

    -- signal it's done
    accept Finish;
    Put_Line("RSA prod got the finish.");

  end rsa_prod;

  -- Serpent Consumer
  task s_cons is
    Entry Finish;
  end s_cons;
  task body s_cons is
    Payload: Raw_Types.Serpent_Pkt;
    A: Interfaces.Unsigned_32;
    P: Interfaces.Unsigned_16;
  begin
    for I in 1..N_S loop
      -- consume one message
      Snd_Rcv_S.in_q.Get(Payload, A, P);
      Put_Line("S msg " &
               Interfaces.Unsigned_8'Image(Payload(Payload'First)) &
               " from " & Interfaces.Unsigned_32'Image(A) &
               ":" & Interfaces.Unsigned_16'Image(P));
      -- do NOT echo it back
    end loop;

    accept Finish;
    Put_Line("S Cons got the finish.");
  end s_cons;

  -- RSA Consumer
  task rsa_cons is
    Entry Finish;
  end rsa_cons;
  task body rsa_cons is
    Payload: Raw_Types.RSA_Pkt;
    A: Interfaces.Unsigned_32;
    P: Interfaces.Unsigned_16;
  begin
    for I in 1..N_RSA loop
      -- consume one message
      Snd_Rcv_RSA.in_q.Get(Payload, A, P);
      Put_Line("RSA msg " &
               Interfaces.Unsigned_8'Image(Payload(Payload'First)) &
               " from " & Interfaces.Unsigned_32'Image(A) &
               ":" & Interfaces.Unsigned_16'Image(P));
      -- do NOT echo back
    end loop;

    accept Finish;
    Put_Line("RSA Cons got the finish.");
  end rsa_cons;
begin
  Put_Line("Test client");
  -- wait for producers/consumers to finish
  rsa_prod.Finish;
  s_prod.Finish;
  rsa_cons.Finish;
  s_cons.Finish;

  -- abort the sender & receiver to be able to finish
  abort Sender_S, Receiver_S, Sender_RSA, Receiver_RSA;
end Test_Client;

One important issue to note here is the way in which exceptions (hence: potential issues) will be handled in this specific implementation of the Snd_Rcv package: since the Sender and Receiver are tasks and don't handle any exceptions themselves, it means that an UDP "eggog" aka exception3 will have as effect the silent death of the Sender/Receiver in which it happens4. I did consider ways of handling such exceptions rather than letting them kill the task silently but so far at least I don't quite see what the task itself can do other than re-trying whatever it was trying to do when it failed. While this could perhaps be considered a better option than not handling exceptions at all, it's been pointed to me that UDP errors mean almost always some hardware failure and as such a re-try is not going to help at all. Moreover, re-trying means also that the failure remains hidden from the calling task since there is no way in which one would be able to tell whether a task is just stuck re-trying or actually proceeding with its work just fine. Considering all this, I decided to leave it for now to the higher level task to monitor its subtasks if/when desired and take action accordingly (e.g. check perhaps periodically if a Sender/Receiver is Terminated or in other abnormal state). This doesn't mean of course that the code can't be changed at a later date to provide a different approach to handling this - all it means is that currently this is the best decision I can see given what I know so far.

With this chapter, the SMG Comms code provides everything that is needed to build on top of it a basic client for Eulora that is compliant with the published communication protocol. So I'd suggest to anyone interested in this to give it a go since starting now means that they would have some time to tinker with it before everything else is in place! At any rate, the SMG Comms series takes at least a break for now (at a later stage there should be a few bits and pieces to add) as I'll focus for a while more on the server-side moving parts that need to be done before Eulora can finally fully work on a sane protocol. The full .vpatch for this chapter and my signature for it:

  1. It's a pleasure to use Ada's implementation of threads aka tasks. But it still can take me a while to be able to say I get something to some reasonable degree - even when the something in question is actually a good implementation of threads, what can I do. ↩

  2. And note that even if I'd implement this with a select to allow some sort of "Shutdown" option, the task could still be stuck waiting on a queue or on UDP since the calls to those are blocking when there is nothing to retrieve yet/no place to write to; so the caller would STILL have to do an abort or at least to be prepared to do abort if the shutdown is not obeyed within some interval. In a nutshell, a shutdown option would therefore still not work at *all times* and as a result I can't quite see why bother with it really. ↩

  3. There are currently 7 of those with clear names: UDP_Invalid_Text_IP, UDP_Failed_Open, UDP_Failed_SetOpt, UDP_Bind, UDP_Failed_Transmit, UDP_Truncated_Send, UDP_Failed_Receive. ↩

  4. In Ada the exceptions are not propagated to the parent task since they would be potentially too disruptive and I can see that quite clearly. ↩

December 16, 2018

A Week in TMSR: 3-9 December 2018

Filed under: A Week, TMSR — Diana Coman @ 8:11 p.m.

On Monday, 3rd of December 2018:

Hanbot is recovering from an appendectomy.

Spyked publishes a regrind of the various TMSR bots as a single V-tree with two branches. His regrind brings together 4 previously separated V-trunks by Trinque, while aiming to preserve the history of the code. Commenting on this work, Mircea Popescu notes that a history-preserving regrind is certainly worth doing when the author is indeed qualified to do it properly as in this case; alternatively, the most basic sort of regrind (that is less demanding of the author's time and skill) simply collapses all existing work into one single genesis vpatch. Spyked's newly published regrind also serves now as a golden standard for any future regrinds with respect to testing - any new regrind should be tested by its author on ALL noted V implementations. In other V-related empirical results, Lobbes reports that a newer GNAT version fails to work with the existing VK.pl implementation of V. This does not come as a big surprise to anyone anymore.

Diana Coman announces her talk to students at Reading University, UK, scheduled to happen on the 10th of December.

With the help of the forum, Spyked realises that he purchased a boobytrap in the shape of a laptop or otherwise a rather expensive desk decoration/ paperweight. The item in question is called c101pa and it has the rather dubious honour of resisting so far ~all of Asciilifeform's significant attempts at disinfection.

Asciilifeform reveals that his day job is that of a genie caught in a lamp with rather unpredictable rubbing-schedule. As a result of having his lamp rubbed at odd times, his own TMSR schedule might go poof in a cloud of smoke at any time.

Esthlos is noted to have stopped any meaningful activity somewhere in November this year. His log summaries are missed, his fate is unknown, his reputation is - as always and for everyone - on the line.

On Tuesday, 4th of December 2018:

Mircea Popescu announces that he will delay making the reports for Qntra, S.MG and S.NSA due to Hanbot's appendectomy.

Mircea Popescu points out the expensive but inevitable bridge-maintaining work needed to gradually build sanity (as TMSR aims to do) from within an insane environment (as everyone currently finds themselves in). He notes that such bridges, if improperly managed, can potentially leak insanity to such degree as to kill off the attempts at sanity on the other end, as illustrated by a potential shattering of bot-code into incompatible and separate parts depending on the underlying type of database system used. Adding to the complexity of the issue is the fact that several choices (even fundamental ones such as the choice of a database system) may still have to be preserved for as long as none emerges as the clearly best option overall. As an example, Mircea Popescu notes that there is already a potential conflictual case of mysql vs postgres - each of them being best for some specific types of use but significantly worse than the other in other cases.

Mocky comes up for air from his search for a place of 9 to 5, Mon to Fri (or similar) servitude. He notes that the search is eating up his time, apparently delaying his reports on the previous Qatar trip and otherwise requiring vodka access as a prerequisite.

Asciilifeform turns 35 but doesn't believe that to be a prime.

Asciilifeform notices empirically that on the Internet it takes two questions to prove one is human after all: non-humans can xor correctly, but they fail to know - as yet - about the proper serial rate of a Fuckgoat.

Spyked takes over the delivery of republican RSS feeds from Trinque. The takeover means that Spyked's feedbot will announce from now on in #trilema all new posts on republican blogs. This very useful service had been previously provided - in an excellent way - by Trinque's deedbot for lack of any other better-placed bot to do it.

BingoBoingo reports on Qntra about Denmark's plans for criminal and expelled migrants.

On Wednesday, 5th of December 2018:

Mod6 briefly stops by to say hello.

Phf is alive in Moscow, Russia and is preparing to sign the lease of a flat there.

Lobbes and Spyked talk of a takeover of rss feeds in #eulora by feedbot from the existing lobbesbot.

Ben Vulpes completes various data entry tasks for Pizarro.

Bingo Boingo publishes on Qntra a brief summary of the adventures of Tumblr with AI and porn.

Trinque and Asciilifeform note that their respective irc bots (deedbot and pehbot) sometimes fail to reconnect. The reasons for this failure mode are yet unknown and nobody had yet the time to sink into investigating the issue.

A few people from some obscure "L0de Radio Show" join the forum brought in by danielpbarron. They talk of printer scam campaigns, keybase and what the majority seems to think of the latter. The result of this is rather predictable but it will take until next day to fully mature.

On Thursday, 6th of December 2018:

Anniez from L0de re-registers with a key that is at least not held by keybase.

BingoBoingo reports that there is now at least one new case of Leishmaniasis in Uruguay.

Mats reports a failed experiment on the ESP8266 microchip when using the rigol ds1054z instrument, due to insufficient sensitivity of the probe. He also outlines his future plans to return to Hong Kong in spring and attempt to acquire a permanent ID card and a Taiwan passport. The process involves additional steps such as applying for the right to abode but has apparently better chance of success than the experiment with ds1054z since Mats was born in Hong Kong and can reasonably expect therefore to be a good match to local probing.

Diana Coman signs chapters 4 and 5 of Asciilifeform's FFA series. She also publishes on her blog Chapter 11 of the SMG Comms series that makes the changes needed to allow arbitrary (as opposed to single, pre-defined) size of the public exponent for RSA encryption.

BingoBoingo publishes on Qntra two articles: on a confused ransomware guy and the adventures of Huawei's CFO in Canada.

Mircea Popescu publishes the delayed Qntra report for November 2018.

Danielpbarron publishes on his blog some talk he had with another dude on religious beliefs.

Lobbes deploys a fix to his auctionbot so that the messages at the end of an auction reflect correctly whether an item was sold to somebody or by that same somebody.

Some things mature quite quickly - Mircea Popescu negrates the main L0de guy.

Mircea Popescu publishes the delayed S.NSA report for November 2018.

On Friday, 7th of December 2018:

Asciilifeform leaks a few results of his chapter 14 of FFA but notes that he doesn't have any further ways to improve the speed performance of the code going forward. Essentially the performance to be reported in Chapter 14 is as good as it gets from FFA.

Bingo Boingo publishes on Qntra two articles on Leacher's Paradise (a torrent tracker) and France's street protests, respectively.

Feedbot fails to report the new posts from Qntra, prompting Diana Coman to enquire if feedbot doesn't track Qntra and Asciilifeform to note that feedbot is down. Spyked sorts out the issue when he comes online, noting that it was due to a failure of all of his connections to freenode. Spyked also plans to take over the feeds in #eulora from Lobbes.

Mircea Popescu publishes the delayed S.MG report for November 2018.

On Saturday, 8th of December 2018:

Nicoleci publishes on her blog a summary of the TMSR log of 18th November 2018.

Lobbes updates his bot's help page, mainly adding content and more detail to existing explanations of available commands.

Mircea Popescu publishes a guide on the exact ways to piss him off. The guide is essentially a structured model of his internal annoyance mechanism, providing also useful pointers to anyone interested in recognizing intermediate steps - milder levels of annoyance as they build up. Hanbot contributes in the comments noting the non-linear accumulative effect on the intensity of resulting rage of repeated failures of enrager to engage with the attempts at sanity on different levels.

On Sunday, 9th of December 2018:

Mats finds out the original meaning - "I'm your slave" - of many common greetings in Europe (e.g. "schiavo", "salve", "servus"), sparking a discussion with Mircea Popescu on what slavery is ("do what you're fucking told to do") and the fact that it exists mainly because a lot of people have a lot of trouble following with any efficiency the only existing alternative to it ("figure out something for you to do").

Lobbes provides to Spyked the list of rss feeds for #eulora.

Diana Coman provides Trinque with feedback on running his Cuntoo script: the script is successful in producing a bootable installation on the specified drive but fails at producing the expected genesis .vpatch of Cuntoo. Trinque helps her better understand the script, describes a way to speed up a subsequent run by copying already built packages to their place and suggests running the script again with a flag set to get additional information.

BingoBoingo publishes on Qntra an article on the hijacking of Linux.org. He also publishes on Pizarro's blog a weekly update of his business activity.

Mircea Popescu publishes on Trilema a post explaining how the stupidity ratchet works. The article is short, its embedded references are a den of rabbit holes.

Nicoleci publishes on her blog an account of several of her old acquintances turning out to be rather different than she thought they were. The difference seems to be that they thought and think they own her while she had no inkling that she was/is their property. As a result, their persistent attempts at "saving" her from her own choices - indeed from making any choices that are not "approved" - seem rather painful.

December 13, 2018

SMG Comms Chapter 12: Thread-Safe Queues

Filed under: Coding, SMG_Comms — Diana Coman @ 4:53 p.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

The various message types of Eulora's communication protocol are mostly implemented by now1 in chapters 1 - 11 of this series. What remains is the actual UDP communication and for this part I'm importing into SMG Comms Stanislav's neat UDP lib. Since Eulora effectively uses the size of the message to cheaply distinguish between RSA and UDP, it follows that any implementation of the protocol will have to support those two sizes (rather than one single fixed size as the original UDP lib imposes). There are several ways to do this2 but the main thing is that "any arbitrary size" is neither needed by SMG Comms as such nor worth the trouble. At the other extreme, copy/pasting the UDP lib code to have two packages that differ only by a constant is rather idiotic. So the middle ground is to use Ada's generic mechanism that allows one to parametrize code: the UDP lib package will expose the length as a parameter of the package as a whole and the client/server code will then simply create a UDP_RSA and a UDP_Serpent package types with their corresponding, fixed sizes. The only downside to this approach is that it breaks some of the existing restrictions but as previously stated, the point was not in having restrictions for the sake of having them but simply in having a good reason for any removal of any restriction. The need for UDP lib to handle those 2 different packet sizes counts in my eyes as a good reason to allow the generic mechanism.

With the UDP lib code, the most basic implementation of the protocol specification could be considered complete3. Basically the code provides everything that is needed to send and receive the messages in the format and through the communication means specified. However, in practice, Eulora will likely require a type of thread-safe queues for any processing of messages after they came in through the UDP socket and/or before they are sent. The reason for this is two-fold: first, it makes sense to decouple the send/receive via UDP from message processing; second, the encryption/decryption part required by the protocol can be relatively slow (especially for RSA) and as a result it's not the sort of thing that makes sense handled directly by the Sender/Receiver immediately on top of the UDP lib. Ideally, the Receiver would simply pass on a new message as soon as it gets it from UDP so that there is no queue forming at the lower level (since messages can then get lost/discarded simply). Arguably the Sender side can just fire the messages as soon as it receives a request to send them, since there is no additional delay there - although it could end up then either firing too many too quickly resulting in more losses than usual or otherwise becoming a bottleneck for the whole application on top of it. At any rate, the exact designs of Senders and Receivers are not part of the protocol itself - any Eulora client can implement those as it sees fit. Still, since I'm quite sure I'll need multiple tasks on the server, I'll include at least the means to decouple send/receive of rsa/serpent messages from any processing, namely a queue that can safely handle put/get calls from any number of tasks (e.g. the receiver will put messages into this queue while any number of worker tasks can then be created to process those messages in any way).

A thread-safe queue is of course nothing new and as such Ada seems to provide different mechanisms for implementation but the one that seems the most straightforward and adequate for this taks is the "protected variable": according to my Ada reference book4, the whole point of a protected variable is precisely to control the access to some protected resource. The resource itself (in this case the actual queue) is declared as private to the protected object and access to it is provided only through two procedures, Get and Put, with the important specific characteristic that calls to those procedures (and in general to any of the procedures of a protected object) are mutually exclusive. So no matter how many tasks call Get and Put and in what order, at any given time, only one task will proceed with one of Get or Put and moreover, no other task will get to enter either of those before the first task has finished. Moreover, the entry to both Get and Put can be effectively guarded by additional conditions: Get should proceed only when there is at least one item in the queue while Put should only proceed when the queue is NOT full. Using all this together with modular types for the index makes it quite easy to define a circular queue of fixed size that allows only mutually exclusive calls of Put and Get:

 -- FIFO queue of UDP messages with protected read/write (thread-safe).
 -- S.MG, 2018

with Raw_Types;
with Interfaces;

generic
  -- exact length of payload of items in the queue (1470 or 1472 in Eulora)
  Payload_Len: in Positive;

package Msg_Queue is

  -- payload type
  subtype Payload_Type is Raw_Types.Octets(1..Payload_Len);

  -- maximum length of the message queue
  Max_Q_Len: constant := 1024;

  -- index in the queue will be modular type
  type Index is mod Max_Q_Len;

  -- an item in the queue; IP/port stored only for the "other" (variable) end.
  type Msg is
    record
      Payload    : Raw_Types.Octets(1..Payload_Len);
      IP_Address : Interfaces.Unsigned_32;
      IP_Port    : Interfaces.Unsigned_16;
    end record;

  -- an array of messages
  type Msg_Array is array(Index) of Msg;

  -- the actual queue of messages, as a protected type
  protected type Queue is
    -- adds the given entry to the queue if there is space; BLOCKS if no space
    entry Put(Payload : in Payload_Type;
              Address : in Interfaces.Unsigned_32;
              Port    : in Interfaces.Unsigned_16);

    -- reads next entry from queue when available; BLOCKS if no entries
    entry Get(Payload : out Payload_Type;
              Address : out Interfaces.Unsigned_32;
              Port    : out Interfaces.Unsigned_16);

  private

    Q: Msg_Array;
    Read_Pos, Write_Pos: Index := Index'First;
    Count: Natural range 0..Max_Q_Len := 0;
  end Queue;

end Msg_Queue;

You might have noticed in the code above that I made the Msg_Queue type generic, with a single parameter that represents the length of the payload of a message that is stored in the queue: this is needed for the same reason as the generic at UDP lib previously - to allow one to store Serpent messages or RSA messages while maintaining them clearly differentiated5. Note that the package defines internally types for the items stored in the queue but the Put/Get methods use 3 parameters rather than the "Msg" record structure - the reason for this is that it makes it easier for callers to use the queue with existing types rather than having to create a new structure just for reading from the queue/writing to the queue. The corresponding implementation in msg_queue.adb:

 -- S.MG, 2018

package body Msg_Queue is
  protected body Queue is
    entry Put(Payload : in Payload_Type;
              Address : in Interfaces.Unsigned_32;
              Port    : in Interfaces.Unsigned_16)
          when Count < Max_Q_len is
      M: Msg;
    begin
      -- fill the Msg item structure
      M.Payload    := Payload;
      M.IP_Address := Address;
      M.IP_Port    := Port;
      -- add it to queue and update counter + pos
      Q( Write_Pos ) := M;
      Write_Pos := Write_Pos + 1;
      Count := Count + 1;
    end Put;

    entry Get(Payload : out Payload_Type;
              Address : out Interfaces.Unsigned_32;
              Port    : out Interfaces.Unsigned_16)
            when Count > 0 is
      M: Msg;
    begin
      M := Q( Read_Pos );
      Payload := M.Payload;
      Address := M.IP_Address;
      Port    := M.IP_Port;
      Read_Pos := Read_Pos + 1;
      Count := Count - 1;
    end Get;

  end Queue;
end Msg_Queue;

As you can see above, a call to Put may proceed only when Count < Max_Q_Len, meaning that the queue is NOT full. Symmetrically, a call to Get may proceed only when Count > 0, meaning that the queue is not empty. The Count variable is updated by both Put and Get to reflect at any given time the exact number of elements in the queue - note that this is needed as a separate variable because the queue is circular and so the indices in the queue are modular types. The modular type is very handy here as the indices will simply roll over when they get to the maximum value and so there is no possibility of attempting to read/write out of the queue boundaries. Combined with the guards for Get/Put on Count, there is no possibility of overwriting items in the queue either. Win.

The only potential downside to choosing this approach to implement the queue as a protected object is that there are yet more restrictions to relax: max_protected_entries can't be 0 anymore, since we are introducing... a protected object, yes; no_protected_types can't remain either for the obvious reason; simple_barriers can't remain because the code uses the variable Count to guard Put/Get (i.e. not a constant) - however, pure_barriers remains in place since the variable is local to Msg_Queue (and I think that how it should actually be at any rate). There are also the task-related restrictions that need to be relaxed when the Msg_Queue is used since there will be presumably some... tasks defined: max_task_entries, no_task_hierarchy, max_tasks, max_select_alternatives. Obviously, the exact list of task-related restrictions that you can keep depends on what tasks you define. The previous list is what I had to relax for some testing.

To see the above in action and at the same time to get a bit more practice with Ada tasks (aka threads), I wrote a basic test that just "reads" and "writes" stuff to a Msg_Queue. The test makes use of worker tasks but it doesn't throw delays in as well. At any rate, any "testing" for multi-threaded code is a rather dubious proposition since one can't even quite control what code gets called when exactly. So as I was saying: more of an exercise/example than a test but anyway, here's the definition of the Worker task type in tests/q_pkg.ads:

 --S.MG, 2018

with Msg_Queue;
with Ada.Text_IO; use Ada.Text_IO;
with Interfaces; use Interfaces;
with Raw_Types;
with RNG;

package Q_Pkg is
  procedure Start;
private
  package MQ is new Msg_Queue(Payload_Len => 10);
  Q: MQ.Queue;

  task type Worker is
    Entry Start(A: in Unsigned_32;
                P: in Unsigned_16;
            Times: in Positive;
            Write: in Boolean);
  end Worker;

  W: array(1..10) of Worker;

end Q_Pkg;

The corresponding implementation in tests/q_pkg.adb:


package body Q_Pkg is
  procedure Start is
    U32: Unsigned_32 := 100;
    U16: Unsigned_16 := 1;
    Write: Boolean;
  begin
    Put_Line("Start");
    for I in 1..W'Length loop
      if I mod 2 = 0 then
        Write := False;
      else
        Write := True;
      end if;
      W(I).Start(U32, U16, W'Length-I+10, Write);
      U32 := U32 + 1;
      U16 := U16 + 1;
    end loop;
  end Start;

  task body Worker is
    Pay: MQ.Payload_Type := (others => 0);
    Address: Unsigned_32;
    Port : Unsigned_16;
  begin
    accept Start(A: in Unsigned_32;
                 P: in Unsigned_16;
             Times: in Positive;
             Write: in Boolean) do
      if Write then
        for I in 1 .. Times loop
          Pay(Pay'First) := Unsigned_8( I mod 256 );
          Pay(Pay'First+1) := Unsigned_8( (I / 256) mod 256);
          Q.Put( Pay, A, P );
          Put_Line(Integer'Image(I) & "." &
                   Unsigned_32'Image(A) & "." & Unsigned_16'Image(P) &
                   " WROTE: " & Unsigned_8'Image(Pay(Pay'First)) &
                   Unsigned_8'Image(Pay(Pay'First+1)));
        end loop;
      else
        for I in 1 .. Times loop
          Q.Get( Pay, Address, Port );
          Put_Line(Integer'Image(I) & "." &
                   Unsigned_32'Image(A) & "." & Unsigned_16'Image(P) &
                   " read: " & Unsigned_8'Image(Pay(Pay'First)) &
                   Unsigned_8'Image(Pay(Pay'First+1)) &
                   " from " & Unsigned_32'Image(Address) & "." &
                   Unsigned_16'Image(Port));
        end loop;
      end if;
    end Start;
  end Worker;

end Q_Pkg;

The "test" itself simply calls Q_Pkg.Start that will iterate through a set of Worker tasks and ask those on even positions to read and those on odd positions to write to the queue (the address/ip parts of "messages" are used here to identify the workers so that one can follow the whole output):

 -- Basic example of using a message Queue with multiple readers/writers
 -- S.MG, 2018

with Q_Pkg;
procedure Test_Queue is
begin
  Q_Pkg.Start;
end Test_Queue;

The next step would now be to actually implement an example of sender/receiver. I've been thinking on this for a while but it's still not very clear to me what would be most useful here or how the sender and receiver should exactly look like. For starters, I'm not even sure whether a sender/receiver should be in fact part of smg_comms or whether it's better to leave them out, as they are strictly speaking outside of the communication protocol itself and a matter for the client/server to decide. The argument for including them (or at least some version/example of them) is that some version of them will be needed at any rate so having an example to start from is likely to help anyone who wants to implement a client. However, what is "basic" version can be a thorny issue and moreover, once some "basic" version is there, there's also a great pull towards using that one as "default" rather than basic. As to more specific questions: should both the sender and the receiver use their own Msg_Queue (i.e. outbound queue for sender to read from and inbound queue for receiver to write into) and otherwise share a socket? Or should just the receiver use a queue since sending is a much thinner layer here (messages are already fully prepared so it's just a call to the UDP lib with the already bound socket)? What should sender/receiver even do with exceptions thrown by the UDP lib, should they handle them all/ just some/none? How should sender/receiver handle potential blocks on writing to/reading from the queues (do they wait a certain amount of time, do they check and signal when queue is fuller than some percentage, so they do something else - what)?

In a word, the sender/receiver part still requires some discussion it would seem since some choices need to be made even if just for now. At any rate, here's the .vpatch for this chapter, together with my signature for it, as usual:

  1. The few that are not yet implemented are also not yet fully specified - some practical experience with an implementation, even partial, can help to flesh those out better too. ↩

  2. The way it's usually done in C being the least beneficial one - it forces upon the code all the potential issues of "any size *could* be fine" without much advantage to claim for it since no program is really likely to use all possible sizes at the same time anyway. ↩

  3. And it's about time, too! Looking at the date for Chapter 1, it's been 3 months almost to the day already. ↩

  4. John Barnes, "Programming in Ada 2012", Cambridge University Press, 2014, p. 515 ↩

  5. Since sizes are quite similar, one could otherwise simply use the larger size when defining the item type and then just fill in only as much as needed. The unpleasant effect of this however is that one then needs to also store somewhere else the actual used length. And given that I'm already using generic for the UDP lib, there's really no point in going to such lengths to avoid it here. So the Msg_Queue is a generic type - any code using it will first have to create a specific type out of it by providing a concrete length of payloads stored. ↩

December 10, 2018

My Talk of Bitcoin at Reading Uni

Filed under: TMSR — Diana Coman @ 11:01 p.m.

As mentioned already in the logs1, today I gave a talk for students at Reading University (UK) about Bitcoin. Following the very useful advice from the forum, I centred my talk on "What Is Bitcoin?" and then I built on that with a discussion of actual innovation in Bitcoin (WoT, V, Deedbot) as opposed to the attempts at subversion of Bitcoin ("blockchain technologiezzz") that are usually touted about as "innovative!!" by all those "involved" through mere talking and repeated mentioning of all the buzz words in whatever happens to be the current consensus-approved sequence. Since there is no recording of the talk2 I decided it's best to do this write-up now, while everything is still quite clear in my memory. Who knows - it might help someone else who is looking to actually understand this Bitcoin thing rather than just copy/paste the most commonly copy/pasted stuff.

Initially the University had this bright approach to schedulling of talks and rooms so that they could not even guarantee that I'd have some sort (any sort!) of board on which to write during the talk and so I decided to make the most of it and make some supporting slides, despite the fact that I rather hate doing those things. Nevertheless, I've made them now so you can download the slides for this talk - just be aware that they are meant precisely to support the talk not to replace it so they don't actually contain even 10% of the talk, they just help *illustrate* what I say. So no, here as everywhere else, you can't really replace the person doing the talk and not see any difference, such is life, totally unfair. I'll try though to summarize in here the main ideas I covered - keep in mind that the whole talk was 90 minutes, of which I'd say about 30 minutes were Q&A throughout (i.e. not all at the end, as I took questions as and when they were able to actually ask something).

dc_reading_uni_2018_560px

After an initial brief presentation of myself and Minigame establishing some basis on which I am actually giving this talk, I collected the existing "definitions" that students had about Bitcoin. As expected, they were not much in terms of meaning but surprisingly enough they were also rather difficult to extract - perhaps shyness or in a more optimist interpretation some well-placed notion that no, they don't actually have any idea what the thing is. Anyway, for the record, I got only two definitions: "a very volatile currency" and "an immutable ledger of transactions." So I noted that those touch a teeny tiny bit on what Bitcoin has in common with "currency" and "ledger of transactions" respectively while missing at the same time the much wider part of what is actually new, different and essentially disruptive about it. And I noted that the only way to actually get an idea of a complex notion (such as Bitcoin) as a whole, is by starting from its basic principles and following their characteristics and implications. So without further digression, I moved on to enumerating the 4 basic principles of Bitcoin (asymmetric cryptography, databases, peer-to-peer (p2p) and anonymity/pseudonimity) and then presenting each of them in turn.

For explaining asymmetric cryptography, I used Mircea Popescu's excellent approach of starting with the simple example of obtaining the 2 prime factors of a given number vs obtaining the number itself from those 2 prime factors when they are known. I took though rather bigger numbers because my audience was - it is to be hoped at least - made of people able to use their computers for such a task at short notice (I hope!). Based on this, I introduced the notion of public and secret (private) key, using the slides to graphically build up the whole thing piece by piece, aka keys, arrows, locks and all that bling. Basically I showed them a mini animated cartoon, what! And I'd say it worked - I could clearly see some ears pricking up!

Following on from the above, I discussed how that makes it possible to send messages without divulging the content but also without divulging the intended target either! Cue more animated cartoons with people simply trying to decrypt messages from a pile until they find the one (or at least one) that they *can* decrypt with their private key - hence, if they can decrypt it, then (and only then), it is meant for them! At this point they actually started to sit straighter and even nod from time to time as it - apparently - made sense. I finished this first part with a summary of what I had said so far, essentially the slide behind me in that picture.

The database and p2p principles I covered much faster as it was really just the main ideas that were of most interest to me for the task at hand: a database is essentially structure applied to data and that means any structure; p2p allows participants to find and obtain all the different parts of information out of a given set, without having to rely on any central authority /middle man but simply by communicating and interacting with the other participants. Nevertheless, p2p was also good for another mini-cartoon to illustrate that one needs to be able to assess what part of the whole they might have at any given time, what parts they are missing and to what degree a proposed part is what they are looking for.

After this brief interlude of databases+p2p I got back to building on the asymmetric cryptography introduced earlier aka introducing anonymity and pseudonimity. As the students were by now rather intently listening, I asked them what they could tell about those 3 messages on slide 30: there are 3 signed messages (encrypted with secret keys) of which 2 verify (decrypt correctly) and one doesn't verify with one given public key. After a few hesitations and false starts, they actually got it - yes, one can say that both that decrypt correctly were actually encrypted with the *same* private key! So I took this further and pointed out that this implies authorship (the one controlling the pair of this public key is the author of the first 2 messages), continuity (whoever signed the first message, also signed the second) and repudiability - one can easily prove that they did NOT sign the 3rd message! All very useful things since together they allow one to obtain pseudonimity - in other words, to have a way of gaining authorship and continuity without having to give up privacy and mix up in there personal information such as date of birth or where they hang out with friends on a Saturday night. So no more of "verify your identity by giving us the keys to your house and generally whatever else we might think of asking". Instead, pseudonimity by means of public/private keypair - one neat and simple way of enjoying the results of your work (aka building your reputation) without having to give up more private information than you choose to. This was the first time I mentioned reputation and I took the time to give a few examples since I was preparing the ground for the WoT later on. Judging by the large amount of questions that the WoT sparked, I'd say it hit home.

Putting everything together, I showed this boring slide 36 with classic bullet points. To make up for it and to check to some extent how much of the head-nodding actually meant some sort of understanding, I asked the students to identify in that definition what exactly "caused" each of those main characteristics listed there. They were fine with p2p -> no middle/central authority and they worked their way on pseudonimity. I explained the other two (reputation vs blind trust; irreversibility) in more detail since there were some important aspects there that weren't apparently all that obvious - including the "trust" in banks and governments and parents and what-not. Basically when many still trust without even realising that they really just ..."believe" and nothing more.

For the next step, I pointed out that innovation with/in Bitcoin is simply something that *maintains* all those main characteristics of Bitcoin while attempting otherwise to adapt the world ("how things are done") to it and NOT the other way around. By contrast, anything that tries to chip away even in the slightest at *any* of those main characteristics is nothing more but an attempt at subversion. As soon as one tries to change Bitcoin to fit the world ("what people expect/need/want"), one is - whether they admit it or not, whether they know it or not - trying to subvert Bitcoin, not innovate with it. So the difference being clear and easy enough to make hopefully, one can tell for anything whether it's innovation or subversion and they can then *also* tell whether the author is at least honest about it! At which point of course, they can therefore update the reputation of this author... even quite publicly if they are using the WoT!

From this point on I went online to show the WoT and V directly, exploring it a bit and discussing the actual meaning and use of the WoT and then linking it to deedbot and the voice system + wallet in #trilema. By this point there were several hands raised at the same time so questions started to really pickup. Quite a good number of questions focused on the "but it's not fair" aspect of an existing WoT + a signed-patches-only V:

  • So a newcomer needs to trust implicitly the deedbot owner in order to get into the WoT?
    • No, a newcomer needs first to find at least one person in the WoT who is willing to give them a rating. Unless and before they do that, they are just not part of the WoT and therefore there isn't much that they can do really. This is NOT meant to be (and nothing can be) a replacement for talking to people or getting to know people!
  • But then it's not fair for those that are not in the WoT!
    • Sure, life is not fair, nor will it ever be fair. Yes, it can be hard to get in the WoT if nobody knows you. So make yourself a key and start building up your reputation so that people CAN get to know you!
  • How do I know you did not rate yourself?
    • You talk to those people who rated me. If you can't talk to them, their rating is meaningless for you anyway. And yes, one can make as many "identities" (aka private keys) as they want to but remember that building up each of them takes time and effort. So why exactly would I waste this time and effort in bogus identities instead of increasing my real reputation?
  • But what if everyone negrates me for a tiny mistake I make in a patch and I can't participate to the community anymore3?
    • What of it? You TALK to them and see what you need to do to get them to change their rating; and if they are indeed as unreasonable as you assume them to be then why would you actually want to work with them?
  • Why isn't the WoT stored in the blockchain so that there isn't a "single point of failure" in the person owning deedbot?
    • You are confusing there the WoT with its *representation*! Deedbot simply stores a representation of the WoT but it is NOT the WoT, nor does it own the WoT. It is participating people who actually make and own the WoT. So if deedbot goes away or its owner turns rogue -as it actually happened with assbot before4 - the WoT simply moves/splits/migrates following the people. The ratings I give are neither fixed in stone nor anyone else's property and it is always and everywhere the people that matter, not the bots. And for that matter there can be *any number of WoTs*, what.
  • But what if the owner of deedbot cheats with the ratings?
    • Well, how exactly? People who gave those ratings know what ratings they gave and they will shout/negrate as soon as they notice any foul play. As for "bots" - what of them, their ratings are worthless since nobody rates *them* (and pure circular ratings you to bot and bot to you won't get you anywhere really, at least not with any sort of thinking people, no).
  • Why is the WoT implemented through deedbot, with a central model? Wouldn't it be better5 to rely on the blockchain only for the WoT so you don't have to trust a single person?
    • Why would the WoT need the blockchain? Ratings are NOT immutable. And at any rate, I'd much rather trust a person (who I can negrate if things go wrong!) than a... network as a whole (which for that matter is not even all that trustful, read about mining cartels to start getting some idea on this matter). Once again: there is no substitute to talking to, interacting with and ultimately trusting (or NOT trusting) individual people!
  • But what stops deedbot's owner from running with all the money?
    • Nothing can ever stop anyone from doing what they want to do. But the incentives here are stacked *against* this: his considerable reputation is on the line; and he can much more easily earn money AND increase his reputation by simply charging a fee for services rather than running away with the piggy banks. It's the same with any sort of contracts too, really: what stops the other party from breaking their contract? "The law!" No, the law doesn't and can't *stop* anything. You may trust some particular state that it will enforce that law and therefore give you some compensation for the breach of contract but it is just that - trust, blind-faith.

There were in fact many other questions but those are the ones that I think are rather usual so the answers can help others perhaps as well as the students who asked them this time. Obviously, the answers given above are those given at a talk - without the time to go deeper into details and without the luxury of linking related logs/posts from the abundant existing materials. I still did point out to students that there is much more to read on the subject both in the logs (which I demonstrated a bit) and on the TMSR blogs but whether they will do any reading of this sort or not is entirely up to them - since the benefits to reap for doing so as well as the losses to enjoy for failing to do so will also be for each of them personally and only for them.

All I can say is that I'm genuinely rather curious whether any of them gets to actually register a key and do something with it, now that they nodded at the talk, they asked the questions, they came at the end to me to say that they liked the talk and now that they have in front of them this opening, this way to get into it all6 - for as long as they actually have it, of course, since nothing is for ever!

  1. Yes, that's where pretty much everything worth the mention is... mentioned. Where else? ↩

  2. And barely any photos as the prof I asked to take some photos followed the talk so intently that he...forgot to take any! So he had to take a couple at the very end, while students were still asking questions, what can I do. ↩

  3. I let pass the whole community participation as there were bigger fish to fry this time. ↩

  4. Yes, I did give them the summary of the #bitcoin-assets debacle. ↩

  5. I honestly think he just-about-said "fairer" ↩

  6. Yes, I did tell them about #eulora and #ossasepia and even that I will rate them if they show up and convince me that they were at the lecture. Not that it makes in itself a difference - those who would come would come and find their way anyway and those who won't still won't, I know that. ↩

December 6, 2018

SMG Comms Chapter 11: Arbitrary Size of Public Exponent

Filed under: Coding, EuCrypt, SMG_Comms — Diana Coman @ 10:59 a.m.

~ This is a work in progress towards an Ada implementation of Eulora's communication protocol. Start with Chapter 1.~

The change from fixed-size to arbitrary size of the public exponent "e" of RSA keys1 turned out to be of course more than just introducing a parameter for e's size - all I can say about it is that I was rather ready for this development by now, given the known mess that is the MPI lib. So the only surprise here was the exact way in which MPI fails rather than the fact that it does fail. Let's take it slowly and in detail since it is - like everything else related to encryption - rather important to have it as clear as one can make it.

On the SMG Comms side, a shorter "e" is really no trouble at all: basically there is no such thing as "shorter" since it can perfectly be the same size as it always was, only starting with whatever number of 0s it needs to make up to the expected length, big deal. And this is in fact already handled and handled well in the wrappers I wrote previously for the RSA and MPI C code, since it was already clear that yes, any set of octets might start at any time with 0s but that doesn't make them fewer octets or anything of the sort. So at first look, there really isn't any need to change anything, since the "change" required is neatly and natively handled as what it is - just a specific case of the more general operation that is implemented. Still, for clarity and further use downstream, I decided to add a constant and a new subtype to Raw_Types simply for the purpose of providing an explicit way of using the exactly-8-octets-long e (neither the new constant nor the new subtype are put to use so far by any of the message pack/unpack or read/write methods): 

    -- RSA public exponent (e) size in octets
    -- NB: this should normally match the E_LENGTH_OCTETS in smg_rsa.h
    -- NOT imported here for the same reason given at RSA_KEY_OCTETS above
  E_LENGTH_OCTETS : constant Positive := 8;

  subtype RSA_e is Octets( 1 .. E_LENGTH_OCTETS);

On the RSA side, the same constant "length of e in octets" goes into include/smg_rsa.h since it is not exactly a knob of the whole thing but rather a parameter for RSA key generation:

/**
 * This is the length of the public exponent e, given in octets.
 * TMSR standard e has KEY_LENGTH_OCTETS / 2 octets.
 * Eulora's communication protocol uses however e with 8 octets length.
 * New keypairs generated will have e precisely this length.
 * Change this to your preferred size of e for generating new keys with that size of e.
 * NB: this impacts key generation ONLY! (i.e. NOT encrypt/decrypt).
 */
static const int E_LENGTH_OCTETS = 8;

As the comments above stress, the "length of e" should normally be a concern in the code only when generating a new key pair; at all other times (encrypt/decrypt), the e that is provided will be used, whatever length it might be. Looking at the key generation code, the change to make is minimal since the code is sane - simply replace a local variable that specified the length of the required prime with the new global constant that now specifies the user's choice of length for e (in rsa/rsa.c, function gen_keypair):

	/* choose random prime e, public exponent, with 3 < e < phi */
	/* because e is prime, gcd(e, phi) is always 1 so no need to check it */
	do {
		gen_random_prime( E_LENGTH_OCTETS, sk->e);
	} while ( (mpi_cmp_ui(sk->e, 3) < 0) || (mpi_cmp(sk->e, phi) > 0));

Following the changes above, a re-read of all my rsa and smg_comms code confirmed that no, there is nothing else to change - it is after all just a matter of exposing a constant to the user, not any change of the underlying algorithm, so that's surely all, right? Well, no, of course it's not, because at the lower level, all those 0-led smaller "e" go into the MPI lib of gnarly entrails. And as it turns out, a quick test whipped out to see the whole thing in action got...stuck, going on for ever somewhere in the MPI code. Where? Well, the stack trace goes 8 levels deep into the MPI code and it looks (at times, as it rather depends on where one stops the neverending run...) like this:

#0  0x000000000040b492 in mpihelp_addmul_1 ()
#1  0x0000000000407cd4 in mpih_sqr_n_basecase ()
#2  0x0000000000407e68 in mpih_sqr_n ()
#3  0x0000000000408098 in mpih_sqr_n ()
#4  0x0000000000407d6b in mpih_sqr_n ()
#5  0x0000000000408098 in mpih_sqr_n ()
#6  0x0000000000407d6b in mpih_sqr_n ()
#7  0x0000000000405693 in mpi_powm ()

Claiming that one fully knows what goes on in 8 piled levels of MPI calls is rather disingenous at best so I won't even go there. However, a closer look at all that code, starting with mpi_powm and following the code seems to suggest that the issue at hand is that MPI simply can't handle correctly 0-led numbers. To which one should add of course "in some cases" so that one can't just say fine, wtf is it doing permitting any 0-led numbers then?? No, that would be too easy so the reality is that it permits and it probably even *requires* in places 0-led numbers but *in other places* it gets stuck2 on them. Aren't you happy to have followed this mess so far to such amazing conclusion? At any rate, going through the MPI code yields some more fun of course, such as this fragment in mpi-pow.c:

    /* Normalize MOD (i.e. make its most significant bit set) as required by
     * mpn_divrem.  This will make the intermediate values in the calculation
     * slightly larger, but the correct result is obtained after a final
     * reduction using the original MOD value.  */
    mp = mp_marker = mpi_alloc_limb_space(msize, msec);
    count_leading_zeros( mod_shift_cnt, mod->d[msize-1] );
    if( mod_shift_cnt )
  mpihelp_lshift( mp, mod->d, msize, mod_shift_cnt );
    else
  MPN_COPY( mp, mod->d, msize );

The obvious clue there is that at least one culprit of "can't handle 0-led numbers" is that divrem function that is indeed called as part of the exponentiation. But the added joke that's for insiders only is that the normalization there is done ad-hoc although there exists a function precisely for...normalizing aka trimming the leading 0s from an mpi! Eh, so what if it exists - by the time something gets as big and tangled as MPI, chances are nobody remembers everything there is but that's not a problem at all, right? But wait, just ~10 lines further down, there is another normalization and at *that* place, the author somehow remembered that there is even a macro defined for this purpose! And oh, another ~10 lines further, there is yet another way in which normalization is done on the spot (shifting the bits directly!). So what is it already that makes one write code like this, some misplaced purple-prose inclination, let's not repeat the same expression or what exactly? Frankly the only logical answer is that it's done on purpose - anything and everything to increase the number of lines of code. Increase productivity3 !!

Moving further, it turns out that this very same function actually *does* trim the leading 0s off the exponent at some point! Which of course begs now the question of just how and why is then a problem to give it a 0-led exponent? Essentially it trims it but too late/not fully/not for everything and not everywhere that it should do it, that's the best I can say about it. And overall, the fact of the matter is simply that MPI just doesn't correctly handle 0-led MPI values4, end of story. To quote from MPI code and comments themselves, the author's explanation:

/****************
 * Sometimes we have MSL (most significant limbs) which are 0;
 * this is for some reasons not good, so this function removes them.
 */

So it is "for some reasons not good", mmkay? It reminds me of the other display of great expertise in "reasons". Without wasting even more time on the MPI code of wonders5, the solution for SMG Comms is essentially a work around: the C wrappers get another job, namely to ensure that the values passed on to MPI are normalized. Note that the symmetrical opposite of this, namely adding missing leading 0s is already implemmented where needed (in the Ada code that actually deals perfectly fine with 0-led values since they are not oh-so-special, really). Thankfully, this is a very simple thing to do: instead of using directly the mpi_set_buffer method to set the value of an mpi number, define an mpi_set_normalized method that calls mpi_set_buffer + mpi_normalize: 

void mpi_set_normalized(MPI m, const char *buffer,
                        unsigned int noctets, int sign) {
  mpi_set_buffer( m, buffer, noctets, sign );
  mpi_normalize( m );
}

Using the above code, all the mpi_set_buffer calls in c_wrappers are replaced by mpi_set_normalized calls and so there are no more 0-led mpi values passed on to the C code when calling rsa from Ada (since this is the purpose of those c_wrappers: to provide a sane interface for Ada to work with the insanity of C for RSA needs). Obviously, if you insist on calling the C rsa encrypt/decrypt methods directly, it's up to you to make sure you don't pass them 0-led values. While I could change the encrypt/decrypt methods themselves to normalize all the keys' components before doing anything, I think that's a very ugly and ultimately incorrect thing to do: the encrypt/decrypt should use precisely what they are given, not go about tampering with the inputs, regardless of "reasons". Yes, it is ugly and incorrect that MPI forces this normalization nonsense but that's not a justification for messing the encrypt/decrypt functions to cover up for it.

Note also that I specifically chose NOT to include the normalization in the existing method mpi_set_buffer because on one hand it's not the job of mpi_set_buffer to trim its inputs and on the other hand there is a need for mpi_set_buffer precisely as it is: there is code in there relying on being able to set the buffer of an mpi to anything, including 0-led vectors (even if at times, that doesn't remain 0-led for long). So no, modifying mpi_set_buffer is not a good option, even without considering the fact that MPI is better thrown away6 than changed.

The rest of the .vpatch for this chapter of SMG Comms contains simply the 2 additional tests (and changes needed for them in the test environment) that I wrote: one for the RSA c code, to flag the issue and one for the Ada code to ensure that there is at least one test with an exponent of 8 octets. I've used first the rsa code with the length of e set to 8 to generate a pair of RSA keys that are used then for the new test. So there is now a new file, "8_keys.txt" containing this new set of keys and the Ada test is simply another call with different parameters to read its input from this file as opposed to another.

Given that the arbitrary size of e touches essentially EuCrypt code, I also packed those minimal changes to smg_rsa.h and to key generation, together with the new test using a shorter e, into a .vpatch for EuCrypt. I've also added in there stern warnings at the encrypt/decrypt regarding the 0-led issue since it is the responsibility of the caller to either make sure they don't provide 0-led values or otherwise deal with a potentially-blocking call. Both .vpatch files and their corresponding signatures are on my Reference Code Shelf as well as linked here for your convenience:

  1. In practice this is more like 8-octets size for Eulora and otherwise 256 octets for TMSR rather than all sorts of sizes. Nevertheless, the big change is from one single size to ~anything more than one size since that means that the size can't be relied on to be anything specific at any given time. ↩

  2. Or as good as stuck: I did not have the patience to wait on it for more than 3 minutes, given that the operation in question should have barely taken a couple of seconds at maximum. ↩

  3. And probably surpass the 5-years plan too, since it's hardly distinguishable already. ↩

  4. Or, apparently, only values with "too many" 0 at the front aka at least one limb. ↩

  5. Yes, I did actually dig even deeper into the MPI code following this because I don't really have a choice not to as long as MPI remains part of SMG Comms and EuCrypt. The conclusion however is still the same, so I'm jumping here to the conclusion and you are free to tell me in detail in the comments section the results of your investigation on this. ↩

  6. Hopefully there will soon be a replacement for it and so it WILL be thrown away, I can't wait for that! ↩

Older Posts »

Theme and content by Diana Coman