Ossabot is a clone of Stanislav's logger, consisting of 2 separate parts:

1. An IRC bot that connects to specified channels and records everything it sees to a local Postgres database.
2. A "reader" that takes everything from the database and formats it as handy html pages of the sort you can admire at http://logs.ossasepia.com.

The main reason to run a logger in the first place is simply that I quite need it in order to keep track of and reference conversations in my #ossasepia channel. But on top of this and as mentioned previously, providing a public logger for tmsr channels is a basic contribution and one where redundancy is key in the long term. Not to mention that running a bot as part of a synchronised fleet is the closest concrete step I can see at the moment towards Gossipd - as soon as you stop to consider the issue of how to synchronize with others, you stumble towards/upon Gossipd quite inevitably, even if you had no previous idea as to what that was. In one word, running a logger is needed, useful and a stepping stone in the desired direction so it jumped to the front of the (long) queue of work waiting to be done.

For lack of a better option, my logger clone is currently running on CentOS 6. For once, it turned out that my choice of environment was too old even for republican code - the logger and especially its dependencies¹ require Python 2.7 and Postgres 9 or 10 while CentOS 6 runs with Python 2.6 and Postgres 8 as default version. Given that the logger had been previously deployed and tested only on Gentoo and combined with the fact that software versions are rather meaningless across different environments, the initial attempt was an exercise in frustration that nearly ended in total abort of the attempt to cross the Gentoo-CentOS divide. Nevertheless, a second attempt was successful, taking advantage of the experience and of a different set of version numbers graciously made public by lobbes. Here's what it took to have Ossabot running on CentOS 6:

1. Postgres 9.6 and related packages (-devel, -contrib² ) installed, configured, initialised and started as a service:

   sudo rpm -Uvh http://yum.postgresql.org/9.6/redhat/rhel-6-x86_64/pgdg-redhat96-9.6-3.noarch.rpm
   
   sudo yum install postgresql96-server postgresql96
   Installed:
     postgresql96.x86_64 0:9.6.15-1PGDG.rhel6
     postgresql96-server.x86_64 0:9.6.15-1PGDG.rhel6                                         
   
   Dependency Installed:
     postgresql96-libs.x86_64 0:9.6.15-1PGDG.rhel6        
   
   sudo yum install postgresql96-devel postgresql96-contrib
   
   Installed:
     postgresql96-contrib.x86_64 0:9.6.15-1PGDG.rhel6
     postgresql96-devel.x86_64 0:9.6.15-1PGDG.rhel6
   
   sudo service postgresql-9.6 initdb
   sudo service postgresql-9.6 start
   sudo chkconfig postgresql-9.6 on
   

   The path for pg_config was for some reason not correctly set/updated when installing the 9.6 as opposed to default 8 Postgres, hence:

   sudo ln -s /usr/pgsql-9.6/bin/pg_config /usr/sbin/pg_config
   

2. Creating the db and user for the bot according to its handy README:

   # su - postgres
   -bash-4.1$ psql
   psql (9.6.15)
   Type "help" for help.
   
   postgres=# create user ossabot createdb;
   CREATE ROLE
   postgres=# alter role ossabot superuser;
   ALTER ROLE
   postgres=# create database ossalog;
   CREATE DATABASE
   postgres=# grant all privileges on database ossalog to ossabot;
   GRANT
   

3. To set local connections to Postgres as trusted so that the various scripts may run without requiring passwords/authentication:

   sudo nano /var/lib/pgsql/data/pg_hba.conf
   changed in pg_hba.conf (trust for local connections)
   sudo service postgresql-9.6 restart
   

4. And creating the database for the logger, according to its README, simply run from logger's dir:

   sh init_db.sh
   

5. The first dependency of the bot is psycopg2 that works fine when installing the default system version, i.e. simply:

   sudo yum install python-psycopg2
   

6. For Python 2.7 on CentOS 6, one needs the software collections (scl) repository:

   sudo yum install centos-release-scl
   sudo yum install python27
   

7. Once scl and python27 are installed, the best approach is to enable it in a separate session of its own, to avoid mixing stuff and going nuts. For example:

   $ scl enable python27 bash
   $ python --version
   Python 2.7.16
   

   The scl will install a whole bunch of executables specifically for the Python2.7 environment. It's unclear to me how the exact place of those is chosen and I didn't bother to dig that up but at any rate, on my machine at least it ended up in /opt/rh/python27 so that the bin dir in there contained for instance:

   $ ls /opt/rh/python27/root/usr/bin/
   easy_install      pydoc             python-config       rst2s5          sphinx-quickstart
   easy_install-2.7  pygmentize        rst2html            rst2xetex       virtualenv
   nosetests         python            rst2latex           rst2xml         virtualenv-2.7
   nosetests-2.7     python2           rst2man             rstpep2html     wheel
   pip               python2.7         rst2odt             sphinx-apidoc
   pip2              python2.7-config  rst2odt_prepstyles  sphinx-autogen
   pip2.7            python2-config    rst2pseudoxml       sphinx-build
   

8. Within the Python2.7 enabled session, you can start installing the various dependencies required by the bot, specifying the precise version required (note the export of LD_LIBRARY_PATH when sudo-ing):

   sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install Flask==0.12.2
   Successfully installed Flask-0.12.2 Jinja2-2.10.1 MarkupSafe-1.1.1 Werkzeug-0.15.5 itsdangerous-1.1.0
   
   sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install requests
   Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
   

9. With all the above installed (and the unused cache stuff removed from the bot's code), the eat_dump.py, reader.py and bot.py finally run, hooray! Note that you might want an up-to-date dump from one of the older bots to get historical data. If you use such a dump, you'll need to change manually in it the database user name to match what you have (or the restore/import will fail). Still, you need now to take care of installing and configuring Apache (or similar) too:

   sudo yum install httpd
   sudo nano /etc/httpd/conf/httpd.conf
   

   To configure virtual hosts on apache, you need to change the httpd.conf file above, adding for instance:

   < VirtualHost *:80 >
     ServerName logs.ossasepia.com
     KeepAlive On
     ProxyPreserveHost On
     ProxyPass "/" http://127.0.0.1:5002/
     ProxyPassReverse / http://127.0.0.1:5002/
   < / VirtualHost >
    

   And then start Apache and set it so it starts automatically on reboot:

   sudo service httpd start
   chkconfig httpd on
   

10. The last step is to write scripts to backup the database and to start the bot + reader parts of the logger as well as set up cron jobs to call those scripts. Note that the scl use means that scripts for starting the bot/reader on CentOS will have to first enable python27, for instance:

    scl enable python27 "python /home/youruser/bot.sh /home/youruser/ossabot.conf >> /home/youruser/bot_log.out"
    

As it turns out, standing up the logger on CentOS 6 is an overnight success, quite literally. The above steps should hopefully make it more of an 1-hour task but if they don't, kindly let me know in the comments below. Meanwhile the bot has been running quite nicely and even turned out some interesting issues regarding loglines numbering and sync between several bots. Interesting means of course that they aren't yet solved either so just in case you were looking for something to do, here's your chance.

The code of the logotron is mirrored together with Stan's and my own signatures, on my Reference Code page.

1. My full list as gathered from the installation notes on CentOS is this: postgresql96-server, postgresql96, postgresql96-libs, postgresql96-devel, postgresql96-contrib, python-psycopg2, python27 (centos-release-scl), Flask (Jinja2, MarkupSafe, Werkzeug, itsdangerous), requests (idna, chardet, certifi, urllib3). Infuriatingly, the python uninstaller does *not* care about the very dependencies that the python installer brought in so if you want to uninstall stuff, you need to do it manually for *all* the above packages. Such progress, such automation, such (w)ow. ↩

2. NB: those are needed e.g. -contrib contains the pg_trgm extension that provides fuzzy string matching and is used by the logger ↩

Category: Coding

Comments feed: RSS 2.0