Ossa Sepia

September 2, 2019

Ossabot and Its Flask of Python 2.7 on CentOS 6

Filed under: Coding — Diana Coman @ 7:57 p.m.

Ossabot is a clone of Stanislav's logger, consisting of 2 separate parts:

  1. An IRC bot that connects to specified channels and records everything it sees to a local Postgres database.
  2. A "reader" that takes everything from the database and formats it as handy html pages of the sort you can admire at http://logs.ossasepia.com.

The main reason to run a logger in the first place is simply that I quite need it in order to keep track of and reference conversations in my #ossasepia channel. But on top of this and as mentioned previously, providing a public logger for tmsr channels is a basic contribution and one where redundancy is key in the long term. Not to mention that running a bot as part of a synchronised fleet is the closest concrete step I can see at the moment towards Gossipd - as soon as you stop to consider the issue of how to synchronize with others, you stumble towards/upon Gossipd quite inevitably, even if you had no previous idea as to what that was. In one word, running a logger is needed, useful and a stepping stone in the desired direction so it jumped to the front of the (long) queue of work waiting to be done.

For lack of a better option, my logger clone is currently running on CentOS 6. For once, it turned out that my choice of environment was too old even for republican code - the logger and especially its dependencies1 require Python 2.7 and Postgres 9 or 10 while CentOS 6 runs with Python 2.6 and Postgres 8 as default version. Given that the logger had been previously deployed and tested only on Gentoo and combined with the fact that software versions are rather meaningless across different environments, the initial attempt was an exercise in frustration that nearly ended in total abort of the attempt to cross the Gentoo-CentOS divide. Nevertheless, a second attempt was successful, taking advantage of the experience and of a different set of version numbers graciously made public by lobbes. Here's what it took to have Ossabot running on CentOS 6:

  1. Postgres 9.6 and related packages (-devel, -contrib2 ) installed, configured, initialised and started as a service:
    sudo rpm -Uvh http://yum.postgresql.org/9.6/redhat/rhel-6-x86_64/pgdg-redhat96-9.6-3.noarch.rpm
    
    sudo yum install postgresql96-server postgresql96
    Installed:
      postgresql96.x86_64 0:9.6.15-1PGDG.rhel6
      postgresql96-server.x86_64 0:9.6.15-1PGDG.rhel6                                         
    
    Dependency Installed:
      postgresql96-libs.x86_64 0:9.6.15-1PGDG.rhel6        
    
    sudo yum install postgresql96-devel postgresql96-contrib
    
    Installed:
      postgresql96-contrib.x86_64 0:9.6.15-1PGDG.rhel6
      postgresql96-devel.x86_64 0:9.6.15-1PGDG.rhel6
    
    sudo service postgresql-9.6 initdb
    sudo service postgresql-9.6 start
    sudo chkconfig postgresql-9.6 on
    

    The path for pg_config was for some reason not correctly set/updated when installing the 9.6 as opposed to default 8 Postgres, hence:

    sudo ln -s /usr/pgsql-9.6/bin/pg_config /usr/sbin/pg_config
    
  2. Creating the db and user for the bot according to its handy README:
    # su - postgres
    -bash-4.1$ psql
    psql (9.6.15)
    Type "help" for help.
    
    postgres=# create user ossabot createdb;
    CREATE ROLE
    postgres=# alter role ossabot superuser;
    ALTER ROLE
    postgres=# create database ossalog;
    CREATE DATABASE
    postgres=# grant all privileges on database ossalog to ossabot;
    GRANT
    
  3. To set local connections to Postgres as trusted so that the various scripts may run without requiring passwords/authentication:

    sudo nano /var/lib/pgsql/data/pg_hba.conf
    changed in pg_hba.conf (trust for local connections)
    sudo service postgresql-9.6 restart
    
  4. And creating the database for the logger, according to its README, simply run from logger's dir:

    sh init_db.sh
    
  5. The first dependency of the bot is psycopg2 that works fine when installing the default system version, i.e. simply:

    sudo yum install python-psycopg2
    
  6. For Python 2.7 on CentOS 6, one needs the software collections (scl) repository:
    sudo yum install centos-release-scl
    sudo yum install python27
    
  7. Once scl and python27 are installed, the best approach is to enable it in a separate session of its own, to avoid mixing stuff and going nuts. For example:
    $ scl enable python27 bash
    $ python --version
    Python 2.7.16
    

    The scl will install a whole bunch of executables specifically for the Python2.7 environment. It's unclear to me how the exact place of those is chosen and I didn't bother to dig that up but at any rate, on my machine at least it ended up in /opt/rh/python27 so that the bin dir in there contained for instance:

    $ ls /opt/rh/python27/root/usr/bin/
    easy_install      pydoc             python-config       rst2s5          sphinx-quickstart
    easy_install-2.7  pygmentize        rst2html            rst2xetex       virtualenv
    nosetests         python            rst2latex           rst2xml         virtualenv-2.7
    nosetests-2.7     python2           rst2man             rstpep2html     wheel
    pip               python2.7         rst2odt             sphinx-apidoc
    pip2              python2.7-config  rst2odt_prepstyles  sphinx-autogen
    pip2.7            python2-config    rst2pseudoxml       sphinx-build
    
  8. Within the Python2.7 enabled session, you can start installing the various dependencies required by the bot, specifying the precise version required (note the export of LD_LIBRARY_PATH when sudo-ing):
    sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install Flask==0.12.2
    Successfully installed Flask-0.12.2 Jinja2-2.10.1 MarkupSafe-1.1.1 Werkzeug-0.15.5 itsdangerous-1.1.0
    
    sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH ./pip2.7 install requests
    Successfully installed certifi-2019.6.16 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
    
  9. With all the above installed (and the unused cache stuff removed from the bot's code), the eat_dump.py, reader.py and bot.py finally run, hooray! Note that you might want an up-to-date dump from one of the older bots to get historical data. If you use such a dump, you'll need to change manually in it the database user name to match what you have (or the restore/import will fail). Still, you need now to take care of installing and configuring Apache (or similar) too:

    sudo yum install httpd
    sudo nano /etc/httpd/conf/httpd.conf
    

    To configure virtual hosts on apache, you need to change the httpd.conf file above, adding for instance:

    < VirtualHost *:80 >
      ServerName logs.ossasepia.com
      KeepAlive On
      ProxyPreserveHost On
      ProxyPass "/" http://127.0.0.1:5002/
      ProxyPassReverse / http://127.0.0.1:5002/
    < / VirtualHost >
     

    And then start Apache and set it so it starts automatically on reboot:

    sudo service httpd start
    chkconfig httpd on
    
  10. The last step is to write scripts to backup the database and to start the bot + reader parts of the logger as well as set up cron jobs to call those scripts. Note that the scl use means that scripts for starting the bot/reader on CentOS will have to first enable python27, for instance:

    scl enable python27 "python /home/youruser/bot.sh /home/youruser/ossabot.conf >> /home/youruser/bot_log.out"
    

As it turns out, standing up the logger on CentOS 6 is an overnight success, quite literally. The above steps should hopefully make it more of an 1-hour task but if they don't, kindly let me know in the comments below. Meanwhile the bot has been running quite nicely and even turned out some interesting issues regarding loglines numbering and sync between several bots. Interesting means of course that they aren't yet solved either so just in case you were looking for something to do, here's your chance.

The code of the logotron is mirrored together with Stan's and my own signatures, on my Reference Code page.


  1. My full list as gathered from the installation notes on CentOS is this: postgresql96-server, postgresql96, postgresql96-libs, postgresql96-devel, postgresql96-contrib, python-psycopg2, python27 (centos-release-scl), Flask (Jinja2, MarkupSafe, Werkzeug, itsdangerous), requests (idna, chardet, certifi, urllib3). Infuriatingly, the python uninstaller does *not* care about the very dependencies that the python installer brought in so if you want to uninstall stuff, you need to do it manually for *all* the above packages. Such progress, such automation, such (w)ow. 

  2. NB: those are needed e.g. -contrib contains the pg_trgm extension that provides fuzzy string matching and is used by the logger 

6 Comments »

  1. Thanks for writing this cookbook! Please consider to also put it in as patch to the "readme", I will sign & mirror.

    Comment by Stanislav Datskovskiy — September 2, 2019 @ 10:18 p.m.

  2. 1 minor nitpick : bot and reader log to the path given in the config, rather than to stdout.

    Comment by Stanislav Datskovskiy — September 2, 2019 @ 11:16 p.m.

  3. Seemed a bit too big to stuff in a patch and possibly of relatively narrow interest. So maybe just a link in the README would work just as well?

    Right you are re log path.

    Comment by Diana Coman — September 3, 2019 @ 12:56 p.m.

  4. I will put in link to this page, then.

    Comment by Stanislav Datskovskiy — September 4, 2019 @ 12:09 a.m.

  5. Works.

    Comment by Diana Coman — September 4, 2019 @ 1:53 p.m.

  6. [...] Intended as a companion guide to http://ossasepia.com/2019/09/02/ossabot-and-its-flask-of-python-27-on-centos-6/ [...]

    Pingback by logs.ericbenevides.com is born: compiled notes on the cloning process « lobbesblog — September 8, 2019 @ 3:46 a.m.

RSS feed for comments on this post. TrackBack URL

Leave a comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <pre> <q cite=""> <strike> <strong>

Theme and content by Diana Coman