As part of recent refactorings of code in Euloran lands, I took a short break to reconsider as well the testing infrastructure such as it is (read: minimalistic in the extreme) and the whole ever-growing pile of tests for Ada code on both client and serverside¹. At which point I realised that I have never done any write up of anything related to existing support for automated Ada tests and as a result I don't even recall exactly just *what* put me off last time from using any of the supposedly available "testing frameworks" and the like - because yes, I am aware they exist and yes again, I have looked at them before but having failed to document it, the work done then needs at least a partial re-doing now. On the bright side, a fresh look at what is out there never really hurt anything - and in this particular case, it turns out as well that... things haven't changed that much really in the meanwhile or rather they haven't changed ~at all!

While I searched around yet again just in case there might be something new and useful just waiting to be discovered², the situation seems to be quite exactly as I recall it: there is one single open source unit testing library, Ahven, that claims to be simple, small and useful on its own, alongside a bunch of proprietary tools (e.g. GNATTest from Adacore or VectorCAST from Vector - previously known/written about as AdaCAST) that eerily bring to mind the spittoon³ of "it's all in one strand" fame, given their insistence on saddling the user thus caught with a whole additional environment that the vendor currently sells. But never mind that and silly me, of course such saddling of the caught user is not at all and under any circumstances what really happens, no, most emphatically NO! The actual true and real reality must be that those tools and their vendors simply and purely help and empower the user - you - each and every step of the way! Yes, exactly so, those huge tools simply have no choice but to pull in their own large environment just to offer you the great solution of solving the automated testing problem that you started with by importing a brand new set of other problems that are however the "problems that everybody has" and therefore come (for an additional but ongoing price) with known solutions, too! Be that as it may, since I somehow never quite have the same problems that everybody has, I'll pass on those great offers of help and empowerment by means of giving up control over my own development environment and look therefore in more detail only at the other single option left: Ahven.

At first glance, Ahven would seem to take the right approach and thus be for once useful, perhaps: it's supposedly small and simple, following some well-known and usual formats for unit testing, what could possibly be a problem with it, right? Well, for starters, merely trying to get the full documentation to show triggers the weirdest of unexpected WTFs: the documentation needs "compiling" but moreover, it *requires* something called Sphinx. A look as to what the Sphinx might be doing all of a sudden in Ada testing frameworks reveals that the mythical creature stands among the open sores tribes for a text search server with "a variety of text processing features" and SQL support and what-not. Do you still recall that initial claim of Ahven's of being small and simple - is that so small and simple that you need a whole text search server for its documentation? Or is it so small and simple that it forces on to your system a full text search server only for the purpose of displaying a bunch of text files in some readable format? Which of those is exactly your small and simple because neither quite fits into my definition of either small or simple. I suppose I might have gotten it all wrong and it's not small *and* simple but rather in some parts small and in other parts simple, except the small is never simple and the simple is never small - is this perhaps it?

Leaving though the Sphinx aside and keeping it still out of my systems, I am nothing if not truly stubborn and so I just read the docs as they were, as annoying as reading "source" of plain text turns out to be, on the off chance that the code itself is perhaps the promised small and simple. It turns out that the small comes in at about 4.5k lines of code (the docs claim 3k but by wc -l disagrees) and the simple includes a lot of support for all sorts that I never intend to have anywhere near my development environment ever again - windows and junit and xml formats included. Moreover, there are all the possible software patterns one might want in there - whether the problem solved truly requires or justifies the additional complexity of implementing those patterns is never a question or never answered apparently with "no". At which point I realise that I'm probably even lacking enough common ground with the code's author to be able to meaningfully discuss why using this framework would be no asset but a liability to me: their focus is on compatibility with all sorts of tools, environments and anything else, while my focus is on having the smallest and most effective way of structuring and running some automated tests. Basically it strikes me that the whole of "small and simple" is literally a matter of having advanced less (for lack of resources, not for lack of want) on the *same road* as those proprietary frameworks and not in the slightest a matter of traveling on a different road entirely. After all, what different road can there even *be*, right?

Just in case that the above is just my adverse reaction to a heavy dose of software patterns in the morning, I read through Ahven's own tests as well, as a reasonable example of a relatively significant set of tests. I ran them too and I got precisely the sort of output that my 115 lines of code (that's my own "framework" - a single Ada package) spit out just the same. The only part I could add (but I didn't yet need) is a timing of each test - as it's really not something I need for *every* test, I left it for now up to individual tests to time the code they run, where relevant.

Overall, I suppose it may very well be that my code base is just not yet big enough to justify anything more complex than my tiny "testing framework" that somehow manages to report test results uniformely and even grouped neatly where desired. This may change indeed and so my 115 lines of code supporting test writing and running may very well grow gradually too, as required. For what it's worth, I don't even necessarily rule out the introduction of some patterns in there at some point if and when there is some benefit to it other than ticking some boxes that it follows best practices or it's "as it should be" or whatever else of this sort. The trouble is that those 115 lines of code support already 8k lines of tests and if it is to grow to 4.5k, I expect it to do way, way more than just add timings for every test too! As the whole code base stands currently, I rather suspect that I'll trim down the non-testing code some more before even considering any expansion of the test-support into thousands and thousands of lines of code. Then again, I keep not having those same problems that everybody has - somehow!

1. For the curious, there are currently about 4k loc of tests on each side (client and server), so about 8k total. And no, they are not mostly duplicates of one another because even the "common protocol part" is truly common only at the very basic level of packing/unpacking of messages really - as soon as one moves even a tiny bit higher up, the requirements of server vs client are so different as to pull the design and implementation of the solution into significantly different directions. ↩

2. I'm ever the optimist and no cure has worked so far, despite plenty attempts. ↩

3. Archived. ↩

Category: Coding, Open Sores

Comments feed: RSS 2.0