Notes on Computer Graphics: Formats, Exporters and Blendering

January 14th, 2020 by Diana Coman

Blender is true to its name at least with respect to forever mixing up1 its API so no exporter you ever write can work reliably for more than one frozen Blender version. I had already some experience with this back in 2017 when I was still naively thinking that it really is the lack of direct path from Blender to CS that keeps all those supposedly eager artists from contributing anything at all to Eulora's client. At the time the plan was to pick one Blender version and actually support that fully. Meanwhile, experience informed that the whole pile is rotten in all the places where one cares to look, from a weird sort of anti-management at the top to a full embrace of Python's own wrong choices, ant-like overspecialisation and overfitting of artists and the resulting impossibility of any productive work coming through meaningful interaction from those quarters. And given the stark choice at the beginning of this year together with the return to Eulora's 0.1.2b client as part of a wider return to pragmatism, my new take on Blender's blendering is to simply look at potential options for setting out one *working* path for *extraction* of assets to the extent that this is even feasible, more like a way of *interfacing* at most, rather than supporting anything anymore2.

Taking the simpler route of interfacing and extracting rather than supporting anything, I had a look at existing graphics formats and existing exporters for various tools in which the overspecialised merrily toil at one thing or another. There are of course the specific formats of each of those tools (e.g. Blender has .blend3, 3D Studio Max has .3ds, Unity has .unity, Unreal has .unreal), as well as various attempts of all sorts to define some "standard" format. For all those standardisation attempts though, the end result really seems to be more of a "everyone makes their own format" than any sort of standard so that there's no lack of formats to look at, if one really wants to review the whole mess. For instance and to go through only a brief list: .x3d and .x (vrml) are xml-based and supported by the Web3D Consortium so potentially with working exporters perhaps although unclear if fit for purpose (and spec is quite large since it wants to supports all sorts); .ply focuses on one object at a time described as a collection of vertices, faces and corresponding properties; .dae builds up on the older collada format and it's xml-based again without any clear gain compared to .x3d; .obj supports geometry only; .dxf seems mostly fit for CAD systems and it has by now so many versions since its beginning in 1982 that it's even dubious to call it "one" format really.

At any rate, in addition to the above, I further looked at quite a few other formats, from the ISO/IEC 86321-1:1999 standard (revised and confirmed in 2016) that is better known as CGM4 to the binary, proprietary and quite widespread5 FBX6, IGES7 and its contenders (e.g. sat, catia, step, parasolid), as well as the newer OpenGEX 2.08 that seems at least to have been designed out of practical necessity combined with frustrating experience with the state of Collada and glTF. Can't beat such root causes as far as I can see it and the result is quite comprehensible and sanely documented too with some plugins provided for 3D Studio Max, Maya and Blender. So off I went and tested the Blender plugin, what's there to lose for some tests anyway?

To start with and since I was getting into Blender again, I had a more systematic look this time at the various versions of Blender9 & Python: it turns out that last Blender version using Python 2.x is Blender 2.49b that is also the last version of Blender that ships without its own Python so that it uses (rather: attempts to use) whatever is installed locally; from Blender 2.5 on, it's Python 3.x that comes also packed within the .tar.gz10 so at least relatively contained aka without requiring further infection of your environment11 with an actual installation of Python 3.x. Summarising, the current status of Blender & Python versions12 that I looked at13:

  1. Blender 2.49b - this is the last version of Blender that ships without Python bundled in and expects otherwise Python 2.6 installed on the machine where you attempt to run it. While you can go the "purist" way and work with this because Python 2.x, I don't quite see much point to this stance here since anyway there's plenty of other unknown stuff you are getting in there just by not compiling Blender from source for instance. And so might as well just stand the thing on a toilet box, use whatever version of Blender works best for your graphical needs, export the result and then consider it done. In fewer words: use Blender as if it were the enemy, because it is.
  2. Blender 2.5Alpha (20MB) - this is the very first version of Blender that ships with Python bundled in (3.1.1) and that starts the "will work with Python 3.x ONLY" current cycle.
  3. Blender 2.62 (44MB) - comes with Python 3.2.2
  4. Blender 2.66 (64MB) - comes with Python 3.3.0. This was the version I considered back in 2017, mainly because there was the cal3d exporter that sort of worked with it. From a user point of view, the jump from 2.5 to 2.6 series is directly visible, the whole damned gui of Blender changes and supposedly there is some useful functionality added too for drawing and rigging.
  5. Blender 2.72 (87MB) - comes with Python 3.4.0. This is the version for which the OpenGEX current exporter was written and as such, the one I looked at in more detail, see below.
  6. Blender 2.79 (137MB) - this already failed to work on my test machine because it wanted a newer GLIBC! On my CentOS 6.10 machine14 for graphics and all that jazz, ldd --version reports GNU libc 2.12 while Blender 2.79 looks for 2.14, 2.15 or 2.17. It turns out that it's the GLIBC version the limit in the end for what version of Blender I even try out, what Python.
  7. Blender 2.81a (161MB) - this is currently the latest version of Blender, hence why it made this list. Of course it fails just like 2.79 above, as it wants the same newer versions of GLIBC.

Going further with the practical tests, I settled on Blender 2.72 since that was the one for which the OpenGEX exporter was made to start with and I don't see all that much to be gained by going back to Blender 2.66 or something since Python will still be 3.x and otherwise what's the big difference anyway. The exporter is simply this file and it installed without error from within Blender, quote my notes: File->User Preferences->Addon->Install from File + navigate and select the .py file; it goes into Import-Export and it NEEDS to be ticked to activate.

To see the exporter in action, I had first an attempt to model some basic character as a way to know exactly what is in there before looking afterwards at what gets exported. So I followed some tutorials for Blender and I even got to obtain a sort of head of something but it bored me out of my skull and with such intensity as I haven't experienced in quite a while, so that I exercised more my self-control than anything else. While most of the issue really is the activity itself, there was the added and steadily increasing annoyance at what seems to be the "blender-approach" of basically having to remember a set of arbitrary shortcuts for doing this and doing that15.

Given that the point was after all to test the exporter rather than my full patience to the extremes, I left my "model" in its half-made state and simply downloaded some .blend files that are apparently abundant under "creative commons, personal use" only with all the bells and whistles that one might wish for. It kind of boggles the mind that the same people who are not interested apparently in doing some work for gaining something are nevertheless happy in such numbers to do possibly even more work just as long as it's branded "for personal use". Anyways, confronted with a whole set of animated characters looking quite nicely even and ranging from furry spiders to aliens, I just picked one at random, downloaded the .blend file that weighs in at 34MB - it has a ton of polygons too so not exactly a low-poly shape at all - and opened it up in Blender (it reports 171 bones in this alien), obtaining a view of both alien and helpful notes of the author, Dennis Haupt, on the side:

Some playing around with the model revealed among other surprises that the alien is rather hollow inside although nicely textured nevertheless so supposedly that's just how alien overgrown cockroaches develop, with the teeth visible through their bottoms, let me illustrate with a view from behind the thing:

Hollow or not, exporting it went quite smoothly: a simple File->Export->OpenGEX + selecting the location for the output file resulted in one single file that proved quite readable as a description of the "scene" but at the same time missing entirely all sorts (animation most obviously). This was not entirely unexpected but more of a test - there's this annoying thing that what the plugin exports depends on the "scene" that you have selected to see in Blender16 and since the default scene for this Alien was not the one containing the animations as such, they were not exported.

For take 2 of the export test, I switched the scene to "Baked Actions"17 and run the exporter again for what proved to be a slow and painful ride: it took some 30 minutes during which Blender was unresponsive as the exporter worked and at some point the whole system was unresponsive too for a while. At any rate, the result of this take 2 was another .ogex file weighing however 41MB . A quick look at it revealed the sort of expected sparse values thing: all the bones and materials and animations but with lots and lots of 0 values, what can one do.

Since I did 2 exports, might as well do the 3rd one on the remaining scene, namely the one containing "the animated model for editing". This took about 5 minutes although it still blocked Blender for most of that time. The result was yet another .ogex file ~8MB in size. Bones, materials and animations seemed to be still in place but more condensed - presumably the animations here contain only the various poses as opposed to intermediate steps or at least this is my current understanding of it all as I haven't sunk in that much time to fully figure it out since it's not yet even clear if this is going to be useful at all going ahead. There was no error reported on any of the 3 runs18 and the .ogex files as far as I could see appear well formatted and complete so I'd class the exporter as working - as in it does what it knows how to do, although the extent to which that is useful depends of course on the as-yet-not-fully-specified what one wants it to do exactly.

Being at testing stage, I went ahead and attempted to export the same alien with the x3d exporter that Blender ships with. This was quick and resulted in all the usual glory of an xml file but exported only the lights and transforms from the scene that supposedly contains "the animated model for editing". So yeah, the result of exporting even from the same scene will of course depend on each specific implementation of an exporter, not like there's some way to standardise that either.

In addition to the above, my notes mention a few tests with some glTF exporters but those failed with some complaints about materials not set as it expected and otherwise a final error spew in the console. It still produced as output a few .png files for textures but I didn't really sink more time into looking in any more detail at this. Moreover, it would seem that one can just get the same assets directly in something other than .blend format if going for the formats for which Blender supposedly ships directly with exporters. So I just weighed the files19 provided for various formats of this same alien directly from where I downloaded the .blend file though be warned that a comparison is not straightforward here since each "export" will contain different stuff really:

  1. .dae (3.32MB .zip)
  2. .x (1.53MB .rar)
  3. .fbx (24.7MB .zip)
  4. .unitypackage (31.46MB .zip)
  5. .3ds (0.7MB .zip)
  6. .obj (1.11MB .zip)
  7. .dxf (2.78MB .zip)
  8. .blend (33.65MB .zip)

To conclude this adventure with Blender, exporters and various graphics formats, I'll only add that the OpenGEX format is at least well structured20, easy to follow even with the naked eye and otherwise apparently having a working exporter for Blender 2.72 if not for anything else (I doubt it's really *possible* to have an exporter working properly for more than one version of Blender unless literally changing it to accommodate each time something else gets blenderised yet again).

Looking at the above from the opposite side though, it could equally well be argued perhaps that one can then go directly for extracting stuff out of the .blend format, relying on that metadata that ensures compatibility across various Blender versions and bypassing the Blender environment and its Python mess altogether. After all, if it gets to cutting idiotic knots, might as well go for a more crucial knot and be done with it. Just use a bigger and sharper sword if you get into sword wielding anyway, right?

  1. Blendering perhaps? I think I'll call this blendering, yeah. As in: you are totally blendering your life putting up with that shit. 

  2. I guess the whole experience of those past years can be easily described as nothing more than a stream of repeated rejections (if in different forms and shapes, sure) of all and any attempts to offer support. So fine, support is not wanted, I finally get it. 

  3. As a side note, this is a "format" in the true blendering style since the structure of a .blend file is quite specific to the version of Blender from which it was exported; supposedly this is mitigated through the addition of metadata in the file to maintain compatibility between files imported/exported on different versions of Blender. 

  4. Computer graphics metafile. The whole thing is not only HUGE as specifications go - the darned spec has 470 pages and the file itself weighs 3.4MB archived, do you hear me? - but also confusing to the point that ISO/IEC itself came afterwards with "Minimum Recommended Capabilities" since nobody would ever implement that full behemoth anyway. Even so, the result is that various implementations will choose various bits and parts resulting in ...incompatibilities of course, what else. Isn't this such a useful standard? The type of "standard" that one could not imagine if not for all previous experience with similar glorious achievements in this vein really. 

  5. It's ok, ants don't mind it if they are owned, what's it to them? 

  6. FBX supposedly stands for filmbox; it's been owned by Autodesk since 2006. 

  7. International Graphics Exchange Specification - this is meant and used mainly for /from CAD systems and as such it's perhaps not terrible but not all that suited for S.MG's current needs either; for one thing there is simply the difference of domain interest (e.g. lacking animation support really) and for another thing it focuses on surface definition and rather ignores most else, volume properties included. 

  8. Eric Lengyel, 2017 

  9. Pre-compiled versions as downloaded from the release repository. I really did not see the point in sinking now whatever many days into getting all sorts of Blender versions to compile locally since no, it's not that straightforward, of course not. Currently Blender claims that they will maintain forever this repository with all previous versions always available. 

  10. Do note how the archive keeps growing and only growing from version to version! 

  11. If you are running Blender on it, to my mind it's already quite infected since you have to have the full graphics stack and all that so not exactly the cleanest of computers anyway. 

  12. While the console within Blender in principle reports the version of Python at the top, I still did the import sys followed by print (sys.version) to find out what is reported as Python version. 

  13. At least if you already have a machine for testing, it's quite painless to have as many different versions of Blender as you want: just keep each of them in its own dir and they are quite self-contained so you can even run several of them at the same time without any clash (I ended up with 3 running at some point mainly because I had forgotten to close some before starting the others.). 

  14. GCC is 4.9.4, Python is 2.6.6 

  15. As written in my notes: Steps for mirror modifier:
    zoom; tab to go to "edit mode" (instead of object mode); A to select all faces; w key to get the full menu -> subdivide (+ change in the menu on the left-hand side toolbar the number of subdivides ie to get 2 or 3 etc); !!!needs face-select from the toolbar at the bottom of the screen to show those dots and allow select of divisions /faces; shift for multiselect; to switch perspective, either numpad 5 for ortho or from bottom left "view" -> view persp ortho; with press s -> scale and can sort-of extrude the cube into a weird sphere; 1 numpad for front view; loop cut from left-handside bar but click + right click to have it EXACTLY in place; there is "limit selection to visible" on bottom bar (to select/deselect); b key for "box select"; x to delete (and can do it in several "modes"); modifiers are UNDER THE WRENCH TAB in properties window (aka right hand side, gah); the "apply" button makes the modifier permanent; NEED to check "clipping" in the mirror modifier to avoid holes and the like. 

  16. This exclusive focus on "scenes" in the CG world drives me nuts. It's all scene this and scene that, can't have just the full damned thing to use otherwise wherever and however one wants, it has to be directly preset just-so. 

  17. Note that those names are given by the model's author, they are not standard in any way, so there is effectively no way of knowing for each model just what scene is the one containing what you are after. 

  18. There were though 2 warnings namely that "region type 5 missing in name:File" and "region type 6 missing in - name:File" but those didn't seem to break in any way the exported file as such. 

  19. Note that they archived mainly for fun because there is very little the compression does for those formats. 

  20. It comes with a simple grammar too as well as an even simpler, more general underlying format, OpenDDL -Open Data Description Language- described in an Appendix. 

Comments feed: RSS 2.0

6 Responses to “Notes on Computer Graphics: Formats, Exporters and Blendering”

  1. > to simply look at potential options for setting out one *working* path for *extraction* of assets to the extent that this is even feasible

    Yup. Liberation of whatever's still usable from the moron swamp stands as the only viable avenue of interaction with the moron swamp.

  2. Diana Coman says:

    Myeah. Just why so many swamps around really.

  3. hanbot says:

    I lol'ed at the shortcut/procedural notes footnote --indeed much like photoshop and pretty much every other "art program" I've ever used, blender feels more like context memorization than item creation, and it's towards the finish line with the "scenes" and "baking" and whatnot that it really became a useless headache for me. One hundred seventy one bones! Holy god....

    Thanks especially for the version work --it's nice to know there's at least a concrete reason for a specific evil.

    I'm not entirely sure I still have my old character model, my poor documentation and frequent moving about making such determination a considerable dig, but I'm curious if 2.66-generated files would be meaningfully eaten by 2.72, and furthermore whether I can actually export something usable with a view to learning how to contribute chars/maybe help guide others to contribute. I've been reading along with the ongoing development discussions, though, and it's not really clear to me that either of these pursuits would be at all useful. So, would they?

  4. Diana Coman says:

    According to Blender itself, any .blend file should work from 2.66 to 2.72 (they claim they always preserve backwards compatibility). So far I haven't ever experienced a .blend file that failed to open in whatever version of Blender I happened to try it on but I haven't specifically checked this claim. So at the very least for that, it can't hurt to see if indeed a 2.66 .blend opens fine and complete in 2.72.

    Re longer term plan, there aren't really any options we have so Blender is back on the table for lack of replacement. It's not yet fully clear whether it'll be worth to update that converter / use that exporter / just write a converter directly for .blend files and atm it's kind of low priority since the main thing is to have the new version of the client working properly and able to download files. But I'd say it's really not the exporter part that should block/stop anything, at the very least having simply the skeleton out should be simple enough (and should work with either the 2.66 converter to cal3d or anything else really). Maybe ask me specifically if/when you get stuck on something and/or when you want to give this part a go?

  5. I don't think I want any republican hands touching the insanity of de-novo creation of blender files, at least not until we've clearly exhausted both 1. the automated processing of already created material ; 2. the systematic processing of the existing unaware hordes with no possible utility past by-hand parameter fiddling ; 3. the systematic working out of independent, untermenschen-driven efforts at systematization. This is a field whence marginals a la lengyel or katauri might spring into novitiate (ie, 3 above) and wherein 2018 nicoles may be applied such that they're applied towards something while the outer layers of mange are abraded off (ie, 2 above).

    So, if you wanna do something here, do 1.

  6. hanbot says:

    Gotcha, thanks.

Leave a Reply