Notes on Computer Graphics - Transfer vs. Display

January 4th, 2020 by Diana Coman

Some traveled path splits at times into such directions that one gets to wonder all over again what choice really means. My most recent experience with this was the "choice" that I got to ponder on during those past few days falling on either side of the past and present year fence:

mircea_popescu: diana_coman, the way this coming to a head is working out in my head is as follows : you have practically speaking the option to either a) go trawl the entire internet, drag out ~everything~ that's conceivably useful (submit expenses report for stuff that's behind reasonable paywalls i guess) and then we systematize the pile ; or else write a possible-lifeform-generation machine and see what you want it to save.

To start with, it seems to me that the option a) is nothing more than "filter some sense out of ALL of this here Internet" while option b) is nothing more than "filter some sense out of ALL of this here cube of pixels." So which shall it be, filtering the swamps and the rubbish dumps or filtering the medium and the space of all possibilities itself? Do you think there's still something quite fundamental to extract out of pseudo-random people's pseudo-random walks over this space or do you bank it all on harnessing the machine to walk the unyielding space for the sparse usefulness instead?1 Do note that in both cases, time remains limited, effort will be spent regardless of the result and no guarantee can be sanely provided upfront as to any exact outcome really.

Since the above is perhaps not yet quite enough of a tough choice for your taste, let me add to it the warnings from the logs, first for option a) and then for option b) :

mircea_popescu: but this aside : our position is remarkably vulnerable, because of the countless ways in which people are morons, from the pure bobeckistan of blender (and let's not forget how python got indexed in the first place) to a very thick stripperweb-style tarabostes idiocy.
mircea_popescu: do you even recall how many "artists" i commissioned to date whose idea of "commission" and "artist" was... take some money and RUN!!!!!
mircea_popescu: gotta be at least a dozen, including both the expert who made one splash screen and the contest winning kid on tardstalk who made the other.
diana_coman: I didn't keep track but I do recall at least a few off the top of my head, yes.
mircea_popescu: anyway, i'm not about to go paying idiots a whole lotta money so they're a little bit less idiotic, what the fuck.

mircea_popescu: now, the risk with choosing b is that it can readily turn into the grave ; in the hands of any being an engineer that'd be exactly the necessary outcome. it can take forever, yes ?
diana_coman: certainly; and part of why I'm not all-that-enthusiastic about it; otoh my current experience with ~equivalent of a) makes me also-not-that-enthusiastic; so I'm looking at "choice".

One way to go about choosing one of a) or b) above is to try the "easier" option before the harder one: since supposedly a) is perhaps more clearly/lower bounded in terms of effort to spend to complete (with even some guarantees of pretty pictures) and therefore less of a potential grave to walk in merrily just like that, one should go with it first anyway, for completeness and peace of mind if for no other reason. I can see the appeal of having this done (as in: in the past, done and dusted) but I have to admit that I can't summon much of an appetite for it as stated, for reasons that include past experience with and knowledge of even a small-scale trial of a), artists in this field, existing tools for computer graphics and existing methods and approaches for the same. On the other hand, I could easily see my potential preference for b) as just a matter of having so far perhaps less experience with the states of art & practice in this area, since otherwise the current definition of "AI" as little more than overfitted statistical models with a lot of parameters2 seems to me to mirror quite closely the definition of computer graphics as little more than overfitted pixel models with a lot of parameters.

While the above similarity of approach between modern "AI" and modern "CG"3 doesn't spell better for either a) or b) options, it still suggests to me that the correct path might be found perhaps by taking a step back and re-evaluating the full situation: both that of what SMG actually needs and that of the wider CG and AI situation. So back I walked those paths for a while and it seems to me that the still-sane point of theory and practice might be just about advanced enough to give us some better way to start that ratchet in this area. Let me though elaborate a bit as a first step on what is my current trouble with the seemingly "easier" option a) as it was stated - or as I interpret the current statement, namely trawling the Internet for all "art" artefacts at first, to then gradually systematize and filter/bring them to a sane representation.

First of all, I am quite adverse to importing now - even through "systematization of existing outputs" - any of the CG rot really, mainly because I think it IS rot and as such quite negative in value to even consider in the first place. What exact sane knowledge or structure am I supposed to extract and systematize out of the semi-random outputs of pixel-futzing? And why exactly should the time be sunk into attempting this anyway, since there isn't any hope anymore of those same pixel-futzers actually doing any useful work? Sure, there are any number of swamps available and they are all multi-layered too so one can sink in this swamp and the next swamp and the ever after next swamp as much of themselves as they like, that's the whole "strategy" of the horde to matter anyway - sink and drown anyone else. But really now, wasn't there (and isn't there still) enough of this already so far?

Second, even assuming that it is worth sinking all that time and effort into extracting whatever sanity one can extract out of the swamps to define a universal model for graphics, it dawns on me that there might be an important distinction that I failed to make explicit so far: what we need currently is a universal model for graphics *data transfer* rather than for graphics full stop. This is important because going at it from graphics artefacts such as they are and from all the theory such as it is, the implicit angle is quite always display rather than transfer and this angle brings with it a lot of display-specific complexity that should be solely a client concern, nothing to do directly with SMG or the data communication protocol. While the client (if a graphical client) is necessarily concerned with displaying graphics and therefore with all the mess this brings currently, this concern does not directly map - nor do I think it should directly map - onto that of transferring graphics over the network. And more importantly perhaps, it doesn't even have to map to a single specific model since there can be as much variety as each client desires with respect to graphics display. The client's display concern is not really directly SMG's concern: from the point of view of the SMG data communication protocol, all that is needed is an efficient way of transferring everything, graphics included - there's no absolute need though to marry this to one or another display approach. Moreover, I think it is inherently a bad idea to even contemplate such marriage precisely because it adds unnecessary complexity where it doesn't belong: let the protocol model explicitly only the structural data to be encoded for transfer and then let the client side deal with its own choice of specific display mess, figure out and carry out whatever conversions it needs to use this graphics data in whatever way it requires.

Focusing on graphics transfer rather than display seems to me to reduce significantly the trouble at least at this stage, since the whole of rasterization for instance is of no direct concern4. Moreover, the only real choice for both 2D and 3D shapes seems to me to be vectorial representation - as far as I'm aware, there are even approaches to modeling of shapes through combining and interpolating vector graphics as opposed to "intuitive sculpting" so I'm not exactly sure that there's any loss choosing this option anyway. As for the rest of "graphics" characteristics such as textures, materials, animations and light, I think that one is better served by investigating existing frameworks5 that capture the key structural characteristics, letting the actual representation be a consequence of those characteristics in whatever way each specific client or artist decides.

In the light of all of the above, I'm not quite convinced there's much to be expected as useful information from filtering either the Internet swamps (aka a) option) or the space of possibilities (aka b) option). There might still be perhaps some gain from a) if one treats it strictly as a dump of stuff to be beaten into the shape already defined and decided upon. Even in such case, I think I'd rather focus on exploring the space of vectorial art - mainly because it holds to my eye more potential of sanity since it's at least not the sort of rotten approach (think Python!) that directly maps onto useless or worse.

As to the b) option, I think even this one would benefit again from going at it not as a means to extract the framework for graphics transfer but precisely the opposite, as a result and as informed by the chosen framework(s). There's more that could be expanded and looked into in this direction too, for sure, but it seems to me that it can still wait for now - at least if I am indeed right about the actual gains of making this crucial distinction between the scope and focus of graphics transfer vs graphics full stop.

  1. Should I go all the way with the restatement of this "choice" so it ends up as a choice between "aggregate of bipeds" vs "aggregate of bits"? 

  2. See Marvin Minksy and Noam Chomsky on it, as they certainly knew what they were talking about. If you have the stomach for it, read also Norvig's stand for the other side so that you have the chance to figure out where you belong, if nothing else. In any case, for the task at hand, let me cite here this tidbit that hits one big nail on the head, from Chomsky's answer to Pinker's question at the Brains, Minds and Machines symposium in 2011: "There is a succ- notion of success which has developed in uh computational cognitive science in recent years which I think is novel in the history of science. It interprets success as uh approximating unanalyzed data." 

  3. Computer Graphics 

  4. It's up to the client entirely and it can be non-standard for all SMG cares. If desired, I don't even see a problem as such to provide for a choice of "raster profile" that comes as whatever collection of files that author thinks are needed. Not like the protocol doesn't support file transfer otherwise, regardless of the file's format. 

  5. Perhaps looking at Laban's framework for describing movement for instance? 

Comments feed: RSS 2.0

9 Responses to “Notes on Computer Graphics - Transfer vs. Display”

  1. This was quite a pleasure to read (and not merely, nor even mainly, for the sadness of the context it came to rescue the reader from).

    let me add to it the warnings from the logs

    Let me add to your adding an archival copy, because the links on the terms are imo very helpful.

    since supposedly a) is perhaps more clearly/lower bounded in terms of effort to spend to complete (with even some guarantees of pretty pictures) and therefore less of a potential grave to walk in merrily just like that, one should go with it first anyway, for completeness and peace of mind if for no other reason

    I confess this is pretty much my heuristic here, yes.

    the semi-random outputs of pixel-futzing?

    The problem is that it's not actually random. They work, the problem is that they work in narrow contexts ad-hoc defined (if we can stretch the term thusly). But every single game has in fact graphics, and lots of games have nice graphics. Pornhub&co are chock-full of quite workable cgi models doing remarkably stupid, meaningless shit, impelled by a dedication worthy of higher goals. Second life's probably yottabytes worth of the same damned tiara re-done over and over and over again.

    since there isn't any hope anymore of those same pixel-futzers actually doing any useful work?

    If nothing else, because your kid's gonna be stuck living among [whatever's left of] them.

    what we need currently is a universal model for graphics *data transfer* rather than for graphics full stop.

    I dunno that this point was lost on anyone ; it certainly permeates the design document we've been working off of.

  2. Diana Coman says:

    Thank you!

    Re pixel-futzing yes, not actually random, just...overfitted basically. But precisely because of this overfitting, I don't think they are useful to inform the model in any way really. They may be useful later on, to be converted and used *with* the model, but that's about it as far as I can see.

    What the kid is stuck with though, kind of has to be his problem, can't quite enter into anything as such.

    I dunno that this point was lost on anyone ; it certainly permeates the design document we've been working off of.

    It does. Perhaps I just got for a while too much into the client details to such extent that I needed to explicitly do this split again so as to figure out which parts matter (structural) and which don't (all the display, with shaders and textures and whatnots included).

    Now the question still remains a bit re what to look at as very next step.

  3. Overfitting is the real problem ; it's becoming ever more self-evident that's the real red thread throughout the (seemingly lengthy) list of "what the fuck happened to the world".

    I think the next step may be re-reading said design document, and thinking about it for a day or two. What exactly is missing such that server can't complete ? (I don't mean, "say first item that comes to mind".)

  4. Oh and also, ding me on these on irc ; I do check blogs periodically, but that period's more like weekish, some items could benefit of being in the same-day list.

  5. Diana Coman says:

    I went ahead and investigated some more the less mentioned part of "what is there", namely formats and approaches esp on the vector-not-raster side. Read another ton, got some things that I still have to structure and write up. I should also check those fabled exporters for Blender & various versions.

    Once the above write-up is done, I was planning precisely to have another think on both the doc and the full current state of where we are with everything, to figure out what would be the best next step really.

    Ok, will ding.

  6. Diana Coman says:

    Meanwhile, the path I refused to take at all here has been pursued by the more fashionable approach to AI, perhaps predictably since it fits them so well otherwise, indeed. It's even noted by others too, as one description of the approach reads, quite correctly, 'it doesn't necessarily "understand" their relationship at a high level, but the results can still be stunning and surprising, making inferences and style combinations that seem very intelligent.'

    The first result making waves seems to be called Stable Diffusion, is based on statistical inference (what else is there), trained on hundreds of GPUs and some 5 billion images (the LAION-5B dataset) and it works essentially as a resource-intensive mixer of existing images with the pick and mix roughly directed by text descriptions.

    The results from what I see are quite the usual stuff that AI delivers, namely "anything" (as long as you keep to what it knows and adapt yourself to speak the language it understands).

    There's already an outcry from the artists' side and there are protections and exclusions and rights to be talked about. I expect there will be more of all this, too.

  7. [...] initial pondering of the choice regarding which way to go best when looking for sense in computer graphics (CG) turned quite quickly into a rather concentrated read of ~everything I could find relevant to CG [...]

  8. [...] a hunt to still find somewhere those older and more useful editions of various reference books on Computer Graphics, I shrugged at first at the note attached to one of them - "disc included", it said and I couldn't [...]

  9. [...] and nobody wants to have to wait more than absolutely necessary. Which is all nice and well, except totally at odds with what the rasterised approach to computer graphics requires, namely huge listings of [...]

Leave a Reply