Category Archives: media design

presets and cue building – a beyond basics checklist | TouchDesigner 099

from the facebook help group

Looking for generic advice on how to make a tox loader with cues + transitions, something that is likely a common need for most TD users dealing with a playback situation. I’ve done it for live settings before, but there are a few new pre-requisites this time: a looping playlist, A-B fade-in transitions and cueing. Matthew Ragan‘s state machine article (https://matthewragan.com/…/presets-and-cue-building-touchd…/) is useful, but since things get heavy very quickly, what is the best strategy for pre-loading TOXs while dealing with the processing load of an A to B deck situation?

https://www.facebook.com/groups/touchdesignerhelp/permalink/835733779925957/

I’ve been thinking about this question for a day now, and it’s a hard one. Mostly this is a difficult question as there are lots of moving parts and nuanced pieces that are largely invisible when considering this challenge from the outside. It’s also difficult as general advice is about meta-concepts that are often murkier than they may initially appear. So with that in mind, a few caveats:

  • Some of suggestions below come from experience building and working on distributed systems, some from single server systems. Sometimes those ideas play well together, and sometimes they don’t. Your mileage may vary here, so like any general advice please think through the implications of your decisions before committing to an idea to implement.
  • The ideas are free, but the problems they make won’t be. Any suggestion / solution here is going to come with trade-offs. There are no silver bullets when it comes to solving these challenges – one solution might work for the user with high end hardware but not for cheaper components; another solution may work well across all component types, but have an implementation limit. 
  • I’ll be wrong about some things. The scope of anyone’s knowledge is limited, and the longer I work in ToiuchDesigner (and as a programmer in general) the more I find holes and gaps in my conceptual and computational frames of reference. You might well find that in your hardware configuration my suggestions don’t work, or something I suggest won’t work does. As with all advice, it’s okay to be suspicious.

A General Checklist

Plan… no really, make a Plan and Write it Down

The most crucial part of this process is the planning stage. What you make, and how you think about making it, largely depends on what you want to do and the requirements / expectations that come along with what that looks like. This often means asking a lot of seemingly stupid questions – do I need to support gifs for this tool? what happens if I need to pulse reload a file? what’s the best data structure for this? is it worth building an undo feature? and on and on and on. Write down what you’re up to – make a checklist, or a scribble on a post-it, or create a repo with a readme… doesn’t matter where you do it, just give yourself an outline to follow – otherwise you’ll get lost along or forget the features that were deal breakers.

Data Structures

These aren’t always sexy, but they’re more important than we think at first glance. How you store and recall information in your project – especially when it comes to complex cues  – is going to play a central role in how your solve problems for your endeavor. Consider the following questions:

  • What existing tools do you like – what’s their data structure / solution?
  • How is your data organized – arrays, dictionaries, etc.
  • Do you have a readme to refer back to when you extend your project in the future?
  • Do you have a way to add entries?
  • Do you have a way to recall entries?
  • Do you have a way to update entries?
  • Do you have a way to copy entries?
  • Do you have a validation process in-line to ensure your entries are valid?
  • Do you have a means of externalizing your cues and other config data

Time

Take time to think about… time. Silly as it may seem, how you think about time is especially important when it comes to these kinds of systems. Many of the projects I work on assume that time is streamed to target machines. In this kind of configuration a controller streams time (either as a float or as timecode) to nodes on the network. This ensures that all machines share a clock – a reference to how time is moving. This isn’t perfect and streaming time often relies on physical network connections (save yourself the heartache that comes with wifi here). You can also end up with frame discrepancies of 1-3 frames depending on the network you’re using, and the traffic on it at any given point. That said, time is an essential ingredient I always think about when building playback projects. It’s also worth thinking about how your toxes or sub-components use time.

When possible, I prefer expecting time as an input to my toxes rather than setting up complex time networks inside of them. The considerations here are largely about sync and controlling cooking. CHOPs that do any interpolating almost always cook, which means that downstream ops depending on that CHOP also cook. This makes TOX optimization hard if you’re always including CHOPs with constantly cooking foot-prints. Providing time to a TOX as an expected input makes handling the logic around stopping unnecessary cooking a little easier to navigate. Providing time to your TOX elements also ensures that you’re driving your component in relationship to time provided by your controller.

How you work with time in your TOXes, and in your project in general can’t be understated as something to think carefully about. Whatever you decide in regards to time, just make sure it’s a purposeful decision, not one that catches you off guard.

Identify Your Needs

What are the essential components that you need in modular system. Are you working mostly with loading different geometry types? Different scenes? Different post process effects? There are several different approach you might use depending on what you’re really after here, so it’s good start to really dig into what you’re expecting your project to accomplish. If you’re just after an optimized render system for multiple scenes, you might check out this example.

Understand / Control Component Cooking

When building fx presets I mostly aim to have all of my elements loaded at start so I’m only selecting them during performance. This means that geometry and universal textures are loaded into memory, so changing scenes is really only about scripts that change internal paths. This also means that my expectation of any given TOX that I work on is that its children will have a CPU cook time of less than 0.05ms and preferably 0.0ms when not selected. Getting a firm handle on how cooking propagates in your networks is as close to mandatory as it gets when you want to build high performing module based systems.

Some considerations here are to make sure that you know how the selective cook type on null CHOPs works – there are up and downsides to using this method so make sure you read the wiki carefully.

Exports vs. Expressions is another important consideration here as they can often have an impact on cook time in your networks.

Careful use of Python also falls into this category. Do you have a hip tox that uses a frame start script to run 1000 lines of python? That might kill your performance – so you might need to think through another approach to achieve that effect.

Do you use script CHOPs or SOPs? Make sure that you’re being carefully with how you’re driving their parameters. Python offers an amazing extensible scripting language for Touch, but it’s worth being careful here before you rely too much on these op types cooking every frame.

Even if you’re confident that you understand how cooking works in TouchDesigner, don’t be afraid to question your assumptions here. I often find that how I thought some op behaved is in fact not how it behaves.

Plan for Scale

What’s your scale? Do you need to support an ever expanding number of external effects? Is there a limit in place? How many machines does this need to run on today? What about in 4 months? Obscura is often pushing against boundaries of scale, so when we talk about projects I almost always add a zero after any number of displays or machines that are going to be involved in a project… that way what I’m working on has a chance of being reusable in the future. If you only plan to solve today’s problem, you’ll probably run up against the limits of your solution before very long.

Shared Assets

In some cases developing a place in your project for shared assets will reap huge rewards. What do I mean? You need look no further than TouchDesigner itself to see some of this in practice. In ui/icons you’ll find a large array of moviefile in TOPs that are loaded at start and provide many of the elements that we see when developing in Touch:

icon_library.PNG

icon_library_example.PNG

Rather than loading these files on demand, they’re instead stored in this bin and can be selected into their appropriate / needed locations. Similarly, if your tox files are going to rely on a set of assets that can be centralized, consider what you might do to make that easier on yourself. Loading all of these assets on project start is going to help ensure that you minimize frame drops.

While this example is all textures, they don’t have to be. Do you have a set of model assets or SOPs that you like to use? Load them at start and then select them. Selects exist across all Op types, don’t be afraid to use them. Using shared assets can be a bit of a trudge to set up and think through, but there are often large performance gains to be found here.

Dependencies

Sometimes you have to make something that is dependent on something else. Shared assets are a kind of single example of dependencies – where a given visuals TOX wouldn’t operate correctly in a network that didn’t have our assets TOX as well. Dependencies can be frustrating to use in your project, but they can also impose structure and uniformity around what you build. Chances are the data structure for your cues will also become dependent on external files – that’s all okay. The important consideration here is to think through how these will impact your work and the organization of your project.

Use Extensions

If you haven’t started writing extensions, now is the time to start. Cue building and recalling are well suited for this kind of task, as are any number of challenges that you’re going to find. In the past I’ve used custom extensions for every external TOX. Each module has a Play(state) method where state indicates if it’s on or off. When the module is turned on it sets of a set of scripts to ensure that things are correctly set up, and when it’s turned off it cleans itself up and resets for the next Play() call. This kind of approach may or may not be right for you, but if you find yourself with a module that has all sorts of ops that need to be bypassed or reset when being activated / deactivated this might be the right kind of solution.

Develop a Standard

In that vein, cultivate a standard. Decide that every TOX is going to get 3 toggles and 6 floats as custom pars. Give every op access to your shared assets tox, or to your streamed time… whatever it is, make some rules that your modules need to adhere to across your development pipeline. This lets you standardize how you treat them and will make you all the happier in the future.

That’s all well and good Matt, but I don’t get it – why should my TOXes all have a fixed number of custom pars? Let’s consider building a data structure for cues let’s say that all of our toxes have a different number of custom pars, and they all have different names. Our data structure needs to support all of our possible externals, so we might end up with something like:

{
      "cues": {
           "cue1": {
                "Tox": "Swank",
                "Level_1": 0,
                "Noise": 1,
                "Level3": 4,
                "Blacklvl": 0.75
           },
           "cue2": {
               "Tox": "Curl",
               "Bouncy": 0.775,
               "Curve": 100.0,
               "Augment": 13,
               "Blklvl": 0.75
           },
           "cue3": {
               "Tox": "Boop",
               "Boopness": 0.775
           }
      }
}

That’s a bummer. Looking at this we can tell right away that there might be problems brewing at the circle k – what happens if we mess up our tox loading / targeting and our custom pars can’t get assigned? In this set-up we’ll just fail during execution and get an error… and our TOX won’t load with the correct pars. We could swap this around and include every possible custom par type in our dictionary, only applying the value if it matches a par name, but that means some tricksy python to handle our messy implementation.

What if, instead, all of our custom TOXes had the same number of custom pars, and they shared a name space to the parent? We can rename them to whatever makes sense inside, but in the loading mechanism we’d likely reduce the number of errors we need to consider. That would change the dictionary above into something more like:

{
      "cues": {
           "cue1": {
                "Tox": "Swank",
                "Par1": 0,
                "Par2": 1,
                "Par3": 4,
                "Par4": 0.75
           },
           "cue2": {
               "Tox": "Curl",
               "Par1": 0.775,
               "Par2": 100.0,
               "Par3": 13,
               "Par4": 0.75
           },
           "cue3": {
               "Tox": "Boop",
               "Par1": 0.875,
               "Par2": None,
               "Par3": None,
               "Par4": None
           }
      }
}

Okay, so that’s prettier… So what? If we look back at our lesson on dictionary for loops we’ll remember that the pars() call can significantly reduce the complexity of pushing dictionary items to target pars. Essentially we’re able to store the par name as the key, and the target value as the value in our dictionary we’re just happier all around. That makes our UI a little harder to wrangle, but with some careful planning we can certainly think through how to handle that challenge. Take it or leave it, but a good formal structure around how you handle and think about these things will go a long way.

Cultivate Realistic Expectations

I don’t know that I’ve ever met a community of people with such high standards of performance as TouchDesigner developers. In general we’re a group that wants 60 fps FOREVER (really we want 90, but for now we’ll settle), and when things slow down or we see frame drops be prepared for someone to tell you that you’re doing it all wrong – or that your project is trash.

Waoh is that a high bar.

Lots of things can cause frame drops, and rather than expecting that you’ll never drop below 60, it’s better to think about what your tolerance for drops or stutters is going to be. Loading TOXes on the fly, disabling / enabling containers or bases, loading video without pre-loading, loading complex models, lots of SOP operations, and so on will all cause frame drops – sometimes big, sometimes small. Establishing  your tolerance threshold for these things will help you prioritize your work and architecture. You can also think about where you might hide these behaviors. Maybe you only load a subset of your TOXes for a set – between sets you always fade to black when your new modules get loaded. That way no one can see any frame drops.

The idea here is to incorporate this into your planning process – having a realistic expectation will prevent you from getting frustrated as well, or point out where you need to invest more time and energy in developing your own programming skills.

Separation is a good thing… mostly

Richard’s killer post about optimization in touch has an excellent recommendation – keep your UI separate. This suggestion is HUGE, and it does far more good than you might intentionally imagine.

I’d always suggest keeping the UI on another machine or in a seperate instance. It’s handier and much more scaleable if you need to fork out to other machines. It forces you to be a bit more disciplined and helps you when you need to start putting previz tools etc in. I’ve been very careful to take care of the little details in the ui too such as making sure TOPs scale with the UI (but not using expressions) and making sure that CHOPs are kept to a minimum. Only one type of UI element really needs a CHOP and that’s a slider, sometimes even they don’t need them.

I’m with Richard 100% here on all fronts. That said, be mindful of why and when you’re splitting up your processes. It might be temping to do all of your video handling in one process, that gets passed to process only for rendering 3d, before going to a process that’s for routing and mapping.

Settle down there cattle rustler.

Remember that for all the separating you’re doing, you need strict methodology for how these interchanges work, how you send messages between them, how you debug this kind of distribution, and on and on and on.

There’s a lot of good to be found how you break up parts of your project into other processes, but tread lightly and be thoughtful. Before I do this, I try to ask myself:

  • “What problem am I solving by adding this level of additional complexity?”
  • “Is there another way to solve this problem without an additional process?”
  • “What are the possible problems / issues this might cause?”
  • “Can I test this in a small way before re-factoring the whole project?”

Don’t Forget a Start up Procedures

How your project starts up matters. Regardless of your asset management process it’s important to know what you’re loading at start, and what’s only getting loaded once you need it in touch. Starting in perform mode, there are a number of bits that aren’t going to get loaded until you need them. To that end, if you have a set of shared assets you might consider writing a function to force cook them so they’re ready to be called without any frame drops. Or you might think about a way to automate your start up so you can test to make sure you have all your assets (especially if your dev computer isn’t the same as your performance / installation machine).

Logging and Errors

It’s not much fun to write a logger, but they sure are useful. When you start to chase this kind of project it’ll be important to see where things went wrong. Sometimes the default logging methods aren’t enough, or they happen to fast. A good logging methodology and format can help with that. You’re welcome to make your own, you’re also welcome to use and modify the one I made.

Unit Tests

Measure twice, cut once. When it comes to coding, unit tests are where it’s at. Simple proof of concept complete tests that aren’t baked into your project or code can help you sort out the limitations or capabilities of an idea before you really dig into the process of integrating it into your project. These aren’t always fun to make, but they let you strip down your idea to the bare bones and sort out simple mechanics first.

Build the simplest implementation of the idea. What’s working? What isn’t? What’s highly performant? What’s not? Can you make any educated guesses or speculation about what will cause problems? Give yourself some benchmarks that your test has to prove itself against before you move ahead with integrating it into your project as a solution.

Document

Even though it’s hard – DOCUMENT YOUR CODE. I know that it’s hard, even I have a hard time doing it – but it’s so so so very important to have a documentation strategy for a project like this. Once you start building pieces that depend on a particular message format, or sequence of events, any kind of breadcrumbs you can leave for yourself to find your way back to your original thoughts will be helpful.

Advanced Instancing | Puzzle Pieces | TouchDesigner

Part 1

Core Concepts

  • Instancing geometry
  • Replicators and Clones
  • The Sort SOP
  • Images composed out of component pieces
  • Real time rendering


Part 2

Core Concepts

  • Instanced geometry from pixel data
  • Texture Arrays – the 3D Texture TOP
  • The Sort SOP
  • Images composed out of component pieces
  • Real time rendering

Advanced Instancing | Instancing with the Animation COMP | TouchDesigner

Once upon a time this had an audio track, and then suddenly it didn’t. A fix coming by the end of the week. Sorry for the delay.

Core Concepts

  • Instancing geometry
  • Working with the Animation COMP
  • Building Animation Channels
  • The Shuffle CHOP
  • Real time rendering

Understanding Referencing | TouchDesigner

referencingReferencing is one of the most powerful tools at the programmer’s disposal in TouchDesigner. Referencing creates a direct link between two or more floats or integers. This allows you to link operators that are outside of their respective families – normally you can only connect CHOPs to CHOPs and TOPs to TOPs, but referencing allows you to create connections between nearly any operators. There are a number of way to create these links with references or expressions. In many of the other posts that I’ve written I often write about using expressions and references, but haven’t taken much time to talk in depth about what this is, how how it all works. Let’s change that.

I started thinking about this when I saw a post on the Derivative forum from a new user struggling with understanding what I’d written in some earlier tutorials. Expressions are something that I continue to learn more about, and open up all sorts of opportunities for faster, more streamlined, and more elegant programming. Let’s start by looking at the typical kinds of referencing that you might do on any project. Specifically, let’s look at how we might connect one family of operators to another. In this example we’ll look at connecting a CHOP to a TOP, and all of the different ways we might do that.

all reference typesIn the image above we actually are referencing the same CHOP in four different ways. We can start this by first talking about how we connect two operators from different families. In this example I’m going to use a noise CHOP and a circle TOP. I want to use the sudo random noise from the noise CHOP to drive the vertical position of my circle in my circle TOP. There are a two major ways that we can make this connection: dragging and dropping the CHOP onto the TOP, or writing an expression that connects the two of them. I often opt for writing the expression – I do this because I think it’s good practice, and has helped me better understand the syntax and structure of using expression in references. We’ll take a look at both of these methods.

Let’s start with the drag and drop method. To use the drag and drop method we need the source operator to be viewer active (there are some exceptions, but it’s a good rule of thumb that your source probably needs to be viewer active to do this). We can make an operator viewer active by clicking on the + symbol in the bottom right corner, or by holding down the alt key (holding alt will make all operators viewer active). You can tell an operator is viewer active because the color coded border disappears, and usually a portion of it is highlighted when you mouse over the operator.

viewer activeLet’s take a moment to better understand the anatomy of an operator while we have this example handy. On the left upper corner of the operator we have a few different toggle switches – viewer, clone immune, bypass, and lock. Along the very bottom of our operator we have it’s name (you can make changes to this field), any flags associated with the operator family, and the viewer active toggle. Having a solid sense of the anatomy of your operators becomes increasingly important the longer you work with TouchDesigner.

Alright, now that we know how to toggle our viewer active mode on and off, and know a little more about our operators’ anatomy lets look at how to build a reference. Let’s make a noise CHOP in our network as well as a Circle TOP. With the Noise CHOP viewer active, click on the name “chan1” in the viewer, and drag it to the Y parameter of the Circle TOP (I’ve made my circle a little smaller, to make this easier to see):

exportAs you do this you should see a drop down menu appear, let’s select “Export CHOP” from the list. You should now see the Y position (or center 2) changed to a green color. You should also see some text that shows up as well. Here’s a closer look at just the paramater we’ve changed in the circle TOP:

export CHOPLooking closer we can see that the text reads: noise_active:chan1. Great, but what does that mean?! Well, if we take a closer look at our Noise CHOP we can see that I changed the name of that operator to “noise_active” – we also see that the name of our noise channel is “chan1”. If we were to abstract what we’re seeing in the export language we might write something like this:

source_operator_name:source_channel_name

Exporting is a fine way to connect operators, but it’s not my personal favorite. I say this because exporting creates a locked relationship. Once you’ve done this you can’t change the text in the target operator. Exporting creates a much more permanent relationship between your operators. To remove the export you’ll need to right click on the parameter filed and select “remove export.” Surely there’s a better way to connect operators?!

In fact, there are still three more ways to connect operators. Taking a closer look at the drop down menu that appears when we use the drag and drop method we see the following that there are several Methods:

current valueAll of these are ways that we can connect two operators together, and many of them yield the same results, so what gives:

  • Export CHOP – we’ve already seen this method, and we know that one of its limitations is that it creates a fixed relationship between two operators. This is excellent for creating something more finished for locked in nature.
  • Text – Text exports the pathway to a particular channel.
  • Current CHOP Value – this exports the value of the operator in question at the precise moment that you drag and drop. Rather than a continually updating value this is just a single float or integer.
  • Relative CHOP Reference – the relative reference exports a python expression that points to the operator being referenced. A relative makes for easy cutting and pasting so long as the network hierarchy relationships remain constant between operators.
  • CHOP Reference (sometimes called Absolute Reference) – the absolute reference also creates a python expression pointing to an operator. The difference here is that it includes the precise pathway to the operator in question making cutting and pasting a bit more frustrating.

For now we’ll take a pass on the “Text” and “Current CHOP Value” options as these have more limited uses. Let’s now take a close look at our Relative and Absolute Reference options.

Relative Referencing

Let’s go ahead and make another circle in our network, and this time let’s create a relative reference between our noise CHOP and our circle TOP.

relative referenceTaking a closer look at our expression we can see that it reads:

op(“noise1”)[“chan1”]

Okay, what does this mean? Lets start by looking at the syntax of this expression. First we can see that we’re looking for an operator. We know this because our expression starts with op(). Next comes the name of the operator in quotation marks. As an important note, Python doesn’t care if you use double quotes or single quotes so long as they match. This means that “noise1” and ‘noise1’ are both equal and produce the same results; ‘noise1″ or “noise1′ however will not work. Finally we see the name of the channel in question in brackets and in quotes – [“chan1”]. This means that our syntax looks something like operator(“exact_name_of_operator”)[“desired_channel”]. Okay, let’s look at another example to make sure we have a firm understanding of how relative referencing works.

Let’s make a Constant CHOP. Let’s name change the name of the constant to “fruits” and name the first three channels, “apple” “pear” and “lemon”. You should have something that looks like this:

fruit constantAlright, now lets add a circle TOP to our network. This time, instead of using the drag and drop method we’ll write out the Python expression to create a reference to our constant. We’ll start by referencing our apple channel. This means our expression is going to be:

op(‘fruits’)[‘apple’]

You can write this directly in the expression directly in the target parameter field for the target operator. When you’re done you should have something like this:

fruits topIf you drag the slider in the constant CHOP to the right, you should now see the circle move up in the viewer. So we’ve successfully connected our circle TOP to the apple channel, why is this any better than just exporting? Well, let’s say that for whatever reason you change your mind while you’re programming and decide that instead you’d prefer for the circle TOP to be connected to the “pear” channel? Written as an expression we can make that change simply by deleting “apple” and replacing it with “pear” or “lemon”. Our expressions then would be:

op(‘fruits’)[‘pear’]
op(‘fruits’)[‘lemon’]

Additionally, if we’ve written a reference as an expression we write some math directly into our reference. We might, for example, only want half of the value coming out of the apple channel. In this case we’d write the expression:

op(‘fruits’)[‘apple’] * 0.5

This would divide ever value in half, changing our scaling from 0 – 1 to 0 – 0.5 instead. We can also use this method to multiply a channel by another channel. For example maybe we want to create a relationship between two different channels from our fruits constant. We might write the expression:

op(‘fruits’)[‘apple’] * op(‘fruits’)[‘lemon’]

You could just as easily do this with a Math CHOP, but you might find that just writing the expression is faster, simpler, or more tidy.

Before we move on, there are two more modifiers that we need to know when writing relative references:

./
../

What on earth are these all about? Well, these are handy directory pointers. At some point you will surely end up wanting to reference an operator that is another part of your network – a control panel, a material, a slider, you name it – if you program in Touch long enough, you’re gonna need these. So what do they mean:

./  – this modifier means the network inside of me
../  – this modifier means in the network above me

If you’re scratching your head, that’s okay. Let’s look at an example. Let’s say that we have a Geometry component, and inside of it we have placed a material – a constant that’s red. A relative reference for that material would be “./constant1”. This means, look inside of me for the material called “constant1”.

constant1So how does ../ work then? Imagine that you’d like the alpha of the constant in our geo to be connected to the noise in our parent network. Here we can write a reference that looks like this:

op(‘../noise1′)[‘chan1’]

Here’s what that would look like:

dot dot slash

Absolute References

Now that we understand what relative references are, what are absolute References? Unlike relative references, absolute references require the entire network path to an operator. In the case of our noise and circle example, that means that our reference looks like this:

op(“references/noise1”)[“chan1”]

absoluteAbsolute references require that you know exactly where in your network you’re referencing an operator from, because you have to use the entire network path. That sounds like a pain, so why use them? Well, let’s imagine that you’re building a complex program and you’re trying to be as tidy and organized as possible. You might build a large change of your user interface in a single location. This means that all of the sliders, buttons, and menus that are being called all live in the same container. In this case, using an absolute reference makes good programming sense. Relative references will leave you constantly trying to figure out how many ../ to use when referencing your user interface. Absolute calls don’t require this, as they point to a very specific place in the network. You can even simplify this by making sure that all of your buttons and sliders are joined with a merge CHOP.

To get a better sense of how this works, download the example .toe file and look at the last example that’s driven by sliders that control the level TOP. There’s lots more to learn about expressions, but practicing your references will help you begin to understand the syntax and logic of how they work.

Download this example toe to learn and explore some more – referencing toe

 

 

 

Let’s Make this Table Data Move | TouchDesigner

14285147532_9f75f63f29_nWorking with live streaming data is about as good as it gets when it comes to programming – this is especially true if you’re working on a project that looks to create a recognizable relationship between live data and media. What, then, is a person to do when placed in a situation without access to a live source of data? Whatever the project, the best way to tackle this problem is to find a source of prerecorded information. If you’re working on something like motion tracking, using a pre-recorded video is an excellent solution to this problem. What about sensors that aren’t image based? What if I’m dealing with a series of floats? What happens if those floats just come to me in a table? How can I take a series of recorded data points that live in an text file and make them move? That’s exactly one of the problems that came up for me recently, and I’ve got a handy trick that will make it easy to work with a data set in a table as though it were streaming into your TouchDesigner network live.

To get started, we need a file with some data in it. I made a quick spreadsheet with two columns. One starts at 0.01 and goes up to 1, awhile the other column starts at .99 and counts down to 0. If you’re following along, you can download that text file here (tabledata). In broad terms, what were going to make is a simple network of operators that moves through a table, pulling one row of data at a time, and then converting that table information into CHOP data. We can see where we’re headed if we look at our whole completed network:

whole networkSo what’s happening here? In the DAT called “data” we have a table of recorded values. Next I use a select to remove the header row from the data, and another select to move through the rows of data. Using another table, a transpose, and a merge gives us a table that’s easy to convert into a CHOP. Now that we have a general sense of what’s happening in this network, let’s dig-in and get to work.

We’ll start by adding a Table DAT to an empty network. Rather than entering data by hand, we can instead just point TouchDesigner to a file that we want it to use. In the Table DAT’s parameters dialogue we’ll click on the plus button to the right of the “file” field and then locate the file that we’re looking to use.

tableDATIn order to see our table data we need to click on the button “Reload File” so that our table will be populated with the information from the file that we’re using. Next we’re going to use a few Select DATs to manipulate the contents of our table. We’re going to use the first select to remove the header row of our table. To set this up, we’ll set our select to extract rows by index, starting at 1.

select1

You’ll also notice that specifying that we’re extracting rows by index turns on a End Row Index value that’s driven by an expression (me.inputs[0].numRows – 1). We’re going to use the logic from this expression a little later on, so tuck that into the back of your mind for just a moment.

Next we’ll use another Select Table to move through the rows of our table. In adding another Select, let’s again set it up to extract Rows by Index. This time, however, we’re going to change the value of the start row and end row index to be the same. Doing things, you should notice that we get only one row of our table. Try changing the values of these parameters – as long as both fields contain the same number you’ll see only one row of information. We’ll animate that in just a moment taking advantage of this.

select2

The next operator that we’ll add to this network is a Transpose DAT. A transpose will change rows into columns, giving us a result that’s two rows, rather than two columns of data.

transpose

While just these changing values are ultimately what I’m after, I would also like my values to have names. To do this I’m going to add another table DAT, creating two rows: xPos and yPos. I’m going to use a Merge DAT to combine these two tables – to make this work properly we’ll need to set the Merge to append columns. When we’re doing we should have something that looks like this:

merge

Alright, not that we have our DAT string’s set up, let’s animate this table, and look at how to get some CHOP data out of these DATs. First let’s start by adding a Constant CHOP to our network. Let’s give our first channel a useful name, and then call our absolute frame count (me.time.absFrame).

constantWhy use absolute frame? I’d like a steadily increasing integer that can be used to drive our progression through the rows of our table. Our absolute frame is an excellent candidate for this need – except that I don’t want to exceed the maximum number of rows in my table. To do this let’s add a Limit CHOP. First up I’ll need to set this Operator to Loop, I’ll also want to set this operator to start at 0 (Minimum).

lmit

For the maximum value, I want to use the total number of rows in our second table (the table that contained only data, without a header). I could hard-code this by entering 200 into the Maximum parameter of our Limit, but then I have to change this number whenever my table changes. A better solution would be to use an expression to pull this number from the table in question – which is exactly what that expression we saw earlier does. The expression we want to use then for our Maximum parameter is: op(‘select1’).numRows.

limit2

Now it’s the moment we’ve been waiting for. Lets make that table move! To do this we’ll use the row counter in to drive our location in our table – we’ll write some relative references in our select2 DAT to make this happen. In the Start Row and End Row Index values let’s use the reference op(‘limit1’)[‘row’] to drive the change in our table.

limit3

The last step here is to add a DAT to CHOP to our network. We’ll add this at the end of our network, and drag the target DAT onto the CHOP.

dat to CHOP

There we have it. We’ve just taken a static table full of data, and turned it into a channel data that changes over time. For extra credit, add a Trail CHOP to the DAT to see what your data looks like.

trail

 

Inspired by Rutt Etra | TouchDesigner

Fall of the House of EscherBack when I was working on The Fall of the House of Escher I became obsessed with live z-displacement in the media. That particular show was wrestling with questions of quantum physics, time, reality, and all manner of perceptual problems and so it made a lot of sense to be love the look and suggested meaning inherent in the displacement of an image in a third dimension. This particular technique is perhaps most well known to us because of the Rutt Etra Video Synthesizer. This video synth was an attempt to begin manipulating video in same manner as sound was being manipulated in the 1970s, and was truly a ground breaking examination of our relationship to video – live video especially. One of the most famous elements of the Rutt Etra Synth was z displacement, the change in the depth of an image based on its luminance.

You can get a better sense of what this kind of effect looks like by playing with the Rutt Etra-izer online. This simple tool lets you play with one slice of what the original video synth did. Additionally, if you’re a mac user you might want to check out what v002 had to share when it comes to plug-ins, as well as some thoughts from Bill Etra. You can find more from v002 here: Rutt Etra v002

So that’s all well and good, but what am I after? Well, in the pursuit of better understanding how to program with TouchDesigner, as well as how to explore some of the interesting ideas from the 1970s, I wanted to know how to replicated this kind of look in a TouchDesigner network. There are plenty of other people who have done this already, and that’s awesome. I, however, happen to subscribe to the kind of art philosophy asks students to copy the work of others – to practice their hands at someone else’s technique, to take what works and leave what doesn’t. Art schools have often required students to copy the work of masters in order to foster a better appreciation and understanding of a particular form, and I think there’s a lot too that in programming. So today, we’ll look at how to make this kind of effect in TouchDesigner and then ask how we might manipulate this idea in ways that differ from how the original method was intended to  work.

Rutt Etra almostTo begin, this idea started when I was looking through the wiki page about Generative Design. Specifically, the sample network talking about image manipulation really got my attention. Buried in this example network is something that’s Rutt Etra flavored, and it’s from this example that I started to pull out some ideas to play with. As we work our way through this example it’ll be easy to get lost, frustrated, or confused. Before that happens to you, take a moment to think about what we’re trying to do. Ultimately, we want to take an image (video later, but for now an image) and add together the RGBA values of an individual pixel and use that number to transform said pixel in a z dimension. In other words, we’re really after creating a faux depth map out of an image. It’s also important to remember that using an image that’s 1920 x 1080 means we’re talking about 2,073,600 pixels. That’s a lot of points, so we’re going to start by creating a grid that simplifies those dimensions – partially for our own sanity, partially for the sake of the processing involved, and partially to remain true the aesthetic of the original idea. Once we replicate the original, then we can start to talk about where we can start to play. That’s enough disclaimers for us to get started, so let’s do some programming.

Let’s start by looking at the whole network:

From this vantage point we can see that we’re going to make a network that uses a little bit of everything – some texture operators (TOPs), some channel operators (CHOPs), and some surface operators (SOPs).

Starting with Texture Operators

First things first, let’s make a new container and dive inside. To our empty container let’s start by adding an In TOP as well as a Movie In TOP. We’re going to start by connecting the Movie In to the second input on the In TOP. Why? The Movie In TOP is going to allow us to pass a video stream into this container. We may, however, want some image to show up while we’re working or when we don’t have a video stream coming in; this is where that second input comes in handy. This gives us a default image to use when we don’t have a stream coming into the container. Here’s what you should have so far:

in

Next, we want to down sample some of this image. This step is actually helping us plan ahead, we’re going to use some expressions to set the parameters of some of our other operators, and this step is going to help us with that. To do this down-sampling, we’re going to use the Resolution TOP. After that we’ll end our TOP string in a Null TOP. All of that should look like this:

Rutt-TOPs

Before we move on, let’s make on change to the resolution TOP in our string. Open up the parameters window for your resolution TOP and on the common page change the Output Resolution parameter to Quarter:

resolution

 

Channel Operators

There are a number of different transformations that we’ll need to do with the data from our image, we’ll start this process by first moving to Channel operators. If we think back to where we started this process with an intent to transform luminance into depth, then we know that one of the operations we need to complete is to add together the RGBA values of our pixels in order to have a single number we can use to manipulate a surface. To get us started we first need to convert our image into channel data that we can manipulate. To do this we’re going to add a TOP to CHOP to our network. This operator is going to allow us to look at our TOP as if it were all Channel data. We’ll see what that looks like in a moment. First, however, let’s make a few changes to our operator. In the Image page for the TOP to make sure that you’re set to “Next Frame” as the download type. On the Crop page you’ll also want to make sure that you’re set to the full image:

Next we need to assign a TOP so we have a texture that we’re converting into Channel data. You can do this by dragging the TOP onto this CHOP, or you can enter in the name of the target TOP on the image page. Your TOP to CHOP, should now look something like this:

top to chop

This viewer on this operator can be taxing on your system, so at this point I’d recommend that you click the small bulls-eye in the upper left hand corner of your CHOP to turn off the viewer. This will save you some system resources as you’re working in your network and keep you from seeing too much lag while you’re editing this network.

The next series of channel operators is going to allow us to make some changes to our data. First we’ll use a Shuffle CHOP to reorganize our samples into sets of channels, then we’ll use some Math CHOPs to add together our pixel values and to give us some control over the z-displacement, and finally we’ll use a Rename CHOP to make it easy to use this data.

Let’s start this process by connecting our Top to CHOP to a Shuffle CHOP and setting our method to be Sequence Channels by Name:

shuffle

Next we’ll add our Math CHOP and set the combine channel operation to Add, let’s also set the multiply value to 1 /3:

math

Next we’re going to add a slider, and another Math CHOP. Why? Great question. For starters, I want to be able to control the strength of this effect with a slider. At some point I’m going to want to be able to drive the strength of this effect, and a slider is a great way for us to do that. Why another Math CHOP? Another excellent question. While we could just use one Math CHOP and apply our slider there, that also means that there’s no way to isolate the effect of the slider. There’s some redundancy here, but there’s also a little more flexibility in the isolation of applying different alterations to our data set. Is this the best way to code a final component, maybe not, but it is a fine way to work with something that’s still being developed. Alright, let’s add our slider and second Math CHOP:

slider math

Next we need to write a simple expression in our math2 operator in order to be able to use the slider as an input method. On the Multi-Add page we’re going to use the out value from the slider as the value that the incoming channel information is multiplied by – if we think about the structure of our slider’s output we’ll remember that it’s a normalized value ranging from 0 – 1, and we can think of this as being the same as 0 – 100%. Alright, here’s our simple reference expression:

op( ‘slider1/out1’ ) [ ‘v1’ ]

In plain English this expression reads: give me the number value of the channel called ‘v1’ from operator called ‘out1’ that’s inside of ‘slider1’. If you click on the + sign on the bottom right of your slider (making it viewer active) you can now move it left and right to see the change in the number value of your math2 CHOP.

simple referecne expression

Before we move away from this portion of our network we need to add one final operator, a Rename CHOP. The Rename CHOP allows us to rename the channels inside of an operator. Later we’ll want to be able to use this number to replace an value from a surface operator chain – in order to do that easily we need to rename this value. In the to field of the rename CHOP type tz:

rename

 

Surface Operators

Now that we have started the process of converting our Texture into a channel data, we need to think about what we’re going to do with that information. We’re going to start by adding a Grid SOP to our network. We’d like this operator to pull some of its dimensions from the our video – this will make sure that we’re dealing with a surface that’s the correct aspect ratio. In our Grid SOP we’re going to use some simple expressions to pull some information from our Null1 TOP (remember that our Null1 is at the end of our TOP chain). First we’ll use the height and width of our Texture to set the number of rows and columns – we can use the dot call method to ask for the height and width of our TOP with the following expressions for rows and cols:

op( ‘null1’ ).width
op( ‘null1’ ).height

Next we can use internally call the number of rows and columns to for our height and width with the expressions:

me.par.cols
me.par.rows

We’ll also want to make sure that we’ve set our grid to be a Polygon with Rows as the connectivity type.

Now we’ll need to get ready for our next step by adding a CHOP to SOP. This operator is going to allow us to create some geometry out our CHOP data.

chopto

What gives? Well, in order for this operator to work properly, we need to have some feed it some CHOP data.

 

Channel Operators Again

Now we’re finally making progress, even if it doesn’t feel like it just yet. For starters, we have our Texture Operators converted into channel data, we have a piece of geometry that we can alter based on the dimensions of the input Texture, and we’re ready to combine our Channel data with our Surface data, we just have some final house keeping to do.

Let’s start by first converting our Grid SOP to a CHOP with a SOP to CHOP.

sopto

Like we did before we’re going to use a Shuffle CHOP, but this time we’re going to set it to Sequence all Channels:

shuffle2

Next we’ll use a Math CHOP to find the absolute value of our shuffle – we can do this by setting the Channel Pre Op to be Positive:

absolute value

Next we’ll use an Analyze CHOP to find the maximum value coming out of our Math CHOP.

analyze

Now we’re going to normalize our data against itself. We’re going to add another Math CHOP to our network. We’ll connect it’s input to our sopto1 CHOP and in this case we’ll use the Multi-Add page and use 1 / the maxim from our analyze CHOP. We can do this with a simple reference expression:

1 / op( ‘analyze1’ ) [ ‘tx’ ]

All of that should look something like this:

MORE math

We can finally start putting all of this together with one more Math CHOP. This final Math CHOP is going to combine our SOP to string, and our TOP to string. You’ll want to make sure that the SOP string is in the first place on the Math CHOP, with the TOP string coming in underneath. Next make sure that Combine Channels is set to Add, and that Match By is set to Channel Name.

SOP TOP

Now let’s end this set of operations in a Null, and move back to where we left off with our surface operators. Back in our Chop to SOP, let’s set our CHOP reference to be our last CHOP null (in my case it’s Null2). All of that should finally look like this:

chop to 1

At long last we’ve finally transformed a grid in the z direction with the information from an input Texture. Ba da bing bang boom:

grid transformed

 

 

Rendering

Dear flying spaghetti monster help us… this is has been a lot of work, but so far it’s not very fancy. What gives? Well, if we really want to make something interesting out of this, we need to render it – that is, we need to change it from being just some geometric information back into some pixels. If we think back to what we learned while we were playing with Instancing, rendering isn’t too hard, we just need a few things to make it work (some geometry, a camera, and a light source… this time we’ll skip that last one).

Let’s start by adding a camera, and a geometry component to our network.

geo camera

For our render to work properly, we need our chopto SOP to be inside of our Geo. At this point you can use your favorite piece of profanity and get get ready to remake this network OR you can HOLD YOUR HORSES and think about how we might solve this problem. My favorite way to address this kind of issue is to jump inside of the geo component, add an In SOP and set it to render and display. This means that we can pass our geometry into this component without needing to encapsulate all of the geometry inside.

insop

Now let’s connect our chopto to our inlet on the Geo comp:

geo connect

Ideally we want the image from our original texture to be used when rendering our z transformation. To do this, let’s jump back inside of our Geo COMP and add a constant Material. A Constant, unlike a phong, isn’t shaded. In other words, it doesn’t render with shadows. While this isn’t great in some respects, the pay off is that it’s much cheaper to render. While we’re getting started, this is a fine material to start with. We’ll also want to make sure that our Constant is using our original TOP as a color Map. In the Color Map field we can tell this operator to look for the operator called in1 in the directory above with the following call:

../in1

constant mat

Next we need to apply this material to our Geo. To do that let’s jump out of the Geo comp and head to the render page of the parameters window. We can tell our Geo to look for the material called constant1 that’s inside of the geo comp like this:

./constant1

geoconstant

Holy macaroni, we’re almost there. The last step we need to take is to add a Render TOP to our network:

renderTOP

At long last we have finally replicated the z-translation aesthetic that we set out to emulate. From here you might consider changing the orientation of the geo comp, or the camera comp for a more interesting angle on your work.

 

Play Time

That’s all well and good, but how can we turn it up a little? Well, now that we have a portion of this built we can start to think about what the video stream going into this operation looks like, as well as how we modify the video coming out of it. Let’s look at  one example of what we can do with the video coming out.

I’ve made a few changes to our stream above (some simple moving animation, and changed the resolution op), to have something a little more interesting to play with but it’s still not quite right. One thing I want to look at is giving the lines a little more of a neon kind of look and feel. To do this I’m going to start by adding a Blur TOP to my network:

blur

Next I’m going to add an Add TOP and plug both of my TOPs into it, adding the blur back to the original render:

addblur

Finally, I’m going to add a Constant TOP set to black, and a Composite TOP. I’ll composite my Add TOP and my Constant TOP together to end with a final composition:

composite

Now it’s your turn to play. What happens when you play with the signal processing after your render, what happens when you alter the video stream heading into this component. Also, don’t forget that we built a slider that controls the strength of the effect – play and make something fun.

rutrut

 

Looking to take a closer look at what makes this process work? Download the tox file and see what makes this thing tick: rut

Vertical Sliders Please | TouchDesigner

vertical sliderIf you’ve done any interface building in TouchDesigner, you’ve surely noticed that there is a button and a slider Comp at your disposal. These are outstanding building blocks to use when you’re creating interfaces, and give you a ton of flexibility and dependability. In the past I’ve taken some time to talk about making some changes to the Button Comp, but what about the slider Comp? Won’t someone think of the slider comps?! Perhaps most importantly, what about making a vertical slider – the standard slider comp is horizontal, and wouldn’t it be nice to have one that worked up and down instead of left to right? Well, today is your lucky day – let’s look at how to make a few changes to our stock horizontal slider and turn it into a vertical slider instead.

Let’s start by first looking at what’s inside of our slider, and what makes it tick so to speak.

slider insides

Unlike the button comp, the slider looks much simpler – after all it’s only made up of a container (the knob) a Panel CHOP and an Out CHOP. That seems pretty bare bones, but if we take a closer look at the knob we can see some interesting things at work here. Of the most interest to us is the expression in the x parameter of the knob. Here you should see an expression that reads:

me.parent().panel.u.val*me.parent().par.panelw-me.par.panelw/2

What on earth does that mean?! Well let’s take a closer look at this by first experimenting with a Constant CHOP that will let us see these numbers individually. In our Constant I’m going to break this longer expression into its component parts so we can see what’s happening.

me.parent().panel.u.val

me.parent().par.panelw

me.par.panelw

At this point you should be seeing an error message. What gives?

error

The answer to that questions lies in the last expression that we’re using – me.par.panelw. A constant, doesn’t have a panel that’s associated with it, so by asking for this value we’re getting an error message telling us that we are in fact asking for the impossible. If we want the panel width of our knob, we need to change this expression slightly so instead it reads: op(‘knob’).par.panelw. Once you make that change, you should now see something like this in your Constant.

fixed

 

Okay, so what are all of these values.

Our first expression (me.parent().panel.u.val) is looking for the u value from the parent’s control panel. We can confirm this value by looking at the Panel CHOP in this slider. At the middle position of the container, the panel u value is .5 – all the way to the right is 1, all the way to the left is 0. So far so good.

samesies

Our second expression (me.parent().par.panelw) is looking for the width parameter of the parent operator. If we look at our parent’s panel values we can see that that parameter panelw is in fact 100. That’s two values down, one to go.

sliderparent

Our third expression (me.par.panelw) is asking for the width parameter of the current operator. We can verify this by looking at our width parameter of the knob, and can see that they match.

panelw

Okay. Now let’s look at what is happening in the full expression:

me.parent().panel.u.val*me.parent().par.panelw-me.par.panelw/2

In English this might read – my parent’s panel u value (.5) * my parents width (100) – my panel’s width / 2. Great. Why? Well by multiplying the u value (a normalized value) with the width of the panel I get my position in terms of pixels – almost. The last thing I need to do is to take into consideration the width of the knob (divided by 2 so you split it’s value equally left and right). So what can we do with this new information?

Let’s make a vertical slider.

First lets start by taking our Slider COMP and swap the width and height values:

setp1vertslider

Next, let’s dive into our slider, and make a few changes. Inside of the slider, on the knob copy the expression from the x parameter and paste it into the y parameter. Let’s also swap the width and height values:

xtoy

Hang on to your boots, because now we need to pay close attention to some details. We did some swapping around with our expressions, so before this is working correctly we need to make sure we change the expressions so they’ll work they way we expect them to. First up we need to change our u value to v, and then we need to look at panelh instead of panelw – remember that we’ve swapped our directions, instead of moving left and right we’re moving up and down and our expression needs to reflect that change. Here’s what your expression should look like:

me.parent().panel.v.val*me.parent().par.panelh-me.par.panelh/2

newknob

Finally, we need to change the Panel CHOP to output the v value instead of the u value:

utov

Congratulations, you should now have a vertical slider that works just like the horizontal ones.

vsliders

 

Inside Wonder Dome | TouchDesigner

first test gridIn approaching some of the many challenges of Wonder Dome one of the most pressing and intimidating was how to approach programming media playback for a show with a constant media presence. One of the challenges we had embraced as a team for this project was using Derivative’s TouchDesigner as our primary programming environment for show-control. TouchDesigner, like most programming environments, has very few limitations in terms of what you can make and do, but it also requires that you know what it is that you want to make and to do. Another challenge was the fact that while our team was full of bright and talented designers, I was the person with the broadest TouchDesigner experience. One of the hard conversations that Dan and I had during a planning meeting centered around our choices of programming environments and approaches for Wonder Dome. I told Dan that I was concerned that I would end up building an interface / patch that no one else knew how to use, fix, or program. This is one of the central challenges of a media designer – how to do you make sure that you’re building something that can be used / operated by another person. I wish there were an easy answer to this question, but sadly this is one situation that doesn’t have simple answers. The solution we came to was for me to do the programming and development – start to finish. For a larger implementation I think we could have developed an approach that would have divided some of the workload, but for this project there just wasn’t enough time for me to both teach the other designers how to use / program in TouchDesigner and to do the programming needed to ensure that we could run the show. Dan pointed out in his thesis paper on this project that our timeline shook out to just 26 days from when we started building the content of the show until we opened.

The question that follows, then, is – how did we do it? How did we manage to pull of this herculean feat in less than a month, what did we learn along the way, and what was an approach that, at the end of the process, gave us results that we used?

Organization

organizationMake a plan and stay organized. I really can’t emphasize this enough. Wonder Dome’s process lived and died in our organization as a team, and as individuals. One of the many hurdles that I approached was what our cuing system needed to be, and how it was going to relate to the script. With three people working on media, our cue sheet was a bit of a disaster at times. This meant that in our first days working together we weren’t always on the same page in terms of what cue corresponded to what moment in the play. We also knew that we were going to run into times when we needed to cut cues, re-arrange them, or re order them. For a 90 minute show with 20 media cues this is a hassle, but not an impossibility. Our 25 minute long kids show had, at the beginning, over 90 media cues.

In beginning to think about how to face this task I needed an approach that could be flexible, and responsive – fast fast fast. The solution that I approached here was to think about using a replicator to build a large portion of the interface. Replicators can be a little intimidating to use, but they are easily one of the most powerful tools that you can use in TouchDesigner. Here the principle is that you set up a model operator that you’d like subsequent copies to look like / behave like. You then use a table to drive the copies that you make – one copy operator per row in the table. If you change the table, you’ve changed / remade your operators. In the same way if you change your template operator – this is called your “Master Operator” – then you change all of the operators at once. For those reasons alone it’s easy to see how truly powerful this component is, but it’s also means that a change in your table might render your control panel suddenly un-usable.

button replicator set-up

Getting started here I began by first formatting my cue sheet in a way that made the most sense for TouchDesigner. This is a great time to practice your Excel skills and to use whatever spreadsheet application / service that you prefer to do as much formatting as possible for you. In my case I used the following as my header rows:

  • Cue Number – what was the number / name for the cue. Specifically this is what the stage manager was calling for over headset. This is also the same name / number for the cue that was in the media designer script. When anyone on the team was talking about M35 I wanted to make sure that we were all talking about the same thing.
  • Button Type – Different cues sometimes need different kinds of buttons. Rather than going through each button and making changes during tech, I wanted to be able to update the master cue sheet for the replicator, and for the properties specified to show up in the button. Do I want a momentary button, a toggle, a toggle down, etc. These things mattered, and by putting these details in the master table It was one less adjustment that I needed to make by hand.
  • Puppet – Wonder Dome had several different types of cues. Two classifications came to make a huge difference for us during the tech process. Puppet entrances / exits, and puppet movements. Ultimately, we started to treat puppet entrances and exits as a different classification of cue (rather than letters and numbers we just called for “Leo On” and “Leo Off”, this simplified the process of using digital puppets in a huge way for us), but we still had puppet movements that were cued in TouchDesigner. During the tech process we quickly found out that being able to differentiate between what cues were puppet movements and what cues were not was very important to us. By adding this column I could make sure that these buttons were a different color – and therefore differentiated from other types of cues.

Here I also took a programming precaution. I knew that invariably I was going to want to make changes to the table, but might not want those changes to be implemented immediately – like in the middle of a run for example. To solve this problem I used a simple copy script to make sure that I could copy the changed table to an active table when we were in a position to make changes to the show. By the end of the process I was probably fast enough to make changes on the fly and for them to be correctly formatted, but at the beginning of the process I wasn’t sure this was going to be the case. The last thing I wanted to do was to break the show control system, and then need 25 minutes to trouble shoot the misplacement of a 1 or 0. At the end of the day, this just made me feel better, and even if we didn’t need it in place I felt better knowing that I wasn’t going to break anything if I was thinking on my feet.

replicator in action

Above you can see a replicator in action – looking at an example like this I think helps to communicate just how useful this approach was. Method, like organization, is just a way to ensure that you’re working in a way that’s meaningful and thoughtful. I’m sure there are other methods that would have given us the same results, or even better results, but this approach helped me find a way to think about being able to quickly implement cue sheet changes into our show control environment. It also mean that we standardized our control system. With all of the buttons based on the same Master Operator it gave the interface a clean and purposed look – staring down the barrel of a 25 show run, I wanted something that I didn’t might looking at.

Thinking more broadly when it comes to organization, beyond just the use of replicators for making buttons I also took the approach that the show should be modular and organized as possible. This meant using base and container components to hold various parts of the show. Communication to lighting and sound each had their own module, as did our puppets. For the sake of performance I also ended up placing each of the locations in their own base as well. This had the added bonus of allowing for some scripting to turn cooking on and off for environments that we were using or not using at any given point in the show. We had a beast of a media server, but system resources were still important to manage to ensure smooth performance.

notThatStory_fullMap

If you want to learn more about replicators you can read through this post about getting started using them.

Show Control

Show control, however, is about more than just programming buttons. Driving Wonder Dome meant that we needed a few additional features at our fingertips during the show. Our show control system had two preview screens – one for the whole composite, and one for puppets only. One of the interesting features of working in a dome is how limited your vision becomes. The immersive quality of the projection swallows observers, which is downright awesome. This also means that it’s difficult to see where all of the media is at any given point. This is one of the reasons that we needed a solid preview monitor – just to be able to see the whole composition in one place. We also needed to be able to see the puppets separately at times – partially to locate them in space, but also to be able to understand what they looked like before being deformed and mapped onto the curved surface of the dome.

show_control

The central panel of our control system had our cues, our puppet actions, our preview monitors, and a performance monitor. During the show there were a number of moments when we had a dome transformation that was happening while nearly simultaneously a puppet was entering or exiting. While originally I was trying to drive all of this with a single mouse, I quickly abandoned that idea. Instead I created a simple TouchOSC interface to use on an iPad with another hand. This allowed me to take a double handed approach to diving the media added some challenge, but paid itself back ten fold with a bit of practice. This additional control panel also allowed me to drive the glitch effects that were a part of the show. Finally it also made for an easy place to reset many of the parameters of various scenes. In the change over between shows many elements needed to be reset, and but assigning a button on my second interface for this task I was able to move through the restore process much faster.

2014-04-09 14.44.46

If you’d like to learn more about using TouchOSC with TouchDesigner there a few pages that you might take a glance at here:

TouchOSC | Serious Show Control
Sending and Receiving OSC Values
Visualizing OSC Data

Cues

Beyond creating a system for interacting with TouchDesigner, a big question for me was how to actually think about the process of triggering changes within my network. Like so many things, this seems self evident on the face of it – this button with do that thing. But when you start to address the question of “how” then the process becomes a little more complicated. Given the unstable nature of our cue sheet, I knew that I needed a name-based approach that I called from a central location. Similar to my module based approach for building the master cue sheet, I used the same idea when building a master reference sheet.

With a little push and guidance from the fabulous Mary Franck, I used an evaluate DAT to report out the state of all of the buttons from the control panel, and name them in a way that allowed for easy calling – specifically I made sure that each cue maintained it’s name letter and number convention from our cue sheet.

master ref

 

On the face of this it seems like that’s an awful lot of scripts to write – it is, but like all things there are easier and harder ways to solve any problem. My approach to here was to let google spreadsheets do some work for me. Since cue sheet was already set-up as a spread sheet, writing some simple formulas to do the formatting for me was a quick and easy way to tackle this. It also meant that with a little bit of planning my tables for TouchDesigner were formatted quickly and easily.

excel script formattingIt was also here that I settled on using a series of Execute DATs to drive the cooking states of the various modules to control our playback performance. I think these DATs were some of the hardest for me to wrap my head around – partially because this involved a lot of considered monitoring of our system’s overall performance, and the decisions and stacking necessary to ensure that we were seeing smooth video as frequently as possible. While this certainly felt like a headache, by the time the show was running we rarely dropped below 28 frames per second.

cooking on and off

If you want to read a little more about some of the DAT work that went into Wonder Dome you can start here:

Evaluate DAT Magic
These are the DATs You’ve Been Looking For

Communication

All of the designers on the Wonder Dome team had wrestled with the challenges of communication between departments when it comes to making magic happen in the theatre. To this end, Adam, Steve, and I set out from the beginning to make sure that we had a system for lights, media, and sound to all be able to talk with one another without any headache. What kind’s of data did we need to share? To create as seamless a world as possible we wanted any data that might be relevant for another department to be easily accessible. This looked like different things for each of us, but talking about it from the beginning ensured that we built networks and modules that could easily communicate.

Screenshot_032314_115125_AM

In talking with lighting, one of our thoughts was about passing information relative to the color of the environment that we found ourselves in at any given point. To achieve this I cropped the render to a representative area, then took the average of the pixel values in that area, then converted the texture data to channel data and streamed lighting the RGBA values over OSC. We also made a simple crossfader in our stream for the times when we wanted the lighting in the scene to be different from the average of the render.

WD_AdamThis technique was hardly revolutionary, but it did create very powerful transitions in the show and allowed media to drive lighting for the general washes that filled the space. This had the added benefit of offloading some programming responsibility from lighting. While I had done a lot of work in the past to coordinate with sound, I hadn’t done much work coordinating with lights. In fact, this particular solution was one that we came up with one afternoon while we were asking questions like “what if…” about various parts of the show. We knew this was possible, but we didn’t expect to solve this problem so quickly and for it to be so immediately powerful. Through the end of the run we continued to consistently get positive audience response with this technique. Part of the reason this solution was so important was be cause Adam was busy building a control system that ultimately allowed him to control two moving lights with two wacom tablets – keeping the washing lighting driven by media kept both of his hands free to operate the moving lights.

Screenshot_032314_115026_AM

The approach to working with sound was, of course, very different from working with lights. Knowing that we wanted to use spatialized sound for this show Stephen Christensen built an incredible Max patch that allowed him to place sound anywhere he wanted in the dome. Part of our conversation from the beginning was making sure that media could send location data bout puppets or assets – we wanted the voice of the puppeteers to always be able to follow the movement of the puppets across the dome. This meant that created an OSC stream for sound that carried the location of the puppets, as well as any other go or value changes for moments where sound and media needed to be paired together.

Screenshot_032314_114946_AM

Communicating with sound wasn’t just a one way street though. Every day the Wonder Dome had a 90 minute block of free time when festival visitors were allowed to explore the dome and interact with some of the technology outside of the framework of the show. One of the components that we built for this was a 3D environment that responded to sound, animating the color and distribution of objects based on the highs, mids, and lows from the music that was being played. Here sound did the high, mid, low processing on its end, and then passed me a stream of OSC messages. to get a smoother feel from the data I used a Lag CHOP before using this to drive any parameters in my network.

Components and Reuse

Perhaps the most important lesson to be learned from this project was the importance of developing solid reusable components. This, again, isn’t anything revolutionary but it is worth remembering whenever working on a new project. The components that you build to use and reuse can make or break your efficiency and the speed of your workflow. One example of this would be a tool that we created to make placing content on the dome. Our simple tool for moving images and video around the dome would be used time and again throughout the project, and if I hadn’t take the time early on to create something that I intended to reuse, I would have instead spent a lot of time re-inventing the wheel every time we needed to solve that problem.

Screenshot_032314_114332_AM

In addition to using this placement tool for various pieces of media in the show, this is also how we placed the puppets. During the development phase of this tool I thought we might want to be able to drive the placement of content from a iPad or another computer during tech. To make this easier, I made sure that there was a mechanism embedded in the tool to allow for easy control from multiple inputs. This meant that when we finally decided to adapt this tool for use with the puppets, we already had a method for changing their location during the show. There are, of course, limits to how much anyone can plan ahead on any project but I would argue that taking the time to really think about what a component needs to be do before developing it makes good sense. I also made use of local variables when working with components in order to make it easier to enable or disable various pieces of the tool.

Screenshot_032314_114451_AM

You can read more about some of this process here:

3D Solutions for a 2D World
Container Display

Documentation and Comments

comment exampleI nearly forgot to mention one of the most critical parts of this process. Documentation and commenting. If I hadn’t commented my networks I would have been lost time after time. One of the most important practices to develop and to continue is good commenting. Whenever I was working on something that I couldn’t understand immediately by just looking at it, I added a comment. I know that some programmers use the ability to insert comments with individual operators, but I haven’t had as much success with that method. Personally, I find that inserting a text DAT is the best way for me to comment. I typically write in a text editor using manual carriage returns. I also make sure that I date my comments, so if I make a change I can leave the initial comments and then append the comment with new information. I can’t say enough about the importance of commenting – especially if you’re working with another programmer. Several times during the process I would help lighting solve a problem, and good commenting helped ensure that I could communicate important details about what was happening in the network to the other programmer.

I think it’s also important to consider how you document your work. This blog often functions as my method of documentation. If I learning something that I want to hold onto, or something that I think will be useful to other programmers then I write it down. It doesn’t do me any good to solve the same problem over and over again – writing down your thoughts and process help you organize your approach. There have been several times when I find shortcuts or new efficiency in a process only when I’m writing about it – the act of taking it all a apart to see how the pieces connect make you question what you did the first time and if there’s a better way. At times it can certainly feel tedious, but I’ve also been served time and again by the ability to return to what I’ve written down.

 

 

TouchOSC | Serious Show Control

TouchOSC BuffetI know. You love your iDevice / Android. You love your phone, your tablet, your phablet, your you name it. You love them. Better yet, you’ve discovered Hexler’s TouchOSC and the thought of controlling your show / set / performance set you on fire – literally. You were beside yourself with glee and quickly set yourself to the task of triggering things remotely. Let’s be honest, it’s awesome. It’s beyond awesome, in some respects there are few things cooler than being able to build a second interface for a programming environment that works on your favorite touch screen device. But before we get too proud of ourselves, let’s have a moment of honesty. True honesty. You’ve dabbled and scrambled, but have you ever really sat down to fully understand all of the different kinds of buttons and switches in TouchOSC? I mean really looked at them and thought about what they’re good for? I know, because I hadn’t either. But it’s time, and it’s time in a big way.

TouchOSC_Editor_-_Untitled_1_touchoscFirst things first, make sure you download a copy of the TouchOSC editor for your OS of choice. If you happen to be using windows, save yourself a huge hassle and make sure that you download the version of the editor (32 or 64 bit) that corresponds to the version of Java that you’re running – otherwise you’ll find yourself unable to open the editor and be grumpy. Great. Now in the editor create a new control setup. Make sure you’re configured to work on the device that you’re planning on playing with / use the most. In my case I’m working with an iPad layout. I prefer to use my iPad in a horizontal orientation most of the time for show control. I’m also only working with one page for now, and happy to leave it as named “1”. You’ll notice for now that the box next to auto for OSC is checked – we’ll leave that for now. Alright, now we’re going to do something that most of us have never done.

In your empty control panel, right click and add one of every different kind of object. Yep. Add one of each so we can look at how they all work. You might choose to skip the repetition of vertical and horizontal sliders – that’s a-okay, but make sure you pick one of them. By the end you should have something that looks like this:

TouchOSC_Editor_-_Untitled_1_touchosc

That’s certainly not sexy, but it’s going to teach you more than you can imagine. Next upload that interface to your device of choice. If you’ve never uploaded a new interface to TouchOSC you’ll need to know the IP address of your computer. If you’re using a windows machine you can use the command prompt and ipconfig to find this information, on a mac use the network pane in System Preferences. Next you’ll want to transfer this layout to your device, Helxer has a wonderful piece of documentation about do get this done, and you can read it here.

Next take a moment to read through what all of those lovely widgets do and how they talk to your programming environment. After that the real fun starts. Start listening to your network with your programming environment of choice, and look at what kinds of messages you get from all of these different kinds of buttons and sliders.

Isadora

If you’re working with Troikatronix Isadora you can start to see the signal flow coming from your layout by first going to the communications drop down menu, and then selecting “Stream Setup.”

Communications_and_Menubar

Next you’ll want to select “auto detect input.”

Menubar_and_Untitled

Now you can start moving sliders and toggling buttons so you can record their address. Make sure that you select “Renumber Ports” and click “OK” on the bottom of the page.

Menubar_and_Untitled

Now you’re ready to use the OSC listener actor to use those inputs to drive any number of elements inside of your scene. You’ll notice that the “channel number” on the listener actor corresponds to the port number on the Stream-Setup dialog.

Menubar_and_Untitled

 

Now you’re off to the races. Do something exciting, or not, but play with all of the buttons, sliders, and gizmos so you know how they all work. For extra credit, remember that you can also send messages back to your layout. To do this you need to use the OSC Transmit actor. You’ll also need to know the IP address of your device, and the port that you’re transmitting to – you can find all of this information on the same page in TouchOSC where you initially set your target IP address.

Quartz Composer

If you’re working with Apple’s Quartz Composer you have to do a little more work to get your OSC stream up and running. For starters, you’ll need to know a little bit more about your OSC messages. You’ll notice in the TouchOSC editor that each object has an address that’s associated with it. For example, my fader is ” /1/fader1 “. To Touch OSC this reads as Fader 1 on Page 1 of your layout. In order to read information from this slider in Quartz, we’ll need to know this address, along with the type of information that we’re sending. In the case of a slider we’re sending a float (a number with fractional parts). To get started let’s open up a new Quartz Composer patch, and add an “OSC Receiver” from the library.

Menubar_and_Untitled_-_Editor

Now let’s take a closer look at that node. If we look at the inspector, and at the settings page we can see lots of useful information.

Menubar_and_OSC_Receiver_and_Untitled_-_Editor

Here we can see what port we’re set to receive information on by default, as well as an example of how we need to format the key’s from our layout so Quartz can properly receive the signals coming over the network. Let’s set up our fader to be properly decoded by Quartz. First we’ll need to remove the test key. Next we’ll want to add a new key by typing in the dialog box /1/fader1 and designating this as a float. Finally, click the plus sign to add this key the receiver. I usually add something else to my Quartz patch to make sure that I’m passing values through my network, this is optional.

Menubar_and_OSC_Receiver_and_Untitled_-_Editor

There you go. Now, to add the other buttons and sliders from your layout, you’ll need to similarly add keys for each of the buttons, sliders, and gizmos. Remember that you can find this information in your TouchOSC editor for this layout:

TouchOSC_Editor_-_Untitled_1_touchosc

Now you’re cooking with gas. Experiment with your layout, and how you can use this buttons and sliders to drive your Quartz Composer creations. For extra credit remember that you can also transmit back to your device. In this case you’ll need to use the “OSC Sender” object in Quartz. You’ll need to know the IP address of your target device, as well as the port that you’re communicating on. Have fun, and making something interesting… or if nothing else, learn something along the way.

TouchDesigner

If you’re working with Derivative’s TouchDesigner you have two options for receiving OSC information – CHOPs or DATS. Both of these methods work well but might have different uses for different situations.

Screenshot_032514_083023_PMLet’s start by looking at the OSC in CHOP. First we need to specify what port we want to listen to. Take a look at the TouchOSC layout to make sure the ports and addresses match. Then start moving sliders and buttons to see them appear in TouchDesigner.

Screenshot_032514_083345_PM

To use these to drive various parts of your network you can either use a select CHOP or reference these channels with expressions.

Next let’s look at what we see coming from the OSC In DAT.

Screenshot_032514_083545_PMHere instead of a changing single we see table with a message header and then with our float values coming in as static messages. Depending on the circumstance one or the other of these methods is going to help you drive your network.

For extra credit use the OSC out Chop to push values back to your TouchOSC layout – remember that you’ll need the IP address of your device, the port you’re receiving messages on, and you’ll need to format the header of the message correctly so that it corresponds to the right slider or button. Have fun learning and playing with your layout of all of the different kinds of controls.

TouchDesigner | Evaluate DAT Magic

13185491723_10766c1574_oLet’s say you’re an ambitions person, maybe too ambitious, and all of a sudden you have a control panel with 68 buttons that need to trigger various scenes, effects, and transitions. Let’s also say that your TouchDesigner network is  vast enough that you need (and I mean in a big way) a simple method for getting information about what’s active in your network. If you find yourself programming the media for a live theatrical event with these same kinds of concerns, the Evaluate DAT might just be the best friend that you didn’t know that you needed.

My example above might sound a little far fetched, so let’s instead look at a more concrete example. Let’s imagine that we have a digital puppet that we’d like to trigger to enter and exit with a button set to toggle. Just to make this more interesting, let’s say that you actually have three puppets and three visual effects that you’d like to all control with buttons from a single control panel. If all of your puppets and effects are all located in a similar place this might not be a huge hassle for you, if however, you’re building a complicated network it will soon become very important to be able to call the states of these buttons across your network.

Here’s my example set-up for us – a control panel that has all of our controls, a base component where we’re going to store the status of our buttons, and a scene where our puppet needs to make an entrance and an exit based on our button’s active state.

Screenshot_031614_012640_AM

I’m going to assume that you’re already familiar with working with control panels in TouchDesigner, but if that’s not the case you can learn a little more here before you keep reading. Lets start by taking a look inside of our “trigger_ref” base component where we are going to store the status of our buttons to use other places in our network.

Screenshot_031614_013152_AM

This should look pretty impressively boring at first glance. To really get a feel for what’s happening here, let’s take a closer took how how this is being driven by our control panel.

evaluate DAT in action

Alright, so here we can see that Puppet 1, 2, and 3 all act as Toggle switches, VFX 1 acts as a momentary trigger, and VFX2 and VFX3 both act as toggles. Before we start to look at why this is important, let’s first see how this is working.

To really make our Evaluate DAT sing we need to start by feeding a table with some instructions about what information we would like it to evaluate. In my case I’ve made a table with a name for the item that I’d like to call in one column, and the pathway to the operator and channel that’s being evaluated in the other column.

Screenshot_031614_013937_AM

If we were to write these instructions in English they might read something like: “TouchDesigner, please watch operator ‘button0’ in ‘ctrl’ and tell me when it’s v1 changes, and give that the name ‘Puppet1’. It’s important note that the formatting and syntax of your request to the Evaluate DAT matters. In my case I needed to put the names in the first column in quotations. I also made sure to tell the evaluate DAT that I wanted an integer (you’ll notice that in front of the operator’s pathway I’ve int() to specify that I want an integer returned to me). Next in the Evaluate DAT I need to check a few settings in the operators parameters.

Screenshot_031614_014828_AM

The most important parameter to take note of here is that I’ve told the Evaluate DAT that I’d like the output in Expressions. This ensures that my input table is being evaluated.

Okay, but why is this interesting / important / worth your attention? While it might be tempting to avoid an organization method like this, it means that when you’re working with a large control panel, or a large cue stack, that you can refer to a button by its name rather than by a complicated network pathway. Better yet, once you write the expression to call this value from our Evaluate DAT it becomes easy to copy and paste that expression and only need to change the name of the value that you’re calling. Let’s go back to our puppet example.

Screenshot_031614_020356_AMHere I have my video that I’d like to change with my puppet button. I’ve already set up my network with a few operators. First I have a constant CHOP, and a trigger CHOP. Next I have a Movie in TOP, a Level TOP, and a Null TOP. I’d like to use my trigger CHOP to change the opacity of my Movie.

To start let’s make sure that our constant CHOP is set up with a channel that we’ll name “Puppet1” .

Screenshot_031614_021044_AM

Let’s also make sure that our trigger is set to have a sustain value of 1, and let’s make sure that we’re happy with how the trigger is going to fire and release. For now I’m only interested in making sure that my sustain is changed.

Screenshot_031614_020704_AM Next we’re going to write a quick expression that will allow our level TOP to reference the trigger CHOP. In your level top we’re going to use the expression on the opacity Parameter on the Post page of Level TOP:

op( “trigger1” )[ “Puppet1” ]

Screenshot_031614_021154_AM

Now we’re going to go back to our constant CHOP to see how this really makes our network programming fun. Here for the value of the Puppet1 channel we’re going to write the following expression:

op( “../trigger_ref/buttons” ) [ “Puppet1” , 1 ]

Screenshot_031614_021552_AM

Now, let’s finally see what this means when we use this in conjunction with our control panel.

puppet inout

Great, so now we toggle the opacity of a video on and off with a button. What’s important here is that what if you suddenly decide that you want this action to happen with the Puppet 2 button instead? Now, rather than having to look up where your original button was, along with it’s reference you can instead just change the expression in the Constant CHOP to be:

op( “../trigger_ref/buttons” ) [ “Puppet2” , 1 ]

Right? We’ve changed all of those button values into a table that we can centrally reference by name. If you’re just getting started with TouchDesigner this might not seem like a huge revelation, but once you start to build more complicated networks the ability to call something by a name (maybe even a name that’s in your cue sheet) becomes hugely important.