I recently had the good fortune of being able to collaborate with my partner, Zoe Sandoval, on their MFA thesis project at UC Santa Cruz – { 𝚛𝚎𝚖𝚗𝚊𝚗𝚝𝚜 } 𝚘𝚏 𝚊 { 𝚛𝚒𝚝𝚞𝚊𝚕 } Thesis work is strange, and even the best of us who have managed countless projects will struggle to find balance when our own work is on the line – there is always the temptation to add more, do more, extend the piece a little further, or add another facet for the curious investigator. Zoe had an enormous lift in front of them, and I wanted to help streamline some of the pieces that already had functioning approaches, but would have benefited from some additional attention and optimization. Specifically, how cues and states operated was an especially important area of focus. I worked closely with the artist to capture their needs around building cues / states and translate that into a straightforward approach that had room to grow as we made discoveries, and needed to iterate during the last weeks leading up to opening.

The Big Picture

{ remnants } of a { ritual } is an immersive installation comprised of projection, lighting, sound, and tangible media. Built largely with TouchDesigner, the installation required a coordinated approach for holistically transforming the space with discrete looks. The projection system included four channels of video (two walls, and a blended floor image); lighting involved one overhead DMX controlled instrument (driven by an ENTEC USB Pro), and four IoT Phillips Hue lights (driven by network commands – you can find a reusable approach on github); sound was comprised of two channels driven by another machine running QLab, which required network commands sent as OSC. The states of each of these end points, the duration of the transition, and the duration of the cue were all elements that needed to both be recorded, and recalled to create a seamless environmental experience.

Below we’re going to step though some of the larger considerations that led to the solution that was finally used for this installation, before we get there though it’s helpful to have a larger picture of what we actually needed to make. Here’s a quick run-down of some of the big ideas:

• a way to convert a set of parameters to python dictionary – using custom parameters rather than building a custom UI is a fast way to create a standardized set of controls without the complexity of lots of UI building in Touch.
• a reusable way to use storage in TouchDesigner to have a local copy of the parameters for fast reference – typically in these situations we want fast access to our stored data, and that largely looks like python storage; more than just dumping pieces into storage, we want to make sure that we’re thoughtfully managing a data structure that has a considered and generalized approach.
• a way to write those collections of parameters to file – JSON in this case. This ensures that our preset data doesn’t live in our toe file and is more easily transportable or editable without having TouchDesigner open. Saving cues to file means that we don’t have to save the project when we make changes, and it also means that we have a portable version of our states / cues. This has lots of valuable applications, and is generally something you end up wanting in lots of situations.
• a way to read those JSON files and put their values back into storage – it’s one thing to write these values to file, but it’s equally important to have a way to get the contents of our file back into storage.
• a way to set the parameters on a COMP with the data structure we’ve been moving around – it’s great that we’ve captured all of these parameters, but what do we do with this data once we’ve captured it? The need here is thinking through what to do with that data once you have it captured.

Cuing needs

One of the most challenging, and most important steps in the process of considering a cuing system is to identify the granularity and scope of your intended control. To this end, I worked closely with the artist to both understand their design intentions, as well their needed degrees of control. For example, the composition of the projection meant that the blended floor projection was treated as a single input image source; similarly, the walls were a single image that spanned multiple projectors. In these cases, rather than thinking of managing all four projectors it was sufficient to only think in terms of the whole compositions that were being pulled. In addition to the images, it was important to the artist to be able to control the opacity of the images (in the case of creating a fade-in / out) as well as some image adjustments (black level, brightness, contrast, HSV Offset). Lighting, and sound had their own sets of controls – and we also needed to capture a name for the cue that was easily identifiable.

As lovely as it would be to suggest that we knew all of these control handles ahead of time, the truth is that we discovered which controls were necessary through a series of iterative passes – each time adding or removing controls that were either necessary or too granular. Another temptation in these instances is to imagine that you’ll be able to figure out your cuing / control needs on your feet – while that may be the case in some situations, it’s tremendously valuable to instead do a bit of planning about what you’ll want to control or adjust. You certainly can’t predict everything, and it’s a fool’s errand to imagine that you’re going to use a waterfall model for these kinds of projects. A more reasonable approach is to make a plan, test your ideas, make adjustments, test, rinse, repeat. An agile approach emphasizes smaller incremental changes that accumulate over time – this requires a little more patience, and a willingness to refactor more frequently, but has huge advantages when wrestling with complex ideas.

Custom Pars

In the past I would have set myself to the task of handling all of these controls in custom built UI elements – if was was creating an interface for a client and had sufficient time to address all of the UI / UX elements I might have taken that approach here, but since there was considerable time pressure it was instead easier (and faster) to think about working with custom parameters. Built in operators have their own set of parameters, and Touch now allows users to customize Component operators with all of the same parameters you find on other ops. This customization technique can be used to build control handles that might otherwise not need complete UI elements, and can be further extended by using the Parameter COMP – effectively creating a UI element out of the work you’ve already done while specifying the custom parameters. The other worthwhile consideration to call out here is your ability to essentially copy parameters from other objects. Consider the black level, contrast, and brightness pars included above. One approach would be to create each par individually, and set their min, max, and default values. It would, however, be much faster if we could just copy the attributes from the existing level TOP. Luckily we can do just that with a small trick.

We start by creating a base Comp (or any Component object), right clicking on the operator, and selecting customize component.

This opens the customize component dialogue where we can make alterations to our COMP. Start by adding a new page to your COMP and notice how this now shows up on the components parameters pages:

For now let’s also add a level TOP so we can see how this works. From your level TOP click and drag a parameter over to the customize component dialogue – dragging specifically to the Parameter column on the customize component dialogue:

This process faithfully captures the source parameter’s attributes – type, min, max, and default vals without you needing to input them manually. In this style of workflow the approach is to first start by building your operator networks so you know what ops you will want to control. Rather than designing the UI and later adding operator chain, you instead start with the operator chain, and only expose the parameters you’ll need / want to control. In this process you may find that you need more or fewer control handles, and this style of working easily accommodates that kind of workflow.

Capturing Pars

Creating a component with all of your parameters is part of the battle, how to meaningfully extract those values is another kettle of fish. When possible it’s always handy to take advantage of the features of a programming language or environment. In this case, I wanted to do two things – first I wanted to be able to stash cues locally in the project for fast retrieval, second I wanted to have a way to write those cues to disk so they were’t embedded in a toe or tox file. I like JSON as a data format for these kinds of pieces, and the Python equivalent to JSON is dictionaries. Fast in TD access means storage. So here we have an outline for our big picture ideas – capture the custom parameters, stash them locally in storage, and write them to disk.

One approach here would be to artisanaly capture each parameter in my hipster data structure – and while we could do that, time fighting with these types of ideas has taught me that a more generalized approach will likely be more useful, even if it takes a little longer to get it right. So what does that look like exactly?

To get started, let’s create a simple set of custom parameters on a base COMP. I’m going to use the trick we learned above to move over a handful of parameters form a level TOP: Black Level, Contrast, and Opacity:

To create a dictionary out of these pars I could write something very specific to my approach that might look something like this snippet:

At first glance that may seem like an excellent solution, but as time goes on this approach will let us down in a number of ways. I wont’ bother to detail all of them, but it is worth capturing a few of the biggest offenders here. This approach is not easily expanded – if I want to add more pars, I have to add them directly to the method itself. For a handful, this might be fine, but over ten and it will start to get very messy. This approach also requires duplicate work – the key name for values means that I need to manually verify if the key name and parameter name match (we don’t have to do this, but we’ll see later how this saves us a good chunk of work), if I misspell a word here I’ll be very sorry for it later. The scope of this approach is very narrow – very. In this case the target operator variable is set inside of the method, meaning that this approach will only ever work for this operator, at this level in the network. All of that and more largely mean that while I can use this method very narrowly, I can use this approach, but I’m going to be sorry for it in the long run.

Instead of the rather arduous process above, we might consider a more generalized approach to solving this problem. Lucky for us, we can use the pars() method to solve this problem. The pars() method is very handy for this kind of work, the major catch being that pars() will return all of the parameters on a given object. That’s all well and good, but what I really wanted here was to capture only custom parameters on a specific page, and to be able to ignore some parameters (I didn’t, for example, need / want to capture the “save cue” parameter). What might his kind of approach look like, let’s take a look at the snippet below.

Abstract Reusable code segment

What exactly is happening here? First off, this one is full of documentation so our future selves will know what’s happening – in fact this is probably more docstring than code. The big picture idea is rather than thinking about this problem one parameter at a time, we instead what to think of entire custom pages of parameters. Chances are we want to re-use this, so it’s been made to be fairly general – we pass in an operator, the name of the page we want to convert to a python dictionary, the name of our newly made preset, and a list of any parameters we might want to skip over. Once we pass all of those pieces into our function, what we get back is a dictionary full of those parameters.

Capture to Storage

Simply converting the page of parameters to a dictionary doesn’t do much for us – while it is a very neat trick, it’s really about what we do with these values once we have them in a data structure. In our case, I want to put them into storage. Why storage? We certainly could put these values into a table – though there are some gotchas there. Table values in TouchDesigner are always stored as strings – we might think of this as text. That matters because computers are notoriously picky about data, and find the challenge of differentiating between words, whole numbers, and numbers with decimal values very difficult. Programmers refer to words as strings, whole numbers as integers or ints, and numbers with decimal values as floats. Keeping all of our parameter values in a table DAT means they’re all converted to strings. Touch mostly does an excellent job of making this invisible to you, but when it goes wrong it tends to go wrong in ways that can be difficult to debug. Using storage places our values in a python dictionary where our data types are preserved – not converted to strings. If you’re only working with a handful of cues and a handful of parameters this probably doesn’t matter – but if you’re thinking about 20 or more parameters it doesn’t take many cues before working in native data types will make a big difference. For reference, an early iteration of the cuing system for this would have needed the equivalent of a table DAT with over 1000 rows to accommodate the stored parameters. These things add up quickly, more quickly than you first imagine that they might.

Okay, so what’s an example of a simple and reusable function we might use to get a dictionary into storage:

Write to file

Similar to the above, we likely want a simple way to write our stored cues to disk in the same format we’re using internally. Python dictionaries and JSON are nearly interchangeable data structures and for our needs we can think of them as being the same thing. We do need to import the JSON module to get this to work correctly, but otherwise this is a straightforward function to write.

What you end up with will look like this:

Reading from file

We’re close now to having a complete approach for working with cues / states. Our next puzzle piece here would be a way to read our JSON from disk, and replace what we have in storage with the file’s contents. This means that whatever is in the file can be used to replace what we have in storage.

What you end up with here might look like this:

Loading pars – does it work

This part is the most tricky. Here the the big idea is to create a duplicate operator that has all of the same custom parameters in our preset maker. Why? Well, that would mean that all of the parameter names match – so which would make loading parameters significantly easier and more straightforward. The other trick here is to remove any of the ignored pars from our ignore list – thinking back this is to ensure that we don’t use any of the parameters that we don’t want / need outside of recording them. We can start this process by making a copy of our operator that’s being used to capture our parameters and then deleting the pars we don’t need. Next we need to write a little script to handle moving around all of the values. That should look something like this:

Making a Module

All of this is a good bit of work, and if you’ve been following along, you probably now have a handful of text DATs doing all of this work. For the sake of keeping tidy, we can instead put all of this into a single DAT that we can use as a python module. Wrapping all of these pieces together will give us something like this:

If you want to see how this works and pull it apart from there you can pull an example TOE file form this repo.

TD JSON – another alternative

There’s another alternative to this approach – which is the new TD JSON elements that are in the TouchDesigner. You can read about them on Derivative’s wiki here. These tools are a promising alternative, and you can do very similar pieces here. In particular we might use something like pageToJSONDict()to do what we’re after. That might look something like this:

That’s slick, but what we get back is almost 75 lines worth of JSON. This feels a little overkill to me for what we’re after here – there’s lots of potential here, but it might be a little more than we need for this actual implementation. Maybe not though, depending on how you want to change this process, it might be just perfect.

Safety Rails

There are some pieces missing in the approach above that I ended up including in the actual implementation for the artist – I’m not going dig into some of these pieces, but it is worth calling attention to some of the other elements that were included. The biggest pieces that needed to be addressed were how we handle failure, duplicates, provided confirmation on an operation, or warned the user about possibly unintended operations.

The artist, for example, wanted to both have the UI flash and to get a message printed to the text port when a preset was successfully saved. The artist also wanted to make sure that a preset wasn’t automatically overwritten – instead they wanted to see a pop up message warning that a preset was going to be overwritten, allowing the user confirm or cancel that operation.

That may seem unnecessary for a tool you build for yourself… until it’s 2am and you haven’t slept, or you’re working fast, or there’s a crit in 10 minutes and you want to make one more adjustment, and and and, or or or. Handling these edge cases can not only add piece of mind, but also ensure you keep your project on the straight and narrow.

Additionally, how you handle failure in these situations is also important to plan – we never want these pieces to fail, but having a gracefully solution for how to handle these moments are tremendously important to both work through and plan. If nothing else, it’s elegantly handling the failure and printing a message – better still is if give yourself a clue about what went wrong. A few breadcrumbs can go a long way towards helping you find the trail you lost. In my opinion, failing code gets a bad wrap – it’s something we grumble over, not something we celebrate. The truth of the matter, however, is that failures are how we make projects better. Being able to identify where things went wrong is how you can improve on your process. It’s a little thing, but if you can shift (even if only slightly) how you feel about a failing process, it will allow you some room to embrace iterative process more easily.

Conclusions

Managing states / cues is tricky. It’s easy to consider this a rather trivial problem, and it isn’t until you really take time to think carefully about what you’re trying to achieve that you uncover the degree of complexity in the questions around how you manage the flow of information in your network. You wont get it right the first time, but chances are you didn’t ride a bike without a few falls, and you probably didn’t learn to play that instrument without getting a few scales wrong. It’s okay to get it wrong, it’s okay to refactor your code, it’s okay to go back to the drawing board as you search to find what’s right – that’s part of the process, it’s part of what will ultimately make for a better implementation.

No matter what, hang in there… keep practicing, leave yourself breadcrumbs – you’ll get there, even if it takes you longer than you want.

Happy programming.

Zoe Sandoval’s { remnants } of a { ritual }

You can see { remnants } of a { ritual } and the work of the DANM MFA Cohort through May 12th at UC Santa Cruz.