Category Archives: Design

Multiple Windows | TouchDesigner

For an upcoming project that I’m working on our show control needs to be able to send out video content to three different projectors. The lesson I’ve learned time and again with TouchDesigner is to first start by looking through their online documentation to learn about what my options are, and to get my bearings. A quick search of their support wiki landed me on the page about Multiple Monitors.

To get started I decided to roll with the multiple window component method – this seemed like it would be flexible and easy to address out the gate. Before I was ready for this step I had to get a few other things in order in my network. Ultimately, the need that I’m working to fill is distortion and blending for the interior surface of a dome using three projectors that need to warp and edge blend in real time. First up on my way to solving that problem was looking at using a cube map ) in order to address some of this challenge. In this first network we can see six faces of a cube map composited together, exported to a phong shader, and then applied to a dome surface which is then rendered in real time from three different perspectives.

Screenshot_121913_105543_PM

A general over view of the kind of technique I’m talking about can be found here. The real meat and potatoes of what I was after in this concept testing was in this part of the network:

Screenshot_121913_105619_PM

Here I have three camera components driving three different Render TOPs, which are in turn passing to three Null TOPs that are named P1, P2, and P3 – projector 1 – 3. As this was a test of the concepts of multiple monitor outs, you’ll notice that there isn’t much difference between the three different camera perspectives and that I haven’t added in any edge blending or masking elements to the three renders. Those pieces are certainly on their way, but for the sake of this network I was focused on getting multiple windows out of this project.

If we jump out of this Container Comp we can see that I’ve added three Window Components and a Button to my network. Rather than routing content into these window elements, I’ve instead opted to just export the contents to the window comps.

Screenshot_121913_105514_PM

If we take a closer look at that parameters of the Window Comp we can see what’s going on here in a little more detail:

Screenshot_121913_111605_PM

Here we can see that I’ve changed the Operator path to point to my null TOP inside of my container COMP. Here we can see that the path is “/project1/P1”. The general translation of this pathway would be “/the_name_of_container/the_name_of_the_operator“. Setting Operator path to your target operator will export the specified null when the window is opened, but it will not display the contents of the null in the node itself. If you’d like to see a preview of the render on the window node, you’ll also need to change the node pathway on the Common Page of the Window Comp. Here we can see what that looks like:

Screenshot_121913_111619_PM

Finally, I wanted to be able to test using a single button to open and close all three windows. When our media server is up and running I’d like to be able to open all three windows with a single click rather than opening them one window comp at a time. In order to test this idea, I added a single button component to my network. By exporting the state of this button to the “Open” parameter of the window on the Window Page I’m able to toggle all three windows on and off with a single button.

TouchDesigner | Import from a System Folder

One of the handy building blocks to have in your arsenal as you’re working with any media system is an understanding of how it works with system folders. Isadora, for example, pulls assets from the folder specified when you load the original file. This means that you can change an asset, save it with the same name and in placing it in the proper system folder your changes show up without having to re-import any files. What then, if I want to automatically pull files from a folder on my computer? In TouchDesigner that’s a relatively simple thing to accomplish without too much work. Better yet, the underlying principles open up a lot of much more interesting possibilities.

Alright, let’s start with a simple problem that we can solve with TouchDesigner. I want to build a simple patch where I can use two control panel buttons to navigate through a list of images (or movies) in a specified folder – essentially I want to be able to add my photos or videos to a folder on my computer, and then display them in TouchDesigner without having to drag them into the network or having to add a Movie-In TOP.

Let’s start by looking at the full network.

Our system is made of three strings – a series of CHOPs, a series of DATs, and a TOP. We’re going to use two buttons to control a series of channel operators to give us a number. We’re going to use this number to select a specific file from a folder, and then we’re going to use that file name in a Movie-In Texture Operator to be displayed.

Let’s start with our CHOP string.

Button – Trigger – Count – Math – Null

First things first, the important place to start in this particular string is actually in the middle at the Count CHOP. The Count CHOP, in this case, is going to watch for a change in a signal and use that as a mechanism for determining when to increment or reset. There are lots of different ways to count, but in this particular case the Count CHOP is an excellent method for incrementing from a button press. Alright, so if we know that we’re going to start with a count, then we don’t actually need the Trigger CHOP – especially as I’m going to watch a Button for a signal change. That said, the Trigger CHOP might be handy if I’m watching something that doesn’t have a specific binary, or a signal where I want a certain threshold to act as the trigger. You’ll notice that my Trigger is bypassed, but I’ve left it in my network just in case I want to use a different kind of triggering method. Alright, with some of those things in mind, lets move back to the beginning of this string.

First off I start by adding to Button COMPs to my network (If you want to learn more about Buttons and Control Panels Start here). Zooming into the button COMP I can set the text for the button itself. First I want to find the Text TOP labeled “bg.” Then I want to open the parameter window for this TOP and change the Text to “Forward” (you can name this whatever you want, I just wanted to know that this button was going to move me forward through my set of photos).

We’ll do the same thing for the other button, and in my case I both changed it’s name and it’s color. You can do both from the Text TOP in your second Button COMP.

Moving back out of the Button COMP I’m going to make a few other changes to my buttons. I want to change their position on the control panel, and I want to make them a little larger to accommodate the longer words inside of them. To do this I’m going to open up the parameter window of the Button COMPs and change the width of both to 66. I’m also going to change the Y location of button labeled “Forward” to 55 to make sure that my buttons aren’t on top of one another.

Before moving on, I also want to change the button Type for both of these COMPs to “Momentary.”

Next I’m going to connect my Button called “forward” to the top inlet on my Count CHOP, and the Button “reset” to the middle inlet on the Count CHOP. This means that every time I click the forward button I’ll increment the count up by 1, and when I click reset it will set the counter back to 0.

In my Math CHOP the only change that I’m going to make is to Pre-Add 1 to the integer coming form my count. I’m going to do this because TouchDesigner counts the first row of a table as 0. When we get to our DATs we’ll see that the 0 line in our table is actually the header of a table. While I could tell TouchDesigner not to show the header, that doesn’t feel like the best solution – especially as I want to be able to see what columns are referring to.

Last but not least, I’m going to end this string of CHOPs in a Null. If you’re new to TouchDesigner, it’s important to know that ending a string in a Null can be a true life safer. Ending in a Null allows you to make changes up stream of the Null, without any headache. Let’s imagine in a complicated network that you’ve connected 20 or 50 different operators to a CHOP. You decide that you want to change something in your string, and you delete the final CHOP (the one that’s connected to everything) leaving you to reconnect all of those patch chords in order to get your network back up and running again. The better solution is to end the string in a Null, and then connect the Null to those 20 or 50 or 1000 other operators. Changing things up stream of the Null is easy, especially as you don’t have to reconnect all of your patch chords again.

Now let’s take a look at our DAT stream.

(Text) – Folder – Select – Null

Here the Text DAT isn’t an essential part of my network, at least in terms of operation. As a general rule of thumb, I try to add comments to my networks as I’m working in them in order to make sure that I can remember how I did something. I highly recommend having some kind of commenting system, especially as this can prove to be a helpful tool to communicate with collaborators about what you’re doing with a specific part of a network. All of that to say, that we can ignore my first Text DAT.

The Folder DAT will pull the contents of a specified system folder. Ultimately, I’ll need the pathway of the files in this folder. I can set the Folder DAT to give me all of that information, and specify if I want it to include the content of sub folders, or the names of sub folders. In my case I only want the names and locations of the files in this single folder.

Next I want to select a single line out of this whole table. To do this, we’re going to connect the Folder DAT to a Select DAT.

First up I want to set this DAT to extract Rows by Index. I want to make sure that both Include First Row, and Include First Column is turned off. Next I want to export the Null CHOP to both the Start Row Index and the End Row Index. Finally, I want to set the Start Column Index to 5, and the End Column Index to 5 – I’m telling the Select DAT that I’m only interested in the sixth column in this table (but you set it to 5, whats the deal? remember 0 counts as a column, so when you’re counting columns in your table 0 is the new 1). I’m also telling it that I only want to see the row that’s specified by the CHOP string.

Last but not least, I’m going to pass this to a Null.

Finally, let’s add a Movie-In TOP to our network. Now, in the Movie-In parameter dialog we’re going to do a little typing. We need to tell the Movie-in TOP that it should reference a cell in a table in a DAT. Easy. Right? Sure it is. We’re going to first click the little plus that shows up as your pointer hovers over the left side of the parameter window.

Next we’re going to toggle the field from a constant into a expression by clicking on the little blue box.

Finally, we’re going to write a short expression to tell TouchDesigner where to look for the pathway of a file to load. Our expression looks like this:

op(‘null1’)[0,0]

If we were writing that in English it would read – Look at the Operator named “null1” and find the contents of the cell in the 0 column position and in the 0 row position. It’s important to note that the name in single quotes should be the name of the DAT Null that we created before, otherwise this won’t act as expected.

There you have it. We’ve now made a network that will allow us to load the contents of a folder, sequentially based on how we use two buttons in our control panel.

WonderDome | Workshop Weekend 1

WonderDome

In 2012 Dan Fine started talking to me about a project he was putting together for his MFA thesis. A fully immersive dome theatre environment for families and young audiences. The space would feature a dome for immersive projection, a sensor system for tracking performers and audience members, all built on a framework of affordable components. While some of the details of this project have changed, the ideas have stayed the same – an immersive environment that erases boundaries between the performer and the audience, in a space that can be fully activated with media – a space that is also watching those inside of it.

Fast forward a year, and in mid October of 2013 the team of designers and our performer had our first workshop weekend where we began to get some of our initial concepts up on their feet. Leading up to the workshop we assembled a 16 foot diameter test dome where we could try out some of our ideas. While the project itself has an architecture team that’s working on an portable structure, we wanted a space that roughly approximated the kind of environment we were going to be working in. This test dome will house our first iteration of projection, lighting, and sound builds, as well as the preliminary sensor system.

Both Dan and Adam have spent countless hours exploring various dome structures, their costs, and their ease of assembly. Their research ultimately landed the team on using a kit from ZipTie Domes for our test structure. ZipTie Domes has a wide variety of options for structures and kits. With a 16 foot diameter dome to build we opted to only purchase the hub pieces for this structure, and to cut and prep the struts ourselves – saving us the costs of ordering and shipping this material.

In a weekend and change we were able to prep all of the materials and assemble our structure. Once assembled we were faced with the challenge of how to skin it for our tests. In our discussion about how to cover the structure we eventually settled on using a parachute for our first tests. While this material is far from our ideal surface for our final iteration, we wanted something affordable and large enough to cover our whole dome. After a bit of searching around on the net, Dan was able to locate a local military base that had parachutes past their use period that we were able to have for free. Our only hiccup here was that the parachute was multi colored. After some paint testing we settled on treating the whole fabric with some light gray latex paint. With our dome assembled, skinned, and painted we were nearly ready for our workshop weekend.

Media

There’s healthy body of research and methodology for dome projection on the web, and while reading about a challenge prepped the team for what we were about to face it wasn’t until we go some projections up and running that we began to realize what we were really up against. Our test projectors are InFocus 3118 HD machines that are great. There are not, however, great when it comes to dome projection. One of our first realizations in getting some media up on the surface of the dome was the importance of short throw lensing. Our three HD projectors at a 16 foot distance produced a beautifully bright image, but covered less of our surface than we had hoped. That said, our three projectors gave us a perfect test environment to begin thinking about warping and edge blending in our media.

TouchDesigner

One of the discussions we’ve had in this process has been about what system is going to drive the media inside of the WonderDome. One of the most critical elements to the media team in this regard is the ability to drop in content that the system is then able to warp and edge blend dynamically. One of the challenges in the forefront of our discussions about live performance has been the importance of a flexible media system that simplifies as many challenges as possible for the designer. Traditional methods of warping and edge blending are well established practices, but their implementation often lives in the media artifact itself, meaning that the media must be rendered in a manner that is distorted in order to compensate for the surface that it will be projected onto. This method requires that the designer both build the content, and build the distortion / blending methods. One of the obstacles we’d like to overcome in this project is to build a drag and drop system that allows the designer to focus on crafting the content itself, knowing that the system will do some of the heavy lifting of distortion and blending. To solve that problem, one of the pieces of software that we were test driving as a development platform is Derivative’s TouchDesigner.

Out of the workshop weekend we were able to play both with rendering 3D models with virtual cameras as outputs, as well as with manually placing and adjusting a render on our surface. The flexibility and responsiveness of TouchDesigner as a development environment made this process relatively fast and easy. It also meant that we had a chance to see lots of different kinds of content styles (realistic images, animation, 3D rendered puppets, etc.) in the actual space. Hugely important was a discovery about the impact of movement (especially fast movement) coming from a screen that fills your entire field of view.

TouchOSC Remote

Another hugely important discovery was the implementation of a remote triggering mechanism. One of our other team members, Alex Oliszewski, and I spent a good chunk of our time talking about the implementation of a media system for the dome. As we talked through our goals for the weekend it quickly became apparent that we needed for him to have some remote control of the system from inside of the dome, while I was outside programming and making larger scale changes. The use of TouchOSC and Open Sound Control made a huge difference for us as we worked through various types of media in the system. Our quick implementation gave Alex the ability to move forward and backwards through a media stack, zoom, and translate content in the space. This allowed him the flexibility to sit away from a programming window to see his work. As a designer who rarely gets to see a production without a monitor in front of me, this was a huge step forward. The importance of having some freedom from the screen can’t be understated, and it was thrilling to have something so quickly accessible.

Lights

Adam Vachon, our lighting designer, also made some wonderful discoveries over the course of the weekend. Adam has a vested interest in interactive lighting, and to this end he’s also working in TouchDesigner to develop a cue based lighting console that can use dynamic input from sensors to drive his system. While this is a huge challenge, it’s also very exciting to see him tackling this. In many ways it really feels like he’s doing some exciting new work that addresses very real issues for theaters and performers who don’t have access to high end lighting systems. (You can see some of the progress Adam is making on his blog here)

Broad Strokes

While it’s still early in our process it’s exciting to see so many of the ideas that we’ve had take shape. It can be difficult to see a project for what it’s going to be while a team is mired in the work of grants, legal, and organization. Now that we’re starting to really get our hands dirty, the fun (and hard) work feels like it’s going to start to come fast and furiously.


Thoughts from the Participants:

From Adam Vachon

What challenges did you find that you expected?

The tracking; I knew it would hard, and it has proven to be even more so. While a simple proof-of-concept test was completed with a Kinect, a blob tracking camera may not be accurate enough to reliably track the same target continuously. More research is showing that Ultra Wide Band RFID Real Time Locations System may be the answer, but such systems are expensive. That said, I am now in communications with a rep/developer for TiMax Tracker (an UWB RFID RTLS) who might be able to help us out. Fingers crossed!

What challenges did you find that you didn’t expect?

The computers! Just getting the some of computers to work the way they were “supposed” to was a headache! That said, it is nothing more than what I should have expected in the first place. Note for the future: always test the computers before workshop weekend!

DMX addressing might also become a problem with TouchDesigner, though I need to do some more investigation on that.

How do you plan to overcome some of these challenges?

Bootcamping my macbook pro will help on the short term computer-wise, but it is definitely a final solution. I will hopefully be obtaining a “permanent” test light within the next two weeks as well, making it easier to do physical tests within the Dome.

As for TouchDesigner, more playing around, forum trolling, and attending Mary Franck’s workshop at the LDI institute in January.

What excites you the most about WonderDome?

I get a really exciting opportunity: working to develop a super flexible, super communicative lighting control system with interactivity in mind. What does that mean exactly? Live tracking of performers and audience members, and giving away some control to the audience. An idea that is becoming more an more to me as an artist is finding new ways for the audience to directly interact with a piece of art. On our current touch-all-the-screens-and-watch-magic-happen culture, interactive and immersive performance is one way for an audience to have a more meaningful experience at the theatre.

 

From Julie Rada

What challenges did you find that you expected?

From the performer’s perspective, I expected to wait around. One thing I have learned in working with media is to have patience. During the workshop, I knew things would be rough anyway and I was there primarily as a body in space – as proof of concept. I expected this and didn’t really find it to be a challenge but as I am trying to internally catalogue what resources or skills I am utilizing in this process, so far one of the major ones is patience. And I expect that to continue.

I expected there to be conflicts between media and lights (not the departments, the design elements themselves). There were challenge, of course, but they were significant enough to necessitate a fundamental change to the structure. That part was unexpected…

Lastly, directing audience attention in an immersive space I knew would be a challenge, mostly due to the fundamental shape of the space and audience relationship. Working with such limitations for media and lights is extremely difficult in regard to cutting the performer’s body out from the background imagery and the need to raise the performer up.

What challenges did you find that you didn’t expect?

Honestly, the issue of occlusion on all sides had not occurred to me. Of course it is obvious, but I have been thinking very abstractly about the dome (as opposed to pragmatically). I think that is my performer’s privilege: I don’t have to implement any of the technical aspects and therefore, I am a bit naive about the inherent obstacles therein.

I did not expect to feel so shy about speaking up about problem solving ideas. I was actually kind of nervous about suggesting my “rain fly” idea about the dome because I felt like 1) I had been out of the conversation for some time and I didn’t know what had already been covered and 2) every single person in the room at the time has more technical know-how than I do. I tend to be relatively savvy with how things function but I am way out of my league with this group. I was really conscious of not wanting to waste everyone’s time with my kindergarten talk if indeed that’s what it was (it wasn’t…phew!). I didn’t expect to feel insecure about this kind of communication.

How do you plan to overcome some of these challenges?

Um. Tenacity?

What excites you the most about WonderDome?

It was a bit of a revelation to think of WonderDome as a new performance platform and, indeed, it is. It is quite unique. I think working with it concretely made that more clear to me than ever before. It is exciting to be in dialogue on something that feels so original. I feel privileged to be able to contribute, and not just as a performer, but with my mind and ideas.

Notes about performer skills:

Soft skills: knowing that it isn’t about you, patience, sense of humor
Practical skills: puppeteering, possibly the ability to run some cues from a handheld device

TouchDesigner | Buttons for Interfaces

Building any system that requires an operator who isn’t the system designer often involves creating a control panel to operate the patch or network in question. A control panel can also be useful for controlling various elements of a system that might otherwise be too complicated to manipulate directly.

In TouchDesigner one of the handy built in components that you might think about using is a button. Buttons have various states of operation that offer different types of control. Let’s star this process by looking at some actual buttons in TouchDesigner so we can see some how these work in practice.

Lets start by creating a new network, and deleting the template operators – if you’re new to touch you can draw a rectangular selection by clicking and dragging with the right mouse button. Select all off the operators and hit delete or backspace. You can also close your palette browser. You should now be left with an empty network.

To get started we’re going to double click the empty network or hit TAB on your keyboard. This brings up the Operator Dialog window. Make sure that you’re in the COMP (component) tab of the dialog window, and select a “Button” from the “Panels” column.

Place this button anywhere on your network. If we look at the parameters of this component we can start to get a sense of how it works, and how we can customize its operation for whatever you might be designing.

Looking at the parameter window we can see how this component is organized. This component has several tabs of parameters that all relate to different types of functions and methods for manipulating the button. Below is a quick summary what each tab deals with:
The Layout page gives us the ability to manipulate how the button is placed and represented on the control panel itself.

  • The Panel page gives us some control over how the button behaves in the panel.
  • The Button page allows for manipulating the kind of button operation methods.
  • The Color page allows for some alteration in the color of the button.
  • The Drag page controls the drag and drop behavior of the button.
  • The Common page controls for some of the higher order mechanisms for this component – for now we don’t need to worry about this page.

To learn more about the Button COMP make sure that you look at Derivative’s documentation here.

Before we move on, let’s open up our control panel so we can see what our button looks like, and where it’s situated. In the upper left hand corner of your network there’s a small square shaped button. Mousing over this button will reveal the help text “open viewer.” Clicking on this button will bring up the panel viewer for the container that you’re working inside of.

Clicking on your open viewer button should mean that you see something like this:

At this point you should be able to click on your button in the viewer. To help us understand what’s happening with the button itself let’s attach a Null to the CHOP out from the container. In your network add a Null Chanel Operator (CHOP). Attach this to the CHOP out from the button (you can tell that it’s a Chanel Operator Out because of the color). With the null attached, now when we click on our button in the viewer we can see what kind of signal is being generated by this button component.

Before we take a closer look at button modes, let’s first see how we can position our button in the viewer. First select the button component to bring up the parameters window. Here on the Layout page we can see X and Y as variables that we can manipulate. These allow us to change the position of the button. The values for X and Y are in pixels. For now, I’m going to leave my button in the bottom left corner.

Next lets take a look at the Button page for this component. Two of the most commonly used states for buttons are Momentary and Toggle. Momentary transmits a signal as a pulse. Clicking the button sends an on, releasing the button sends an off. A toggle enables a button state to be activated or deactivated. In this case, clicking the button toggles it to the on position where it stays until you click the button again sending a signal of off. There are additional button types to choose from, but for now these two different types will help us enough to work through some basic ideas about how you might use and change a button. I’m going to leave my button set as a toggle down for now.

Now that we have a basic understanding of how this component works, lets start customizing it’s appearance. To do this we need to dive inside of our Button. We can do this by zooming in with the scroll wheel until we dive into the component, or we can select the button and hit the “i” key on your keyboard (i in this case is a shortcut for “in”).

Inside of our button we can see the Chanel operators, Texture Operator, and Data Operator that are driving this interface element. One of the most interesting elements to take a note of us the panel1 CHOP. Here you can see that TouchDesigner tracks several different kinds of interaction for our button. We can trigger actions based on the state of our button (is it on or off), the rollover position of our mouse (is the mouse hovering over the button), and if the button is being actively selected (the moment when the mouse button is being actively clicked).

Let’s start our customization of this button by first looking at the Text TOP (Texture Operator). If you click on the text TOP labeled “bg” we can see the parameters associated with this TOP. On the first page we see “Text.” I’m going to change the name of my button here to “Play.”

Next I want to adjust the color of this button. Before we do that, lets first draw our attention to the fact that our button changes it’s color depending on if we’re mousing over it, or have turned it on or off. A closer look at the Text TOP and the associated DAT and CHOP we can see that something very specific is happening. An expression CHOP is watching the button and reporting out the state of our button as a variable called “i” which is then being used by the DAT to dynamically change the color of our button. The table DAT defines what color values should be used for the button in the states: off, on, rollover, rollover on. If we want to keep the established convention of different colors for different states, we then need to edit the table DAT rather than the text TOP.

Let’s start by zooming into the table DAT. We can see here that we’re only using a single color value here for the dynamic content. This means that our options for changing the color of the button right now are only limited to gray scale. Bummer. Let’s make some changes to our table so we can have some color. To do this we’re first going to make our DAT active by clicking on the small + sign in the button right corner of this operator.

With this DAT in active mode we can now edit the table. Color values for our Text are generated off of three different numbers – values for red, green, and blue. We’re going to add two more columns and change the name of one column so we can use three different numbers to generate some different colors. With our DAT active, right click on the column labeled “background” and select add right. Repeat this step so that you have two empty columns to the right of the background column. Next let’s edit the text in the “background” cell, changing it to “bg_r.” In the columns next to this one add the names “bg_g” and “bg_b.” This is going to break our button for a moment, and that’s okay, don’t panic. You should now have a table that looks like this:

For now copy from bg_r to the other columns. Now your table should look like this:

Next lets fix our text TOP. Click on the Text TOP and then click on the Color page. On this page we can see that our values for the background color are all 0s and in red, telling us that something is broken. If we click on the text “Background Color” on this page we can see the expressions driving the dynamic changes for this TOP.

Next we’re going to edit the text in these expressions to account for our new columns in table DAT. We can see that there are three fields that correspond to our bg_r, bg_g, and bg_b with bgcolorr, bgcolorg, and bgcolorb. We’re going to change only the last reference in the expressions in these cells to represent the changes in our table. What we’re after here is to make sure that:

  • bgcolorr is looking at the bg_r
  • bgcolorg is looking at the bg_g
  • bgcolorb is looking at the bg_b

We’ll do that by changing the term “background” in the expressions to match the columns that we’ve added.
Our edited expressions should look like the following (your button should be working again):

Now we need to change our table DAT in order to generate the colors that we want for our button. To help me determine the float values for the colors that I want, I’m going to add a constant TOP to the network momentarily. With my constant added to the network, I’m click on the white rectangle to bring up the color selector. This window will also give me the float values that I want to know so I can edit my table.

After a little bit of exploring I’ve found some colors that will work as a place to start. My resulting table DAT looks like this:

We now have a Button that we’ve given a unique name to, and made changes to a table DAT so that we can see a custom set of colors on our button that dynamically change as we operate our control panel. Now that we’ve made one, try making some buttons on your own.

Sending and Receiving OSC Values with TouchDesigner

The other day I posted look at how to send and receive TouchOSC data with Troikatronix’s Isadora. In an effort to diversify my programming background I’m often curious about how to translate a given task from one programing environment to another. To this end, I was curious about repeating this same action in Derivative’s TouchDesigner. If you’re one of a growing number of artists interested in visual programming, VJing, media design, interactive system design, media installation, or media for live performance than it’s well worth your time to look at both Isadora and TouchDesigner.

That said, I initially started thinking about control panel interfaces for TouchOSC when I saw Graham Thorne’s instructional videos about how to transmit data from Isadora to Touch OSC (If you have a chance take a look at his posts here: Part 1 & Part 2). Last year I used TouchOSC accelerometer data to drive an installation at Bragg’s Pie Factory in Downtown Phoenix, but I didn’t do much with actual control panels. Receiving data from TouchOSC in TouchDesigner is a piece of cake, but sending it back out again didn’t work the first time that I tried it. To be fair, I did only spend about 3 minutes trying to solve this particular question as it was a rather fleeting thought while working on another project.

Getting this to working Isadora, however, gave me a renewed sense of focus and determination, so with a little bit of experimenting I finally got this working the way that I wanted. While there are any number of things you might do with TouchDesigner and TouchOSC, I wanted to start with a simple deliverable: I wanted the position of Slider 1 to inversely change the position of Slider 2, and vise versa. In this way, moving Slider 1 up moves Slider 2 down, and moving Slider 2 up moves Slider 1 down. In performance setting it’s unlikely that I’d need something this simple, but for the sake of testing an idea this seemed like it would give me the kind of information that I might need.

For the most part this is pretty easy, but there are a few things to watch for in order to get the results that you want.

Here’s the network stream of Chanel Operators (CHOPs):

OSC In – Select – Math – Rename – OSC Out

OSC In

Derivative Documentation on the OSC In CHOP 

The OSC in CHOP is pretty excellent out the gate. We need to do a little configuring to get this up and running, but after that we’re truly off to the races. First off it’s important to make sure that your TouchOSC device is connected to your network and broadcasting to your TouchDesigner machine. If this process is new to you it’s worth taking a little time to read about out to set this all up – start on this page if this is new to you. Once your TouchOSC device is talking to your TouchDesigner network we’re ready to start making magic happen. One of the first things that I did was to activate (just adjust up or down) the two faders that I wanted to work with. This made sure that their IDs were sent to TouchDesigner, making it much easier for me to determine how I was going to rename in them in the future.

Screenshot_100913_094345_PMSelect

Derivative Documentation on the Select CHOP

When you’re working with the OSC In CHOP, all of your TouchOSC data comes through a single pipe. One of the first things to do here is to use the select CHOP to single out the signal that you’re looking for (as a note, you could also export the Channel in question to a Null, or handle this a variety of other ways – in my case a Select CHOP was just the first method I tried). This operator allows you to pull out a single Channel from your OSC In.

Screenshot_100913_094428_PM

Math

Derivative Documentation on the Math CHOP

Next I wanted to remap the values of my incoming signal to be inverted. A Math CHOP lets us quickly remap the range for our data by changing a few values in the Range Tab of that parameters window. You’ll notice that you first need to specify the from range (what’s the range of incoming values), and then specify the to range (what’s the range of outgoing values). In my case, the from values are 0 and 1, and the to values are 1 and 0.

Screenshot_100913_094528_PM

Rename

Derivative Documentation on the Rename CHOP

The Rename CHOP is one of the most important steps in your whole network. When communicating with TouchOSC you need to know exactly what object you’re trying to drive with your data. To accomplish this you need to know the UDP address of your device, the port number that you’re board casting to, and the name of the slider or button that you’re wanting to change. While we can specify the UPD address and port number with another CHOP, we need to use the Rename CHOP (a quick aside – there are, of course, other ways to do this – this just happens to be the method that I got working after a bit of trial and error) to specify the name of the object that we’re driving. In my case I wanted fader 1 to drive fader 2. You’ll notice that formatting of the fader is important. In this particular OSC layout there are multiple tabs. We can see this as evidenced by the “1/” preceding the name of our fader. In changing the name of this input we need to enter the precise name of the fader that we want to change – this is why we activated both faders initially, this gave us the names of both objects in the TouchOSC panel. You’ll notice that changed the name of my fader to “1/fader2”.

Screenshot_100913_094606_PM

OSC Out

Derivative Documentation on the OSC Out CHOP 

Now that we’ve processed and named our signal we’re ready to send it back out to TouchOSC. In this CHOP we need to input the UDP address of the TouchOSC device (you can find this in the info tab of the TouchOSC app) as well as the port that we’re sending to. We also want to make sure that we’re sending this as a sample, and finally we want to turn off “send events every cook.” This last step is important because it ensures that we’re only sending values when they change. If we send messages for every cook we won’t be able to see the inverse relationship between our two sliders. Correction, we’ll be able to see the relationship so long as we don’t set up slider 2 to drive slider 1. In order to create an inverse relationship with both sliders we only want to transmit data as it’s changed (otherwise we’ll find that we’re fighting with the constantly streamed position data from our slider that isn’t being activated).

Screenshot_100913_094705_PM

The last step in this process is to copy the whole string and set up fader 2 to transmit to fader 1, this last step allows both faders to drive one another in turn.

There you have it. You’ve now successfully configured your TouchDesigner network to receive and send data to a mobile device running TouchOSC.

Cue Building for Non-Linear Productions

The newly devised piece that I’ve been working on here at ASU finally opened this last weekend. Named “The Fall of the House of Escher” the production explores concepts of quantum physics, choice, fate, and meaning through by combining the works of MC Escher and Edgar Allen Poe. The production has been challenging in many respects, but perhaps one of the most challenging elements that’s largely invisible to the audience is how we technically move through this production.

Early in the process the cohort of actors, designers, and directors settled on adopting a method of story telling that drew its inspiration from the Choose Your Own Adventure books that were originally published in the 1970’s. In these books the reader gets to choose what direction the protagonist takes at pivotal moments in the drama. The devising team was inspired by the idea of audience choice and audience engagement in the process of story telling. Looking for on opportunity to more deeply explore the meaning of audience agency, the group pushed forward in looking to create a work where the audience could choose what pathway to take during the performance. While Escher was not as complex as many of the inspiring materials, its structure presented some impressive design challenges.

Our production works around the idea that there are looping segments of the production. Specifically, we repeat several portions of the production in a Groundhog Day like fashion in order to draw attention to the fact that the cast is trapped in a looped reality. Inside of the looped portion of the production there are three moments when the audience can choose what pathway the protagonist (Lee) takes, with a total of four possible endings before we begin the cycle again. The production is shaped to take the audience through the choice section two times, and on the third time through the house the protagonist chooses a different pathway that takes the viewers to the end of the play. The number of internal choices in the production means that there are a total of twelve possible pathways through the play. Ironically, the production only runs for a total of six shows, meaning that at least half of the pathways through the house will be unseen.

This presents a tremendous challenge to any designers dealing with traditionally linear based story telling technologies – lights, sound, media. Conceiving of a method to navigate through twelve possible production permutations in a manner that any board operator could follow was daunting – to say the least. This was compounded by a heavy media presence in the production (70 cued moments), and the fact that the scrip was continually in development up until a week before the technical rehearsal process began. This meant that while much of the play had a rough shape, there were changes which influenced the technical portion of the show being made nearly right up until the tech process began. The consequences of this approach were manifest in three nearly sleepless weeks between the crystallization of the script and opening night – while much of the production was largely conceived and programmed, making it all work was its own hurdle.

In wrestling with how to approach this non-linear method, I spent a large amount of time trying to determine how to efficiently build a cohesive system that allowed the story to jump forwards, backwards, and sidewise in a system of interactive inputs, and pre-built content. The approach that I finally settled on was thinking of the house as a space to navigate. In other words, media cues needed to live in the respective rooms where they took place. Navigating then was a measure of moving from room to room. This ideological approach was made easier with the addition of a convention for the “choice” moments in the play when the audience chooses what direction to go. Have a space that was outside of the normal set of rooms in the house allowed for an easier visual movement from space to space, while also providing for visual feedback that for the audience to reinforce that they were in fact making a choice.

Establishing a modality for navigation grounded the media design in an approach that made the rest of the programming process easier – in that establishing a set of norms and conditions creates a paradigm that can be examined, played with, even contradicted in a way that gives the presence of the media a more cohesive aesthetic. While thinking of navigation as a room-based activity made some of the process easier, it also introduced an additional set of challenges. Each room needed a base behavior, an at rest behavior that was different from its reactions to various influences during dramatic moments of the play. Each room also had to contain all of the possible variations that existed within that particular place in the house – a room might need to contain three different types of behavior depending on where we were in the story.

I should draw attention again to the fact that this method was adopted, in part, because of the nature of the media in the show. The production team committed early on to looking for interactivity between the actors and the media, meaning that a linear asset based play-back system like Dataton’s Watchout was largely out of the picture. It was for this reason that I settled on using troikatronix Isadora for this particular project. Isadora also offered opportunities for tremendous flexibility, quartz integration, and non-traditional playback methods; methods that would prove to be essential in this process.

Fall_of_the_House_of_Escher_SHOW_DEMO.izzIn building this navigation method it was first important to establish the locations in the house, and create a map of how each module touched the others in order to establish the required connections between locations. This process involved making a number of maps to help translate these movements into locations. While this may seem like a trivial step in the process, it ultimately helped solidify how the production moved, and where we were at any given moment in the various permutations of the traveling cycle. Once I had a solid sense of the process of traveling through the house I built a custom actor in Isadora to allow me to quickly navigate between locations. This custom actor allowed me to build the location actor once, and then deploy it across all scenes. Encapsulation (creating a sub-patch) played a large part in the process of this production, and this is only a small example of this particular technique.

Fall_of_the_House_of_Escher_SHOW_DEMO.izz 2

The real lesson to come out of non-linear story telling was the importance on planning and mapping for the designer. Ultimately, the most important thing for me to know was where we were in the house / play. While this seems like an obvious statement for any designer, this challenge was compounded by the nature of our approach: a single control panel approach would have been too complicated, and likewise a single trigger (space bar, mouse click, or the like) would never have had the flexibility for this kind of a production. In the end each location in the house had its own control panel, and displayed only the cues corresponding to actions in that particular location. For media, conceptualizing the house as a physical space to be navigated through was ultimately the solution to complex questions of how to solve a problem like non-linear story telling.

Book Keeping

Not all of the work of media design is sexy. In fact, for all of the excitement generated by flash of stunning video, interactive installations, and large scale projection there is a tremendous amount of planning and paper work involved in the process. In the case of a theatrical project, one of the designer’s crucial pieces of work comes in the form of creating a cue sheet. For the uninitiated, a cue sheet is a list of all of the moments in a show that contain media. These documents help the designer, stage manager, and director establish a common understanding about how media / lights / sound are going to be used in a given production.

Drafting a useful cue sheet is often more a matter of preference, but it warrants mentioning some things that can help the media designer get organized as she / he wrestles with a growing list of content to generate. While one could certainly use a word-processing program in order to generate a cue sheet, I prefer to working a spreadsheet. Excel or Google Spreadsheets are a fine medium for writing cues, and have features that can be extremely helpful.

Cue Sheet Must-Haves

In my opinion there are a few must-have columns in organizing your cue sheet:

The Cue – Whatever you’ve settled on using, numbers or letters or some combination of the two, you need a column that puts the cue name / number in plain sight.

The Page Number – If you’re working from a traditional script, keep track of the page number. At some point you’ll be struggling to find the place in the script where a particular cue is getting called, and knowing the page number can ensure that you stay organized in the planning and technical rehearsal process.

Duration – How long does the cue last? In talking with the director it’s important to have a shared understanding of what’s happening in a given moment in a production. Specifying how long a particular effect or video is going to last can help provide some clarity for the designer as she/he is editing, animating, or programming.

From the Script – What’s in the source material that’s driving you to design a particular look? Did a character say something in particular? Is there any stage direction that’s inspiring your choice? Having a quick reference to what’s inspiring you can be a godsend while you’re crafting the content for a production.

Notes – For all that times you say “I’ll remember,” you will invariably forget something. Write it down. Did the director say something inspiring? Write it down. Did the lighting designer mention something about the amount of ambient light during a particular moment? Write it down. Write down what you’re thinking, or brainstorming. You’re never obligated to keep anything, but having a record of what you’ve been thinking gives you something to start from when you sit down to edit / animate / program.

Shooting Notes – If you’re going to need to record video for a production, make note of what particulars you need to keep in mind at the shoot. Do you need a green screen? A particular lighting effect? A particular costume? Keeping track of what you need for a particular moment is going to make the filming process that much easier.

Checklists. At the end of my cue sheet I keep a checklist for each cue. Three columns that help me keep track of what work I’ve done, and what work is left to be done.

Filmed / Animated – Is this cue filmed or animated?
Edited – Is this footage cut and prepped for the playback system?
Installed – Is this footage installed in the playback system?

Working with a Spreadsheet

Simple Formulas

One of the truly magical parts of working with a spreadsheet is one’s ability to use simple formulas to automate a workflow. A simple example comes out of the show that I’m currently working on, The Fall of the House of Escher. A new work that’s come out of the collaborate process of the 2014 ASU Graduate Acting / Directing / Design cohort, this show is built around a structural focus on giving the audience as much agency as possible. Central to this endeavor is a choose-your-own-adventure model for a the production. Given this structure, our production has sections that are distinguished from another another by a alpha-numeric prefix – 1A, 2A, 3A, et cetera. Additionally, I want to pair the section code with the number of a particular cue. This, for example, might look like 3A-34 (section 3A, cue 34). This combination of letters and numbers could make renumber cues, should one change, a difficult process. To simplify this particular problem I can use a simple formula for combining the contents of columns.

House_of_Escher_Media_Cue_Sheet

First I start by creating separate columns for the section of the show, and the cue number. Next I created a third column intended to be a combination of these other two columns. Here I inserts the following formula: =A24&”-“&B24

Here the Google Spreadsheets (or Excel for that matter) will read this formula as: display the contents of cell A24 and insert a “-” symbol and take the contents of cell B24. This may not seem like a big deal until you consider the time saved when a cue is added or removed, forcing a change in the numbering convention of all the following cues.

Conditional Formatting

Conditional formatting largely comes in varieties that are relative to changing the background of a particular cell. In this case Excel has a much wider range of possibilities for this kind of implementation. For me, however, the simplicity of automatic color coding is tremendously helpful. For example, let’s consider the final three checklist categories that I talked about earlier. Let’s say that I want every cell that contains the word “No” to be color coded red. In order to achieve this look first I’d highlight the cells that I want the formula to apply to.

Next I’d click on the background formatting button in the toolbar and select “Conditional Formatting” from the bottom of the list.

House_of_Escher_Media_Cue_Sheet 2

Finally I’d write my simple formula to control the change of color for the Cell’s selected.

House_of_Escher_Media_Cue_Sheet 3

Multiple Tabs

Last by not least, maintaining multiple tabs on a worksheet saves time and organizational energy. Additionally this allows you to cross reference notes, cells, and thoughts in your workbook. You might, for example, maintain a working cue sheet where you can brain storm some ideas and be a little less tidy. You can then use simple reference formulas to pull the relevant data to your final cue sheet that you give to the Stage Manager. Should you have the good fortune of having an assistant, you might make a separate page in your work book to outline her/his responsibilities on a project.

A cleanly organized cue-sheet is far from exciting, but it does ensure that you stay focused and productive as you work.