After looking at how to work with accelerometer data in Isadora in an earlier post, I thought it might also be worth looking at how to approach the same challenge working with Derivative’s TouchDesigner. In the Spring of 2013, for an installation piece, I used TouchDesigner to create a sculpture with a reactive projection component. While I learned a lot about working with TouchOSC in the process, I didn’t spend much time really digging into understanding what kind of data I was getting out of my sensor – in this case I’m using an iPod Touch running TouchOSC, broadcasting data over a wireless network. This type of sensor is one that I hope to use in furutre live performances, and so spending some time really digging into what the detects is an area of interest for me.

TouchOSC to TouchDesigner

To start we need to first capture the OSC data with TouchDesigner. I’m using TouchOSC for this process. With TouchOSC installed on your mobile device you first need to make sure that’ve identified your IP address, and chosen a port number that corresponds to the port number you’ve selected in TouchDesigner. Next we need to ensure that TouchOSC is broadcasting the accelerometer data. You can do this by first tapping on the “Settings” icon in the upper corner of the TouchOSC panel. From here selection “Options” Under options we need to make sure that “Accelerometer (/xyz)” is enabled. Back in TouchDesigner we’ll need to do a little work to get started. You’ll need to create a new network, and toss out all of the template OPs so we can get started fresh. You can choose to create a new container or to use the one that’s present in the template file. Once we’re in this first container we’ll start by creating an OSC In Channel Operator (CHOP). I’ve set my network port to 5001 (the same as in TouchOSC). Once your port numbers are set you should see thee bars in the CHOP reporting out the data from TouchOSC. We can see with a quick glance here that we have three channels of information: accxyz1, accxyz2, and accxyz3. Once we dive into our network we’ll come back to this in order to control our visualization.

The Big Picture

First let’s take  look at the full patch and talk through what our next steps are going to be. In order to have a fuller sense of what the accelerometer data is doing, we’re going to create a simple visualization. We’ll do this by creating a small sphere for each data stream, and map their change to the vertical axis, while mapping a constant change to the horizontal axis. We’ll then set up a simple feedback effect so we can see a rough graph of this data over time that will automatically be cleared each time our visualization wraps around the screen. This approach should allow us to see how the three data channels from the accelerometer correlate to one another, as well as giving us a chance to consider how we might work with this kind of data flow inside of a TouchDesigner network. Start to finish we’ll work with all of the different classes of operators (Components, Textures, Channels, Surface, Materials, and Data), as look at connecting operators and exporting paraemters.

Getting our OSC Data Ready

While the process of programming often bounces back and forth between modules, I’m going to start us out by looking at how I’ve parsed the OSC data so that it can be useful for us as we transition to other operators in our network. The flow of operators looks like this:

OSC In – Select – Lag – Math – Null

The same OSC in is passed to two copies of the “select – lag – math – null” string. While there are plenty of other ways to accomplish this same flow of data, this will allow us to make changes to the mapping of data coming out of individual channels (if we happen to need that option). We start with the OSC In, and then pass this value to a Select CHOP. The Select CHOP helps us to pull a single stream of data out of our OSC In Chop. In adding the Select CHOP make sure that in your parameters window (short-cut key “p”) you’ve selected the appropriate channel. You’ll notice that in the example below, I’ve selected “accxz1” as the first select operator. Next I’ve added a Lag CHOP. This operator will allow us to control the rate at which the data is being processed from our sensor. In essence, this allows us to smooth out the noise from the accelerometer, in effect making it a little less sensitive. Your mileage may vary here, but for my current configuration I’ve set the Lag values to 0.5 and 0.5. After you get your system up and running you may well want to return to these parameters, but for the time being this is a fine place to start. The Math CHOP allows us to remap the incoming values to a different range. This is another place that will require some individual adjustments once you get your whole network set up. To get started we can begin by setting the from range to -2, and 2. The To Range should be set to -1.5, and 1.5. While you’re not likely to see values of -2 or 2 coming from your accelerometer when you get started, this will make sure that there isn’t any data that’s outside of our measurement range. We’ll end our string with a Null. It’s considered good TouchDesigner programming practice to always end your strings with a Null. This ensures that if you want to change your string, add operators, or in any way alter your work, you don’t need to re-export any data. So long as you have ended your work with a Null, you can always make changes upstream of this object. While it feels cumbersome when you’re first starting in TouchDesigner, it’s hands down a practice that will save you time and headaches as you make increasingly complicated networks. Last, but not least, don’t forget to add a Text DAT to the front end of your string. This operator doesn’t do anything in terms of function, but it does allow you a space to write comments to yourself (or whoever is going to be working with your network). Making a few notes about how your network is working, and your underlying thoughts in setting up your chain of operators will help refresh your memory when you come back to a portion of your network after focusing on other areas.

Working with Surface Operators

Anytime we’re drawing some geometry in TouchDesigner we need to Surface Operators (SOPs). While we could do this same process with an image and by only using Texture Operators (TOPs) that means that we’re relying on something other than TouchDesigner to create our artwork. For this particular exercise, I think it’s worth thinking about how we might use TouchDesigner exclusively to create the visualization we’re going to see. Let’s take a look at the whole SOP network before we dive into how to put this all together.

As a general practice we can see out the gate that we’re going to use a single sphere and then create three copies of that single geometry that we’ll then change by exporting our TouchOSC CHOPs to change the behavior of our spheres. We’ll end the whole process by passing our SOPs into a Geometry component, and applying a Phong shader.

Let’s start by first creating a single Sphere SOP. Next we’ll connect that to a Transform SOP. Next we’ll pass our sphere into a Geo Component. To do this, we’ll need to make a few changes to the standard Geo Component. Start by creating a new Geo Component. Double click on the Geo. Once inside, delete the place holding Torus and add an In SOP. Make sure that you turn on both the blue display flag and the purple render flag on the the In SOP. By replacing the Torus with an In we’ll now be able to pass in the SOP operator strings outside of the Geo Component. Exit the Geo by hitting the “u” key (u for up), or by using the mouse wheel to zoom out of the component. You should now notice that there’s an inlet on the left side of our Geo Component. You can now connect the Transform SOP to the Geo, and you should see a picture of the sphere in the window.

Next create a Material called a Phong. Our Phong is going to allow us to add some color to our Geo. Drag the Phong on top of the Geo – you should see an arrow with a plus sign appear to tell you that you’re applying this Phong shader to this Geo Component. In the options for the Phong you can now select any color you’d like. With one whole string completed, you can select the string from Transform through Phong, copy and paste the string two more times for a total of three spheres. Make sure that you connect the Sphere SOP to the two additional strings, and you should be in business.

Next we’re going to export the CHOPs from our accelerometer channels to change the location of our spheres. To make this tutorial readable, I’m going to forgo detailing how to export CHOPs. If this isn’t something you’ve done before, you can look for more information about exporting CHOPs in TouchDesigner by first looking at Derivative’s official documentation here.

Export the Null CHOPs for accxyz1-3 to the Transform SOPs of the three respective Spheres. Make sure that you export the Nulls to the Y position field of the Transform SOPs. All three speheres should now be moving up and down based upon the change in accelerometer data  for each channel.

We’re also going to want to be able to change the size of our spheres based upon the over all look of our visualization. To do this we’re going to set up a simple string of CHOPs that we’ll then export to all three Transform SOPs. Start by creating a Constant CHOP and connect it to a Null CHOP. The Constant outputs a constant number in a channel and will allow us to change the dimensions of our spheres collectively. Next export the Null from this string to all three size attributes (x, y, z) of all three transform SOPs. In the parameters of the Constant change the value of the channel to 0.025.

Finally, don’t forget to add a Text DAT. Again, this is a great place to add some notes to yourself, or notes for your operator about what you’ve done and what’s important to consider when coming back to work with this string later.

Changes over Time

Moving up and down is only a part of this challenge. While what we’ve made so far is interesting, it still lacks a few key elements. For starters, we want to be able to see this change over time. In our case that means seeing the spheres move from left to right across the screen. Here’s a look at the string of operators that’s going to help us make that happen:

Wave – Math – Null 
and
Wave – Math – Trigger – Null

We’re going to use a single Wave CHOP to drive the lateral movement of our spheres, and to act as the trigger to erase our video feedback that we’ll make once we’ve set our geometry to render.

The first thing we need to do is to create a Wave CHOP. This operator will create waves a regular intervals which we can control with the attributes of this operator. First we want to change the type of wave to be ramp. A ramp will start at 0 and increase to a specified value, before returning to 0. Next we’ll set the the Period and the Amplitude to 30.

Next we need to look at the Channel Tab of this operator’s parameters. On the Channel tab we need to set the End value to 30 seconds. The values that we’re currently changing are time it takes for the wave to cycle. You may ultimately find that you’d like this to happen faster or slower, but we can use these values at least as a place to start. We also need to set the End and REnd values to be 1800 ( 60 frames per second * 30 seconds = 1800 frames). This will ensure that we have enough time for our animation to actually wrap from right to left.

The first Math CHOP that we’re going to change is on the top string. This operation is going to scale our wave values to make sure that we’re starting just off the screen on the left, and traveling to just off the screen on the right. In the From Range insert the values 0 and 30. In the To Range change the values to -2.1 and 2.1. Connect this to the Null. Next export the Null CHOP to the X position field of all three Transform SOPs.

One of the things we can do to plan ahead, is to build a trigger based on this same wave that will erase our video feedback that’s going to be generated in the rendering process. Create a new Math CHOP and connect it to the same Wave CHOP. In the From Range change the values to 0 and 30, and in the To Range change the values to 0 and 1.

Next we’re going to add a Trigger CHOP to our second string. A trigger can be used for any number of purposes, and in our case it’s going to help ensure that we have a clean slate for each time our spheres wrap from the right side of the screen back to the left. With the Math CHOP connected to the trigger, the only change we should need to make is to ensure that the Trigger Threshold is set to 0. Connect the trigger to a Null. We’ll come back to this CHOP once we’re building the rendering system for our visualization.

Rendering

Rendering a 3D object is fairly straight forward provided that you keep a few things in mind. The process of rendering an object requires at three components: at least one Geometry (the thing being rendered), a Camera (the perspective from which it’s being rendered), and a Light (how the object is being illuminated). We’ve already got three Geo components, now we just need to add a light and a camera. Next we’ll add a Render TOP, and we should see our three spheres being drawn in the Render TOP window. If you want to know a little more about how this all works, take a moment to read through the process of rendering in a little more detail. Let’s take a look at our network to get a sense of what’s going to happen in the rendering process.

There are a couple of things happening here, but the long and short of it is that we’re going to create a little bit of video feedback, blur our image, add black background, and create a composite out of all of it.

We’re going to start by making these two strings:

Render – Feedback – Level – Composite – Blur
and
Render – Level – Over – Over – Out

Here’s a closer look at the left side of the strings of operators that are rendering our network (as a note, our Render TOP is off the screen the left):

Once our string is set up, make sure that your Feedback is set to target Comp1. Essentailly what we’re creating here a layer of rendered video that is the persistent image of the spheres passing from left to right. Additionally, at this point we want to export the trigger CHOP to the feedback TOP to the bypass parameter. When the bypass value is greater than 0 it turns off the feedback effect. This means that when our trigger goes off the screen will clear, and then the bypass value will be set back to 0.

Now let’s look at the right side of our rendering network:

Here we can see our over TOPs acting as composite agents. The final Over TOP is combined with a Constant TOP acting as a black background. Finally all of this is passed to an Out TOP so it can be passed out of our container.

That’s it! Now we’ve built a simple (ha!) visualization for the three channels of data coming out of an iPod Touch accelerometer passed over a wireless network. The next steps here are to start to play. What relationships are interesting, or not interesting. How might this be used in a more compelling or interesting way? With a programming environment like TouchDesigner the sky is really the limit, it’s just a matter of stretching your wings.