Tag Archives: TouchDesigner

Maintaining Perspective with Multiple Cameras | TouchDesigner

File this away under “interesting theoretical concepts that I’ll never use… or will I?”

At some point while making realtime generative art for massive installations you may find that you’re beyond the capabilities of traditional realtime rendering in Touch. Let’s say, for example, that you need to render 12 hd outputs for a 3 x 4 array of screens – a resolution of 7680 x 3240 certainly can be done with a single render TOP, but delivering that final texture is a little more tricky.

I’m well aware that there are a number of possible solutions to this problem but before you find yourself composing that email to me about how to hack a way to a solution… what if it wasn’t 12 ouputs, what if it was 120? 200? What if every output was 4k? The answer we’re really after here is how to draw a scene with consistent perspective across multiple machines… because at some point you’ll have to use multiple machines. So, what do we do?

Forget what we do… what does that even look like? I’m still so confused.

Okay, so let’s first look at some examples of what it looks like as reference.

In this example we can see one large canvas that spans multiple screens. This is great – it’s huge and beautiful. This example shows a large desktop, which is also great… but what if we’re after some real-time rendering? This is a great illustration of the problem we might encounter. What if these displays were all 1920 x 1080. It’s a 7 x 4 array, so that’s going to be a definite challenge for a complex scene on a single machine. At this point we probably can’t realistically produce a single pixel to pixel texture for this array on a single machine. Instead we’d have to have a system of distributed rendering machines. Okay, that’s pretty straightforward and we can do some hip flat rendering that’s all orthographic no problem. What if we want perspective? If you want perspective in your real time rendering you need a way to conceptualize what the entire “screen” is, and then how to selectively render just a portion of that larger scene.

Huh?

Consider the beautiful work of Refik Anadol. I can’t speak to exactly what technique is being used here, but it’s a good illustration of the same challenge. How can you maintain the illusion of perspective if you need to render your generative art on multiple machines? That’s the real question we’re trying to answer… and now we can look at some ideas to help us better understand that challenge.

The process and methodology described below aim to solve that problem. For this example I’m going to work in a scale that’s unrealistic… but will allow those without a commercial license to play along from home. If you have a commercial license feel free to turn up the resolution as long as you keep mathematics involved in mind.

First things first, let’s build out a simple proof of concept that will make sure we understand this problem more completely.

Let’s imagine that we have a large composition that we need to cut up (for the sake of rendering) into 4 smaller slices. That might look something like this:

multi_perspective

Remember, this is just a proof of concept so we’re going to start with a very easy implementation first, before we start to dig into the more complex questions. An important lesson to consider when it comes to programming is to start by reducing the problem to its basic elements, then when you have a foundational understanding of the issue start to scale up – don’t worry, we’ll get there we just have to start small.

Okay, so we’ve got our 2 x 2 array that we want to render. Let’s see how we can set up some cameras to render just one of those squares a piece, but all from the same point of view.

Wait! Why do they need the same point of view? We’re after the same point of view so we can maintain perspective. It’d be easy enough to use four different cameras that were translated into positions to only see their section of the larger quad, but the results from lighting and perspective calculations wouldn’t match. You can use this multi-camera transformation technique if you’re doing orthographic rendering with emissive lighting, but not if you want to maintain perspective and use non-emissive lighting. It’s okay if you don’t believe me – I didn’t believe me either, and I had to set it up and test it bunch of times before I really understood what’s happening.

What’s that going to look like? Well, for the perspective of a single camera that can see the whole scene – the effect we want to recreate eventually – we might see something like this:

full_camera_view.PNG

From the vantage point of a single camera, we want to be able to zero in on just a single quadrant in our scene, something like this:

single_quad.PNG

Eventually, we want to reassemble the view of four cameras to look like our original single camera view.

Matt, I still don’t get it. That’s okay. Keep reading, and if by the time we get done with this example it’s still not a useful technique you can stop reading. If, however, you want a means to do perspective based illusions across multiple machines for massive installations, keep reading to the end.

We’re going to set up our test by by using a part of our camera COMP that you may not have used before. Specifically, when it comes to the view page, we’re going to use the Viewing Angle and Method called “Focal Length and Aperture.” I wish I could tell you exactly what this means to TouchDesigner – spoiler, I can’t – what I can help you understand is how these values relate to one another in order to achieve our particular illusion.

We’re going to start by setting up a simple example. Add a geo to your scene, and replace the torus inside with a grid. Set your sizex parameter to be 16/9. If you want to follow along step for step , change your grid to be a polygon, and add a noise SOP with a period of 0.02 and an amplitude of 0.5. Connect that to a facet SOP with unique points and computed normals. Connect your chain of operators to a null SOP and make sure that your display and render flag are turned on for your null. You should have a simple network that looks like this:

SOPs.png

Outside of your geo add two moviefile in TOPs. In the first you can use the supplied quad arrangement in the assets portion of the git repository that accompanies this post. It’s called multi_perspective.png.In your second moviefile in TOP select the FiledGuide.tif. Composite these two together with a composite TOP, and change the operand method to Add. Connect this to a null TOP, and finally assign this null to a phong material as the color map. When you’re done you should have something that looks like this:

texture_grid.png

Whew. Alright, now we can finally get to the interesting part. Let’s add a light and a camera to our network.

Now, in our camera comp let’s set the tz parameter to 10 units:

cam_tz.PNG

Next let’s move over the view page of the camera COMP. Here we’re going to leave the projection as perspective, but we’re going to change the viewing angle method to “Focal length and Aperture. Next we’ll change our Focal Length to 10 and our aperture to 16/9:

full_view_cam

This is our camera that can see the entire scene. Let’s add a render TOP to our network so we can see what our camera sees:

full_view_cam_view

If we were to bypass our noise SOP the view of this piece of geometry would fit exactly within our view-port. From here forward it’s going to get a little interesting. We want to maintain the perspective calculations from this vantage point, but we want only a single quadrant at a time to fill our view-port. It’s almost like zooming in and cropping to only a sub-section of our view. How do we do that?

Let’s copy our first camera, and then make a few adjustments. I’m going to call my new camera cam_single_p1. Next we’re going to leave our Focal Length and Aperture settings just as they were. We are, however, going to change our window x/y parameters to be:

winx -0.25
winy 0.25

We’re also going to change our window size to be 0.5.

cam_single_p1

Let’s render that camera and see what we get:

single_cam_p1_rendered.PNG

Woah!! That works just like we wanted! Thinking through our other cameras, we can quickly see that the combination of our windows size and offsets act as a zoom and translation mechanism. Try adding 3 more cameras with the following tx/y settings:

Cam2

  • tx 0.25
  • ty 0.25

Cam3

  • tx -0.25
  • ty -0.25

Cam4

  • tx 0.25
  • ty -0.25

If you render these cameras you should see something like this:

all_cams.PNG

Okay. That’s all pretty slick, but how do all of these parameters relate?

What does it mean?!

The meat and potatoes of this technique is to define a view-port’s aspect ratio, the number of vertical slices that a single window represents, and then to specify where the offsets sit that represent the center of a given window.

Huh?!

Let’s think through our simple example. We made our rendered quad a 1.7778 x 1 rectangle – a rectangle with a 16:9 aspect ratio. This was the same value we used for our aperture. We also set the distance of our camera from our geo to be 10 units, which was the same value we used to define our focal length. The window size is a ratio of 1 over the number of vertical sections… in our case we had two vertical sections, so 1 over 2 is 0.5. Our xy window offsets represent the center of our windows in our sections. That’s a little harder to wrap our heads around, and a better way to think about it is the UV coordinates of the center of a given window into our scene. Let’s break that out a little more.

Window Size

  • 1 / number of vertical windows that can fit within our viewport

Focal Length

  • the distance of our camera to our scene window

Aperture

  • the aspect ratio of our scene window

Win X

  • ( U * 2) – 1 ) / 2

Win Y

  • – ( ( U * 2) – 1 ) / 2 )

To really dig into the power of this technique we need to push beyond just a quad based set up we need a more abstract configuration of windows in our scene. Now that we understand the mechanics of our set up, let’s look at some arbitrary configuration of windows that might be spread across a large number of machines. What if our window arrangement looked something like the below:

multi_cam_perspective_output_map

What’s going on here?

Well,  let’s imagine that you’re working on a large format LED installation where you need to slice up your scene into uniform HD chunks that then feed an LED controller. In some cases those regions overlap even though only a single LED screen is outputting the content. Relationally, however, you still need to be able to control and designate the regions for the screens. Output 5 and 7 are a good example of these. The final shape of that screen is going to be the combined outline of the two displays, but the video feed to the LED controllers needs to be consistent HD cutouts. All of the dimensions in this example could probably be managed by doing a single rendering of the full scene then cropping to a given region – but at some point your ability to render the full scene and then use TOPs to crop out pixels is going to fail. This example has 7 outputs, but it’s not hard to imagine a project that had 20 or more – as reference, the need to understand this technique came out of working on an installation with 68 discrete outputs.

Okay, okay, okay… fine. So what are all these targets and numbers about?

We’ll remember in our first proof of concept example that we were able to take advantage of a simple 0.25 offset – that makes sense right?

If our geometry is placed with it’s bottom left corner at the origin (0,0), then the center of our first slice is going to be at (0.25, 0.25):

2016-12-13 13.35.19.jpg

That’s great Matt, but that doesn’t make any sense… following this logic, our slices would have been more like:

  • Slice1 ( 0.25, 0.75 )
  • Slice2 ( 0.75, 0.75 )
  • Slice3 ( 0.25, 0.25 )
  • Slice4 ( 0.75, 0.25 )

So what gives?!

Well, we have to remember that we set up our geometry to have it’s center at the origin. If we take this into account, what we see is more like:

2016-12-13 13.36.58.jpg

Looking at this, it should make sense why we used the translation coordinates that we did. The other interesting thing to understand in this look into our geometry is to consider the following.

If the full extent of our scene represents the bounds of a complete window, then we can begin to think about our winx and winy coords as being more akin to a UV – a normalized coordinate on our full window. We need to do some additional math to compensate for the translation of our origin, but that’s pretty straightforward.

If you’re still scratching your head, that’s okay. Let’s look at how to take our new window map and create a programmatic means of slicing up that full scene.

Thinking back to our proof of concept test we need a few things in place in order for this all to work as we expect:

  • We need the transforms for all our cameras to be the same – these are on the xform page: tx, ty, tz.
  • We also need several pieces for the view page:
    • Focal Length
    • Aperture
    • Winx
    • Winy
    • Winsize

Given that our cameras aren’t going to be doing any moving once we set up our calibration, we can safely use python expressions in our parameter fields. In terms of optimization, if an op does a lot of cooking it’s often better to use exports instead of expressions – expressions end up getting complied on demand and evaluated when op op cooks. Since we want fixed cameras looking into a moving world, we can use expressions for our cameras – it’s also going to be less of a hassle to set up, which is great.

For starters let’s add our reference template into our scene. We can start by dragging in our texture, connecting a Null TOP, and then assigning this to the color of a constant Material.

reference_plate.PNG

Next let’s add a geometry to our scene. We can replace the torus inside with a rectangle that has our source window’s aspect ratio, which in this case is 16:9 or 1.778 : 1.

scene_window.PNG

Next I’m going to use a Null COMP to hold the transforms of our camera system. I’m going to set this to have a tz value of 5, and otherwise leave this alone.

Null_pars.PNG

Let’s also add an object CHOP to help us with determining the distance between the null and geometry – to be clear, we don’t need to do this with an object CHOP, we could do this with a math CHOP, or with Python.

In the Object CHOP I’m going to set our null as the target object, geo1 as the reference object, and I’m going to set this to compute distance.

object_chop.png

I’m also going to add some tables that hold some reference information for us. I want to know the width and height of our scene, as well as the width and height of a given cut-out.

ref_values.PNG

We’re also going to need a table with all of our pixel space coordinates:

output_coords.PNG

Now that we have all our primary ingredients ready, we can build out a system to convert our pixel space coordinates into a set of winx and winy translation values.

Let’s start by looking at this process in general. We can first start with our coords in pixel space:

Pixel Space

output x y
output1 640 360
output2 205 130
output3 237 518
output4 525 130
output5 752 593
output6 1013 188
output7 1012 483

These values represent the actual center of these windows in the full pixel scene. I started by making this template in Photoshop, and then measured the location of the center of each given viewport.

Once we have these values, we need to convert them into a normalized values. In other words, how do these pixel coordinates translate to UV coordinates. This is a pretty straightforward calculation – the pixel value divided by it’s respective dimension for the full scene:

  • outputx / full_scene_x
  • outputy / full_scene_y

We can set up a quick eval DAT to do all of this for us:

convert_to_uv

The two python expressions that drive this in the table2 DAT are:

me.inputCell / op( 'table_scene' )[ 'scene_x', 1 ]
me.inputCell / op( 'table_scene' )[ 'scene_y', 1 ]

Our results from this can be found in the table below.

UV Space

output x y
output1 0.5 0.5
output2 0.16015625 0.18055555
output3 0.18515625 0.71944444
output4 0.41015625 0.18055555
output5 0.5875 0.82361111
output6 0.79140625 0.26111111
output7 0.790625 0.67083333

Now that we have the UV coords that represent the center of each window, we need to convert these values into a scale that takes into account that the center of our geometry is located at the origin. For our x values we multiply our value by 2, subtract 1, and divide by 2. For our y values we use the same operation and multiply by negative 1. We can use another eval DAT to do just this for us.

convert_to_winxy.PNG

The two python expressions that drive this in table3 are:

( ( me.inputCell * 2 ) - 1 ) / 2
( ( ( me.inputCell * 2 ) - 1 ) / 2 ) * -1

Now we have taken our original pixel coords and then converted them into our winx and winy transforms.

Converted into winx winy transfroms

output x y
output1 0.0 -0.0
output2 -0.33984375 0.31944444
output3 -0.31484375 -0.21944444
output4 -0.08984375 0.319444444
output5 0.08750000 -0.323611111
output6 0.29140625 0.238888888
output7 0.290625 -0.170833333

With these values set, we just need to make sure that we compute our window size, aperture, and focal_length. Looking back to the above, calculating these remaining values should be a snap.

Window Size

Window size is 1 over the number of windows that can fit into our full scene. This also means that our total scene’s width should be n x the width of a given window.

  • In our case a single window (measured in Photoshop) is 320 pixels, and our full scene is 1280 pixels.
  • 1280 / 320 = 4
  • 1 / 4 = 0.25
  • Window Size = 0.25

Focal Length

Focal Length is the distance between our point of view (in our case the null), and our full scene. We’ve used an object CHOP to compute this distance.

Aperture

Our aperture is the aspect ratio of our full scene.

  • 1280/720 = 1.778
  • Aperture = 1.778

I’m using an eval DAT to do the computation and organization of all of this:

cam_attr_calculations.PNG

The python for these looks like this:

1 / ( op( 'table_scene' )[ 'scene_x', 1 ] / op( 'table_render_attr' )[ 'width', 1 ] )
op( 'table_scene' )[ 'scene_x', 1 ] / op( 'table_scene' )[ 'scene_y', 1 ]
op( 'object1' )[ 'dist' ]

Python in touch is name dependent, so I’d recommend looking at this example network if you’re trying to replicate this effect.

Now we can set up our cameras to correctly crop out a given viewport of the entire scene. I’m going to rely on the digits of a camera to be correspondences to the output. So in my case output1 and camera1 should be the same thing. I’m also going to use the translation values of our null to set the location of the camera.

All of that said, our expressions for our camera should look like this:

camera_expressions.PNG

Again, all of our expressions are name dependent, so I’d recommend looking over how I’ve organized this in the sample file in order to make sure you know exactly what’s referencing what.

Now we can copy past our camera 6 more times – I’m using digits in many of these expressions to match the camera digit to the output digit. Looking over my results it looks like we’re right on the money.

view_ports.PNG

To really appreciate what’s happening here, I’m going to turn off the rendering on our calibration plate, and turn on a sample piece of geometry.

view_ports_real_geo.PNG

In geo2 you can see the entire geometry, and in each of our viewports we can see that we’re only rendering the region of our geometry that falls into single view.

“That’s a mess… I don’t get it.” You might well be saying. That’s okay. Let’s rearranged our TOPs to mirror more closely what’s in our template:

view_ports_real_geo_rearranged.PNG

Hopefully, doing this we should better be able to see how these various pieces work together.

At this point we now have a means of rendering a complex scene across multiple machines (or GPUs if you’re able to use affinity on Quadro cards), and maintain perspective. That unlocks a whole new avenue for realtime rendering that breaks you away from the limitations of single machine configurations or reliance on baked media for distributed realtime rendering.

Download this example from github


It’s always lovely to get an email from Derivative headquarters.

In this case I just got a lovely ping from Malcolm to let me know that the crop parameters on the render TOP can be used for the same functions described above.

Let’s look back at our first example to understand how that might work. In this case I’m going to use the same initial camera that we set up – our cam_full_scene COMP. I’m going to use this one already since I know that it’s correctly configured to capture the entire width and height of our reference plate. Next I’m going to add a render TOP, and under the CROP page I’m going to change my crop right and crop bottom pars to 0.5. For the sake of understanding the concept I’m going to leave this in a fractional unit space, but we could just as easily determine these values as absolute pixel measures. The result of this looks like this:

render_crop1.PNG

Next up, rather than using another render TOP I’m going to use a render pass – there’s lots of good reasons to use the render pass but one of the most important considerations here is that it’s a very efficient rendering operation. We do, however, need to make a few other adjustments. We need to target our render2 as our render TOP on the Render Pass page, and we also need to toggle on clear to camera color, and clear depth buffer:

render_crop2.PNG

On our render pass we’ll need to use the following crop parameters:

render_crop2_2.PNG

As we add additional render pass tops we need to target the previous render pass – a given render or render pass TOP can only have a single render pass assigned to it.

Our crop parameters should look like this:

renderpass3

  • crop left 0.0
  • crop right 0.5
  • crop bottom 0.0
  • crop top 0.5

renderpass4

  • crop left 0.5
  • crop right 1.0
  • crop bottom 0.0
  • crop top 0.5

All in all when we’re done we should have something that looks like this:

render_crop_full.PNG

Here the result is the same, the methodology going into it all is just a little different. Like all things TouchDesigner, there are multiple means of solving the same challenge and the “right” one ultimatly comes down to the choice that’s best for your particular installation.

Happy Programming everyone.

* The git repository and support files have been updated to reflect this additional material.

Building a Calibration UI | Reusing Palette Components – The Stoner | TouchDesigner

Here’s our second stop in a series about planning out part of a long term installation’s UI. We’ll focus on looking at the calibration portion of this project, and while that’s not very sexy, it’s something I frequently set up gig after gig – how you get your projection matched to your architecture can be tricky, and if you can take the time to build something reusable it’s well worth the time and effort. In this case we’ll be looking at a five sided room that uses five projectors. In this installation we don’t do any overlapping projection, so edge blending isn’t a part of what we’ll be talking about in this case study

stoner

As many of you have already found there’s a wealth of interesting examples and useful tools tucked away in the palette in touch designer. If you’re unfamiliar with this feature, it’s located on the left hand side of the interface when you open touch, and you can quickly summon it into existence with the small drawer and arrow icon:

pallet

Tucked away at the bottom of the tools list is the stoner. If you’ve never used the stoner it’s a killer tool for all your grid warping needs. It allows for key stoning and grid warping, with a healthy set of elements that make for fast and easy alterations to a given grid. You can bump points with the keyboard, you can use the mouse to scroll around, there are options for linear curves, bezier curves, persepective mapping, and bilinear mapping. It is an all around wonderful tool. The major catch is that using the tox as is runs you about 0.2 milliseconds when we’re not looking at the UI, and about 0.5 milliseconds when we are looking at the UI. That’s not bad, in fact that’s a downright snappy in the scheme of things, but it’s going to have limitations when it comes to scale, and using multiple stoners at the same time.

stoner

That’s slick. But what if there was a way to get almost the same results at a cost of 0 milliseconds for photos, and only 0.05 milliseconds when working with video? As it turns out, there’s a gem of a feature in the stoner that allows us to get just this kind of performance, and we’re going to take a look at how that works as well as how to take advantage of that feature.

stoner_fast

Let’s start by taking a closer look at the stoner itself. We can see now that there’s a second outlet on our op. Let’s plug in a null to both outlets and see what we’re getting.

stoner_nulls

Well hello there, what is this all about?!

Our second output is a 32 bit texture made up of only red and green channels. Looking closer we can see that it’s a gradient of green in the top left corner, and red in the bottom right corner. If we pause here for a moment we can look at how we might generate a ramp like this with a GLSL Top.

glsl_vuvst

If you’re following along at home, let’s start by adding a GLSL Top to our network. Next we’ll edit the pixel shader.

out vec4 fragColor;

void main()
{
 fragColor = vec4( vUV.st , 0.0 , 1.0 );
}

So what do we have here exactly? For starters we have an explicit declaration of our out vec4 (in very simple terms – our texture that we want to pass out of the main loop); a main loop where we assign values to our output texture.

What’s a vec4?

In GLSL vectors are a data type. We use vectors for all sorts of operations, and as a datatype they’re very useful to us as we often want variable with several positions. Keeping in mind that GLSL is used in pixeltown (one of the largest burrows on your GPU), it’s helpful to be able to think of variables that carry multiple values – like say information about a red, green, blue, and alpha value for a given pixel. In fact, that’s just what our vec4 is doing for us here, it represents the RGBA values we want to associate with a given pixel.

vUV is an input variable that we can use to locate the texture coordinate of a pixel. This value changes for every pixel, which is part of the reason it’s so useful to us. So what is this whole vec4( vUV.st, 0.0, 1.0) business? In GL we can fill in the values of a vec4 with a vec2 – vUV.st is our uv coordinate as a vec2. In essence what we’ve done is say that we want to use the uv coordinates to stand in for our red and green values, blue will always be 0, and our alpha will always be 1. It’s okay if that’s a wonky to wrap your head around at the moment. If you’re still scratching your head you can read more at links below

Read about more GLSL Data Types

Read about writing your own GLSL TOP

Okay, so we’ve got this silly gradient, but what is it good for?!

Let’s move around our stoner a little bit to see what else changes here.

pushingpoints

That’s still not very sexy – I know, but let’s hold on for just one second. We first need to pause for a moment and think about what this might be good for. In fact, there’s a lovely operator that this plays very nicely with. The remap TOP. Say what now? The remap top can be used to warp input1 based on a map in input2. Still scratching your head? That’s okay. Let’s plugin a few other ops so we can see this in action. We’re going to rearrange our ops here just a little and add a remap TOP to the mix.

remapTOP.PNG

Here we can see that the red / green map is used on the second input our our remap top, and our movie file is used on the first input.

Okay. But why is this anything exciting?

Richard Burns just recently wrote about remapping, and he very succinctly nails down exactly why this is so powerful:

It’s commonly used by people who use the stoner component as it means they can do their mapping using the stoners render pipeline and then simply remove the whole mapping tool from the system leaving only the remap texture in place.

If you haven’t read his post yet it’s well worth a read, and you can find it here.

Just like Richard mentions we can use this new feature to essentially completely remove or disable the stoner in our project once we’ve made maps for all of our texture warping. This is how we’ll get our cook time down to just 0.05 milliseconds.

Let’s look at how we can use the stoner to do just this.

For starters we need to add some empty bases to our network. To keep things simple for now I’m just going to add them to the same part of the network where my stoner lives. I’m going to call them base_calibration1 and base_calibration2.

calibration_bases

Next we’re going to take a closer look at the stoner’s custom parameters. On the Stoner page we can see that there’s now a place to put a path for a project.

stoner_path

Let’s start by putting in the path to our base_calibration1 component. Once we hit enter we should see that our base_calibration1 has new set of inputs and outputs:

base_capture1_added

Let’s take a quick look inside our component to see what was added.

inside_base1.PNG

Ah ha! Here we’ve got a set of tables that will allow the stoner UI to update correctly, and we’ve got a locked remap texture!

So, what do we do with all of this?

Let’s push around the corners of our texture in the stoner and hook up a few nulls to see what’s happening here.

working_with_calibration1

You may need to toggle the “always refresh” parameter on the stoner to get your destination project to update correctly. Later on we’ll look at how to work around this problem.

So far so good. Here we can see that our base_calibration1 has been updated with the changes we made to the stoner. What happens if we change the project path now to be base_calibration2? We should see that inputs and outputs are added to our base. We should also be able to make some changes to the stoner UI and see a two different calibrations.

working_with_calibration2.PNG

Voila! That’s pretty slick. Better yet if we change the path in the stoner project parameter we’ll see that the UI updates to reflect the state we left our stoner in. In essence, this means that you can use a single stoner to calibrate multiple projectors without needing multiple stoners in your network. In fact, we can even bypass or delete the stoner from our project once we’re happy with the results.

no_stoner

There are, of course, a few things changes that we’ll make to integrate this into our project’s pipeline but understanding how this works will be instrumental in what we build next. Before we move ahead take some time to look through how this works, read through Richard’s post as well as some of the other documentation. Like Richard mentions, this approach to locking calibration data can be used in lots of different configurations and means that you can remove a huge chunk of overhead from your projects.

Next we’ll take the lessons we’ve learned here combined with the project requirements we laid out earlier to start building out our complete UI and calibration pipeline.

Building a Calibration UI | Software Architecture | TouchDesigner

Here’s our first stop in a series about planning out part of a long term installation’s UI. We’ll focus on looking at the calibration portion of this project, and while that’s not very sexy, it’s something I frequently set up gig after gig – how you get your projection matched to your architecture can be tricky, and if you can take the time to build something reusable it’s well worth the time and effort. In this case we’ll be looking at a five sided room that uses five projectors. In this installation we don’t do any overlapping projection, so edge blending isn’t a part of what we’ll be talking about in this case study.

Big Ideas

Organization Matters

How you build a thing matters. Ask any architect, chef, crafts person, quilter, and on and on and on. The principles and ideas that drive how you’re building your network matter, and as a full disclaimer I am a misery to collaborate with when it comes to messy TouchDesigner networks and messy directories. “As long as it works, it doesn’t matter Matt!” Is the mantra I often hear as a push back to my requests for organized work. There’s a lot of truth in that, but if you ever have to work with another person (or work with your future self) then the organization of your work will matter. Cluttered, messy, or disorganized structures will make a world of misery for you and your collaborators. You can do whatever you want at the end of the day, but my strong recommendation is that you create some guiding principles for your organization structures, network layout, documentation, and pending todo items on a project.

Here are some of my simple suggestions:

Turn off node resizing for TOPs and COMPs, and don’t resize your nodes by hand:

What? Why? Those things might make perfect sense to you, and you might like them a lot – and that’s great. I hate them in my networks because they insert irregularities into the flow of the network that spoof importance. When I look at a network and there is a single node that’s larger, smaller, resized in anyway that’s different from the surrounding nodes  it implies to me that it has some significance or importance. It might well be that you just needed a closer look for a second and then forgot to change its size again, but for the person looking at your code for the first time that it is significant – even if it’s not. Time and again I’ve seen that kind of use of resizing throw off myself, and many of programmers that I respect.

If you need to use size as a defining characteristic in your networks, then use python to make nodes consistent sizes. It all goes back to being organized and consistent. Help your future self, or the other people you’re working with avoid as much confusion as possible… chances are the project is already confusing enough.

resizing

Align your ops:

I like the left bottom corner of my ops aligned with the network grid. I like to start new ops on new grid lines, and I like a full grid line of vertical space between ops. I also like to arrange ops (when possible) so that wires don’t cross unnecessarily. OCD much Matt? Sure, you can say that if you like. For me, the truth of the matter is that I like tidy networks where I can quickly see the flow of operations left to right. Too much space and things feel spread out all over creation, too little space and it can be difficult to see a network’s operation when you’re zoomed out too far. For me, this is the right kind of balance. It might well be different for you, and that’s wonderful. My only encouragement here is to be purposeful. Whatever you do, make sure it’s a choice you’re making not just the happenstance layout that emerged from the creative process. Be messy, be wild, place nodes every which way … but tidy up before you save and commit your work for the day.

alignment

Color code sparingly

I love me some color coding, but make sure it’s done purposefully and give yourself a key. This is another situation where color coding is often a great idea on the face of it, and then it quickly changes into something you can hardly keep track of. That’s okay, and my experience has taught me that less is better in this regard.

Leave those op names alone and only append after the op name

It took a lot to get me on board for this, but it’s now something that I’m whole wholeheartedly behind. Leaving the original operator name in place and only appending after is a tremendous help when it comes to quickly glancing at a network. It helps the viewer know quickly what an operator is doing without having to do any additional inspection.

“I don’t get it Matt.”

That’s okay. Let’s look at this example:

names_nightmare

In 2 seconds or less can you tell if the first operator with the banana is a select or a moviefilein TOP? Can you tell if the composited image is a comp TOP or a multiply TOP? Can you tell if the TOP labeled “done” is a null or an out TOP?

Compare that with:

names

To be fair, the names aren’t nearly as exciting, but they do make it much easier to understand what’s happening at a glance.

Like it or not, you’re an engineer now

As much as we all might like to fancy ourselves Artists with capital letter As and a filigree flourish, the truth is that right now you’re an engineer.  That’s not to say that you’re not an artist too, you are; but some problems don’t need artful solutions, they need thoughtfully engineered solutions based on a complex understanding of computation and computer science. The more you work in TouchDesigner, the more you’ll find that you’re as much engineer as you are artist – that’s not only okay, that’s an important realization to make about your own work. It can be a bifurcating moment to feel at conflict with the art, and the mechanics of a given network – “this approach is beautiful and it works but costs 20 milliseconds” is probably not a viable solution. It’s especially not a viable solution for something like a calibration pipeline that needs to have as little computational overhead as possible. The work that goes into an efficient process isn’t always glamorous, or much to show off; instead, it makes room for the art with a capital A to take more cycles and be more expressive.

Lucky for us, there’s room to be both engineer and artist in our work. Remember to embrace both of those roles, even when faced with the frustration of complex logic problems, or when faced with the questions of aesthetics.

Externalizing Files

So we should externalize files? Yes. For the love of god, yes. If you believe in a world where you don’t tear your hair out, yes. In case you still are wondering, the answer is yes – you should always look for ways to externalize files.

Why? That seems silly.

Once upon a time I didn’t externalize any components, and my projects lived as complete toe files and the world was beautiful. Then I crashed my project because I did something silly, and then it took me hours to track down where the problem was… even opening the crashautosave.toe left me slowly moving through networks in networks trying to figure out what I had done where to make everything crash. Worse yet, because it was just a single toe file there was no way to unit test any of the smaller modules. Open, change, save, open crash… repeat. If you haven’t had this experience yet, it’s only a matter of time.You too will crash one day, and mightily. You’ll wonder why you ever wanted to work with computers in the first place, and why you don’t just open up a bar in on a sandy beach somewhere.

Snark aside, externalizing components in your network has several benefits. For starters, it moves you away from having a single toe file black box that’s difficult to test or debug. Separating out smaller modules allows for more portability between the applications that you build, and it means that you can test just that module. It also means that you can update just that module and not your entire toe file.

Externalizing modules, extensions, and the like also lets you more easily compare files over time if you’re using a piece of version control software – don’t worry, we’ll talk about git in just a second. It also makes it easier to work with text files in something like sublime (my favorite) or Visual Studio Code. That might not seem like a big deal now, but the more you work with extensions and python, the more you’ll want robust text editing tools at your finger tips.

Externalizing files also reduces bloat in your toe file. If you’ve ever locked a texture in your toe file you know all to well that it your slim file size can quickly balloon. External files also makes it easier to save and try multiple configurations. We’re going to work on a building a calibration UI, we might want to be able to save a calibration configuration and then try another calibration. The same goes for configuration files. Rather than a configuration that’s locked into your toe file, an external file can make it easy to load lots of different configurations or other settings.

It might well seem like I’m belaboring this point, and I’m doing that on purpose. Working with external files is more work, takes more organization, takes better planning, and requires a more meticulous approach – but if you want to grow as a developer and programmer, you’ll wrestle with this challenge sooner or later. I’d encourage you to do that sooner rather than later.

git

What is git? Direct from the git website :

Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.

Git is easy to learn and has a tiny footprint with lightning fast performance. It outclasses SCM tools like Subversion, CVS, Perforce, and ClearCase with features like cheap local branching, convenient staging areas, and multiple workflows.

But what does that mean?! Unlike working with art where a tool like Dropbox would be helpful, when we’re developing software it’s often more helpful to be have some specific means of specifying when a file needs to be saved (probably with the same name), along with a message about what our changes have done. Let’s say we’re working on our calibration UI, we’ve made a change that now allows us to save out our calibration configuration. That’s great, but it’s come at the cost of changing a piece of how our UI works. We know for sure that the old way works 100% of the time, and we’re pretty sure that our new way works at least 80% of the time. Git allows us to check in the new version of our software, commit it to our working directory structure, and make a note of what we did.

We’re smart programmers, so we’ve also started to externalize our files. This means that we’ve separated out parts of our network responsible for different tasks. With git we’re now able to check in only the single tox that’s changed while we’ve been working. Better yet, we can even work with another person. Developer 1 might be working on one part of our project, while Developer 2 is working on another. Both developers are saving their respective tox files, and git allows us to track that progress and merge work seamlessly. If we do end up editing the same files, we end up with a merge conflict… which means what Matt? A merge conflict means just that, two files are competing for the same place in our directory, and we need to review them in order to decide which file to keep.

Git also allows us to work in multiple branches on our project.

Huh?

Let’s imagine that we find a bug, we might want to create a branch in our code to deal with resolving this bug.

Why?

Well, let us imagine a situation familiar to us all. We find a bug, squash it mightily. Only to discover that our fix has created another problem… maybe a much bigger problem. The little bug that we killed we could have worked with, but now we have a show stopper bug running a muck, that we don’t know how to stop, and the show starts in 45 minutes. Trouble shooting in another branch allows you to make fixes and then integrate them back into your project when you’re confident that they don’t create other problems.

Best of all, git creates a history that let’s us step back through each one of our commits. At any point we can return to a past state of our project. Taking advantage of modular design and externalized files makes this even more powerful because it means that with a little work we can resurrect ideas that might otherwise have been lost or discarded.

That’s a big commercial for git so far, but no cautionary tales – an cautionary tales there are plenty of. While git is a phenomenal tool, it can be a bit beastly to wrangle when  you’re first starting – know that you’ll get frustrated, know that you’ll curse it, know that you’ll want to give up on using it, then count to 10 and give it another go. Like all things you’re unfamiliar with, take your time to get used to it as a tool set. You might choose to start by using a GUI tool, but at some point you should learn some of the commands to run directly from the command line. You’ll also want to avoid checking in any of your assets. These days I divide up my projects and separate my assets from my code. I have a few exceptions for this practice, but I try to keep them few and far between – only really allowing for template files, or grids. Essentially, I try to make sure that it’s only files that are going to change in frequently.

If you’re going to use git, you’ll probably also use something like github or bitbucket. These are great services and host your code online so you can get to it from anywhere. These services also come with some great tools for making action items, bug tracking, and maintaining documentation for your project. Take advantage of these tools.

learn more about git at https://git-scm.com/
start using git with github or bitbucket

See more comics from XKCD at https://xkcd.com/

Extensions Everywhere

I’ve talked plenty of about extensions, how to use them, and what they’re good for. I’ve had some spirited discussions about when to use Extensions vs. Modules, and there’s a healthy conversation to be had there for those who want to really get into the weeds of it all. The place I’ve finally landed is this – I use extensions anytime I want to extend the capabilities of a given component. Case to case the degree of extension varies – sometimes I take fuller advantage of using classes than others, and that’s okay. To me, it’s okay if I don’t always take complete advantage of the difference between Extensions and Modules. Sticking with extensions means choosing an approach that works for 99% of the use cases where I need more flexibility in my programming. For mean that means a little bit of stability from the Touch paradigm where there are almost always at least a handful of different ways to solve the same problem. You might like Modules better, and that’s okay. In this case study, however, we’ll see the use of Extensions.

We’ll also see a move away from fragmenting our code across our network. It’s often tempting to use a button deep down in a UI somewhere to perform a logical operation. The only catch is that now that script is buried deep down in your project somewhere, and should it ever give you problems, good luck finding it. For the most part, on large projects I aim to keep the lion share of my operations focused in an extension. This takes a little more time to write, but it means that things are more organized, easier to diff between check ins, that I can change my code in a single place, and that I’m writing functions that are callable from other parts of the network. In my experience so far, this little amount of extra work makes for a larger savings across the whole project.

More about Extensions
Python in TouchDesigner | Extensions
Understanding Extensions

documenting your code – what are docstrings?

I’m a broken record. I know, but really – leave yourself notes, voluminous ones. Better yet, when using extensions make sure to add some docstrings.

“What are docstrings?” you ask.

I’m so glad you did! Docstrings are a feature in several programming languages that allow you to embed  comments or information into your code that can be inspected at run time. If you’re still scratching your head that’s okay. Let’s look at an example, this one is from the wikipedia page about docstrings:

"""
Assuming this is file mymodule.py, then this string, being the
first statement in the file, will become the "mymodule" module's
docstring when the file is imported.
"""

class MyClass(object):
    """The class's docstring"""

    def my_method(self):
        """The method's docstring"""

def my_function():
    """The function's docstring"""

“Great. But why are these useful?”

There useful in lots of ways but for me they have some special importance. For starters, they help ensure that I’m documenting my code as I go. The more tips you leave yourself along the way, the happier you’ll be in the long run. It does make your code longer, and it does take more time. But, if you’re really invested in building something reusable that you can come back to and refine over time, these are solid investments to make that process smoother in the future.

You can get to these notes from anywhere. This internal documentation can be called from anywhere in the network, so if you’ve forgotten how many arguments your method takes, or what it is supposed to return you can get some quick answers without having to track down the extension. When you start to write lots of complex methods this can become a real help. It’s probably not something you need when you’re first working on the project, but if you’re collaborating with someone else or coming back to your project over time it’ll be well worth the time and effort.

Finally worth considering is that docstrings are a part of the larger Python style guide. Like it or not you’re an engineer, and at this point you’re turning into a budding Python programmer. Learning about the best practices of other programmers outside of the TouchDesigner silo helps broaden your perspective and exposes you to how other people work. For better or worse I spent a lot of time in the liberal arts, and one of the core tenants of the ideology that underpins the arts is that exposure to other ideas and perspectives is important in its own right. I’ve carried this with me through life, and I think even in the cases of programming languages it’s worth considering. The conventions and practices of any community teaches you about how they think of the world – how it’s organized, how it aught to work, how that group values the world, or what they don’t value. The Python community is one that we participate in, even if only mostly from our gray network of operators.

learn more about docstrings
see google’s python style guide for docstrings

Mapping out our Project

You know the saying – measure twice, cut once.

Before digging in to a new project make sure you ask a lot of questions, map out as many details as possible, and make a plan.

Okay, so what about our project?

For starters

  • We know that we’ll have five projectors and one monitor that acts as a UI.
  • We know the resolution of all of our displays.
  • We know that we’ll externalize lots of files in our network.
  • We know that we’re going to use git as our version control system.
  • We know that we are going to work with other people.
  • We know that we need to document our work.
  • We know that we’ll need a way to control warping for the five different displays
  • We know that we’ll need a few other controls – both for the mapping, and also to make space for the future UI elements to control playback.
  • We know we need to previs the layout.
  • We know that we’re looking to build out a complete application with all of the features that you might expect from that kind of build.

Whew! Okay, so where do we start? Before we start throwing down ops let’s think through a plan.

We’ll need a container that holds our UI, we’ll also need a container for each of our displays. We know that it’s best to just use a single perform window that spans multiple monitors, so we should make sure we plan for that when we’re developing our project.

We also need to think about how we’re going to set up rendering our scene so we can previs it without turning on all the projectors. That will mean a multi camera set up and some live rendering.

We should also think about how we might ingest video to display for calibration and test playback. On top of that we should also build out a test module to help the creative programmer understand the correct formatting for how to work in the framework we’ve created.

We should also plan to make some space in our build to house assets that are needed across our network, as well as a place to keep our calibration data. We’ll take advantage of the stoner, so we better make some time to pull that apart and better understand how it works.

We’ll use a few set of extensions, so we should also make sure that we know how that works before we dig in too deep.

Finally, we should make sure we know how git works, since we’re going to use that along the way.

We’re bound to run into some other things we need to suss out as we go, but this should give us a solid starting point in terms of understanding the project from a distance.

Next we’ll take a closer look at the stoner to make sure we understand this component we’re going to use, then we’ll dig in and start building.

Python in TouchDesigner | Extensions | TouchDesigner

Rounding out some of our work here with Python is to look extensions. If you’ve been following along with other posts you’ve probably already looked over some extensions in this post. If you’re brand new to this idea, check out that example first.

Rather than re-inventing the wheel and setting up a completely new example, let’s instead look at our previous example of making a logger and see how that would be different with extensions as compared to a module on demand.

A warning for those following along at home, we’re now knee deep in Python territory, so what’ we’ll find here is less specific to TouchDesigner and more of a look at using Classes in Python.

In our previous example looking at modules we wrote several functions that we left in a text DAT. We used the mod class in TouchDesigner to treat this text DAT as a python module. That’s a neat trick, and for some applications and situations this might be the right approach to take for a given problem. If we’re working on a stand alone component that we want to use and re-use with a minimal amount of additional effort, then extensions might be a great solution for us to consider.

Wait, what are extensions?! Extensions are way that we can extend a custom component that we make in TouchDesigner. This largely makes several scripting processes much simpler and allows us to treat a component more like an autonomous object rather than a complex set of dependent objects. If you find yourself re-reading that last statement let’s take a moment to see if we can find an analogy to help understand this abstract concept.

When we talked about for loops we used a simple analogy about washing dishes. In that example we didn’t bother to really think deeply about the process of washing dishes – we didn’t think about how much detergent, how much water, water temperature, washing method, et cetera. If we were to approach this problem more programatically we’d probably consider making a whole class to deal with the process of washing dishes:

class Dishes():

    def __init__( self ):
        return

    def Full_cycle( self, list_of_dishes ):
        # what if we want to wash and then dry?

        self.Wash( list_of_dishes, water_temp )
        self.Dry( list_of_dishes )

        return

    def Wash( self, list_of_dishes, water_temp ):
        # what does washing really entail
        # all of that would go here

        number_of_dishes    = len( list_of_dishes )
        detergent_amt       = self.Set_soap( number_of_dishes )
        water_temp          = self.Set_water_temp( water_temp )

        return

    def Dry( self, list_of_dishes ):
        # what does washing really entail
        # all of that would go here

        number_of_dishes     = len( list_of_dishes )
        dry_time             = self.Set_dry( number_of_dishes )
        return

    def Set_soap( self, number_of_dishes ):
        return

    def Set_water_temp( water_temp ):
        if water_temp == 'hot':
            temp            = 98

        elif water_temp == 'medium':
            temp            = 80

        if water_temp == 'low':
            temp            = 60

        return temp

    def Set_dry( self, number of dishes ):
        return

Okay, so the above seems awfully silly… what’s going on here? In our silly example we can see that rather than one long function for Wash() or Dry(), we instead use several smaller helper functions in the process. Why do this? Well, for one thing it lets you debug your code much more easily. It also means that by separating out some of these elements into different processes we can fix a single part of our pipeline without having to do a complete refactor of the entire Wash() or Dry() methods. Why does that matter? Well, what if we decide that there’s a better way to determine how much soap to use. Rather than having to sort through a single long complex method for Wash() we can instead just look at Set_soap(). It makes unit testing easier, and allows us to replace or develop that method outside the context of the larger method. It also means that if we’re collaborating with other programmers we can divide up the process of writing methods.

What’s this self. business? self. allows us to call a method from inside another method in the same class. This is where extensions begin to really shine as opposed to using modules on demand. We an also use things like attributes andinheritance.

Okay, so how can we think about actually using this?

Let’s take a look at what this approach looks like in python:

import datetime

log_file            = op( 'text_log_file' )
log_path            = "example_extensions/log_files/log.txt"

full_text           = '''{now}

Current Year        | {year}
Current Month       | {month}
Current Day         | {day}
Current Hour        | {hour}
Current Minute      | {minute}
Current Second      | {second}
Current Microsecond | {microsecond}
'''

raw_date_time       = "On {month}-{day}-{year} at {hour}:{minute}:{second}"

verbose_log_message = '''============================
VERBOSE MESSAGE

{date_time}
----------------------------
operator            || {operator}
At Network Location || {path}

----------------------------
Logged
{message}
============================
'''

log_message         = '''----------------------------
{now}
----------------------------
{operator}
{path}
{message}
'''

class Ext_example():

    def __init__( self ):
        '''The init function.

        We're not doing anything with our init function in this example
        so we'll leave this empty.

        Notes
        ---------------
        ''' 
        return

    def Log_date( self ):
        year                        = datetime.datetime.now().year
        month                       = datetime.datetime.now().month
        day                         = datetime.datetime.now().day
        hour                        = datetime.datetime.now().hour
        minute                      = datetime.datetime.now().minute
        second                      = datetime.datetime.now().second

        updated_log_date            = log_date.format( 
                                                        month       = month,
                                                        day         = day,
                                                        year        = year,
                                                        hours       = hour,
                                                        minutes     = minute,
                                                        seconds     = second
                                                    )

        return updated_log_date

    def Log_date_time( self ):
        '''Create a formatted time stamp

        A look at how we might create a formatted time stamp to use with
        various logging applications.

        Arguments
        --------------- 
        None

        Returns
        ---------------
        formatted_text( str ) - a string time stamp

        Notes
        ---------------
        '''

        now         = datetime.datetime.now()
        year        = datetime.datetime.now().year
        month       = datetime.datetime.now().month
        day         = datetime.datetime.now().day
        hour        = datetime.datetime.now().hour
        minute      = datetime.datetime.now().minute
        second      = datetime.datetime.now().second
        microsecond = datetime.datetime.now().microsecond

        date_time   = raw_date_time.format( 
                                        month   = month,
                                        day     = day,
                                        year    = year,
                                        hour    = hour,
                                        minute  = minute,
                                        second  = second
                                        )

        return date_time

    def Log_message( self, operator, message, verbose=False, text_port_print=True, append_log=True ):
        '''Logging Method.

        A simple look at how you might start to think about building a logger for a TouchDesigner
        application. A logger is a great way to build out files with time stamped events. The
        more complex a project becomes, the more important it can become to have some means
        of logging the operations of your program. Here's a simple look at what that might
        look like.

        Arguments
        --------------- 
        operator( touch object ) - the touch object whose path you'd like incldued in the log message
        message( str ) - a message to include in the log
        verbose( bool ) - a toggle for verbose or compact messages
        text_port_print( bool ) - a toggle to print to the text port, or not
        append_log( bool ) - a toggle to append to the log file , or not

        Example
        --------------- 
        target_op       = op( 'constant1' )
        message         = "This operator needs attention"

        parent().Log_message( target_op, message )

        also

        parent().Log_message( target_op, message, verbose = True )

        Returns
        ---------------
        None

        Notes
        ---------------
        You'll notice that some arguments receive default values. This is so you don't have
        to include them in the call. This means that by default the message will be compact, 
        will print to the text port, and will append the log file.
        '''

        path        = operator.path
        op_name     = operator.name

        # logic tests for verbose or compact
        if verbose:
            message = verbose_log_message.format(
                                                    date_time   = self.Log_date_time(),
                                                    operator    = op_name,
                                                    path        = path,
                                                    message     = message
                                                )

        else:
            message = log_message.format(
                                            now                 = self.Log_date_time(),
                                            operator            = op_name,
                                            path                = path,
                                            message             = message
                                        )

        # logic tests for text_port_print
        if text_port_print:
            print( message )

        else:
            pass

        # log tests for appending log
        if append_log:
            log_file.write( '\n' + message )

        else:
            pass

        # save the log file to disk - external from the TouchDesigner project   
        self.Save_log()

        return

    def Save_log( self ):
        '''Saves log to disk.

        This helper function saves the log file to disk.

        Notes
        ---------------
        None
        '''

        op( log_file ).save( log_path )

        return

    def Clear_log( self ):
        '''Clears Log File.

        This helper function clears the text dat used to hold the log file.

        Notes
        ---------------
        None
        '''

        op( log_file ).clear()

        return

So getting started we set up a couple of variables that we were going to reuse several times. Next we declared our class, and then nested our methods inside of that class. You’ll notice that different from methods, we need to include the argument self in our method definitions:

# syntax for just a function
def Clear_log():
    return

# syntax for just a method in a class
def Clear_log( self ):
    return

We can also see how we’ve got several helper functions here to get us up and running – ways to add to the log, save the log to an external file, a way to clear the log. We can imagine in the future we might want to add another method that both clears the log, and saves an empty file, like a reset. Since we’ve already broken those functions into their own methods we could simply add this method like this:

def Reset_log():
    self.Clear_log()
    self.Save_log()
    return

Now it’s time to experiment.

Over the course of this series we’ve looked at lots of fundamental pieces of working with Python in TouchDesigner. Now it’s your turn to start playing and experimenting.

Happy programming!