Category Archives: Escher

Cue Building for Non-Linear Productions

The newly devised piece that I’ve been working on here at ASU finally opened this last weekend. Named “The Fall of the House of Escher” the production explores concepts of quantum physics, choice, fate, and meaning through by combining the works of MC Escher and Edgar Allen Poe. The production has been challenging in many respects, but perhaps one of the most challenging elements that’s largely invisible to the audience is how we technically move through this production.

Early in the process the cohort of actors, designers, and directors settled on adopting a method of story telling that drew its inspiration from the Choose Your Own Adventure books that were originally published in the 1970’s. In these books the reader gets to choose what direction the protagonist takes at pivotal moments in the drama. The devising team was inspired by the idea of audience choice and audience engagement in the process of story telling. Looking for on opportunity to more deeply explore the meaning of audience agency, the group pushed forward in looking to create a work where the audience could choose what pathway to take during the performance. While Escher was not as complex as many of the inspiring materials, its structure presented some impressive design challenges.

Our production works around the idea that there are looping segments of the production. Specifically, we repeat several portions of the production in a Groundhog Day like fashion in order to draw attention to the fact that the cast is trapped in a looped reality. Inside of the looped portion of the production there are three moments when the audience can choose what pathway the protagonist (Lee) takes, with a total of four possible endings before we begin the cycle again. The production is shaped to take the audience through the choice section two times, and on the third time through the house the protagonist chooses a different pathway that takes the viewers to the end of the play. The number of internal choices in the production means that there are a total of twelve possible pathways through the play. Ironically, the production only runs for a total of six shows, meaning that at least half of the pathways through the house will be unseen.

This presents a tremendous challenge to any designers dealing with traditionally linear based story telling technologies – lights, sound, media. Conceiving of a method to navigate through twelve possible production permutations in a manner that any board operator could follow was daunting – to say the least. This was compounded by a heavy media presence in the production (70 cued moments), and the fact that the scrip was continually in development up until a week before the technical rehearsal process began. This meant that while much of the play had a rough shape, there were changes which influenced the technical portion of the show being made nearly right up until the tech process began. The consequences of this approach were manifest in three nearly sleepless weeks between the crystallization of the script and opening night – while much of the production was largely conceived and programmed, making it all work was its own hurdle.

In wrestling with how to approach this non-linear method, I spent a large amount of time trying to determine how to efficiently build a cohesive system that allowed the story to jump forwards, backwards, and sidewise in a system of interactive inputs, and pre-built content. The approach that I finally settled on was thinking of the house as a space to navigate. In other words, media cues needed to live in the respective rooms where they took place. Navigating then was a measure of moving from room to room. This ideological approach was made easier with the addition of a convention for the “choice” moments in the play when the audience chooses what direction to go. Have a space that was outside of the normal set of rooms in the house allowed for an easier visual movement from space to space, while also providing for visual feedback that for the audience to reinforce that they were in fact making a choice.

Establishing a modality for navigation grounded the media design in an approach that made the rest of the programming process easier – in that establishing a set of norms and conditions creates a paradigm that can be examined, played with, even contradicted in a way that gives the presence of the media a more cohesive aesthetic. While thinking of navigation as a room-based activity made some of the process easier, it also introduced an additional set of challenges. Each room needed a base behavior, an at rest behavior that was different from its reactions to various influences during dramatic moments of the play. Each room also had to contain all of the possible variations that existed within that particular place in the house – a room might need to contain three different types of behavior depending on where we were in the story.

I should draw attention again to the fact that this method was adopted, in part, because of the nature of the media in the show. The production team committed early on to looking for interactivity between the actors and the media, meaning that a linear asset based play-back system like Dataton’s Watchout was largely out of the picture. It was for this reason that I settled on using troikatronix Isadora for this particular project. Isadora also offered opportunities for tremendous flexibility, quartz integration, and non-traditional playback methods; methods that would prove to be essential in this process.

Fall_of_the_House_of_Escher_SHOW_DEMO.izzIn building this navigation method it was first important to establish the locations in the house, and create a map of how each module touched the others in order to establish the required connections between locations. This process involved making a number of maps to help translate these movements into locations. While this may seem like a trivial step in the process, it ultimately helped solidify how the production moved, and where we were at any given moment in the various permutations of the traveling cycle. Once I had a solid sense of the process of traveling through the house I built a custom actor in Isadora to allow me to quickly navigate between locations. This custom actor allowed me to build the location actor once, and then deploy it across all scenes. Encapsulation (creating a sub-patch) played a large part in the process of this production, and this is only a small example of this particular technique.

Fall_of_the_House_of_Escher_SHOW_DEMO.izz 2

The real lesson to come out of non-linear story telling was the importance on planning and mapping for the designer. Ultimately, the most important thing for me to know was where we were in the house / play. While this seems like an obvious statement for any designer, this challenge was compounded by the nature of our approach: a single control panel approach would have been too complicated, and likewise a single trigger (space bar, mouse click, or the like) would never have had the flexibility for this kind of a production. In the end each location in the house had its own control panel, and displayed only the cues corresponding to actions in that particular location. For media, conceptualizing the house as a physical space to be navigated through was ultimately the solution to complex questions of how to solve a problem like non-linear story telling.

Custom Quartz Compositions in Isadora

The What and Why

The more I work with Isadora, the more I feel like there isn’t anything it can’t do. As a programming environment for live performance it’s a fast way to build, create, and modify visual environments. One of the most interesting avenues for exploration in this regard is working with Quartz Composer. Quartz is a part of Apple’s integrated graphics technologies for developers and is built to render both 2D and 3D content by using the system’s GPU. This, for the most part, means that Quartz is fast. On top of being fast, it allows you access to GPU accelerated rendering making for visualizations that would be difficult if you were only relying on CPU strength.

Quartz has been interesting to me largely as it’s quick access to a GPU-accelerated high performance rendering environment capable of 2D, 3D and transparency. What’s not to love? As it turns out, there’s lot to be challenged by in Quartz. Like all programming environments it’s rife with its own idiosyncrasies, idioms, and approaches to the rendering process. It’s also a fair does of fun once you start to get your bearings.

Why does all of this matter? If you purchase the Isadora Core Video upgrade you have access to all of the Core Imaging processing plugins native to OS X. In addition to that you’re now able to use Quartz Composer patches as Actors in Isadora. This makes it possible to build a custom Quartz Composer patch and use it within the Isadora environment. Essentially this opens up a whole new set of possibilities for creating visual environments, effects, and interactivity for the production or installation that you might be working on.

Enough Already, Let’s Build Something

There are lots of things to keep in mind as you start this process, and perhaps one of the most useful guidelines I can offer is to be patent. Invariably there will be things that go wrong, or misbehave. It’s the nature of the beast, paying close attention to the details of the process is going to make or break you when it all comes down to it in the end.

We’re going to build a simple 3D Sphere in Quartz then prep it for control from Isadora. Easy.

Working in Quartz

First things first, you’ll need to get Quartz Composer. If you don’t already have it, you’ll need to download Quartz Composer. Check out I Love QC’s video about how to do this:

The next thing we’re going to do is to fire up QC. When prompted to choose a template, select the basic composition, and then click “Choose.”

Template_Chooser

One of the first things we need to talk about is what you’re seeing in the Quartz environment. The grid like window that you’re started with is your patch editor. Here you connect nodes in order to create or animate your scene.

Untitled_-_Editor

You should also see a window that’s filled with black. This is your “viewer” window. Here you’ll see what you’re creating in the patch editor.

Untitled_-_Viewer

Additionally you can open up two more windows, by clicking the corresponding icons in the patch editor. First find the button for Patch Library, and click that to open up a list of nodes available for use within the network.

The Patch Library holds all of the objects that are available for use within the Quartz editor. While you can scroll through all of the possible objects when you’re programming, it’s often more efficient to use the search box at the bottom of the library.

Library

Next open open up the patch inspector.
The patch inspector lets you see and edit the settings and parameters for a given object.

Untitled_-_Editor 3

Let’s start by making a sphere. In patch library search for “sphere” and add it to your patch. Out the gate we’ll notice that this sucks. Rather, this doesn’t currently look interesting, or like a sphere for that matter. What we’re currently seeing is a sphere rendered without any lighting effects. This means that we’re only seeing the outline of the sphere on the black background.

Untitled_-_Viewer_and_Untitled_-_Editor

This brings us to one of the programming conventions in Quartz. In Quartz we have to place objects inside of other components in order to tell QC that we want a parent’s properties to propagate to the child component.

To see what that means let’s add a “lighting” patch to our network. Congratulations, nothing happened. In order to see the lighting object’s properties change the sphere, we need to place the sphere inside of that object. Select the sphere object in the editor, Command-X to cut, double click on the Lighting object, then Command-V to paste.

Untitled_-_Viewer_and_Untitled_-_Editor_1

This is better, but only slightly.

Untitled_-_Viewer_and_Untitled_-_Editor 2

Let’s start by changing the size properties of our sphere. Open the Patch Inspector and click on the Sphere object in the editor. Now we can see a list of properties for our Sphere. Let’s start by adjusting the diameter of our sphere. I’m going to change my diameter to .25.

Sphere_and_Untitled_-_Viewer_and_Untitled_-_Editor 2

Next, select “settings” from the drop down menu in the Patch Inspector. Here I’m going to turn up the number of sub divisions of my sphere to give it a smoother appearance.

Sphere_and_Untitled_-_Viewer_and_Untitled_-_Editor 3

With our sphere looking pretty decent I want to add some subtle animation to give it a little more personality. We can do this by adding a LFO (low-frequency oscillator). We’ll use our LFO to give our sphere a little up and down motion. In the Patch Library search for LFO and add it to your editor next to your sphere.

Untitled_-_Editor 4

Next click the “Result” outlet on the “Wave Generator (LFO)” and connect it to the “Y Position” inlet on the sphere.

Wonderful… but this is going to make me sea sick.

Next we’re going to make some adjustments to the LFO. With your patch inspector open, click on the LFO. Let’s the following changes:

Period to 2
Phase to 0
Amplitude to .01
Offset to 0

Wave_Generator__LFO__and_Untitled_-_Viewer_and_Untitled_-_Editor

Now you should have a sphere that’s very gently bouncing in the space.

Next let’s return to the parent lighting patch to make some changes to the lighting in this environment. We can get back to the parent either by clicking on the button “edit parent” or by clicking on the position in the string of objects where we want to go.

Untitled_-_Editor 5

In the root patch let’s click on the lighting object and change some parameters:

Material Specularity to 0.1
Material Shininess to 128
Light 1 Attenuation to .2
Light 1 X Position to -0.25
Light 1 Y Position 0.5
Light 1 Z Position to 1

Lighting_and_Untitled_-_Viewer_and_Untitled_-_Editor

Excellent. We should now have a sphere gently bobbing in space with a light located just barely to the left, up, and away (as a note these are locations in relation to our perspective looking at the sphere).

At this point we could leave this as it is and open it in Isadora, but it wouldn’t be very exciting. In order for Isadora to have access to make changes to a QC patch we have to “publish” the inputs that we want to change. In other words, we have to choose what parameters we want to have access to in Isadora before we save and close our QC patch.

I’m thinking that I’d like this sphere to have a few different qualities that can be changed from Isadora. I want to be able to:

  • Change the Lighting Color (Hue, Saturation, and Luminosity as separate controls)
  • Change the position of the light
  • Change the Sphere Size

In QC in order to pass a value to an object, the parameter in question needs to be published from the Root patch. This will make more sense in a second, but for now let’s dive into making some parameters available to Isadora. First up we’re going to add a HSL to our patch editor. This is going to give us the ability to control color as Hue, Saturation, and Luminosity as individual parameters.

Connect the Color outlet of the HSL to the Light 1 Color inlet on the lighting object.

Untitled_-_Editor 6

Now let’s do some publishing. Let’s start by right clicking on the HSL object. From the pop-up menu select “Publish Inputs” and one at a time publish Hue, Saturation, and Luminosity. You’ll know a parameter is being published if it’s got a green indicator light.

HSL_Color

Next publish the X, Y, and Z position inputs for the lighting object. This time make sure you change the names Light X Pos, Light Y Pos, and Light Z Pos as you publish the parameters.

Untitled_-_Editor

At this point we’ve published our Color values, and our position values, but only for the light. I still want to be able to change the diameter of sphere from Isadora. To do this we need to publish the diameter parameter variable from the “sphere” object, then again from the lighting object.

First double click on the lighting object to dive inside of it. Now publish the diameter parameter on the sphere, and make sure to name it “Sphere Diameter.” When you return to the root patch you’ll notice that you can now see the “Sphere Diameter” parameter.

Untitled_-_Editor

We now need to publish this parameter one more time so that Isadora will be able to make changes to this variable.

Here we need to pause to talk about good house keeping. Like all things in life, the more organized you can keep your patches, the happier you will be in the long run. To this end we’re going to do a little input splitting, organizing, and commenting. Let’s start by right clicking anywhere in your patch and selecting “Add Note.” When you double click on this sticky note you’ll be able to edit the text inside of it. I’m going to call my first note “Lighting Qualities.”

Next I’m going to go back to my HSL, right click on the patch, and select “Input Splitter” and select Hue. You’ll notice that you now have a separate input for Hue that’s now separate from the HSL. Repeat this process for Saturation and Luminosity. I’m going to do the same thing to my lighting position variables that are published. Next I’m going to make another note called “Sphere Qualities” and then split m sphere diameter and drag it to be inside of this note. When I’m done my patch looks like this:

Untitled_-_Editor 8

Right now this seems like a lot of extra work. For something this simple, it sure is. The practice, however, is important to consider. In splitting out the published inputs, and organizing them in notes we can (at a glance) see what variables are published, and what they’re driving. Commenting and organizing your patches ultimately makes the process of working with them in the future all the easier.

With all of our hard work done, let’s save our Quartz patch.

Working in Isadora

Before we fire up Isadora it’s important to know where it looks to load quartz patches. Isadora is going to look in the Compositions folder that’s located in the Library, System Library, and User Library directories. You can tell Isadora to only look in any combination of these three at start up. Make sure that you copy your new quartz composition into one of those three directories (I’d recommend giving your custom quartz comps a unique color or folder to make them easier to find in the future).

With your QC patch in place, and Isadora fired up lets add our custom patch to a scene. Double click anywhere in the programming space and search for the name of your patch. I called mine Simple Sphere. We can now see that we have our composition with all of the variables we published in QC.

Untitled

We can see what our composition looks like by adding a CI projector and connecting the image output form our QC actor to the image inlet on the projector. Let’s also make sure that we set the CI projector to keep the aspect ratio of our image.

Untitled___Stage_1_and_Untitled

When you do this you should see nothing. What gives?!
If you look back at your custom actor you’ll notice that the diameter of the sphere is currently set to 0. Change that parameter to the 0.1, or any other size you choose.

Untitled___Stage_1_and_Untitled 2

You should now see a dim floating sphere. What gives?!
If you look at the position of your light you’ll notice that it’s currently set to 0,0,0, roughly the same location as the sphere. Let’s move our light so we can see our sphere:

Light 1 X Position to -0.25
Light 1 Y Position 0.5
Light 1 Z Position to 1

Untitled___Stage_1_and_Untitled 3

If you’re connected to an external monitor or projector you’ll also want to make sure that you set the render properties to match the resolution of your output device:

Untitled 3

There you have it. You’ve now built a custom quartz patch that you can drive from Isadora.

Book Keeping

Not all of the work of media design is sexy. In fact, for all of the excitement generated by flash of stunning video, interactive installations, and large scale projection there is a tremendous amount of planning and paper work involved in the process. In the case of a theatrical project, one of the designer’s crucial pieces of work comes in the form of creating a cue sheet. For the uninitiated, a cue sheet is a list of all of the moments in a show that contain media. These documents help the designer, stage manager, and director establish a common understanding about how media / lights / sound are going to be used in a given production.

Drafting a useful cue sheet is often more a matter of preference, but it warrants mentioning some things that can help the media designer get organized as she / he wrestles with a growing list of content to generate. While one could certainly use a word-processing program in order to generate a cue sheet, I prefer to working a spreadsheet. Excel or Google Spreadsheets are a fine medium for writing cues, and have features that can be extremely helpful.

Cue Sheet Must-Haves

In my opinion there are a few must-have columns in organizing your cue sheet:

The Cue – Whatever you’ve settled on using, numbers or letters or some combination of the two, you need a column that puts the cue name / number in plain sight.

The Page Number – If you’re working from a traditional script, keep track of the page number. At some point you’ll be struggling to find the place in the script where a particular cue is getting called, and knowing the page number can ensure that you stay organized in the planning and technical rehearsal process.

Duration – How long does the cue last? In talking with the director it’s important to have a shared understanding of what’s happening in a given moment in a production. Specifying how long a particular effect or video is going to last can help provide some clarity for the designer as she/he is editing, animating, or programming.

From the Script – What’s in the source material that’s driving you to design a particular look? Did a character say something in particular? Is there any stage direction that’s inspiring your choice? Having a quick reference to what’s inspiring you can be a godsend while you’re crafting the content for a production.

Notes – For all that times you say “I’ll remember,” you will invariably forget something. Write it down. Did the director say something inspiring? Write it down. Did the lighting designer mention something about the amount of ambient light during a particular moment? Write it down. Write down what you’re thinking, or brainstorming. You’re never obligated to keep anything, but having a record of what you’ve been thinking gives you something to start from when you sit down to edit / animate / program.

Shooting Notes – If you’re going to need to record video for a production, make note of what particulars you need to keep in mind at the shoot. Do you need a green screen? A particular lighting effect? A particular costume? Keeping track of what you need for a particular moment is going to make the filming process that much easier.

Checklists. At the end of my cue sheet I keep a checklist for each cue. Three columns that help me keep track of what work I’ve done, and what work is left to be done.

Filmed / Animated – Is this cue filmed or animated?
Edited – Is this footage cut and prepped for the playback system?
Installed – Is this footage installed in the playback system?

Working with a Spreadsheet

Simple Formulas

One of the truly magical parts of working with a spreadsheet is one’s ability to use simple formulas to automate a workflow. A simple example comes out of the show that I’m currently working on, The Fall of the House of Escher. A new work that’s come out of the collaborate process of the 2014 ASU Graduate Acting / Directing / Design cohort, this show is built around a structural focus on giving the audience as much agency as possible. Central to this endeavor is a choose-your-own-adventure model for a the production. Given this structure, our production has sections that are distinguished from another another by a alpha-numeric prefix – 1A, 2A, 3A, et cetera. Additionally, I want to pair the section code with the number of a particular cue. This, for example, might look like 3A-34 (section 3A, cue 34). This combination of letters and numbers could make renumber cues, should one change, a difficult process. To simplify this particular problem I can use a simple formula for combining the contents of columns.

House_of_Escher_Media_Cue_Sheet

First I start by creating separate columns for the section of the show, and the cue number. Next I created a third column intended to be a combination of these other two columns. Here I inserts the following formula: =A24&”-“&B24

Here the Google Spreadsheets (or Excel for that matter) will read this formula as: display the contents of cell A24 and insert a “-” symbol and take the contents of cell B24. This may not seem like a big deal until you consider the time saved when a cue is added or removed, forcing a change in the numbering convention of all the following cues.

Conditional Formatting

Conditional formatting largely comes in varieties that are relative to changing the background of a particular cell. In this case Excel has a much wider range of possibilities for this kind of implementation. For me, however, the simplicity of automatic color coding is tremendously helpful. For example, let’s consider the final three checklist categories that I talked about earlier. Let’s say that I want every cell that contains the word “No” to be color coded red. In order to achieve this look first I’d highlight the cells that I want the formula to apply to.

Next I’d click on the background formatting button in the toolbar and select “Conditional Formatting” from the bottom of the list.

House_of_Escher_Media_Cue_Sheet 2

Finally I’d write my simple formula to control the change of color for the Cell’s selected.

House_of_Escher_Media_Cue_Sheet 3

Multiple Tabs

Last by not least, maintaining multiple tabs on a worksheet saves time and organizational energy. Additionally this allows you to cross reference notes, cells, and thoughts in your workbook. You might, for example, maintain a working cue sheet where you can brain storm some ideas and be a little less tidy. You can then use simple reference formulas to pull the relevant data to your final cue sheet that you give to the Stage Manager. Should you have the good fortune of having an assistant, you might make a separate page in your work book to outline her/his responsibilities on a project.

A cleanly organized cue-sheet is far from exciting, but it does ensure that you stay focused and productive as you work.

Escher Image and Animation Research

In thinking about what the media and animation for The Fall of the House of Escher might look like I’ve been sifting through the internet this summer looking for images that abstractly represent the universe and the behavior of particles and waves. Some of the more interesting work that I’ve found uses simple geometric shapes, and particle systems to evoke a sense of scale and distance and perspective. The work of the motion graphics designer Mr. Div is a prime example of someone who makes works that are both simple and also strikingly captivating.

The gif to the right is amy attempt at copying his piece “Tri-Heart”. I think copy-art is a practice that can’t be over stated. Recreating a work that you see from scratch teaches you more than simply following a tutorial. You are forced to wrestle with questions of how and why, and solve problems that don’t necessarily have clear solutions. While I don’t think this animation, specifically, is going to find it’s way into Escher, there are qualities of it that I really like and that feel distinctly quantum.

On the other end of the spectrum, in the “just follow along with a tutorial” category is a fascinating how-to create by minutephysics. Their quick After Effects tutorial covers how to create a simple simulation of formation of the universe. While it’s not scientifically accurate, the aesthetic conveys the look and feel of much more complex simulations that look at formation of the universe. Both of their videos are worth a watch, and the result of the tutorial can be seen here. Again, I don’t know that this exact animation will be something that I use, but it has elements that are inspiring and interesting.

In many respects there is a daunting amount of media in this production – interactive elements, large scale animations, moments of world-shifting perspective, and the expression of the abstract inner space of the atom. There’s a life-time of work in this production, and it goes up in October. There’s lots to do.