TouchDesigner | Working Styles | Git

Recently when I was teaching at LDi 2017 a participant asked if I might take some time to document how we use git in our projects at Obscura. Version control isn’t always a sexy topic, but it is a vital piece of our pipeline and process, and one well worth considering if you’re moving away from being a lone developer on projects. It’s also worth getting a handle on version control approaches if you’re looking to join a software team in general.

A Blanket Disclaimer

Git is complicated… ask anyone that’s used git. XKCD has my favorite description of working with git on a regular basis:

Which is to say that I’m not a git expert, often have issues of my own, and have certainly done the dance of copying my flies to somewhere else to ensure that I don’t loose all of my work. I don’t mean to suggest that you shouldn’t use git – you should – but rather that like all things that are new or different to you, this one comes with its own set of challenges. Okay okay… so why should I use it then Matt?! We’ll see below what makes it so powerful, and when it comes to working in teams this far outpaces any other approach – but you have to be patient, thoughtful, and considerate.

What is Git?

Okay so what is git again?

According to Wikipedia:

Git (/ɡɪt/^[7]) is a version control system for tracking changes in computer files and coordinating work on those files among multiple people. It is primarily used for source code management in software development,^[8] but it can be used to keep track of changes in any set of files. As a distributed revision control system it is aimed at speed,^[9] data integrity,^[10] and support for distributed, non-linear workflows.^[11]

Uhhhh… say what now?

Many of you, dear readers, are probably landing here because you’re curious about how this works with TouchDesigner. In Touch, the default behavior is that everytime you save a file you get a new version – project.1.toe, project.2.toe, project.3.toe, and on and on and on. The idea of version control is similar except that it allows you to avoid the process of having additional files – you only ever have one version of your file, but you can reach back in time and find other versions. Similar to Apple’s TimeMachine idea, or Dropbox’s version retrieval. You only ever see one file in your directory, but if you realize that you need a previous one, you can reach back and fetch it.

To achieve this, git uses the convention of a commit. When you’re ready to add something to your history, you commit your changes. That is, when you have a version of your file that you want to hold onto, you specifically choose to commit it to your history along with a message about what changed. If you’re working on a team this is great – it means you can see the history of every commit, and what another developer had to say about what they were doing and why they made the change. Git also helps you differentiate between versions – in text based files you can call a command to see exactly what changed between versions and who made that change.

More than that, git also lets you build branches of your project. Let’s say you’re working on an installation. It’s working great, but you want to be able to try out some other ideas or work on some updates for the project. You don’t want to make this change to the project file that’s running, and you don’t want to you don’t want to interrupt the installation’s operation. Moving to another branch lets you work in a parallel project with all of the same file names where you can make all the changes you want, and then decide when to roll that into your actual project.

This becomes especially interesting when you’re working with multiple people. Once you’re working on a large project it often becomes important to have several folks contributing… git helps organize this distributed work, and keep you from overwriting one another’s contributions.

Further, using something like bitbucket or github means that your project is hosted on an outside server – so even if you’re machine gives up the ghost, your work isn’t interrupted.

Challenges in using TouchDesigner and Git

At this point you might be sold on the idea of using git.

YAY! Welcome to the git party!

Before you get too excited, there are some important issues to consider when it comes to using git with TouchDesigner.

• Most Touch files are binary – what does that mean Matt? Well, it means that unlike a .py or .json, or .py file, your toe or tox file is actually made up of hex strings that are difficult to parse outside of the context of Touch. That makes them very difficult to diff – that is, to tell what’s different between committed versions.
• Toe Files are whole projects – a toe file holds a whole project, which is great, but makes collaborating very difficult. Part of the beauty of git is that multiple people can work together at the same time without overwriting one another’s changes. That doesn’t make a whole lot of sense if you can’t see what’s different inside of a toe file, and if the whole project is stuck inside of a single file.
• Using git means learning shell commands or a tool to use git – If you’re going to use git, you’ll have to learn some shell commands, or learn another tool that interfaces with git (github has a great desktop tool). That’s not terrible, but it’s not the same as using something like dropbox box or google drive.

Our Solution

Okay… so how do we make this work then?

Well, first things first, you start by thinking about toxes for your modules / component work-spaces, and your start externalizing your scripts. That’s a big change in workflow for lots of folks, and if you’re not ready for that change that’s okay. if you’re working on big projects, however, and working with other people now is the time to level up. Let’s look at a simple project build so we can see how this might work in practice.

First we need to set up our git project. I’m going to use github – it’s free to use the public version, and it’s got one of the best desktop utilities. Git has a great tutorial for their app so I’m not going to cover that here, instead we’ll look at how the shell commands work.

For starters, you’ll want to install git, I’m also going to use git bash instead of just windows command line (it’s a little easier to read, though you can use either), so if you want to follow along you’ll want to make sure that you hit that checkbox when you install.

There are lots of ways to start a new repo, but I’m going to start mine from my github account. Once I create an account, I’ll need to use the plus button in the top right corner of the page to create a new repo, then choose my settings for it:

Once I’ve created my repo online, I need to clone this to my computer. Cloning my repository means that I’m going to create a local copy where I can make and track changes, and commit to my online repo. To do this, I need to first navigate to where I want my project to live. I’ve created a dummy directory called example on my D:\ drive:

Here in this directory, I’m going to right click and choose “git bash here” to open up a git bash terminal at this directory location.

In our git bash terminal we’ll enter our first clone commands. We’ll first need to copy the URL from the clone drop down menu on the web-page:

Let’s copy that ULR to our clipboard. Next we’ll use the git clone command in our git terminal window to start our cloning process. Here’s our command:

git clone https://github.com/raganmd/touch_git_example.git

Here’s what it looks like in our terminal window:

Alright! Now we have our git project cloned into our windows directory:

From here on in we need to make sure that our git terminal is inside of our newly added directory. We can repeat the same step we used earlier – navigate into the directory, right click, and open a git bash here; or we can navigate there in our terminal window. Let’s do this right from our existing terminal window. We’ll need to change our directory to touch_git_example, the name of the newly cloned repo. We can do this with the command:

cd touch_git_example/

Now we’re ready to start working! Let’s start by creating a new folder called toxes. All of our toxes are going to go into this directory:

Next let’s open Touch and save a project file in the root of our directory. I’m going to call mine touch_git_example.toe:

Inside of Touch I’m going to start by getting rid of my project component in the root, and I’m going to create a new base that’s called base_project:

Next I’m going to set up a few more things. Inside of base_project I’m going to create a few elements:

• container_display – this will be the display elements for my project
• base_com – this will hold the communication elements for my project

Next I need to externalize these elements. To do this,  we right click on them and choose “Save Component tox…” from the drop down menu:

I’m going to save both of these elements in the toxes directory:

Now we need to make sure both of these components point to their external tox files. We can do this by going to the common page and locating opening our toxes in the external tox parameter field. We also want to make sure that we turn off the “Save Backup of External” parameter:

Notice that these are relative, not absolute paths – this is VERY IMPORTANT. We want our paths to be relative to our project directory. This will help ensure that our externalization process doesn’t break when we move to another computer.

Finally we need to save both of our toxes one more time, and save our project one more time.

YIKES! That’s a lot of steps… what did we do exactly here Matt?

Well, first we set up our project and saved our toe file. Then we created some components that we want in our project, but that we want to be able to edit independently of one another (com and display). After we saved them both the first time we had to point them back to their external files so they open correctly. We saved them a second time to make sure that relative path parameter was saved with our toxes. Finally, we saved the whole project again to make sure that our toxes with external paths were correctly set up in the toe file.

Whew… okay, why?!

Well, at this point, unless we add another component in our base_project layer, we never save our toe file again – we only save the toxes. This also means that the work can be split up… one developer can work in base_com, and another in container_display and they won’t over-write one another’s work. Keep in mind, if we add another component in base_project we’ll need to save the toe file; we’ll also need to change the toe file if we make changes to the project (like the perform window, project settings, and the like).

Let’s go back to our git window to see how we commit all of this to git.

Back in our terminal we can add all of our new elements at once with:

git add -A

This adds all of our files as tracked elements that we’ll now keep an eye one. Next we need to commit these changes. Let’s also add a message so we know what we did:

git commit -m "Our initial commit with toe file and two components"

At this point our changes are committed, and we’re ready to push them back up to our github repo. We can do this with a push command:

git push

NICE WORK!

Now we can head back over to github to see our project:

Better yet, if we click on the commits link we’ll see our entry history of contributions:

At this point it’s time to start working. Now as you work you can create snap shots to return to. That process usually looks first adding your files with git add, then commiting your files with a log message using git commit, finally you push your changes with git push. You can retrieve the work that other team members have done with git pull.

You’ll also notice that with a history of our changes it means we can move back to any of those snapshot moments in our project. If you’ve ever had a time when you made a change that broke everything… and couldn’t figure out how to undo that change, version control is for you. This lets you move back in time to find a working version of the single module that you changed rather than breaking out in a cold sweat of pure panic.

You can also externalize scripts, glsl, channel data, geometry data, and and and. Generally speaking, you don’t put assets in your repo. Git LFS (Large File Storage) helps with some of that, but for the most part you don’t want to fill up a repo with video. We sometimes will put in a single calibration frame, but it’s important to be very careful when adding large files to your repo as that can make for big headaches.

At Obscura we built out a save process that automates a lot of the above. We also make our repos mirror our touch structure. This means that if we know where an element is in touch, we know where it is in our repo.

Like all things, there is a TON more to learn on this front, but hopefully this gives you some ideas about where to get started with a version control system and working across machines and with other developers.

Happy programming everyone!