Lister’s Dank Dark Secrets

Welcome back for the third and final episode of The Big Badass lister. This week’s knowledge store comes to you directly from Ivan himself. He gives us a peek into some of the trickier parts of lister, the power-user secrets, and some of the cooler things you can do with lister when you use a little python. This section stands on its own, but if you’re brand new to lister it might be worth giving the other parts of this long form post a read through. 

And now… none other than Ivan himself:

Notes From The Developer

As the main developer of the lister custom component, I’m excited that it’s getting more use and specifically I’m excited to get to collaborate with Tutorial-master Ragan on this blog post. I’d like to start with a brief history of lister’s development. It’s something I have fond memories of, as this was the very first project I would work on with Derivative.

When I started on it, the listCOMP, the operator lister is built on, was brand new and had only been used internally by Derivative. I was given a very primitive version of lister that had already been through three developers and had no associated design plan. It was built mostly using a Python extension, which was also a relatively new concept in TouchDesigner. I started adding features that were immediately needed, and the lister quickly became an incredibly valuable ui element. We needed more and more features and I kept piling them on as requested. Eventually I realized that I had created a bit of a monster. The lister is incredibly powerful and has a number of great features but, full confession, it does reveal my inexperience with TouchDesigner when I developed it. Some day, there will be a lister 2.0, but until then it’s worth understanding lister’s shortcomings when working with it.

Most importantly, lister is a little slow. Its heavy reliance on Python and listCOMP’s inability to insert single rows and columns means that certain operations, especially full refreshes, can drop frames. While this doesn’t hinder lister as a development tool, it can limit its use in live performances. The second main caveat with lister is that moving beyond simple operations will often involve some Python programming. I created a large number of callbacks in order to facilitate this, but managing lister can still be complicated and will sometimes involve working directly with the listCOMP itself. In fact, I would recommend learning how the listCOMP works by itself to anyone doing heavy lister work. The final disclaimer I’d like to make is about the lister extension itself. Because of the way it was developed, it is overly large and rather confusing, and generally not a great example of clean TouchDesigner Python code.

All that said, lister is still a powerhouse for displaying and working with lists of data. Let’s have some fun and get into the deep features.

The Column Definition Table 

You can do a ton of fancy things with lister using just the column definition table. As you already know, this table is where you set up the big picture layout of your list. There are also a number of hidden gems that you might miss just by skimming the wiki.

Source Data and Source Data Modes

For this example (sourceDataModes.tox), I’ve set up a simple table input with Input Table Has Headers on. For starters, compare the input table, the column definitions, and the lister output. Notice how many more columns the lister has than the input table. That’s because you can use the same source data multiple times in different columns by using the same header string in sourceData. You can also rearrange the columns however you want because when Input Table Has Headers is on, the lister looks up the data by table header, not by column number.

Modes: int, float, string, version

The simplest source data modes are int, float, string, and version. They do not filter or change the text in any way, they are only used for sorting. To see this, click on the column header to sort by that column and notice that the same data is sorted differently depending on the source data mode. Any data that cannot be converted to the chosen mode is sorted alphabetically.

Special Modes: blank, rowNum, constant, repr

The blank mode keeps the text data internally, but doesn’t display it in the cell. This is extremely useful in conjunction with topPath and cellLook methods to show information graphically instead of with text. 

The rowNum mode also keeps the text data, but only displays the row number. This is just a convenient way to display row numbers without coding.

The constant mode puts the literal value in sourceData into each cell in the column. This is mostly used as a quick and easy way to make buttons. For example “X” for delete.

The repr mode displays the sourceData using Python’s repr function instead of converting the object to a string. When using tables (which hold only strings) this just means that the data will have quotes around it. When using objects, lists, or dictionaries as your source data rows, this can provide more detailed information about Python objects. 

Super special mode: color

The color mode is a shortcut for creating colored cells. In order to use this feature, the data in a color column must be formatted properly. Specifically, it must be a list of four numbers, followed by an optional string. The first four numbers will be used to define the cell’s color. The string will be used as text drawn over that color. The example lister shows a number of examples. If the data is formatted incorrectly, it will be used directly as text, with no colors applied. If you have incoming data that you want to color in the lister, the onConvertData callback is probably the best place to change that data into this r, g, b, a, text format.

Expressions in the Column Definition Table and Stretchy Columns

There are a couple places where you can use expressions in the column definition table to avoid using callbacks at all. This powerful technique is illustrated in the sourceDataModeEvalAndHelp example. This example uses a list of operators as its raw data, and its column define and output are as follows:

The first place you can use an expression in the column definition table is to process your source data. If you set sourceDataMode to eval, the sourceData will be treated as an expression, and your row object will be available in the variable object. In the example, you can see type(object) displayed in the middle column, and an even cleaner version of that, object.__class__.__name__, in the right column. This powerful method lets you display just about anything in a Python object.

There is also some secret sauce for the help row, which defines popup help text for a column’s cells. An asterisk by itself will make the popup help display cell contents exactly. This is useful when a column might not be wide enough to show the cell contents because it allows you to hover over it for the full text. If the asterisk is followed by text, that text is used as an expression which will be evaluated and displayed in popup text. That expression gives you access to object just like in sourceData, and also provides text, which is the cell’s text. In the example, the middle column help will show the text representation of the row object, and the right column will display the cell text prepended with “type: “.

One last thing to notice in this example is that the left column is set to stretch. This means it will take up any available horizontal space in the lister display. If more than one column is set to stretch, the available space will be divided equally. That’s pretty obvious stuff, but what you might not know is that the width of the column will be used as the minimum width for stretch columns. 

Editable, selectRow, and clickOnDrag

The last few column definition tricks are pretty simple but worth going over because when you need these features they’re easy. This section will use the editableTable example and is particularly worth trying in TouchDesigner because the features are interactive. Also notice the columns are rearranged from the source table because I don’t want you to forget that great feature!

The editable setting in colDefine is pretty straightforward. A 1 means single-click to edit a cell and a 2 means double-click to edit. What’s so great is that if you use this in combination with the Input Table Has Headers and Auto-sync Input Table parameters, you have an instant table DAT editor with a bunch of nice features. It’s worth noting here that if you are not using auto-sync features, you will have to explicitly propagate changes from your lister into your data by using Python callbacks. For an example of that, see the onDrop Callback section below.

Output DATs and Selection Features

The next two settings we’re going to look at have to do with the lister’s “selection” features. The selection is most easily seen in the output, so it’s a good time to look at the two outputs ports of the lister component: out_table and out_selection. 

The out_table DAT output is a table of all the text in the lister. This is often very similar to an auto-synced table, but is not exactly the same because of various data permutations you can do inside lister. This table can also be useful for reacting to data changes in the lister. 

The other output is the out_selected DAT, and only displays the data from the “selected” rows in the lister. 

In the example, click on cells in the left column to select them and see this in action. 

The reason you can only click on the left column in this example, is the selectRow setting. As you may have guessed, clicking a column will only select rows if this is set to 1. Notice when you click on the middle or right columns in the example, the selection does not change.

The clickOnDrag setting is a little more subtle. To see what this does, click on the left column in the example and drag the mouse up and down to other columns. Each row is selected as you drag because this setting is on. This is useful for things like presets or colors because you can set up a column of choices and drag across to browse through them quickly. This also triggers onClick callbacks for each cell as you drag across them.

Parameter Tricks

The lister component exposes a number of complex list interaction features that are commonly needed. Whenever possible, these are accessible via parameter. Let’s take a look at the possibilities. This example uses the editableTable tox used in the previous section. These examples are worth playing with to understand the exact effects. Let’s walk through them one by one.

• The Header parameter simply turns on and off the header row display. 
• Clickable Header turns on and off the ability to sort rows by clicking on the column header. In later versions of lister, ctrl-click will turn off this sorting. 
• Selectable Rows controls whether rows can be selected with mouse clicks. As explained above, selectRow must be on in the column define table as well.
• Multiple Row Select lets you select multiple rows using the common modifiers in file explorers. That is, holding down ctrl adds or removes selections and shift will select a section of rows.
• Drag To Reorder Rows enables dragging selected rows to new positions in the list.
• Arrow Keys lets you move the row selection by pressing up and down arrows.
• Delete Key lets you delete all selected rows by pressing Delete.
• Highlight Rollover enables/disables highlighting of the row that the mouse is currently hovering over.
• Row Striping lets you differentiate rows with two different styles of highlight, either thin lines between rows or alternating highlights. The settings for these two colors are found in the define DAT in the lister’s config component.

Other Interaction Techniques: Buttons, Toggles, Menus

Now that we’ve looked at how easy it is to set up editable text, let’s explore some other common interaction features you might want to use in your lister. The example lister references a table with rows of “Yay” or “Nay” votes. You can right-click the text to get a menu or push any cell to toggle votes back and forth. The input table setup is the Auto-sync type you’ve seen before in this tutorial, but with the row selection features turned off. To set up the actual interactions, we’ll be using some simple Python callbacks. 

Setting Up Button and Toggle Looks

We’ll go into how each column actually changes the data in the Interaction Callbacks section below, but for now let’s look at how we set up the looks. Setting up the columns in this lister starts, as usual, with the column define table. They all use the “button” cellLook. This means that each cell in the column will use the text and color settings in the config TOPs that start with the word “button”. When setting up a cell look, you must have a TOP with the same name. You can also have an optional name + “Roll” TOP for rollovers and name + “Press” TOP for when the mouse button is down on the cell. In this example, we set up all three. You can have as many of these cell looks as you want in your lister, they just need unique names.

The “Button” and “Toggle” columns go one step further than cell looks by adding images and removing the text. To remove the text, we just use the “blank” sourceDataMode. To add the images, we add a name in the topPath row. The “Button” column does not reflect the data in the table, it is simply a push button that switches the “Vote”. It uses the topPath “btnImage” and has separate TOPs for the roll and press state as well. In this example I’ve used circleTOP, hsvadjustTOP, and levelTOP to create the images, but you can use any TOP techniques you like. Note how the transparent part of the “btnImage” TOPs will reveal the corresponding “button” cellLook tops.

The “Toggle” column uses the same technique as the “Button” column, but will use a different image for each data state. The topPath for this column is “toggle*” and the asterisk at the end indicates that the text in the column will be appended when looking for the TOP to display. This means the image “toggleNay” will be used when the cell’s data is “Nay”, even though you don’t see the “Nay” because the sourceDataMode is “blank”. When using this asterisk mode, the “Roll” and “Press” will still be appended. In this example, we only set up a “Press” state, but notice that the background is still highlighted when rolling over these cells because of the “buttonRoll” cellLook. These toggles were created with the rectangleTOP but again, you can use any TOPs you want, including movieFileIn if you want external files.

Interaction Callbacks

(This section requires some basic Python knowledge) 

def onClick(info):  
    inputTable = info['ownerComp'].par.Inputtabledat.eval()
    cell = inputTable[info['row'], 'Vote']
    if cell.val == 'Yay':
        cell.val = 'Nay'
    else:
        cell.val = 'Yay'

The first callback we’ll look at it is onClick. This is called when the mouse is pressed and released on any non-header cell. Notice that when you click on any cell in the lister, it switches the “Vote” for that row. Because we will be working directly with the input table, the first thing we need to do is get it from the lister’s parameters. All lister callbacks will have the lister itself in the “ownerComp” key of the info dictionary. The first line in the callback gets the DAT stored in ownerComp’s Inputtabledat parameter and stores it in inputTable. The next line gets the specific cell in the inputTable by cross-referencing info[‘row’], which holds the number of the row clicked on, and the “Vote” column of the table.

Now that we have the tableDAT cell, we just need to test if its value is “Yay”. If it is, we set it to “Nay”. If it’s not, it must be “Nay” already so we set it to “Yay”. Again, note that this same code works for every cell in the lister, no matter which column, because they all change the same value in the input table. Because the “Vote” and “Toggle” columns in the lister are based on the input table’s value, they automatically update when we make this change.

Using Popup Menus With Lister

Using popup menus with a lister is an incredibly useful technique. It basically allows you to perform any number of custom functions on a per-row or per-cell basis. This example will be one of the simplest possible cases, but the method can be expanded easily. Because this is a lister tutorial and not a popup menu tutorial, we won’t dive deep into the many features of the TouchDesigner popMenu but the wiki has some good examples and it’s fairly straightforward once you get the basics.

popMenu = op.TDResources.op('popMenu')

In this example we use the system popMenu, which is accessed via the global shortcut op.TDResources. If you want to customize the look of your popup menu, you can get a unique one from the palette UI section. If you don’t mind the generic look, it is generally safe to use the shared system one because there will only ever be one popMenu open at a time.

def onClickRightVote(info):
    popMenu.Open(items=['Yay', 'Nay'], callback=selectVote, 
                    callbackDetails=info)

The next step in using a popup menu is opening it, which we do in the onClickRightVote callback. This is an example of the special cell callback system in lister, where we use the generic name of the callback (onClickRight) followed by the name of the column from the column definition table (Vote) to create a callbacks that is only called when that column is clicked. 

The Open function itself has a number of arguments, but in this case we only use three. The items argument is a list of choices in the menu. For this example, you can select “Yay” or “Nay”. The next argument, callback, is a function that will be called when the user makes a choice. In this case, we call it selectVote and we’ll define that function next. The callbackDetails argument will be passed into the chosen callback to provide extra info. A simple and powerful technique is to just pass the entire info dictionary in this argument, so that the popup menu’s callback will have access to everything that this lister callback knows.

When the Open function is called, the popup menu appears, and TouchDesigner runs normally in the background, waiting for the user to either select something or click somewhere not on the popup menu. If the user clicks somewhere else, the menu closes and the callback function selectVote will never be called. The last step is to define what the system will do if the user does click a selection from the menu.

def selectVote(menuInfo):
    clickDetails = menuInfo['details']
    row = clickDetails['row']
    col = clickDetails['col']
    lister = clickDetails['ownerComp']
 
    lister.SetCellText(row, col, menuInfo['item'])

The selectVote function is a callback from the popup menu, and like lister’s callbacks it takes a single argument that will be filled with a dictionary of information. For the sake of clarity, we’ve named it menuInfo instead of info but you can name it whatever you want. From the menuInfo dictionary we grab the “item”, which is the choice that was clicked on (“Yay” or “Nay”) and store that in menuChoice. We then get clickDetails from  menuInfo[‘details’], which is the original info dictionary in the onRightClickVote callback. Remember, we sent that explicitly using the callbackDetails argument in Open. Those clickDetails contain the row and column of the cell that was clicked on, and the lister itself, so we store those as well. It should be noted that you don’t have to store all these things in separate variables, but it’s a lot easier to read and explain if you do.

The last line is actually the only line in this function that changes anything. We use the lister method SetCellText to (did you guess?) set the cell text at the row and col that was clicked on. We set it to whatever string was clicked on in the menu, which can be found in menuInfo[‘item’]. Because we have Auto-sync on for the lister, setting the cell text will cause the input table to be updated automatically. This in turn causes the Toggle column to be updated automatically. Easy-peasy! So much power with so little code. You can do a whole lot more things with this technique just by adding more choices to the menu and writing different reactions depending on the “item” selected.

Drag And Drop Between Listers

(This section requires some basic Python knowledge)

Creating drag-and-drop systems is never dead simple, but lister does a lot of the heavy lifting for you. In this example we’ll go over the setup and Python code for dragging and dropping between listers and from outside them. The basic setup is things you’ve seen before in this tutorial: two listers with input tables and auto-sync on. For convenience, they’re displayed side by side in the dragDropContainer component. The only significant change in the column definitions is that the columns are set to be draggable. 

The one interesting thing about the set up is that both listers share a single config comp, ddListerConfig, which means they have the same look, the same column definitions, the same callbacks etc. This config comp sharing is not necessary for dragging and dropping between listers, it’s just convenient because both listers in this example work exactly the same. This requires a little extra attention to the callbacks, because they have to be written in a generic way that works for either lister. The good news is that if you stick to always using data from the info dictionary, it will always work. For example info[‘ownerComp’] will always contain the lister component you are working with.

To see the listers in action, just drag rows back and forth. They contain paths to actual nodes in the network. You can also drag a node from the network into either lister to add it to the list. You’ll also notice a line in the list showing where the dropped row will appear. This is controlled by the Drop Highlight parameter on the listers, and the ugly color I used to make it stand out is controlled by the define table inside the config comp. Try the other Drop Highlight settings, just to get a look at them. They can all be useful, depending on how you set up your drag/drop system.

The onDropHover Callback

def onDropHover(info):
    debug('onDropHover', info)
 
    for item in info['dragItems']:
        # make sure all dropItems are operators
        if not isinstance(item, OP):
            return False
    return True

This part of our drag/drop system is really simple. First, we have a debug statement for convenience so you can open a textport and see what info is available. In the info dictionary, we check dragItems and make sure everything in the list is an operator (OP) using the isinstance function. If we find an item that is not an operator, we return False to indicate that it is not an acceptable drop, otherwise we return True. Now, the listers accept either operators or rows from the other list, so why do we take any operators? Because when you pass lister data, the dragItem will still be the lister operator itself. There is more specific drag data elsewhere in the info dictionary, but we don’t care about that in the onDropHover callback. We’ll check that next.

The onDrop Callback

def onDrop(info):
    debug('onDrop', info)
 
    ownerComp = info['ownerComp']
    dropRow = info['row'] # row that was dropped in
    # special cases of dropRow
    if dropRow == 0:
        # dropped on header, put it in row 1
        dropRow = 1
    elif dropRow == -1:
        # was dropped on part of lister without a cell, drop it at bottom
        dropRow = len(ownerComp.Data)

In the first half of the callback, we have a debug statement again, we store the lister in ownerComp, and then we do a little setup to decide where to drop a row. As usual with lister callbacks, the row under the mouse is in info[‘row’] and we store that in dropRow. There are a couple of special cases, though. If dropRow is zero, that’s the header row. We don’t want to drop above that, so we’ll just drop into row 1 instead. If dropRow is -1, that means the area of the lister without any cells, and we’ll just drop it after the last row. To get the last row, we find the length of ownerComp.Data. 

The Data member of lister is worth examining a bit. It’s a Python list with an item for each row in the lister. Each item in the Data list is an ordered dictionary with a key for each column in the list. Those keys correspond to the column row in the column definition table. The value in each key is the text stored in the corresponding cell of the lister. There is one more key in the dictionary: rowObject. The rowObject can be either a list of strings from an input table, or the row’s Python object as provided in the onGetRawData callback or the Raw Data parameter.

    for dragItem in info['dragItems']:
        # handle multiple items being dropped

Now that we know where we want to drop, we need to go through each of the dragItems and create a row for it.

        if dragItem.name in ['dragDrop1', 'dragDrop2']:
            # it's one of our example listers, copy the info from source
            listerInfo = info['fromListerInfo'] # dragged cell
            sourceRow = listerInfo[0] # dragged row
            sourceData = dragItem.Data[sourceRow] # full row data
            ownerComp.Data.insert(dropRow, sourceData) # insert data
            ownerComp.DataChanged() # required after changing lister.Data

This first case is if the dragItem is one of our two listers. This means we’re dragging a row from the lister. The source cell is stored in info[‘fromListerInfo’] and from that we can get the source row number and thus the full Data item. We then simply insert that Data item into our lister at the dropRow we figured out before. One last thing you have to do when changing the Data list directly in Python is call the DataChanged function.

There are a couple extra things worth noting here. One is that the system can get confused between the lister operator itself and a cell from that lister. This will be fixed in later versions of listCOMP, but for now rest assured that it will never be an issue in practice. The other thing it’s important to understand is that our life is made very easy in this case by the fact that the Data item is identical in both listers. When adding a row into Data in more complicated situations, you will often want to use lister’s helper function RowObjectToDataRow, which takes a row object and creates an item that is properly formatted for the Data list. 

            # remove row from source
            if not ownerComp.panel.ctrl:
                dragItem.DeleteRows([sourceRow])

Now that we’ve added the rows to the lister that was dropped on, we need to remove it from the one that it was dragged from. I snuck in a copy feature with the if statement: if ctrl is being held down, we don’t delete the row from the source, which means it will now be in both listers. Also note the way that you can check if ctrl is being held down in onDrop is to check ownerComp.panel.ctrl. This works for shift (and any other modifier/panel value) as well.

        else:
            # it's a node from a network, add it via data table
            inputTable = ownerComp.par.Inputtabledat.eval()
            inputTable.insertRow([dragItem.path, dragItem.__class__.__name__],
                                    dropRow)

This else clause is for when an operator is dropped in the lister from a network. In this case, we use the technique of changing the input table instead of the Data list directly. Either way works, and it’s really up to you to decide which you prefer. The code here is using standard TouchDesigner tableDAT operations. First we get the DAT itself from the lister’s parameters, then we use the DAT’s insertRow function to create a row with path and class name at the location of dropRow.

Wrapping Up

It’s great to finally spell out some of these lister tricks! I’ve been meaning to write something like this for a long time. I hope it shed light on some new useful stuff and that you find lister helpful in your projects. Feel free to tag me directly in the TouchDesigner forums if you get stuck or you have ideas for new features.

You can download the examples from this post here.

Thanks for sticking with us over the last three weeks, and we hope you’ve learned a bit more about lister.