Enabling the User to Create Vizeme Mappings

Up to this point, vizeme mapping has been performed using a single map file, containing a set of vectors created by the author to approximately represent facial expressions for each phoneme. It would be useful, however, to enable each user of the DIVA system to construct their own faces, whether to improve upon the current faces, to fit the user's perception, or to vary the emotion of the face as desired. This is accomplished through the implementation of three interfaces using MAX/MSP, together forming a set of "Vizeme Tools" in the DIVA project:

i) VizemeBuilder - enables the user to create individual facial expressions for each phoneme, and store PC values for these faces in vizeme files.
ii) CreateVizemeDictionary - enables the user to create their own dictionary files, specifying a list of different emotions for the faces. The creation of multiple-emotion dictionaries is useful only in future work - for the present discussion, phoneme -> vizeme mapping is one-to-one, and so only a single emotion (call it neutral) is permitted -- the dictionary file "dictionary.dict" supplies the appropriate default.
iii) CreateVizemeMap - enables the user to create their own map files by selecting a vizeme file for each phoneme.

The interfaces can be accessed from the "Vizeme Tools" drop-down menu in the DIVA main window. Each interface enables the user to create a specific type of file, and so they all share the following behaviours:

i) Load - loads the selected file to the interface, checking first for unsaved changes and giving the user the option to save, if this is the case.
ii) New - refreshes the interface, checking first for unsaved changes and giving the user the option to save changes, if this is the case.
iii) SaveAs - saves current changes to a file, prompting the user to choose a filename
iv) Overwrite - if a file is loaded, and the current user matches the file owner, overwrites the current file with most recent changes. Note that a dictionary file cannot be deleted if there exist any map files associated with it.
v) Delete - if a file is loaded, and the current user matches the file owner, deletes the current file. Note that a dictionary file cannot be deleted if there exist any map files associated with it.
There are a few additional behaviours specific to each interface, described below.

VizemeBuilder

The Vizeme Builder interface launches the KuraFace model and connects to it upon initialization. By adjusting the eight knobs, the user can stretch the face along all 8 principal components, and view the effects. When the desired facial expression is obtained, the PC vector values can be saved to a vizeme file:

These files will then be parsed at performance time to load PC vectors.

CreateVizemeDictionary

The Create Vizeme Dictionary interface allows the user to specify a list of expressions, or different facial emotions. As mentioned above, the extension of vizeme mappings to different facial emotions is not yet supported, and the process of this extension is described in "Future Work". Presently, dictionary files should only contain a single expression, and the file dictionary.dict is the default file:

Dictionary files create categories of sorts for map files -- each map file must be based upon one of the existing dictionary files.

CreateVizemeMap

The Create Vizeme Map interface works in the following way:

i) Choose Dictionary - select the desired file from the list of existing dictionary files - this will specify the expressions in map file entries, and brings up a list of map files for this dictionary.
ii) New Map / Load Map - new map creates an entry in the right-hand table (the map table) for each phoneme, and for each expression, leaving the VizemeFile column blank. Load map loads one of the existing map files associated with this dictionary.
iii) Select Phoneme - selecting a phoneme from the dictionary displayed on the left-hand side brings up a list of vizeme files saved for that phoneme.
iv) Add/Replace Vizeme - by highlighting one of the vizeme files in the left-hand table, and an appropriate row (matching phoneme) in the right-hand table, the two are associated, and the filename is added to the "VizemeFile" column. This means that when the map file is saved, it will specify a mapping of that phoneme & expression to the facial shape specified in that vizeme file.

The existing default map file is map.map, shown below:

The map file associates each phoneme with a vizeme file, such that when it is loaded, the PC vector associated with each phoneme is loaded from the specified vizeme file.

To summarize:

i) Vizeme Tools consists of three interfaces in the DIVA project, accessed from the main window
ii) Together, the tools enable the user to customize the phoneme --> vizeme mapping used in performance mode
iii) VizemeBuilder allows the user to create and save individual facial shapes
iv) At this point in the project, CreateVizemeDictionary is not necessary - just choose the default dictionary.dict file when creating map files
v) CreateVizemeMap allows the user to pick and choose from vizeme files to create a mapping of phonemes to PC vectors specifying facial shapes.

Next, step 6 describes how the map file is loaded at performance time.