This area is for the recording/reporting of progress through the different areas of the project.

Eric

• 20-23 May, 2008
  • New Hardware: completed touchGlove/connectBox hardware components, and drew up documentation detailing its parts and construction ( gave to Bob to photocopy and, presumably, post somewhere ).
  • Hardware Organization: Started a spreadsheet list of all hardware, and still trying to find an appropriate speaker for the wearable system.
  • Putting a Face to it: obtained ArtiSynth software from Ian, will be working to integrate a facial simulation with the DIVA speech system.
• 26-30 May, 2008
  • Worked with ArtiSynth and MXJ (Java-coded Max Objects) and created a MAX patcher which sends principal component to ArtiSynth's kuraFace and controllably stretches the face.
• 2-6 June, 2008
  • Mapped connections on ConnectBox / TouchGlove hardware, documented connection mappings.
  • Shopped speakers and amplifiers -- settled on a 9V battery-powerable amplifier and 300 mW speaker which yield good sound in the speech range.
  • Began assembly of a backup ConnectBox / TouchGlove hardware system
  • Sourced a Pelican Case (1510 carry-on model) and ordered through purchasing@ece, waiting on Sid for approval.
• 9-13 June, 2008
  • Completed backup ConnectBox / TouchGlove hardware system
  • Designed amplifier box / speaker system, built 2 copies, and documented design
• 16-27 June, 2008
  • Investigated the problem of generating a talking face from DIVA data
  • Developed an interface in MAX, and a rudimentary set of vowel and consonant vizemes, to test the talking face
• 30 June-4 July, 2008
  • refactored and improved phoneme testing interface, added a simulated xy-space for testing vowel transitions.
  • cut and assembled set of wires for amplifier / speaker
  • obtained pelican case, communicating with Helene on
    • fitting pieces into case
    • securing connections on the touchGlove
    • assembling a set of components for Marguerite to practice
• 7-11 July, 2008
  • began integration of facial simulation with DIVA code, wrote an interface to create vizemes
  • met with helene to think about touchglove connection issue
  • made a solid plan for first and second stage of integration for vizeme creation and training
• 14-18 July, 2008
  • implemented first stage (nearly complete) of vizeme creation in DIVA code
  • implemented new connection method (metal snaps) for touchglove
• 21-25 July, 2008
  • completed first stage of vizeme creation in DIVA code ( still needs testing ), contains:
    • create vizeme interface
    • create vizeme dictionary interface
    • create vizeme map interface
    • launch artisynth from the perform window after loading map file, and link glove data to mxj object to drive the face
  • with marguerite's help, determined the order of putting on suit components, for the purpose of quick transitions at perform time
• 28-30 July, 2008
  • began investigating training and create accent & planning how to extend these for vizeme training and accent, stage 2 of integration of Artisynth with DIVA code
• 5-8 August, 2008
  • continued working at extension of training and create accent -- training is about done, allowing user to click a tab to choose between phoneme & vizeme training
• 11-15 August, 2008
  • put implementation of stage 2 on hold, cleaning up and fixing bugs on vizeme interfaces, documenting and making preparations for final report
• 18-22 August, 2008
  • created a "Talking Face Summer 08" section in the wiki detailing work on the talking face over the last fourth months, and future directions for the project.
  • prepared presentation slides for final presentation
  • examined bug in system with Sid, whereby the KuraFace, when launched from MAX, stops responding after about 45 seconds. Not resolved at present, but have a few leads.

Allison

• 20-23 May, 2008
  • Further solidified file formats with Ying
  • Continued working on matlab and python code - the code is now reading/writing almost all the files
  • Figured out how to call matlab from C - need to test on another/faster computer, but it seems to be slow
• 26-30 May, 2008
  • Did a test of the workflow with Ying - the file i/o is in place but there are still things that need to be automated:
  • Refactored Matlab code slightly to create trainer and performance objects
  • Learned how to write C objects for Max/MSP - dealt with many issues that came up when compiling, etc
  • Began writing the C object trainer and performance objects
• 2-11 June, 2008
  • Ran into much trouble trying to call Matlab from max - Sid suggested I implement it in python for now and come back to it when I have time
    • Extra python functions are added and working
  • Implemented the session browser
    • Can load a profile, see all of your sessions, create a new accent, select sessions to include in that accent, train that accent, load old accents to retrain them
    • Was actually quite complicated because max has no listbox - ended up using 2 cellblocks with left/right transfer arrows
  • Create accents:
    • Created an accent template folder and a python script that copies the files from that template folder to the new accent folder
  • Started on Sid's scaling warning request:
    • I'm printing timestamps in the weights files and I created a max object for printing the timestamp in the scale override file
  • Created a new simpleDialog object using mxj that looks prettier and doesn't include an input field
    • Can set both the title and the message
  • Further improved the python code
    • Created a centralized class for getting paths from the profile, accent, network, session, etc.
  • Ying and I integrated the entire system, and it works!
    • Still making minor improvements
• 11-13 June, 2008
  • Added in an alert light that lets the user know if they need to retrain the accent (because they rescaled after training)
  • Created an optionDialog object also using mxj, that can be initialized with a list of option names, and will create an outlet for each one. A bang in the inlet pops up the dialog, and when the user selects one of the options, a bang is sent out the corresponding outlet.
  • Took Sid's original stop consonant data, unscaled it, formatted it correctly for our system, and added it back into the system
  • Did a number of bug fixes/improvements that came up as we were testing the system
  • Ran a test of the system - we can get all the way through profile creation, creating sessions, creating and training an accent, override scaling, and speaking!!
• 16-20 June, 2008
  • Created a new mxj dialog specifically for taking filename input - added it in for create profile and create accent
    • Checks to make sure filename contains no illegal characters
    • Checks to make sure file doesn't already exist in given directory
    • Alerts the user if so and requests another filename
    • Works for both files and directories
  • Fixed a bug in scaling due to incorrect column order in scale file
  • Met with Marguerite to show her how the system works
    • Got many suggestions for things to consider improving
  • Demoed system for DIVA group
  • Added in dAllesandro's song waveform
    • Updated the bpatcher to make use of Max 5 interface elements, and put it in a more accessible location
  • Helped Ying integrate Eric's record/playback bpatcher into the rest of the system
  • Modified the dictionary bpatcher to make use of the Tab object
    • Removes arbitrary phoneme-index mapping -- we can now select and retrieve selected in terms of phoneme label (or index, if desired)
    • Added a "next" button for easier transitions between phonemes
    • Working on adding an audio playback for each phoneme that will help the user identify which sound they are currently training
• 23-27 June, 2008
  • Commented lots of my code and made small improvements as I thought of them
  • Switched to the Intel Mac Mini until my new machine arrives
  • Looked at balancing the volume levels but I may need Bob's help for that
  • Reworked the data trap to integrate with the new system
    • Basically gutted all the data receiving/concatenating/synchronization code and redid it with Ying's help - it should do a good job of keeping in sync
    • Tested it by playing a recording through our new playback system and comparing the original file with the one recorded by the data trap
    • Results are 100% perfect
  • Tested the full system with Marguerite -- scaling wasn't working for some reason so we didn't get very far in the test
    • Took lots of photos, though
• 30 June - 4 July, 2008
  • Created a "synth switch" that will allow us to switch between synths by sending a single number
    • It should also be easy to add new synths into the system
  • Built an object that can be used for coll storage/jit.cellblock synchronization
    • Much easier to understand/use than the previous system of messages
  • Styled some of the bpatchers with inset panels that better-separate their various areas
  • Demoed the system for Sid and took down his suggestions
  • Ying and I started implementing the first suggested new feature: presets that will allow the performer to store and reload the entire state of the system, so that they can set it up for a performance with a single click
    • All of the input sliders, dials, etc in a bpatcher can be saved to a preset by simply dropping in 2 objects - max and our custom objects take care of the rest
• 7 - 11 July, 2008
  • Ying and I completed work on the preset feature
  • Redid the "Create Accent" interface to use "Save"/"Save As", as suggested by Sid
  • Removed all mention of scale override from my code
  • New feature: display which phonemes are present in each trained accent
    • Will help user determine if they're missing any
    • Stays up-to-date as user transfers sessions back and forth
  • Created a Delete script and added a Delete Accent button
  • Added a Delete Session button as well
  • Discussed integration of the face stuff with Ying and Eric
  • Added Delete Preset
  • Added a SoundMap selection umenu
  • Participated in demo for high school tour
• 14 -18 July, 2008
  • Transferred everything over to the new MacBook
    • Fixed 1 small bug resulting from using a newer version of Max: "append $1" messages now may only be sent in the left inlet of a message box
  • Created a [delete] help file
  • Bug fixing and interface adjusting
• 21 - 25 July, 2008
  • Was home sick most of the week
  • Cleaned up some of my code to fit with recent changes
  • Changed playback so that it no longer plays the last sound ad infinitum when"Stop" is clicked (on computers without a glove)
  • Modified the phoneme colour display on the Inspector to take into account vowelProb
    • The total of the background alphas for the phonemes now adds up to 1, just like the total of the phoneme probs does in the real system
• 28 July - 1 August
  • Created a new inspector that visualizes phonemes and v/c probabilities in various ways
  • Ying and I redid the Recorder so that the code is easier to follow. The UI also got an overhaul so that it is easier to use.
  • Lots of small UI adjustments
• 5-7 August
  • Added a foot pedal flag to the EricDataTrap
  • Received the repaired CyberGlove, working on getting it to run on my laptop
    • Had to install a driver for the serial-to-usb converter
  • Also working on making a max collective and standalone application from the code
  • Upgraded to Max 5.0.4 - one of the new features was patcherargs for bpatchers, which allowed me to change the following:
    • Phoneme Coverage Indicator now uses bpatcher args to tell each Indicator which phoneme it is
    • The new inspector now uses bpatcher args to tell each levelSlider whether it's a vowel, consonant, or stop
• 8 - 25 August
  • Vacation!!
• 26 - 29 August
  • Worked out kinks in building standalone applications
    • Created a wiki page with instructions
  • Created a Setup patcher from which the user can set the location of their data files, or create a new data file structure
    • Created a new user data template that will be used for this purpose - have to remember to keep it up-to-date!
  • Redid the default presets for the templates
• 1 - 5 September
  • At the suggestion of Sid, created auto-setup functionality that will automatically chose /users/shared/DIVADATA as the DIVA data directory upon installation, unless a different directory is chosen. If DIVADATA didn't yet exist, it will also copy the entire user data filestructure and some files from Templates
  • Built a standalone, got it working on the Intel Mac Mini, set up the equipment in Eric's lab
    • Also gave them a lesson on how to use the system
    • Noted some bugs/suggestions, see the new bug page
• 8 - 12 September
  • Started fixing bugs noted above
  • Working on getting the wired glove/ Polheums Fastrak to work with the new system
    • Wired glove works as is
    • Fastrak will require some work
  • Noted the change in some command strings from Fastrak->Patriot
    • Tried simply changing these in the code, but it didn't work

Ying

• 20-23 May, 2008
  • Improved on the version with no learning
  • Finished the training interface
  • Solved the creating new folder and automatic saving text file problems by using open source external objects
  • Finished implementing saving scaling data
  • Finished conversion to and from .csv file format
  • Started working on loading accent.
• 26-30 May, 2008
  • Changed the Max/MSP code for vowel, consonant and decision networks based on new accent format.
• 2-5 Jun, 2008
  • Integrated with Allison's session browser.
  • Added Perform window.
  • Added saving and loading of user defined variance and weight.
  • Added Inspector window for easy data inspection and debugging.
• 11-13 Jun, 2008
  • Improving the process of loading and creating a new profile and session.
  • Debugging
  • Improving the adding user defined scale interface
• 16-20 Jun, 2008
  • Improved on the scaling window
  • Demoed the system for the DIVA group
  • Improved on training session
    • Added confirmation on whether the user wants to save changes
    • Added feedback on the status of the session
  • Improved on the input control interface
  • Integrated record and playback module initially done by Eric
• 23-27 Jun, 2008
  • Added preset for setting operating hemisphere
  • Tested with different operating hemisphere and it works fine
  • Added instruction for input control and added tooltips
  • Added assist messages for itCyberglove and polPatriot objects
  • Commented some codes (both C and Max/MSP)
  • Added input control for the touch glove
• 30 June - 4 July, 2008
  • Tested the system in the mobile mode
  • Added comments for Max/MSP code and C code
  • Created a repository for the DIVA project on the local machine (need to move to a backed up server)
  • Started on implementation of preset for various settings with Allison
• 7 - 11 July, 2008
  • Completed work on presets
    • Changed preset saving for accent selection menu (use textedit instead of umenu to same the actual name)
  • Changed the way the user define Z bounds
    • Z bounds are define as voice mapping setting instead of saving in the accent file
    • Users can now define the way they want to map the pitch. The mapping function can be non-linear
  • Changed some small aspects of the interface (arrangements and colors)
  • Started using svn for collaboration. The repository is on DIVA1 machine. Haven't heard from Gregor for setting up svn on the server.
  • Demo for high school students
• 14 - 18 July, 2008
  • Fixed a bug in itCyberglove external object.
    • There was a index out of array bug.
  • Added pitch mapping using a table so that user can define their only pitch mapping function.
    • This is currently used for singing mode where the table is a MIDI table
  • Added tabbed interface so that the user can switch from talking to singing which have different controls.
• 21 - 25 July, 2008
  • Completed initial draft on software user manual in work doc.
    • Need further revision to reflect changes in software.
    • Need to put onto wiki.
  • Added vowel space adjustment.
  • Added color indicator in inspector.
  • Changed sound on/off and plosives input selection.
  • Added tooltips and improved the interfaces.
• 28 July - 1 August, 2008
  • Allison and I redid the Recorder so that the code is easier to follow.
  • Started on implementing plane calibration.
  • Helped Marguerite with training.
• 5 - 8 August, 2008
  • Finished one version of implementation of x-y plane calibration
    • This method uses G command of the Polhemus Patriot tracker to change the source mounting frame
  • Helped Allison with creating Max collectives and packages
  • Looked into the USB driver for Polhemus Patriot
    • Haven't found an existing driver yet
• 11 - 15 August, 2008
  • Finished a second version of x-y plane calibration using quaternion and rotation matrix
    • This is the version currently used in DIVA
  • Added comments to max/msp patchers and c code
  • Upgraded the OS of DIVA1 machine to Leopard.
  • Removed the metro for banging output from the two externals polPatriot.mxo and itCyberglove.mxo
    • This is an attempt to solve the hanging problems. This problem is more prominent when OS is upgraded to Leopard.
    • qelem is used in the thread to trigger output.
• 18 - 22 August, 2008
  • Added a small footpedal to the system
    • Added footpedal selection to allow the flexibility of changing foot pedal used
    • Also added preset for footpedal choice.
  • Shot a video of me talking with DIVA
  • Added comments to Max/MSP code and C code.
  • Modified the user manual for new changes.
  • Added preset for plane calibraiton
  • Started technical documentation on wiki.