ABSTRACT
Music inherently conveys several open and interesting scientific questions, which all embed a notion of time. Specifically, musical orchestration is the subtle art of writing musical pieces for orchestra, by combining the spectral properties specific to each instrument in order to achieve a particular sonic goal. For centuries up to this day, orchestration has been transmitted empirically and never a true scientific theory of orchestration has emerged, as the obstacles that this analysis and formalization must surmount are tremendous. Indeed, this question puts forward one of the most complex, mysterious, and dazzling aspects of music, which is the use of timbre to shape musical structures in order to impart emotional impact. Timbre is the complex set of auditory qualities (usually refered as the sound colour) that distinguish sounds emanating from different instruments. Intuitively, an auditory object is defined by several properties which evolve in time. Decades of simultaneous research in signal processing and timbre perception have provided evidence for a rational description of audio understanding. We adress these questions by relying on both solid perceptual principles and experiments on known empirical orchestration examples, and developing novel learning and mining algorithms from multivariate time series that can cope with the various time scales that are inherent in musical perception. In this quest, we seek tools for the automatic creation of musical content, a better understanding of perceptual principles and higher-level cognitive functions of the auditory cortex, but also generic learning and analysis techniques for data mining of multivariate time series broadly applicable to other scientific research fields. In this context, the multivariate analysis of temporal processes is required to understand the inherent variability of timbre dimensions, and can be performed through multiobjective time series matching. This has led us to implement and commercialize the first automatic orchestration system called Orchids, which allows for turning any sound into an orchestral score. Our research is now focused on automatic inference through deep representational learning to allow an automatic deciphering of these dimensions in order to provide optimal features for orchestration, by targeting correlations existing in the work of renowned composers. Finally, although recent advances in machine learning get us closer to strong artificial inteligence, most tasks solved are oriented towards mathematico-logical intelligence. We believe that it is fundamental to attack problems of creative intelligence. In this regard, music provides a unique framework, where learning is mostly unsupervised and objectives are not defined through task-oriented goals. Hence, addressing these questions could give rise to a whole new category of creatively intelligent machines. As a proof-of-concept, we recently developed the first live orchestral piano system. The system provides a way to play on a traditional piano, while the different notes are affected to a full classical orchestra in real-time. This system is based on statistical models of deep learning that were trained on seminal examples found in the repertoire for famous composers through a conditioning of symbolic musical time series.
BIOGRAPHY
Philippe Esling received an M.Sc in Acoustics, Signal Processing and Computer Science in 2009 and a PhD on multiobjective time series matching in 2012. He was a post-doctoral fellow in the department of Genetics and Evolution at the University of Geneva in 2012. He is now an associate professor with tenure at IRCAM, Paris 6 since 2013. In this short time span, he has authored and co-authored over 15 peer-reviewed journal papers in prestigious journals such as ACM Computing Surveys, Publications of the National Academy of Science, IEEE TSALP and Nucleic Acids Research. He received a young researcher award for his work in audio querying in 2011 and a PhD award for his work in multiobjective time series data mining in 2013. In applied research, he developed and released the first computer-aided orchestration software called Orchids, commercialized in the Fall of 2014, which is already used by a wide community of composer. He has supervised six masters interns, a C++ developer for a full year and is currently supervising two PhD students. He is the lead investigator in time series mining at IRCAM, the main collaborator in the international France-Canada SSHRC partnership and the supervisor of an international working group on orchestration.