Beat Spectrum

For a number of years I worked on an approach to analyze audio and video based on self-similarity. Comparing every part of a media file exhaustively with every other part results in a "similarity matrix" that both visualizes the time structure of the media and facilitates further analysis like segmentation or verse/chorus detection in popular music.

A particular innovation is the "beat spectrum:" a measure of self-similarity of rhythmic audio versus lag time, which can describe the rhythmic character such as tempo, time signature, and "swing."

Along with my colleague Matthew Cooper, we produced a fruitful number of publications and patents exploring the applications of this approach.

2005 Temporal Event Clustering for Digital Photo Collections Matthew Cooper, Jonathan Foote, Andreas Girgensohn, and Lynn Wilcox ACM Transactions on Multimedia Computing, Communications, and Applications, August 8, 2005 [PDF (2251 KB)] 2004 Video Segmentation Combining Similarity Analysis and Classification Matthew Cooper ACM Multimedia 2004, October 28, 2004 [PDF (330 KB)] 2003 Temporal Event Clustering for Digital Photo Collections Matthew Cooper, Jonathan Foote, Andreas Girgensohn, and Lynn Wilcox Proc. ACM Multimedia 2003. pp. 364-373, November 2, 2003 [ PDF (392 KB)] Summarizing Popular Music via Structural Similarity Analysis Matthew Cooper, and Jonathan Foote 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, October 19, 2003 [PDF (117 KB)] Automatically Organizing Digital Photographs Using Time and Content Matthew Cooper, Jonathan Foote, and Andreas Girgensohn Proc. IEEE Intl. Conf. on Image Processing, September 14, 2003 PDF (167 KB)] Media Segementation using Self-Similarity Decomposition Jonathan Foote, and Matthew Cooper Proc. SPIE Storage and Retrieval for Multimedia Databases, Vol. 5021, pp. 167–75, January 20, 2003 [PDF (354 KB)] 2002 Summarizing Video using Non-Negative Similarity Matrix Factorization Matthew Cooper, and Jonathan Foote IEEE Multimedia Signal Processing Workshop, December 11, 2002 [PDF (155 KB)] Creating Music Videos using Automatic Media Analysis Jonathan Foote, Matthew Cooper, and Andreas Girgensohn ACM Multimedia 2002, December 1, 2002 [PDF (720 KB)] Audio Retrieval by Rhythmic Similarity Jonathan Foote, Matthew Cooper, and Unjung Nam 2002 International Symposium on Music Information Retrieval, October 13, 2002 [PDF (395 KB)] Automatic Music Summarization via Similarity Analysis Matthew Cooper, and Jonathan Foote 2002 International Symposium on Music Information Retrieval, October 13, 2002 [PDF (1831 KB)] 2001 Scene Boundary Detection Via Video Self-Similarity Analysis. Matthew Cooper, and Jonathan Foote In Proceedings of the International Conference on Image Processing, Thessaloniki, Greece. October 7-10, 2001., October 7, 2001 [ PDF (257 KB)] Visualizing Musical Structure and Rhythm via Self-Similarity Jonathan Foote, and Matthew Cooper Proc. International Conference on Computer Music (ICMC), Habana, Cuba, September 2001., September 12, 2001 [ PDF (651 KB)] The Beat Spectrum: A New Approach to Rhythm Analysis Jonathan Foote and Shingo Uchihashi In Proceedings of the International Conference on Multimedia and Expo 2001 (ICME), Tokyo, Japan. August 22-25, 2001., August 25, 2001 [PDF (432 KB)] 2000 Automatic Audio Segmentation using a Measure of Audio Novelty. Jonathan Foote In Proceedings of IEEE International Conference on Multimedia and Expo, vol. I, pp. 452-455, 2000., July 30, 2000 [PDF (697 KB)] 1999 Visualizing Music and Audio using Self-Similarity. Jonathan Foote In Proceedings of ACM Multimedia ’99, pp. 77-80, Orlando, Florida, November 1999, October 30, 1999 [PDF (828 KB)]

More at http://www.fxpal.com/?p=MediaAnalysis

U.S. Patent 7,068,723: Method for automatically producing optimal summaries of linear media

U.S. Patent 7,284,004: Summarization of digital files

U.S. Patent 7,424,150: Systems and methods for media summarization