Multimedia Technology Curriculum - White Paper

March 1998

Richard Cole, Chair, Computer Science
Marsha Berger, Associate Director, Courant Institute
Jacob Schwartz, Department of Computer Science
Ken Perlin, Director, Center for Advanced Technology

Executive Summary

Continuing advances in multimedia technology present an enormous opportunity, of which NYU is well positioned to take advantage. This document presents two alternatives for improving NYU's presence in multimedia technology. One involves creating a large interdepartmental Center of national prominence, building on NYU's strengths. The second involves building up the focus on multimedia technology within the CS department.


In the last decade, the computer has decisively switched its major use in our society, from being primarily a computational device, to one that supports and promotes communication. "Multimedia" is a catch-phrase for many of the emerging computer-based technologies involved in the evolving communication landscape.

Because of the centrality of communication (including teaching and collaboration) in business, science, and the arts, the development and deployment of multimedia technology are burgeoning activities at NYU, as well as in NYC and the United States as a whole. NYC in particular provides fertile ground, being the nation's center for all media-related content other than feature film (for which it is second only to Hollywood), as well as being the nation's financial center and most international city.

NYU already has key internal strengths. The development of multimedia technology is a growing activity in the Computer Science Department at NYU, and NYU's Interactive Telecommunications Program (ITP) is an internationally recognized leader in multimedia design and production.

If NYU can combine these strengths with growth in teaching the application of multimedia technology in our other academic departments, then the university, situated as it is in the heart of Silicon Alley (or, more accurately, "Media Alley"), is well placed to become a leading Center for Multimedia technology. This document explores several options for establishing a more cohesive multimedia technology presence at NYU. A key ingredient in the growth of this area is the hiring of faculty in multimedia technology. Several options are presented for this as well. We briefly outline a possible educational activity - a new Masters degree program in Multimedia Technology - to help support this effort. Budgetary options are presented.

Multimedia at NYU

We see two potential scenarios for building up multimedia at NYU. In the first, multimedia technology sits in the Computer Science Dept. In the second, NYU establishes a separate Center for Multimedia Technology. Briefly, here are some pros and cons of the two approaches.

Multimedia Technology Division in CS

The CS department is a natural place to house a multimedia technology effort. Ken Perlin is already running the CAT center, and he has just submitted a major STC preproposal to the NSF. Denis Zorin, who has already made key contributions in the field of computer graphics, will be joining us in the fall. Jack Schwartz was the founding director of CAT. Other computer science faculty pursuing multimedia-related research include Chee Yap, Ralph Grishman, Arthur Goldberg, Bud Mishra, Alan Siegel, and Davi Geiger (see the Appendix). We could build on these efforts to attract new faculty. It would surely be easier to hire new faculty at the junior level in a more traditional departmental setting. The CS Masters program already has a specialization in Multimedia. This provides a reasonable environment for ramping up. The drawbacks would be that developing a new educational program is beyond the resources of the CS dept. These resources are already stretched to their limit to cope with the four educational programs the department offers, as well as the large increase in undergraduate C.S. enrollment in the last few years. Additionally, although computer science is a logical and even major part of multimedia technology, it is not the whole story.

Center for Multimedia Technology

The more daring prospect is a Center for Multimedia Technology, separate from any existing departments. The Center could include a research component, a teaching component and a service component, sketched below. A Center approach affords more flexibility. For example, the Center could hire an artist, or graphics designer; something not likely to happen in a CS dept. Conceptually, any significant multimedia effort would need to be quite interdisciplinary. Yet because the focus is on emerging technologies (as opposed to the design focus of ITP), Computer Science faculty would still need to play a central role. We envision a pairwise collaboration between faculty from at least TSOA, Stern, Wagner, School of Ed, and the science departments on the one hand, and CS faculty on the other hand. An MS program, outlined roughly below, fits more into this interdisciplinary mold. One way that such a Center could draw in other parts of NYU is by having faculty in other depts. apply for release time to spend at the Center, to develop multimedia tools for their specialities. This would also provide a mechanism for MS graduate students to find projects.

Drawbacks include the difficulty of hiring junior tenure-track faculty into such a Center. Where would they get tenure? What would the draw be for a new faculty member to set up an MS program? Where would doctoral students come from? A solution to this problem may be to have Center faculty also have a home department where half their teaching is done, in addition to the Multimedia teaching of the Center. There is precedent for this at NYU. For example, all courses the Undergraduate Women's Studies department are cross-listed with other departments, and each of its faculty members is jointly appointed with another department.

While in principle the idea of jointly appointed faculty solves one problem, it may be problematic for tenure track faculty to have two homes. Hiring a senior person will not ultimately solve this problem either. While the right senior person would be ideal for establishing such a Center, we expect that most new faculty hires would still be at the junior level. In fact, the field is so new that there are not many senior people in it, even including people from more established areas such as graphics. Therefore, such people are in high demand, and there would be very substantial startup costs for senior hires in this area.

Possible faculty hiring

The search committee of the CS dept. is looking at a few top multimedia candidates this year. Although we do not yet know whether a Multimedia Technology Center will begin to take form this year, nor what that form would be, we feel it best to begin the process of educating ourselves in terms of hiring in this area. Following is a brief description of some of the top candidates in the MM/CS area.

Michael Black is currently the manager of the multimedia group at Xerox PARC. He specializes in human motion analysis and its application to human-computer interaction and multimedia. We are also looking at Christoph Bregler, who is about to receive his PhD from Berkeley; he also works on human motion analysis, as well as on integrating speech and graphics. This latter work shows considerable promise for facilitating multilingual lip-synching and facial animation. Their resumes are attached. Both are interested in having practical impact, which is likely to lead to interaction with indviduals outside the department. Both candidates are interviewing at top-five departments such as Stanford and MIT. We are also looking at Dennis Korn, a top candidate nationally in the databases area.

Here is a tentative list of areas for proposed hires:

Multimedia and Educational Programs

Here are two possible paths.

Enlarge the CS effort:

The first option is to enlarge the effort in the computer science department (see a description of our current multimedia research efforts in the Appendix). We have already begun to build in this area in the educational program in several exciting ways. Over the last few years, we have developed heavily subscribed courses (DO WE HAVE NUMBERS) in Multimedia, Internet and Intranets, Programming for the World Wide Web, and User Interfaces. This is in addition to our long standing Graphics courses, offered at both the MS and PhD levels. We have also hired a second faculty member in graphics (Denis Zorin), in addition to Ken Perlin. This will enable us to offer the graphics courses once a year, rather than alternating between the MS and PhD courses. Additional hiring, on a small scale, will not bring other departments into a multimedia effort, but will enhance the Multimedia visibility of NYU. This level of hiring is easier to support budgetarily, and has a smaller ramp-up. The overall cost would be for two new faculty, with overhead, offset by increased enrollment in courses, with modest advertising of our recently designed Multimedia specialization within the traditional MS program.

Establish an MS program in Multimedia:

The second, more ambitious, plan is to consider establishing an MS program in multimedia. This would help financially support a Center for Multimedia. As a very preliminary plan to start the discussion going, we propose a two year program, with the following structure. At the center would be Computer Science, and instruction would lead toward enabling students to eventually (in the second year) do projects that allow CS/non-CS pairwise interaction.

The first year (four course semesters) would bring students from various disciplines up to speed on the basics they should know in the various aspects of multimedia technology. This includes simple Java programming, basic linear algebra (enough to manipulate and visualize geometric objects, but not, for example, EigenVectors). It also includes programming for the Web, including Client/Server technology. The students should be required to deliver their assignments over the Web, to accustom them to Web based document/program delivery.

In addition, students should attain basic familiarity with multimedia production tools such as Premiere, MacroMedia Director, PhotoShop, SoundEdit, QuickTime, as well as at least one 3D animation tool, such as Alias, SoftImage, or 3D StudioMax.

Also in these first four course semesters, students should be exposed to some of the key problems that require visualization, data filtering and transmission, and remote collaboration. For this they will need to learn the basics of financial analysis tools, DNA databases and their current interfaces, search tools such as Nexus, and a basic understanding of how a product like the Bloomberg is created and delivered.

Finally, students should learn the basics of the study of the social issues that surround of multimedia technology. This will touch subjects including psychology, education, journalism, changing work-styles and impact on urban centers.

The second year would focus on projects courses with two advisors. One advisor would be from within computer science. This advisor would lead the "development of tools" aspect of the project. The other advisor would from another discipline, and would lead the "domain expert" aspect of the project. For example, one possible advanced course would pair Bud Mishra (CS) with David Schwartz (Chemistry/genomics), and would require the students to build and use tools to visualize aspects of the sequenced genome and its associated functional objects (eg: expressed proteins).

Some examples of existing CS/non-CS research partnerships that could feed into this second year (this list is far from exhaustive) are:

A potential difficulty is that many students would not be adequately prepared to take graduate courses in Computer Science plus a traditionally quite distinct field.

If new faculty taught one course in the new multimedia program, and one in their "home" department, (most likely a PhD level course), this implies 4 faculty form the core of the Center's multimedia teaching. This imposes some additional load on CS as well as one some other departments. Several adjuncts may be necessary too. The project requirement would impose quite a load on the 4 faculty; however we expect that existing CS faculty in graphics, and industrial sources of projects that would need only minimal faculty involvement, would lighten this a bit. Assuming 25 -30 graduate a year, the budget is outlined below.

Appendix - Multimedia in the CS Dept.

Multimedia has been growing in visibility in the CS dept over the last few years, as well as in its collaborations with other departments around the University. We outline the major faculty and their projects to give an idea of the scope of our current activity. Our main strength in this area is in the visual domains. This includes Graphics, Computer Vision, and Computer Human Interfaces.

Ken Perlin in CS is well known for his interdisciplinary interests. Most recently, he has been working with a class of animation students, in collaboration with Peter Weishar of the Animation Area in the Department of Film and Televiion at TSOA. Ken brings his awareness of technical possibilities and both advisors bring their interest and experience in animation. Ken also works with Philip Benfey in the Department of Biology. They are using the Pad multiscale interface as an interface paradigm for an on-line virtual laboratory and teaching tool. Davi Geiger in CS, who works in Computer Vision, has been collaborating with Bob Shapley and Nava Rubin of the Center for Neural Science; he has also been involved in discussions with people involved in Medical Imaging in the Medical School. In another direction, the user interface Pad++, developed by Ken Perlin and his associates, is being deployed in David Schwartz's Optical Mapping project in order to better view DNA data (this is being done by Bud Mishra in CS and a Computer Science doctoral student). Chee Yap, another CS faculty member, has recently begun a new project in this area in "Active Visualization", based on the Internet. The Natural Language project Proteus, by Ralph Grishman in CS, is turning its interest to speech, another major focus of computer-human interfaces and an important part of multimedia activity. As another example, an important activity is arising due to the Projects course by CS clinical faculty member Arthur Goldberg. This course places MS students in projects in local industry. This semester, for the first time, two of these projects are with companies in Silicon Alley. (This is a model for some activities we would like to see develop further). In this work, Arthur has been collaborating with Ajit Kambil in the Information Sciences department at Stern.

It seems clear that if there were more opportunities for interaction, with more faculty and staff to pursue and encourage these interactions, other departments would express even greater interest in using multimedia in their own activities.