Multimedia Technology Curriculum - White Paper

March 1998

Richard Cole, Chair, Computer Science
Marsha Berger, Associate Director, Courant Institute
Jacob Schwartz, Department of Computer Science
Ken Perlin, Director, Center for Advanced Technology

Executive Summary

Continuing advances in multimedia technology present an enormous opportunity, to which all universities including NYU must respond, and of which NYU is well positioned to take advantage. This document presents two complementary activities as means for strengthening NYU's presence in multimedia technology. Together, these will (i) create an interdepartmental Center of national prominence, building on NYU's strengths.(ii) strenghen the multimedia technology program within the CS department. (iii) Create a new Master’s degree in Multimedia technology.


In the last decade, the computer has decisively switched its major use in our society, from being primarily a computational device, to one that supports and promotes communication. "Multimedia" is a catch-phrase for many of the emerging computer-based technologies involved in the evolving communication landscape.

Because of the centrality of communication (including teaching and collaboration) in business, science, and the arts, the development and deployment of multimedia technology are burgeoning activities at NYU, as well as in NYC and the United States as a whole. NYC in particular provides fertile ground, being the nation's center for all media-related content other than feature film (for which it is second only to Hollywood), as well as being the nation's financial center and most international city.

NYU already has key internal strengths. The development of multimedia technology is a growing activity in the Computer Science Department at NYU, and NYU's Interactive Telecommunications Program (ITP) is an internationally recognized leader in multimedia design and production.

A recent Coopers and Lybrand study shows that there is a single gap in human resources that is limiting the growth rate of the NY New Media industry: the lack of qualified technically capable college graduates entering the work force. In fact, that study specifically recommends that the Universities in this area increase their focus on turning out graduate level students who will be capable of filling this gap. Our University, situated as it is in the heart of Silicon Alley (or, more accurately, "Media Alley"), is well placed to become a leading Center for Multimedia technology.

NYU needs to combine its existing strengths, with a focus on turning out technologically literate students to the workforce, who are also well grounded in the real-world applications of the new technologies. This effort will require strong participation by the Department of Computer Science, as well as by various faculty in our other academic departments.

This document proposes to outline such a cohesive multimedia technology presence at NYU. A key ingredient in the growth of this area is the hiring of faculty in multimedia technology. We briefly outline a possible educational activity - a new Masters degree program in Multimedia Technology - to help support this effort. Budgetary options are presented.

Initial Strengths

The CS department is a natural place to focus a multimedia technology effort. Ken Perlin is already running the CAT center, and he has just submitted a major STC preproposal to the NSF. Denis Zorin, who has already made key contributions in the field of computer graphics, will be joining us in the fall. Jack Schwartz was the founding director of CAT. Other computer science faculty pursuing multimedia-related research include Chee Yap, Ralph Grishman, Arthur Goldberg, Bud Mishra, Alan Siegel, and Davi Geiger (see the Appendix). We could build on these efforts to attract new faculty. It would surely be easier to hire new faculty at the junior level in a more traditional departmental setting. The CS Masters program already has a specialization in Multimedia. This provides a reasonable environment for ramping up. The drawbacks would be that developing a new educational program is beyond the resources of the CS dept. These resources are already stretched to their limit to cope with the four educational programs the department offers, in addition to the large increase in undergraduate C.S. enrollment in the last few years. Additionally, although computer science is a logical and even major part of multimedia technology, it is not the whole story.

Plan Component 1: The Proposed MS in Multimedia Technology.

As one of the key elements in our plan we propose creation of a new MS in Multimedia Technology having several goals:
  1. To encourage educational use of this technology by NYU Departments generally;

  2. To add significantly to the stream of persons, trained in multimedia applications and technology, available to New York industry;

  3. To defray some of the costs to the University of an expanded multimedia program, by generating an offsetting revenue stream.
We propose to consider establishing an MS program in multimedia. This would help financially support a Center for Multimedia. As a very preliminary plan to start the discussion going, we propose a two year program, with the following structure. At the center would be Computer Science, and instruction would lead toward enabling students to eventually (in the second year) do projects that allow CS/non-CS pairwise interaction.

The first year (four course semesters) would bring students from various disciplines up to speed on the basics they should know in the various aspects of multimedia technology. This includes simple Java programming, basic linear algebra (enough to manipulate and visualize geometric objects, but not, for example, EigenVectors). It also includes programming for the Web, including Client/Server technology. The students should be required to deliver their assignments over the Web, to accustom them to Web based document/program delivery.

In addition, students should attain basic familiarity with multimedia production tools such as Premiere, MacroMedia Director, PhotoShop, SoundEdit, QuickTime, as well as at least one 3D animation tool, such as Alias, SoftImage, or 3D StudioMax.

Also in these first four course semesters, students should be exposed to some of the key problems that require visualization, data filtering and transmission, and remote collaboration. For this they will need to learn the basics of real-world applications such as financial analysis tools, DNA databases and their current interfaces, search tools such as Nexus, and a basic understanding of how a product such as the Bloomberg is created and delivered.

Finally, students should learn the basics of the study of the social issues that surround of multimedia technology. This will touch subjects including psychology, education, journalism, changing work-styles and impact on urban centers.

The second year would focus on projects courses with two advisors. One advisor would be from within computer science. This advisor would lead the "development of tools" aspect of the project. The other advisor would from another discipline, and would lead the "domain expert" aspect of the project. For example, one possible advanced course would pair Bud Mishra (CS) with David Schwartz (Chemistry/genomics), and would require the students to build and use tools to visualize aspects of the sequenced genome and its associated functional objects (eg: expressed proteins).

Some examples of existing CS/non-CS research partnerships that could feed into this second year (this list is far from exhaustive) are:

A potential difficulty is that many students would not be adequately prepared to take graduate courses in Computer Science as well as in a quite distinct field.

If new faculty taught one course in the new multimedia program, and one in their "home" department, (most likely a PhD level course), this implies four faculty form the core of the Center's multimedia teaching. This imposes some additional load on CS as well as one some other departments. Several adjuncts may be necessary as well. The project requirement would impose quite a load on the four faculty; however we expect that existing CS faculty in graphics, and industrial sources of projects that would need only minimal faculty involvement, would lighten this a bit. Assuming 25 -30 graduates a year, the budget is outlined below.

The goal of encouraging educational use of this technology by NYU Departments is approached by building active student involvement of projects, generally to involve faculty from departments other Computer Science, into the degree curriculum. This curriculum assumes 12 courses of 3 credits each, and is as follows:

Nominal Semester (I):

  1. Introduction to Multimedia Tools and Techniques: HTML, JavaScript, Director and Lingo, Tools for Image, Video, and Sound

  2. Computer Graphics: Basic Graphics in Java

  3. Multiprocess and Object-Oriented programming Threads and Object Concepts in Java

  4. Educational Multimedia at NYU and elsewhere: A series of lectures on the use of multimedia in various NYU departments, other educational settings, and industry.

Nominal Semester (II):

  1. Multimedia Practicum (Project): This will divide the enrolled student body into groups of roughly 3-6, with complementary skills, to work on a semester-long educational or technology-oriented project under the joint direction of (a) a staff member at the proposed Center for Multimedia Educational Technology (see below) and (b) an NYU faculty member involved in the development of a multimedia educational product, or interested in initiating such a project. In some cases, these projects may be organized as ‘group internships’ at industrial projects of special interest.

  2. Advanced Multimedia Techniques
  3. Programming Interactive Web Applications
  4. Electronic Commerce

Nominal Semester (III):

  1. Multimedia Practicum (Project)
  2. Analysis of Images
  3. Analysis of Natural Language
  4. Programming Languages

The last three offerings shown here for the nominal third semester give only one possible student focus. The student will also have the option in this semester to adopt alternate areas of focus. Options can come from courses in ITP, TSOA; TSOA animation and graphic design; Department of Music; School of Business (Multimedia Industry and Policy); School of Education (Music Technology, Educational Multimedia and Multimedia pedagogy.)

The student body to be enrolled would center on persons with undergraduate degrees (or minors) and/or professional experience in programming, but would not be limited to such persons. Students without adequate computer background will be required to take the programming preparatory graduate course already offered by the CS department, in addition to the standard MS degree courses.

Plan Component 2: A New Center for Multimedia Educational Technology

The second component of the plan we propose is to create a new Center for Multimedia Technology, conceptually separate from any existing department, but to be adminitered pro tem by the Computer Science Department. The Center will include a teaching component and a service component, sketched below. This Center will orgainze and support the inter-departmental collaborations which the Masters program sketched above requires, and beyond this provide technical support on a continuing basis for NYU faculty wishing to to undertake the development of multimedia educational products. This entity will require a sutably equipped laboratory space for faculty and project use, and will also support a teaching laboratory for the Multimedia Masters program. It will reuire 3-4 staff members, of which 2 should be senior persons able to steer the Multimedia Practica of the Masters program.

A Center approach affords more flexibility. For example, the Center could hire an artist, or graphics designer; something not likely to happen in a CS dept. Conceptually, any significant multimedia effort would need to be quite interdisciplinary. Yet because the focus is on emerging technologies (as opposed to the design focus of ITP), Computer Science faculty would still need to play a central role. We envision a pairwise collaboration between faculty from at least TSOA, Stern, Wagner, School of Ed, and the science departments on the one hand, and CS faculty on the other hand. An MS program, outlined roughly below, fits more into this interdisciplinary mold. One way that such a Center could draw in other parts of NYU is by having faculty in other depts. apply for release time to spend at the Center, to develop multimedia tools for their specialities. This would also provide a mechanism for MS graduate students to find projects.

Drawbacks include the difficulty of hiring junior tenure-track faculty into such a Center. Where would they get tenure? What would the draw be for a new faculty member to set up an MS program? Where would doctoral students come from? A solution to this problem may be to have Center faculty also have a home department where half their teaching is done, in addition to the Multimedia teaching of the Center. There is precedent for this at NYU. For example, all courses the Undergraduate Women's Studies department are cross-listed with other departments, and each of its faculty members is jointly appointed with another department.

While in principle the idea of jointly appointed faculty solves one problem, it may be problematic for tenure track faculty to have two homes. Hiring a senior person will not ultimately solve this problem either. While the right senior person would be ideal for establishing such a Center, we expect that most new faculty hires would still be at the junior level. In fact, the field is so new that there are not many senior people in it, even including people from more established areas such as graphics. Therefore, such people are in high demand, and there would be very substantial startup costs for senior hires in this area.

Possible faculty hiring

The search committee of the CS dept. is looking at a few top multimedia candidates this year. Although we do not yet know whether a Multimedia Technology Center will begin to take form this year, nor what that form would be, we feel it best to begin the process of educating ourselves in terms of hiring in this area. Following is a brief description of some of the top candidates in the MM/CS area.

Michael Black is currently the manager of the multimedia group at Xerox PARC. He specializes in human motion analysis and its application to human-computer interaction and multimedia. We are also looking at Christoph Bregler, who is about to receive his PhD from Berkeley; he also works on human motion analysis, as well as on integrating speech and graphics. This latter work shows considerable promise for facilitating multilingual lip-synching and facial animation. Their resumes are attached. Both are interested in having practical impact, which is likely to lead to interaction with indviduals outside the department. Both candidates are interviewing at top-five departments such as Stanford and MIT. We are also looking at Dennis Korn, a top candidate nationally in the databases area.

Here is a tentative list of areas for proposed hires:

Impact on the CS Department

Clearly this proposal will require an enlargement of the current effort in the computer science department (see a description of our current multimedia research efforts in the Appendix). We have already begun to build in this area in the educational program in several exciting ways. Over the last few years, we have developed heavily subscribed courses (DO WE HAVE NUMBERS) in Multimedia, Internet and Intranets, Programming for the World Wide Web, and User Interfaces. This is in addition to our long standing Graphics courses, offered at both the MS and PhD levels. We have also hired a second faculty member in graphics (Denis Zorin), in addition to Ken Perlin. This will enable us to offer the graphics courses once a year, rather than alternating between the MS and PhD courses. Additional hiring, on a small scale, will not bring other departments into a multimedia effort, but will enhance the Multimedia visibility of NYU. This level of hiring is easier to support budgetarily, and has a smaller ramp-up. The overall cost would be for two new faculty, with overhead, offset by increased enrollment in courses, with modest advertising of our recently designed Multimedia specialization within the traditional MS program.

Appendix - Multimedia in the CS Dept.

Multimedia has been growing in visibility in the CS dept over the last few years, as well as in its collaborations with other departments around the University. We outline the major faculty and their projects to give an idea of the scope of our current activity. Our main strength in this area is in the visual domains. This includes Graphics, Computer Vision, and Computer Human Interfaces.

Ken Perlin in CS is well known for his interdisciplinary interests. Most recently, he has been working with a class of animation students, in collaboration with Peter Weishar of the Animation Area in the Department of Film and Televiion at TSOA. Ken brings his awareness of technical possibilities and both advisors bring their interest and experience in animation. Ken also works with Philip Benfey in the Department of Biology. They are using the Pad multiscale interface as an interface paradigm for an on-line virtual laboratory and teaching tool. Davi Geiger in CS, who works in Computer Vision, has been collaborating with Bob Shapley of the Center for Neural Science; he has also been involved in discussions with people involved in Medical Imaging in the Medical School. In another direction, the user interface Pad++, developed by Ken Perlin and his associates, is being deployed in David Schwartz's Optical Mapping project in order to better view DNA data (this is being done by Bud Mishra in CS and a Computer Science doctoral student). Chee Yap, another CS faculty member, has recently begun a new project in this area in "Active Visualization", based on the Internet. The Natural Language project Proteus, by Ralph Grishman in CS, is turning its interest to speech, another major focus of computer-human interfaces and an important part of multimedia activity. As another example, an important activity is arising due to Clinical faculty member Arthur Goldbergs Projects course. This course places MS students in projects in local industry. This semester, for the first time, two of these projects are with companies in Silicon Alley. (This is a model for some activities we would like to see develop further). In this work, Arthur has been collaborating with Ajit Kambil in the Information Sciences department at Stern.

It seems clear that if there were more opportunities for interaction, with more faculty and staff to pursue and encourage these interactions, other departments would express even greater interest in using multimedia in their own activities.