This is a discussion document that offers a new research focus for
CISR's collaborative multimedia program. This theme is relevant to the post
production editing of multimedia and traditional video/film, and to more general
questions of multimedia interactivity.
This paper also intends on explicating some of the formal properties of the new
mediums and suggests the application of this understanding through novel software
design principles.
The basic idea is to apply database technology to the creation and
structuring of media art works. The hypotheses are that media art works
can be usefully represented as databases, that such databases ("mediabases")
can be used as practical tools to support shared invention, production, and
delivery of both interactive and linear media works, and that database science
allows for a new class of forms for interactive work.
Code, Structure, Script
A mediabase is a computer database whose purpose is to store a media work or
information pertaining to the structure and creation of a media work.
The mediabase will give access to the formal characteristics of a new
media work and may blend production content, script, and, in some cases,
production data.
In an interactive work a viewer's interventions would be implemented
as database retrieval operations, and the presented images and sounds
would be data extracted from the mediabase.
For a traditional work (film or video) edited in a non-linear editing
suite, the mediabase would, allow for editors to automate some of the
task presently processed through the labours of editing decision lists.
A number of current interactive media works, notably nonfiction works that
deal with archives, are concerned with presenting contents of informational
databases. Such databases, which we call domain databases, are not complete
mediabases in the sense that we are using the term here, because they
contain the content of the work (or part of it), but not any script, production,
or presentation data.
Every media work involves script, production, presentation, and content data.
In current practice, this data is recorded in various ad-hoc ways. Under the
mediabase concept, this data is organised and integrated in machine-readable
form, in a way that allows the use of contemporary database tools.
The extent to which the data can be captured in this way and the extent to
which it must be transformed to be integrated effectively are research issues.
Research Opportunities
Taking the database approach to media works might lead us down a
number of research avenues:
Collaborative Work
Mediabases would be good for supporting collaborative work, in that a
mediabase can have specific elements that are created, maintained, and
shared in organized ways by the various individuals and teams involved
with the creation process.
Research in this area would entail analysis of media works creation processes,
and design of appropriate database structures to support the required collaborative
relationships. An important challenge would be to develop common concepts for
minimizing the number of different database designs needed to support the
various media and styles in current practice.
Database technology is presently highly evolved in the areas of data
sharing and distributed access. This represents a great potential
advantage, one which could help this project produce usable early results.
Navigation
Mediabases are not intended to be only administrative tools for organizing
production teams. The intent is to develop data design and programming principles
that render mediabase systems capable of implementing aesthetic principles in
media works.
One project in the aesthetic area would apply database techniques to the problem of
selecting paths through navigable works. At present, such paths usually must be
completely preprogrammed. As the complexity of interactive genres increases, this
will become less and less feasible. Where discrete works or fragments are merged,
preprogrammed navigation may be completely impossible without re-doing
whole productions. Also, preprogramming may in some cases be aesthetically
undesirable. In general, there is a need for computer assistance
in the making of navigation decisions.
If we think of a linear work as a particular traversal of a navigable one, it
becomes evident that navigating and editing have much in common.
Montage theory describes the meanings of cinematic transitions (editing) in
terms of the attributes of the shots and scenes involved.
This project would seek to apply montage theory to the navigation problem,
using a database implementation approach. The research would first devise
a methodology for representing shot and scene attributes in relational database
form, then develop software to use those attributes for helping make navigation
decisions in interactive works. This software would present to the user, at each
interactive decision point, those navigation alternatives that would tend to have
the most cinematic meaning.
Thus, the interactive work would tend to foster valid viewer experiences,
regardless of how much preprogramming had been done or not done.
In the project plan we wish to develop several stages of montage software
development, beginning with simple algorithmic code, and ending with
mixed-initiative expert systems.
This project concept is based principally on the ideas of Oliver Hockenhull,
a Vancouver filmmaker, writer and communition theorist, who has for
several years been working on the concept of applying montage theory to
interactive works.
-----
The Associative Work & Self-Contextualization
Most of today's interactive media work is based on the
hypermedia concept, in which data elements can contain one or more pointers that
viewers can follow to reach other such elements. Thus, a hypermedia document is a
network, in which data elements form a multiply connected nonhierarchical graph.
Navigation, the process of following pointers from one
data element to another, is a concept akin to hunting, in which the viewer
"goes" from one place to another, seeking an objective.
In order to produce a hypermedia work capable of delivering a rich viewer experience,
its creators must develop and present a large and complex context -- a cyberspace...
Generally speaking, creating cyberspaces is time consuming and expensive.
The hypermedia model does not easily support works based on gathering, rather than
hunting, paradigms. In such works, the counterpart of navigation is associatioon,
in which elements are gathered from global contexts and arranged to form local contexts.
Associative works are works that
offer the viewer the ability to gather elements from a global context and arrange
them into one or more private contexts.
Association is a relational activity, in which the viewer builds
contexts through interaction with the work.
We call an associative work self-contextualizing to suggest that each viewer's
particular context is created afresh from the elements of the work, without
starting from a prebuilt global form.
We envision the primary elements of an associative work as follows:
1. One or more search spaces, which are databases from which the viewer may
extract elements of content. The union of all a work�s search spaces is referred
to as its global context. Many, if not most, works will have only a single search
space in their global contexts.
2. One or more gathering tools that the viewer can use to find elements in the
global context. These tools may be common to all search spaces, or specific to
particular ones.
3. One or more personal workspaces for each viewer. For many works,
these workspaces would be non-volatile, to allow continuity over multiple
sessions. The structure of the workspaces would vary from work to work, and
could vary from simple to complex.
4. One or more arranging tools that the viewer can use to arrange gathered
elements in the workspaces.
It is natural to envision the user interface to such a work as one that would
allow many different objects -- images, animations, sounds, and so on -- to coexist
in a single multimedia presentation at once. Which elements were combined and
juxtaposed would vary, depending on the what objects the viewer had gathered and
how she or he had arranged them in the personal workspace.
Searches, Continuity, and Style
In most associative works, how searches work will be the strongest single
determinant of viewer experience. We believe that most associative works
will support mixed-initiative searching, i.e. a searching in which gathering
choices are made jointly by the viewer and by the work itself.
One class of mixed-initiative systems features searches that depend on the contents
and arrangement of the viewer's current personal context. For example, the montage
approach described above provides a way for searches to be qualified by cinematic
attributes, these choice of an attribute is selected by the viewer. Specifically,
the searches in a montage-based associative work are constrained to find only
(or mainly) those elements that are cinematically meaningful in the current
personal context.
In a mixed-initiative associative system, as a viewer built up her or his own
workspace, successive searches would tend increasingly to find objects that "worked"
in it, contextually. Thus, the system could support continuity and style.
The system would learn the viewer.
Publication and Combination
An attractive outcome of the gathering process in
associative works is that the personal workspaces created by the viewers (or, if you
will, co-created by the viewers and the original authors) are themselves
valid artifacts that may be preserved and shared as searchable contexts.
We say that a personal workspace offered to others as a searchable context
has been published.
Given certain standards, search spaces that had never "met" each other
before could be combined to create new possibilities. For example, CISR
could concatenate an associative Cunningham Archive search space with search
spaces from other artists' archives of the same historical period to allow
viewers to find new associations.
Current database technology is moving toward an object-oriented paradigm,
in which data elements and the programs that use them are bundled into relatively
self-contained units ("objects"). Because they are self-contained, these units
tend to be more easily shared between contexts, and transported into new ones.
Thus, having object-oriented mediabases would facilitate the reuse of published
media material, the combining of disparate media works, and other interesting
integrations.
Making sharing and reuse of work easier has its risks, in that
intellectual property and integrity of creative results can be more easily
compromised. This risk would be offset somewhat by the fact that modern database
tools have highly developed access control facilities which, if properly invoked,
can provide highly selective and secure restrictions.
Project Scope
This R&D project extends from cinematic theory to database theory, with the
results having immediate practical application. The results would depend
critically on effective interweaving of technical and aesthetic disciplines.
A number of outcomes might be supported: Various commercial and/or prototypical
production tools could be developed to serve the different interests mentioned
in Opportunity, above. Sponsorship of exploratory and/or demonstration art works and
the development of a montage software package would be a clear option.
To do the subject justice, collaboration with Banff and other sites inside and
outside Rapport would be indicated. Finally, the idea would seem to offer
good potential for academic research at the graduate and possibly faculty level.
Conclusion:
It is fruitful to consider the digitalization of media creations
(film, video, or interactive multimedia) as a distributed database
problem in which the work is realised as a single shared database ("Mediabase")
that is designed, built, and used in various ways by the creative team.
For interactive works, the database and associated programming codes would also
be the work's final form. For conventional linear works, the final assembly
process would be a specifically formatted extract of the database.
This approach is distinct from merely using a multimedia user interface as a
way to access an informational database in which the information in the database
is viewed as domain data.
Understanding the relationship of aesthetic intent, software structure,
and user interaction is a necessary departure for furthering
eloquence in the new media.
Mediabasediscussiondoc.html/cisr/oliver@cisr.bc.ca