Jonathan Helfman, 3/2000
jon@imagebeat.com

Dub is a sound editor for the Palm handheld that works with Hubbub, a sound instant messenger. Dub was designed and implemented by Jonathan Helfman for AT&T Labs-Research and "The Loop," a research group in Menlo Park, CA focused on technology for teenagers.

The purpose of Dub is to let people create and edit sounds quickly, without requiring a mastery of music notation. Our initial user population were the teenage users of Hubbub, an instant messenger for sending MIDI sound messages.

Dub uses a graphical notation that is much simpler to interpret and manipulate graphically than traditional music notation. Dub's graphical notation is inspired by Morton Subotnick's "Making Music" software. Each note in a sound is represented by a selectable rectangle. The rectangle's length corresponds to the note's duration. The rectangle's vertical position corresponds to the note's pitch. The rectangle's horizontal position corresponds to the note's position within the sound sequence. Another version of this notation is used in the Palm application "miniGrid" by miniMusic.

Although Dub shares a graphical notation with other applications, Dub's user interface differs in significant ways that make it much easier to use. Dub's design emphasizes the similarity between editing sounds and the more familiar activity of editing text. Most of Dub's users will already be comfortable entering, selecting, cutting, and pasting symbol sequences with a conventional text editor. Dub's design attempts to leverage these familiar skills in a new context by treating sounds as a sequence of notes and spaces that can be entered, selected, cut, and pasted.

The rectangles that represent Dub's notes have a larger graphical representation than those in similar applications. (In this document, a note's minimum size will be referred to as MINSIZE; a reasonable value for MINSIZE seems to be about five pixels high and five pixels wide.) Larger notes are easier to select, particularly in a Palm application where notes can be selected by tapping directly on their graphical representation. Having larger notes also makes it possible to provide visible feedback when notes are selected.

Unlike other applications, spaces between Dub's notes have an explicit visible representation (e.g. a hollow rectangle). Providing a visible representation for spaces allows users to select and edit spaces just like notes and reinforces the analogy that a sound is a continuous sequence.

To further extend the analogy between editing sounds and text, Dub provides an "insertion point" and allows users to select a "region," a highlighted sub-sequence of notes and spaces. Dub's insertion point and region behave in the expected way: the insertion point can be positioned between notes and spaces; the region can be cut and then pasted at the location of the insertion point. The logic for selecting and editing notes, spaces, and regions is described in detail below.

Screen Descriptions

The initial screen is called the "list view." The list view displays a list of the sounds contained in a Palm sound database (Palm sound databases are implemented as a collection of standard Midi files). [There is a good chance that the name of each sound will be preceded by an associated icon.] In the top right is a category popup, which lets users switch between SIMs (sound instant messages), SIDs (sound identifiers), Alarms, and a user's own personal sound categories. When displaying a user's personal sounds, a "New" button will appear at the bottom of the list view. Selecting the "New" button will bring the user to the "edit view," which is described below. The figure on the left shows the contents of the "SIMs" database. The sound named "Sci-fi" has been selected. Selecting a sound in the list view loads the associated sound into Dub's "play view."

Dub edit view

The edit view looks almost the same as the play view, but the sound's notes, spaces, and name can be modified. When a user tries to edit a read-only sound (such as one of the HubBub SIMs), Dub copies the sound to the user's corresponding personal sound category. When editing an editable sound, the sound's name is not highlighted, but the text insertion point is positioned at the end of the name to make it easy to extend the name. Selecting the button labeled "Done" will return to the list view. Selecting the button labeled "Delete" will popup an alert form to obtain the user's confirmation before deleting the sound and returning to the list view.

In both the play view and the edit view, sounds are displayed graphically in a portion of the display called the "display area."

The dotted vertical line within the display area is called the "insertion point" (see the play view figure). The insertion point it used to specify a point between notes or spaces.

To the left of the display area is a graphical depiction of a two-octave piano keyboard. The keyboard serves as a visual indication that pitch is specified by the vertical dimension of the display area. In the edit view, the keyboard also serves as a visual guide to help users assign notes to specific pitches.

Above the display area is a horizontal region, which is called the "selection border." Tapping in the selection border changes the position of the insertion point. Dragging in the selection border stretches the insertion point into a "region," which may encompass a range of notes and spaces. The edit view figure shows a region, which is marked by a stipple pattern that extends vertically through the selection border and down into the display area.

Playing Sounds

The black triangles in the selection border are used to play the current sound in various ways.

Tapping the black triangle at the far left of the selection border plays the entire sound, even if there is a region.

Tapping the black triangle at the top of the insertion point plays the sound starting at the insertion point.

If there is a region, tapping the black triangle at the beginning of the region plays the sub-sequence of the sound indicated by the region. When playing a sound indicated by a region, Dub plays the sound repeatedly. Each repetition is separated by a brief delay.

The insertion point moves from left to right when notes are played to indicate the end of the currently playing note. The insertion point appears as a solid black vertical line when moving through the region.

When sounds are playing they can be interrupted in a variety of ways. When sounds are playing the black triangle at the far left of the selection border is replaced by a octagonal "stop sign" icon. Selecting the octagon will interrupt the sound, leaving the insertion point at the interrupted position. Any graffiti command will also interrupt a playing sound, leaving the insertion point at the interrupted position. If there is no region, tapping in the selection border will interrupt the sound and begin playing it again from the tapped position. If there is a region, tapping where the region and the selection border overlap will interrupt the sound and begin playing it again from the tapped position. Tapping outside of the region, but still inside the selection border, will interrupt a playing sound and reset the position of the insertion point. Tapping in the display area will interrupt a playing sound, remove any existing region, and allow the sound to be edited as described below.

Creating Notes and Spaces

In the edit view, notes are created by tapping (and possibly dragging) in the display area. When a note is created, the note is highlighted and played. After the note is played the insertion point is positioned at the end of the note.

The display area is an invisible grid of cells that are each MINSIZE wide and high. When tapping in an empty column (one that contains no existing note) a note is created that is MINSIZE in duration. Although the relationship between the smallest possible rectangle and the duration of it's corresponding note can be varied, a reasonable default would make MINSIZE equal to a sixteenth note. When notes are created by taps that are more than MINSIZE apart, a space between the notes (i.e. a silent note) is created automatically. The new space is wide enough to fill the columns between the new note and the closest previous note.

When tapping in a column that contains an existing note or space, the tap is interpreted as an insertion of a new note. If the existing note or space is only MINSIZE wide, the tap inserts a new note after the existing note or space. The sub-sequence from the insertion point through the end of the sound is shifted to the right to make room for inserting the new note.

When tapping in a column with an existing note or space that is wider than a single column, the tap is interpreted as an insertion of a new note, but the existing note or space is not split. Tapping in a column at the end of an existing note or space moves the insertion point to the end of the existing note or space. Tapping in a column at the beginning of an existing note or space moves the insertion point to the beginning of the existing note or space.

If the existing note or space is longer than two columns wide, tapping in a column other than the ones at either end has no effect unless the stylus is dragged. If the stylus is dragged into the last column of the existing note or space, a new note is inserted at the end of an existing note or space. If the stylus is dragged into the first column of the existing note or space, a new note is inserted at the beginning of an existing note or space. If the user continues to drag the stylus after the new note is inserted, Dub behaves as described below (see Dragging Notes, Spaces, and Regions).

It is possible to create notes and spaces using graffiti commands. The lower-case letters (a-g) create the corresponding notes in the lower octave, while the upper-case letters (A-G) create the corresponding notes in the upper octave. Spaces can be created using the graffiti space command (a left to right gesture). Notes and spaces created using graffiti commands are inserted into the current sound at the location of the insertion point and the insertion point is positioned at the end of the new note or space. The default duration for notes and spaces created with graffiti commands should probably be longer than MINSIZE; a reasonable default duration would probably be the equivalent of an eighth note or a quarter note.

It is also possible to create notes and spaces with the "paste" menu command. When a sequence is pasted from the clipboard, it is inserted into the current sound at the location of the insertion point and the insertion point is advanced to the end of the newly pasted sub-sequence

Dragging Notes, Spaces, and Regions

Once notes and spaces are created or selected, they can be dragged immediately without requiring an additional tap. Notes and spaces can only be dragged in discrete units of MINSIZE length. When notes are dragged vertically, their pitch changes. The note plays repeatedly (with a brief delay between repetitions) as it is dragged to indicate the new pitch. If a note is preceded by a space, the space moves vertically with the note. Although a space may be selected, it can not be dragged vertically (it makes no sense to vary the pitch of a silent note).

When notes or spaces are dragged horizontally their duration changes. A note plays repeatedly (with a brief delay between repetitions) as it is dragged to indicate the new duration. Dragging a note or space to the right lengthens it. The entire sub-sequence to the right of the selected note or space is shifted to the right to make room for the longer note or space. Dragging a note or space to the left shortens it. Once a note or space is shortened to less than MINSIZE, it is removed -- further dragging to the left will not affect any notes or spaces to the left of the selected note or space.

Once a region is specified, by dragging in the selection border, it can be "grabbed" by tapping anywhere in the stippled portion of the display area. A grabbed region can be dragged in MINSIZE increments. Dragging a region vertically changes the pitch of the notes in the region. If the region is preceded by a space, the space moves vertically with the region.

Dragging a region to the right shifts the entire sub-sequence from the beginning of the region to the end of the sound. A new space is inserted in the gap at the beginning of the region. If the region had been preceded by a space, the space is extended to fill the gap left by the displaced region.

A region may be dragged left only if it is preceded by a space. The entire sub-sequence from the beginning of the region to the end of the sound is shifted to the left, shortening the space. If the space is shortened to a width that is less than MINSIZE, it is removed entirely, and the region can be dragged left no further.

Deleting Notes, Spaces, and Regions

Notes and spaces can be deleted by selecting them and dragging left as described above.

In addition, the "cut" menu command and the backspace graffiti command can be used to delete a selected note, space, or region. If the selected note, space, or region had been at the end of the sound, the insertion point is positioned at the end of the sound. Otherwise the gap caused by the deletion is repaired by shifting the end part of the sound (the sub-sequence from the end of the gap to the end of the sound) to the left, filling the gap, and positioning the insertion point where the gap had begun.

Menu Commands

The "copy" menu command copies the selected note, space, or region to the clipboard. If there is no region and no note or space is selected, the "copy" menu command plays an error tone.

The "paste" menu command inserts the contents of the clipboard into the sound at the insertion point, positioning the insertion point at the end of the newly pasted sub-sequence As with any other insertion, any existing sub-sequence to the right of the insertion point is shifted right to make room for the newly inserted sub-sequence The "paste" menu command plays an error tone if nothing is in the clipboard.

The "cut" menu command deletes any selected note, space, or region as described above (see Deleting Notes, Spaces, and Regions). If there is no region and no note or space is selected, the "cut" menu command plays an error tone.

Hardware Buttons

The hardware scroll-up button advances the insertion point to end of next note or space. The hardware scroll-down button moves the insertion point to beginning of previous note or space.