Capture MIDI input

Work in progress

Capturing MIDI input opens the way to "learn­ing" from the per­for­mance of a musi­cian or anoth­er MIDI device. The first step is to use the cap­tured incom­ing NoteOn/Noteoff events, and option­al­ly ControlChange and PitchBend events, to build a poly­met­ric struc­ture that repro­duces the stream.

The dif­fi­cul­ty of this task lies in the design of the most sig­nif­i­cant poly­met­ric struc­ture — for which AI tools may prove help­ful in the future. Proper time quan­ti­za­tion is also need­ed to avoid over­ly com­pli­cat­ed results.

We've made it pos­si­ble to cap­ture MIDI events while oth­er events are play­ing. For exam­ple, the out­put stream of events can pro­vide a frame­work for the tim­ing of the com­pos­ite per­for­mance. Consider, for instance, the tem­po set by a bass play­er in a jazz improvisation.

The _capture() command

A sin­gle com­mand is used to enable/disable a cap­ture: _capture(x), where x (in the range 1…127) is an iden­ti­fi­er of the 'source'. This para­me­ter will be used lat­er to han­dle dif­fer­ent parts of the stream in dif­fer­ent ways.

_capture(0) is the default set­ting: input events are not recorded.

The cap­tured events and the events per­formed on them are stored in a 'cap­ture' file in the temp_bolprocessor fold­er. This file will lat­er be processed by the interface.

(Examples are found in the project "-da.tryCapture".

The first step to use _capture() is to set up the MIDI input and more specif­i­cal­ly its fil­ter. It should at least treat NoteOn and NoteOff events. ControlChange and PitchBend mes­sages can also be captured.

If the pass option is set (see pic­ture), incom­ing events will also be heard on the out­put MIDI device. This is use­ful if the input device is a silent device.

It is pos­si­ble to cre­ate sev­er­al inputs con­nect­ed to sev­er­al sources of MIDI events, each one with its own fil­ter set­tings. Read the Real-time MIDI page for more explanations.

Another impor­tant detail is the quan­ti­za­tion set­ting. If we want to con­struct poly­met­ric struc­tures, it may be impor­tant to set the data to the near­est mul­ti­ple of a fixed dura­tion, typ­i­cal­ly 100 mil­lisec­onds. This can be set in the set­tings file "-se.tryCapture".

Simple example

Let us take a look at a very sim­ple exam­ple of a cap­ture on top of a performance.

C4 _D4 _capture(104) E4 F4 G4 _capture(0) A4 B4

The machine will play the sequence of notes C4 D4 E4 F4 G4 A4 B4. It will lis­ten to the input while play­ing E4 F4 G4. It will record both the sequence E4 F4 G4 and the notes received from a source tagged "104".

Suppose that the sequence G3 F3 D3 was played on top of E4 F4 G4. The cap­ture file might look like this:

Note that all dates are approx­i­mat­ed to mul­ti­ples of 100 mil­lisec­onds. For exam­ple, the NoteOff of input note G3 falls exact­ly on the date 3000 ms, which is the NoteOff of the played note E4.

The record­ing of input and played notes starts at note E4 and ends at note G4, as spec­i­fied by _capture(104) and _capture(0).

An accept­able approx­i­ma­tion of this sequence would be the poly­met­ric expression:

C4 D4 {E4 F4 G4, - - G3 - F3 - D3 - -} A4 B4

Approximations will be cre­at­ed from the cap­ture files at a lat­er stage.

Combining 'wait' instructions

Try:

_script(wait for C3 channel 1) C4 D4 _capture(104) E4 F4 G4 _script(wait for D3 channel 1) _capture(0) A4 B4

The record­ing takes place dur­ing the exe­cu­tion of E4 F4 G4 and dur­ing the unlim­it­ed wait­ing time for note D3. This allows events to be record­ed even when no events are being played.

The C3 and D3 notes have been used for ease of access on a sim­ple key­board. The dates in the cap­ture file are not incre­ment­ed by the wait times.

The fol­low­ing is a set­up for record­ing an unlim­it­ed sequence of events while no event is being played. Note C0 will not be heard as it has a veloc­i­ty of zero. Recording ends when the STOP or PANIC but­ton is clicked.

_capture(65) _vel(0) C0 _script(wait forever) C0

Interpreting the record­ed input as a poly­met­ric struc­ture will be made more com­plex by the fact that no rhyth­mic ref­er­ence has been provided.

Microtonal corrections

In the fol­low­ing exam­ple, both input and out­put receive micro­ton­al cor­rec­tions of the 

_scale(just intonation,0) C4 D4 _capture(104) E4 F4 G4 A4 _capture(0) B4

Below is a cap­ture file obtained by enter­ing G3 F3 D3 over the sequence E4 F4 G4 A4.

The out­put events (source 0) are played on MIDI chan­nel 2, and the input events (source 104) on MIDI chan­nel 1. More chan­nels will be used if out­put notes have an over­lap — see the page MIDI micro­tonal­i­ty. In this way, pitch­bend com­mands and the notes they address are dis­trib­uted across dif­fer­ent channels.

Added pitchbend

In the fol­low­ing exam­ple, a pitch­bend cor­rec­tion of +100 cents is applied to the entire piece. It does mod­i­fy out­put events, but it has no effect on input events.

_pitchrange(200) _pitchbend(+100) _scale(just intonation,0) C4 D4 _capture(104) E4 F4 G4 A4 _capture(0) B4

Again, after play­ing G3 F3 D3 over the sequence E4 F4 G4 A4:

Pitchbend cor­rec­tions applied to the input (source 104) are only those induced by the micro­ton­al scale. Pitchbend cor­rec­tions applied to the out­put (source 0) are the com­bi­na­tion of micro­ton­al adjust­ments (see pre­vi­ous exam­ple) and the +100 cents of the pitch­bend command.

Capturing and recording more events

The _capture() com­mand allows you to cap­ture most types of MIDI events: all 3-byte types, and the 2-byte type Channel pres­sure (also called Aftertouch).

Below is a (com­plete­ly unmu­si­cal) exam­ple of cap­tur­ing dif­fer­ent messages.

The cap­ture will take place in a project called "-da.tryReceive":

_script(wait for C0 chan­nel 1) _capture(111) D4 _pitchrange(200) _pitchbend(+50) _press(35) _mod(42) D4 _script(wait forever)

The record­ing will have a "111" mark­er to indi­cate which events have been received. Only two notes D4 are played dur­ing the record­ing, the sec­ond one is raised by 50 cents and has a chan­nel pres­sure of 35 and a mod­u­la­tion of 42.

After play­ing back the two D4s, the machine will wait until the STOP but­ton is clicked. This gives the oth­er machine time to send its own data and have it recorded.

The sec­ond machine is anoth­er instance of BP3 — actu­al­ly anoth­er tag on the interface's brows­er with the "-da.trySend" project:

{_vel(0) <<C0>>} G3 _press(69) _mod(5430) D3 _pitchrange(200) _pitchbend(+100) E3

This project will start by send­ing "{_vel(0) <<C0>>} " which is the note C0 with veloc­i­ty 0 and dura­tion null (an out-time object). This will trig­ger "-da.tryReceive" which wait­ed for C0. The curly brack­ets {} restrict veloc­i­ty 0 to the note C0. Outside of this expres­sion, the veloc­i­ties are set to their default val­ue (64). In this data, chan­nel pres­sure, mod­u­la­tion and a pitch­bend cor­rec­tion of +100 cents are applied to the final note E3.

The result­ing sound is ter­ri­ble, you've been warned:

However, the 'cap­ture' file shows that all events have been cor­rect­ly recorded:

Captured events (from "-da.trySend") are coloured red. They have been auto­mat­i­cal­ly assigned to MIDI chan­nel 2, so that cor­rec­tions will not be mixed between per­for­mance and reception.

The pitch­bend cor­rec­tions are shown in the "cents cor­rec­tion" col­umn, each applied to its own channel.

The chan­nel pres­sure cor­rec­tions (coloured blue) dis­play the expect­ed val­ues. The mod­u­la­tion cor­rec­tions (in the range 0 to 16383) are divid­ed into two 3-byte mes­sages, the first car­ry­ing the MSB and the sec­ond the LSB.

There is a time mis­match of approx­i­mate­ly 150 mil­lisec­onds between the expect­ed and actu­al dates, but the dura­tions of the notes are accu­rate. The mis­match is caused by the delay in the trans­mis­sion of events over the vir­tu­al port. The data looks bet­ter if the quan­ti­za­tion in the "-da.tryReceive" project is set to 100 ms instead of 10 ms. However, this is of minor impor­tance, as a "nor­mal­i­sa­tion" will take place dur­ing the (forth­com­ing) analy­sis of the "cap­ture" file.

For geeks: The last col­umn indi­cates where events have been record­ed in the pro­ce­dure sendMIDIEvent(), file MIDIdriver.c.

Capture events without the need to perform

The set­up of the "-da.tryReceive" project should be for example:

_capture(99) _vel(0) C4 _script(wait forever)

Note C4 is not heard due to its veloc­i­ty 0. It is fol­lowed with all MIDI events received by the input until the STOP but­ton is clicked. This makes it pos­si­ble to record an entire per­for­mance. The pro­ce­dure can be checked with items pro­duced and per­formed by the Bol Processor.

Note that the inclu­sion of pitch­bend mes­sages makes it pos­si­ble to record music played on micro­ton­al scales and (hope­ful­ly) iden­ti­fy the clos­est tun­ings suit­able for repro­duc­tion of the piece of music.

For exam­ple, try to cap­ture this phrase from Oscar Peterson's Watch What Happens:

_tempo(2) {{4, {{2, F5 Bb4 C5 C4} {3/2, D4} {1/2, Eb4 Db4 D4}, {F4, C5} {1/2, F4, G4} {1/2, F3, G3} {2, Gb3}}}, {4, {C4 {1, Bb3 Bb2} {2, A2}, {D3, A3} {1, Eb3 Eb2} {2, D2}}}}

The result is com­plex, but it's going to lend itself to being analysed automatically:

0 F5 NoteOn 77 64
0 F4 NoteOn 65 64
0 C5 NoteOn 72 64
0 C4 NoteOn 60 64
0 D3 NoteOn 50 64
0 A3 NoteOn 57 64
230 F5 NoteOff 77 0
230 Bb4 NoteOn 70 64
470 Bb4 NoteOff 70 0
470 C5 NoteOff 72 0
470 C5 NoteOn 72 64
470 F4 NoteOff 65 0
470 C5 NoteOff 72 0
470 F4 NoteOn 65 64
470 G4 NoteOn 67 64
470 C4 NoteOff 60 0
470 Bb3 NoteOn 58 64
470 D3 NoteOff 50 0
470 A3 NoteOff 57 0
470 Eb3 NoteOn 51 64
710 C5 NoteOff 72 0
710 C4 NoteOn 60 64
710 F4 NoteOff 65 0
710 G4 NoteOff 67 0
710 F3 NoteOn 53 64
710 G3 NoteOn 55 64
710 Bb3 NoteOff 58 0
710 Bb2 NoteOn 46 64
710 Eb3 NoteOff 51 0
710 Eb2 NoteOn 39 64
960 C4 NoteOff 60 0
960 D4 NoteOn 62 64
960 F3 NoteOff 53 0
960 G3 NoteOff 55 0
960 F#3 NoteOn 54 64
960 Bb2 NoteOff 46 0
960 A2 NoteOn 45 64
960 Eb2 NoteOff 39 0
960 D2 NoteOn 38 64
1730 D4 NoteOff 62 0
1730 Eb4 NoteOn 63 64
1830 Eb4 NoteOff 63 0
1830 C#4 NoteOn 61 64
1880 C#4 NoteOff 61 0
1880 D4 NoteOn 62 64
1960 D4 NoteOff 62 0
1960 F#3 NoteOff 54 0
1960 A2 NoteOff 45 0
1960 D2 NoteOff 38 0

Only sig­nif­i­cant columns are dis­played. The ori­gin of dates is set to the first NoteOn or NoteOff received.

The next task on our agen­da will be to analyse the 'cap­ture' file and recon­struct the orig­i­nal poly­met­ric expres­sion (shown above) or an equiv­a­lent ver­sion. Then we can con­sid­er mov­ing on to gram­mars, sim­i­lar to what we've done with import­ed MusicXML scores (read page).