USB MIDI Control Surface Mixing Console

Introduction

Our project is an 8-track USB MIDI control surface for music production and audio engineering in Ableton, a digital audio workstation. A control surface lets the user manipulate audio tracks without touching the software, allowing for physical interaction with the music and more screen space for other key information when working. Our control surface physically controls two tracks at a time with two sets of motorized faders for volume, knobs for panning (moving the track left or right), and buttons for mute, solo, and arm control. Playback and record control (play, stop, and record) and track selection are also available via buttons on the control surface. Our control surface can also follow parameter automation, where you can program automatic volume or other parameter changes during the track’s playback (only works in receive mode).

Project Objective

Control surfaces can be expensive, costing upwards of thousands of dollars. The objective was to build a high quality control surface that could be affordable to scale up (add physical control to more tracks). In this case, high quality is defined as possessing comprehensive track control (volume, solo, arm, mute) and interface well with DAWs (digital audio workstations). The primary obstacle for creating a high quality control surface is interfacing with the DAW. Many companies use their own proprietary protocols for interfacing with DAWs (I.E Mackie Control). Also DAWs that do possess control surface libraries/APIs typically do not have official documentation.

Design

The control surface uses a Raspberry Pi 4 model B to send and receive MIDI messages to and by the DAW respectively that correspond to the state of each parameter. The board and GUI reflects these state changes accordingly. The control surface has two modes designed for appropriate fader behavior: receive and transmit. In receive mode (default when the software is first run), the board will receive the current state of the volume parameter set in Ableton and adjust the board’s fader level accordingly via motor control. In transmit mode, the volume fader can be manually adjusted to change the volume level in Ableton, sending MIDI messages corresponding to the track’s fader level. Button MIDI messages and rotary encoders can be transmitted and received regardless of the receive/transmit mode as they do not possess a physical moving state on the board.

Hardware Design

In this project, we worked with numerous different hardware elements. Mounting and packaging everything in a way that was secure, ergonomic, and prevented electronic interference was an initial challenge. We used cardboard because we could cut and manipulate it easily for prototyping. We sketched out our initial designs. The angle of the control surface was set to both have space to house all the electronics and be ergonomic (having a slight upward slant). We also cut holes in the back to easily connect cables to the RPi4.

The board has 11 buttons – 3 for each track, 3 for playback/record, and 2 for receive/transmit mode and track selection. It also has two rotary encoders for panning, two faders for volume, a motor controller, and an ADC for the fader readings. The schematic and picture shows how all hardware components were connected. We initially wanted to use the PiTFT as an onboard display for the GUI, but opted not to due to the limited number of GPIO pins. We instead used a computer monitor for the display.

There were three main “control” headers that all buttons, rotary encoders and faders connected to. The main one was the GPIO pin header, which connected to the RPi using the breakout cable from Lab. It provided all GPIO pins as well as a 5V source. All 11 buttons, shown in light blue, had one connection into a GPIO pin and one into ground. The KY-040 rotary encoders, shown in orange, were a little more complex, needing 2 GPIO pins for CLK and DATA, and also required a 5V source and ground (provided by the RPi4). The two faders, also called motorized potentiometers (10 kilo ohms), required the TB6612FNG 5V dual motor controller for the fader movement via PWM signals. We used an ADS1115 16-Bit analog to digital converter (ADC) for the potentiometer readings to manually control the volume, which required I2C connections (SDA and SCL) to the RPi4. We initially were using the PiTFT which uses I2C address 0x48, so we wired the VDD to the ADDR pin to use the 0x49 address for the ADC. We used two ADC inputs, A0 and A1 for the left and right faders respectively.

Software Design

Overview

The core goal of our software is for our RPi4 to act as a MIDI USB device and send and receive MIDI messages in (relatively) real time and update the board and GUI state accordingly.

Packages/dependencies

We used Python3 for our entire project. Some essential packages included Mido for sending and receiving MIDI messages, RPi GPIO for GPIO control, evdev and select for our rotary encoders, threading for concurrency, and PyGame for our GUI. The Ableton _Framework API and the Ableton Live object model were also key in DAW interfacing.

USB MIDI Device and the Device Tree Overlay

Device Tree Overlay was used to register our RPi4 as a USB MIDI device. We had to add “dtoverlay=dwc2” and “modules-load=dwc2,g_midi” in two separate lines to the /boot/config.txt file. This creates a device tree overlay for the dwc2 driver that has USB gadget mode, which allows the RPi4 to act as a USB device on startup. Then, we call “sudo modprode g_midi” in the terminal to load the g_midi kernel module so that it allows the RPi4 to act as a USB MIDI device. After configuring the device, we could see the device as “MIDI Gadget” when plugged into the computer. We then used “aconnect -l” in the terminal to get the MIDI port name, which was titled “f-midi”. This is the port that we must send and receive MIDI information from, so we used the Mido python library to create an inport and outport such that we could send and receive MIDI messages via the “f-midi” port. In Ableton, we ensure to use Link midi, as well as track and sync the input and outputs to the MIDI Gadget. This ensures we will receive and send MIDI information via our device.

MIDI Mapping

MIDI (Musical Instrument Digital Interface) is a protocol widely used for musical instruments and equipment that interfaces with software. MIDI possesses a wide range of message types, but we only used control change messages, which are common for controlling musical hardware. The control change message packet is 3 bytes, where the first byte is the status byte that dictates the message type and channel number (channels range from 0-15), the second dictates the note (0-127), and the third dictates the value (0-127).

Full MIDI Mapping of our Board

• Volume: CC 0 notes 0-7
• Pan: CC 0 notes 8-15
• Mute: CC 1 notes 0-7
• Solo: CC 1 notes 8-15
• Arm: CC 1 and notes 16-23
• Play: CC 1 note 24
• Stop: CC 1 note 25
• Record: CC 1 note 26

We mapped these accordingly in the MIDI_map.py Ableton control script. In the Mido library, these bytes are handled in hexadecimal.

Ableton _Framework, MIDI Control Scripts, Ableton configuration

The Ableton Live Object Model (LOM) is used together with the _Framework to retrieve track information. Although Ableton’s _Framework MIDI control script library is not officially documented, people have uncompiled and reverse engineered existing control surfaces’ pyc files (located in Ableton’s file contents) and made it open source. Hence, we used these MIDI control python scripts from Github user laidlaw42. These MIDI control scripts allowed us to receive MIDI messages when track and playback parameters were updated. Upon following the directions in the readme and placing the directory in /Applications/Ableton Live 11 Suite/App-Resources/MIDI Remote Scripts, These parameters were mapped to MIDI notes and channels specified in MIDI_map.py. We also edited the MIDI_Gadget.py file’s __init__ function to send 8 tracks’ states (volume, mute, solo, arm) information via MIDI to our RPi4 on project startup so that we could save the initial states and keep them updated. The code is provided in the appendix and is commented for reference. The MIDI control scripts allowed for 8 track control, so due to time constraints and poor documentation, we limited the scope of our project to just 8-track control.

Converting Hardware to MIDI Messages

For the faders, we needed to convert ADC readings to MIDI notes. Our ADC is 16 bits, but the max potentiometer value we could get with 5V connected was 26340. The ADC required us to download ADS1115 specific adafruit libraries to be able to get readings. We applied a conversion formula to map values from 0-26340 to a range of 0-127: max(0, round (127*adcValue/26340). RPi GPIO was used for sending PWM signals. For adjusting the fader position to a corresponding MIDI Value via the motor controller, we found that a PWM frequency of 150 Hz and a duty cycle of 52 worked the best for our faders. This required a lot of experimentation. We only moved the faders if there was a difference in current MIDI fader value and Ableton volume MIDI value of greater than 4 to reduce “unnecessary” movement and over corrections. Before moving the faders, we would check if the current fader value is larger or smaller than the newest received Ableton volume fader value and move the fader accordingly within a while loop. We created two helper functions per fader titled faderx_move_up/down(), where x is the fader number 1 or 2. We also set a timer of 0.5 seconds for this while loop to ensure that the fader does not get stuck or make too many movements in a given function call.

The KY-040 rotary encoders also required modifying the /boot/config.txt file. We added “dtoverlay=rotary-encoder,pin_a=X,pin_b=Y,relative_axis=1”, where X and Y are the SCL and SDA pins. We did this for both rotary encoders. We used evdev and select to be able to read events from multiple devices. We filtered out the device list such that only “rotary@e” and “rotary@a”, the names of our rotary encoders, were the devices being read, since not doing this would cause keyboard and mouse events to also be tracked. For each event, we added or subtracted to the MIDI value based on the direction it was turned (CC to increase the MIDI value, CW to decrease). We bounded the value from 0 to 127 (MIDI range).

We used RPi GPIO for buttons and callback functions. For button mapping, we used a button dictionary to easily map each button’s channel number (integer key) to its corresponding MIDI channel and note (tuple value). We had a button dictionary for the tracks and another button dictionary for the playback/record controls.

Main Loop

Our code runs on a while loop with threads in the background for receiving and transmitting MIDI messages, and updating the rotary encoders. Inside the while loop, we handled sending manually changed fader values, receiving Ableton volume fader values and moving the faders accordingly, and updating the GUI. For reading the two faders in separate threads, we used a lock to prevent ADC readings from the wrong channel.

Receiving MIDI Messages from Ableton and State Saving

This function, receive_midi() is run as a thread before the main code loop. It is composed of a while loop that calls Mido’s MIDI message polling function on our defined inport (to prevent blocking calls). We then filter out any messages that do not pertain to the control change channels that we are concerned with (0 and 1).

For any track parameter message, we store the MIDI values in a dictionary (we will refer to this as the MIDI dictionary) where the key is a tuple of (control change number, note), and the value is the corresponding MIDI value. This allows us to easily and continuously receive and save the state for every single parameter.

Sending Track Control MIDI Messages to Ableton

Sending MIDI messages is different for each piece of hardware. We used Mido for sending the messages through defined outport. The buttons’ callback function (on the rising edge) sends a MIDI message to the button’s corresponding CC and note (via the button dictionary) with a value of 127. Ableton’s MIDI control scripts state that buttons should only send MIDI values of 127. The faders use two functions inside of the main loop, update_pan() (for transmit mode) and receive_fader_x() (where x is the corresponding fader number from 1-2). Update_fader() is called when in transmit mode and checks the difference between the current fader values (using our MIDI conversion formula) and the saved MIDI values (via the MIDI dictionary) and sends the appropriate MIDI messages to Ableton. It only sends the value if the current fader position is different from the saved state MIDI value from the MIDI dictionary. The rotary encoders use a thread for update_pan() function (initialized before the main loop), which converts state changes into MIDI values (as described earlier) then updates and sends the MIDI message to the RPi4.

Receive Mode

We use a global variable titled “receiving” combined with a callback function for the receive/transmit button that flips the boolean value on every press to track the state of the mode. In receiving mode, the receive_fader_x() thread queries the MIDI dictionary to check if the volume has been changed on Ableton. If so, it adjusts the fader accordingly to match Ableton’s state.

Transmit Mode

In transmit mode, receive_fader_x() does not query for value changes and does not move the faders upon any change. In the main loop, the update_fader() function is called in transmit mode, and it sends the new MIDI values to Ableton if the fader has been adjusted (as stated in the “Sending Track Control MIDI Messages to Ableton” section).

Track Selection

The two currently controlled tracks are always adjacent in track order. For example, if tracks 1 and 2 are controlled, track 1 is assigned to the left set of controls on the board, and track 2 to the right set. Therefore, only pairs like 1 & 2, 2 & 3, 3 & 4, etc (up to 8), can be controlled simultaneously. We configured this using our “next” button’s callback function update_bank() which increments (and modulo 7 to bound the values from 0-7) a global “offset” variable. To prevent switching tracks and sending the previous track’s fader value, we set the global variable “receiving” to true before switching channels.

Note About MIDI Messages with 8 Tracks

In general, to send MIDI messages for track controls, we add an offset to the note byte to target the currently selected track. For the left track on the board, we add the offset, and for the right track, we add the offset plus one.

Playback Control MIDI Messages to Ableton

Playback/recording control buttons are mapped to the playback callback function which directly sends the value 127 to the corresponding MIDI notes using the playback button dictionary. We made this separate button callback due to the offset additions for the track control button.

MIDI Feedback Loops and Latency

An issue we had was ensuring that MIDI messages sent by Ableton do not override the intended state change that the user is currently trying to make. The receive/transmit mode was the work-around for the fader, since if a fader is being automated and it is manually touched, it is hard for our software to distinguish what to do without these two modes. The rotary encoders faced similar problems, where in update_pan(), if the knob was turned quickly, the updated MIDI messages that Ableton was sending would be saved in the receive_midi() function, overwriting the intended changes. We made a workaround by writing the state to the current value if no event occurred in that while loop iteration (shown in our update_pan() function), allowing for the MIDI messages to sequentially update. Finally, using time.sleep() with small time values was crucial in producing effective and natural state changes and reducing latency.

GUI/Updating the GUI

The GUI followed after developing the underlying MIDI and hardware logic for the board. We used PyGame for our GUI to display the status of all parameters on the board, roughly corresponding to their positions. This GUI is used to understand what tracks we are currently selecting, what mode (receiving and transmitting), and what track controls are selected or disabled. The GUI components access the MIDI dictionary to display the current track states, displaying red if enabled or white of disabled. It also uses the offset logic described in previous sections to get the current track’s corresponding information. The next and stop buttons have momentary color changes to red (~0.3 seconds) using threading.Timer(), as these buttons are momentary switch buttons. We used basic text, rectangles, and stock fonts for our GUI design.

Testing

The design section encapsulates our successful approach to the project. Within this timeframe were numerous development iterations.

Hardware Process and Testing

We first checked that all parts were working as expected. This was basic for buttons (using callbacks). For the rotary encoders, we ensured that the basic KY-040 code would return events when the knob was turned both directions. For the fader, we checked that the potentiometer was wired properly, with ADC values getting larger as the fader was pushed up (to correspond to larger volume level). We developed basic test functions to move the fader to a position that corresponded to a given MIDI/ADC value. After mounting all buttons, faders and encoders to the cardboard base and soldering wires to the relevant inputs, we performed continuity testing to ensure all connections were secure. Once we completed this, we could assign specific GPIO pins to each hardware element.

Software Process and Testing

The first goal of the software was to be able to control one track via MIDI messages without receiving MIDI messages. Then, we scaled it up to two tracks. We began building the GUI, where we had a similar development process. We initially were trying to track the states of the board based only on physical input. However, we then learned about the Ableton MIDI control scripts in week 4 of the project, so we had to go back to reimplement the MIDI receiving thread and update the board accordingly to achieve 8-track control.

We originally wanted to be able to send and receive fader values without having to use a transmit and receive button. However, the best way to do this would be to use capacitive touch sensor caps for the faders to detect a human finger pushing on the fader (in which the fader would allow the fader to be pushed), which was out of our budget. Therefore, we went with the transmit/receive button. We had plenty of issues with getting the motorized faders to work properly. At points, we used too high of duty cycles and frequencies, which lead to plenty of overshooting (moving beyond the fader value) and obscure behavior by the fader.

Another issue when scaling up to two tracks and threading was reading two different fader values at the same time. Because the faders are reading from the same ADC, a threading lock titled “adc_lock” in the code was needed to prevent reading the other fader’s value. This removed any obscure behavior from the motorized faders. Then, we scaled it to 8 tracks using the dictionary described in the software design section.then scaling it to 8, to adjust to the currently selected two tracks. Since we used global variables and dictionaries to hold the tracks’ states, updating the GUI required minimal additional code.

Testing in Ableton

In the earlier stages, we had a 1 track Ableton project to test functionality, followed by a 2 track project once we had implemented full board control. We initially tested that the GUI was updated without the need for Ableton as we were not using the MIDI control scripts at this point. When we redid the implementation using the MIDI control scripts, we reverted back to 1 track, and scaled to 2. We moved to 8 tracks to ensure that states were being updated accordingly on Ableton, the GUI, and the physical control surface (the faders). We introduced the GUI further into this reimplementation and adjusted state updates and element sizes as needed.

Results and Conclusions

Our initial plan of keeping track of the states based on the board's transmission was not successful, as it would not properly align with Ableton. This then required a full revamp of the project to actively listen for MIDI state changes from Ableton using the MIDI control surface scripts to then update the board’s state. Overall, despite its 8-track limitation, this project turned out as expected and provides the desired functionality. In the future, we look to expand our control surface to be able to handle more than 8 tracks via Ableton’s track banks. Time permitted, we could have implemented more features and modes that leverage the full extent of Ableton's MIDI control scripts. However, within the 5-week time frame, we learned a lot about numerous hardware components, protocols, frameworks, and control flows and created a well functioning and affordable control surface.

Future Work

We achieved the goals we had intially set out. That being said, with more time, it would be interesting to expand the control surface to include more tracks, as well as additional functionality with the piTFT.

Parts List

• 1 Raspberry Pi4B Provided by lab
• 11 Buttons Provided by lab
• 5 Cardboard Sheets Provided by lab
• 2 Breadboards Provided by lab
• 1 ADC Provided by lab
• 1 Motor Controller Provided by lab
• 2 Motorized Fader $24.99
• 2 Rotary Encoder Provided by lab

Total: $49.98

References

MIDI Forums
MIDI Linux Sunxi
Adc Data Sheet
Ky 040
Mido docs
Evdev docs
Live Object Model
Ableton Live MIDI Remote Scripts
Ableton _Framework Documentation
Adafruit Motorized Sliding Potentiometer Datasheet