Painting Music – using Explainable AI for music composition

About the Project

These projects are open to students worldwide, but have no funding attached. Therefore, the successful applicant will be expected to fund tuition fees at the relevant level (home or international) and any applicable additional research costs. Please consider this before applying.

Painting Music is a collaborative project with a local artist Kate Steenhauer. The project looks to convert Kate’s painted brushstrokes into music, using AI as the digital composer. The tagline for the project can be thought of as: “Is AI good or bad?”, and the music created can provoke the audience into one feeling or another depending on their own bias and point of view. The project has already resulted in a publication [1], has been given a spotlight by Mathworks through the innovative use of their software program Matlab [2], and has been the focus of a film and numerous art installations across Scotland. The project has also received funding for the development of an app that allows someone to draw a picture on a touchscreen and for the Painting Music AI composition process to then be applied for music generation.

There are a number of steps in the process before the music composition can begin, including: capturing an image of the canvas; undertaking image processing to identify the newly painted elements; comparison of the newly painted element against previously painted elements in the image; and selection of which attributes in the analysis are important for the music composition. The Painting Music process permits a mapping of visual characteristics to musical attributes – e.g. white space on the canvas could be linked to silence in the music, or more complicated object could be linked to higher frequencies.

Following this process, an AI engine is developed that undertakes musical composition. For current AI methods, this is a difficult task, and it exposes the limitations of the current AI methodologies in the areas of explainability, and the ability for an AI to be truly creative. For example, AI cannot learn by itself what jazz music is and be able to explain how this is different from blues, or pop, or classical music. This PhD will therefore explore how AI techniques can be modified to better fit these requirements, and to enhance the music composition process. This again emphasises the limitation in the current process – to genuinely compose, the AI should understand musical composition, and to understand different styles of music.

The outputs of this PhD can therefore have profound applications well beyond the scope of this project and could lead to important developments in the search for truly Explainable AI approaches.

Decisions will be based on academic merit. The successful applicant should have, or expect to obtain, a UK Honours Degree at 2.1 (or equivalent) in Computing Science, Engineering or related disciplines. A degree in Music may also be considered if the applicant can demonstrate numerate ability or experience in AI.

Application Procedure:

Formal applications can be completed online: https://www.abdn.ac.uk/pgap/login.php.

You should apply for PhD in Engineering to ensure your application is passed to the correct team for processing.

Please clearly note the name of the lead supervisor and project titleon the application form. If you do not include these details, it may not be considered for the studentship.

Your application must include: A personal statement, an up-to-date copy of your academic CV, and clear copies of your educational certificates and transcripts.

Please note: you do not need to provide a research proposal with this application.

Informal enquiries can be made by contacting Dr A Starkey at a.starkey@abdn.ac.uk. If you require any additional assistance in submitting your application or have any queries about the application process, please don't hesitate to contact us at researchadmissions@abdn.ac.uk

Funding Notes

This is a self-funding project open to students worldwide. Our typical start dates for this programme are February or October.

Fees for this programme can be found here Finance and Funding | Study Here | The University of Aberdeen

References

[1] Starkey, A., Steenhauer, K., Caven, J.. Painting Music: Using artificial intelligence to create music from live painted drawings. Drawing: Research, Theory, Practice, vol. 5, no. 2, pp. 209-224
[2] Pingel, J., Starkey, A., Caven, J.. Painting Music with Artificial Intelligence. https://blogs.mathworks.com/deep-learning/?p=6672&draftsforfriends=FY0CN4ngY0XhFGAgoTL9Nm0FWk0hrXkD