The unlikely entanglement of AI and Gregorian Plain Chant

The Challenge: Complexities in Cataloguing Medieval Manuscripts

Mediaeval music manuscripts are not only ancient and fragile but also incredibly complex. They contain unique notations, languages, and symbols that require specialised knowledge to interpret. Traditional cataloguing methods are time-consuming and often inconsistent, hindering the accessibility of these cultural artefacts. Scholars and musicologists face barriers such as:

• Fragmented Resources: Manuscripts are scattered across various institutions and private collections.
• Manual Processes: Cataloguing requires meticulous, manual effort to annotate and interpret each manuscript.
• Complex Notations and Annotations: These manuscripts contain intricate musical notations, lyrics, rubrics, marginalia, and other features that require detailed analysis.
• Need for Efficient Cataloguing: Traditional methods are time-consuming and labour-intensive, hindering the progress of research and preservation efforts.
• Data for AI Training: To automate and enhance the cataloguing process, a rich dataset is needed to train advanced AI models for Optical Music Recognition (OMR) and Music Information Retrieval (MIR).

Our Solution: The Repertorium Workspace

Spork led the development of the Repertorium Workspace, a cloud-based, collaborative platform designed specifically to address these challenges. Our goal was to create a tool that not only streamlines the manual cataloguing process but also generates high-quality data to train AI models developed by our consortium partners to automate and streamline the cataloguing process.

Here are the key features and benefits:

User-Centric Design

• Intuitive Interface: Developed with direct input from musicologists to ensure the platform meets the specific needs of its users.

Advanced Annotation Tools

• Polygonal Annotation: Our custom SVG drawing tools enable detailed delineation of complex manuscript features, critical for accurate interpretation.
• Precise Tagging: We built a custom tagging system which allows users to annotate musical notation (notes, lyrics, rubrics, marginalia etc) by adding tags to the polygons.
• W3C Annotation: We store both tags and polygonal information in W3C Annotation format. This adherence to international standards allows for seamless integration with other IIIF-based image viewers and facilitates the sharing of annotations across different platforms.
• GABC/NABC Integration: Musicologists describe the musical information in these manuscripts using a textual musical notation called NABC. A custom tool for parsing and displaying this notation was developed for Repertorium (Scrib.io) which we integrated into our application. This feature allows musicologists to add GABC/NABC notation to groups of annotations, directly linking visual elements to their musical representations. The textual notations are essential for training AI models, providing them with rich, structured data to learn from.

Integration with Existing Databases

• Linked Data Model: Seamlessly connects with established databases like the Digital Image Archive of Medieval Music (DIAMM). We wrote scripts which can import manuscripts from any IIIF presentation compliant archive, by parsing and converting to our own internal database model.
• Cantus Index: We created an integration with Cantus Index, which is a network of medieval music databases linked through a unique identifier called a “Cantus ID”. In the workspace, once musicologists have fully annotated a chant on a manuscript page, they can use this data to “search” the Cantus Index, allowing them to link that chant to an existing Cantus ID and pull in any additional metadata about that particular chant into our database.
• Data Consistency: Ensures information is standardised and prevents duplication, enhancing the reliability of research data.
• Enhanced Discoverability: Linking to external resources broadens the context and depth of available information for each manuscript.

IIIF Image Interoperability Framework

• IIIF Compliance: Utilises the International Image Interoperability Framework to access and display high-quality images from any compatible archive. IIIF enables us to use manuscripts and images from a plethora of global institutions, and allows us to circumvent copyright and ownership restrictions (as the images are streamed to our application using IIIF servers).
• Image Processing Tools: Includes features to enhance readability of IIIF images, such as zooming and panning and applying WebGL filters (such as invert, contrast etc.) critical for examining fine details.

Integration with AI Models

• Creating a Rich Dataset for AI Training:
  • The detailed annotations and tags collected through the workspace serve as high-quality training data for AI models.
  • Our consortium partners use this dataset to develop advanced OMR and MIR models aimed at automating the recognition and interpretation of medieval musical notations.
• AI-Powered Automation:
  • Integrated AI models into the workspace using Gradio, an interface that allows seamless interaction with machine learning models via APIs.
  • One AI model automates the creation of polygons and tags, identifying features on manuscript images with increasing accuracy.
  • Another AI model uses the images, polygons, and tags to automatically generate GABC/NABC notations, translating visual information into textual musical representations.
• Improving Efficiency Over Time:
  • While the AI models are still improving, their accuracy will increase as more data is annotated and fed back into the training process.
  • This creates a positive feedback loop where the AI models become more proficient, further speeding up the cataloguing process.
  • The ultimate goal is to automate the majority of the annotation process, allowing musicologists to focus on higher-level analysis and interpretation.

Impact on the Project

• Accelerated Cataloguing Process:
  • By combining advanced annotation tools with AI integration, we've significantly reduced the time required to catalogue manuscripts.
  • The platform has enabled the systematic processing of over 350,000 digital images.
• Enhanced Data Quality for AI Training:
  • The rich, detailed annotations provide high-quality data that improve the performance of AI models.
  • As the AI models become more accurate, they further reduce manual workloads and increase efficiency.
• Advancement of AI in Musicology:
  • Our contributions support the development of AI technologies in the field of musicology, opening new avenues for research and discovery.
• Global Collaboration:
  • The platform fosters a collaborative environment where researchers worldwide can contribute to and benefit from shared data and tools.
• Preservation and Accessibility:
  • By digitising and annotating manuscripts, we help preserve fragile historical documents and make them accessible to a broader audience.