New AI based FluxMusic helps transform text description into music by leveraging advanced machine learning techniques to bridge the gap between language and music composition. It analyses the text with the help of AI generating music that best suits the textual description.
Available on GitHub, the system employs multiple pre-trained text processing tools to capture subtle details within the description. This software functions by using AI to analyse meaning and emotion of given description. Then it applies a series of transformations to a blurry, noisy version of a song, refining it iteratively based on the text information. This process involves breaking down the music into patches and denoising them using techniques inspired by diffusion models.
The architecture of FluxMusic integrates two models, CLAP-L and T5-XXL, as text encoders for the extraction of conditioned caption features. This dual approach enables the model to capture both coarse and fine-grained textual information effectively. By using multi-modal input …