by Li Yang Ku (Gooly)
(link to the rap machine if you prefer to try it out first)
In my previous post, I gave a short tutorial on how to use the Google AI platform for small garage projects. In this post, I am going to follow up and talk about how I built (or more like an attempt to build) my holiday project, a machine that completes your rap lyrics using the “Transformer” neural network.
Transformer is a neural network model introduced by Google Brain for mostly language related tasks. What is interesting about this architecture is that instead of taking one word at a time, it takes in the whole input sentence at once and learns the relationship between each word. This allows transformers to learn useful relationships such as what a pronoun refers to in a sentence. In the original paper “Attention is All You Need”, this ability to understand relations between words are referred to as attention since it has the ability to focus more on certain pairs of words. I will not go into the details of Transformer since quite a few people already explained it in great length in their blogs (such as this blog and this blog.) My rationale was that Transformer’s ability to learn relationships between words in rap sentences should allow the network to learn what words rhyme well together or have the right flow.
I collected rap lyrics from 14 artists with a total of around 180000 lines of rap. These rap lyrics are further filtered to around 50000 lines of lyrics that I considered to “rhyme” with another line. The first line is the input and the second line is the output. Pairs of these sentences are then split into training and evaluation (90:10 split.) The Transformer architecture I used is mostly based on this Google Cloud Platform tutorial on generating poetry. After a little bit of hyper tuning, I ended up with a Transformer architecture of 2 hidden layers, 4 heads (number of attentions), and 512 hidden layer size. This model is then trained for 75000 iterations. The total cost of experimenting with different models and trial and error cost me a bit more then a hundred bucks (note that you are given $300 to start).
The final result was, well, entertaining but with limited success. Occasionally it will rhyme if your sentence ends with a common word but most of the time its just a weirdo spitting out random sentences with a semi-rapper attitude. The following are a few example results:
Input: This thing I made raps like a machine
Output: It’s all fair in love with me
Input: It generates raps that run deep
Output: That’s what I mean
Input: No rap is free, this thing cost me a fee
Output: I got a whole lotta guap on me
Input: Consider this as a new year gift that raps
Output: Chillin’ on some wildin’ out of that
Input: You might say my input rap is bad
Output: I was gone but now I m back
Input: If you don’t like your output rap
Output: Where the fuck am I going back
You can try out the rap machine here yourself. Thank you all for reading this blog and wish you all an entertaining 2020!
Leave a comment