Training a Rap Machine

by Li Yang Ku (Gooly)

(link to the rap machine if you prefer to try it out first)

In my previous post, I gave a short tutorial on how to use the Google AI platform for small garage projects. In this post, I am going to follow up and talk about how I built (or more like an attempt to build) my holiday project, a machine that completes your rap lyrics using the “Transformer” neural network.

Transformer is a neural network model introduced by Google Brain for mostly language related tasks. What is interesting about this architecture is that instead of taking one word at a time, it takes in the whole input sentence at once and learns the relationship between each word. This allows transformers to learn useful relationships such as what a pronoun refers to in a sentence. In the original paper “Attention is All You Need”, this ability to understand relations between words are referred to as attention since it has the ability to focus more on certain pairs of words. I will not go into the details of Transformer since quite a few people already explained it in great length in their blogs (such as this blog and this blog.) My rationale was that Transformer’s ability to learn relationships between words in rap sentences should allow the network to learn what words rhyme well together or have the right flow.

I collected rap lyrics from 14 artists with a total of around 180000 lines of rap. These rap lyrics are further filtered to around 50000 lines of lyrics that I considered to “rhyme” with another line. The first line is the input and the second line is the output. Pairs of these sentences are then split into training and evaluation (90:10 split.) The Transformer architecture I used is mostly based on this Google Cloud Platform tutorial on generating poetry. After a little bit of hyper tuning, I ended up with a Transformer architecture of 2 hidden layers, 4 heads (number of attentions), and 512 hidden layer size. This model is then trained for 75000 iterations. The total cost of experimenting with different models and trial and error cost me a bit more then a hundred bucks (note that you are given $300 to start).

The final result was, well, entertaining but with limited success. Occasionally it will rhyme if your sentence ends with a common word but most of the time its just a weirdo spitting out random sentences with a semi-rapper attitude. The following are a few example results:

Input: This thing I made raps like a machine

Output: It’s all fair in love with me

Input: It generates raps that run deep

Output: That’s what I mean

Input: No rap is free, this thing cost me a fee

Output: I got a whole lotta guap on me

Input: Consider this as a new year gift that raps

Output: Chillin’ on some wildin’ out of that

Input: You might say my input rap is bad

Output: I was gone but now I m back

Input: If you don’t like your output rap

Output: Where the fuck am I going back

You can try out the rap machine here yourself. Thank you all for reading this blog and wish you all an entertaining 2020!

Tool Tutorial: Google AI Platform for Hobbyist

by Li Yang Ku (Gooly)

In this post I am going to talk about the Google AI platform (previously called Google ML engine) and how to use it if deep learning is just your after work hobby. I will provide links to other tutorials and details at the end so that you can try it out, but the purpose of this post is to give you a big picture of how it works without having to read through all the marketing phrases targeting company decision makers.

Google AI platform is part of the Google cloud and provides computing power for training and deploying deep networks. So what’s the difference between this platform and any other cloud computing services such as AWS (Amazon Web Services)? Google AI platform is specialized for deep learning and is suppose to simplify the process. If you are using TensorFlow (also developed by Google) with a pretty standard neural network architecture, it should be a breeze to train and deploy your model for online applications. There is no need to set up servers, all you need is a few lines of gcloud commands and your model will be trained and deployed in the cloud. (You also get a $300 dollar first year credit for signing up on Google Cloud Platform, which is quite a lot for home projects.) Note that Google AI platform is not the only shop in town, take a look at Microsoft’s Azure AI if you like to shop around.

So how does it work? First of all, there are four ways to communicate with Google AI platform. You can do it 1) locally: where you have all the code on your computer and communications are made through commands directly, 2) on Google Colab: Colab is another Google project that is basically a Jupyter notebook on the cloud which you can share with others, 3) on the AI platform notebook: which is similar to Colab but have more direct access to the platform and more powerful machines, and 4) on any other cloud server or jupyter notebook like webservice such as FloydHub. The main difference between using Colab versus AI platform notebook is pricing. Colab is free (even with GPU access), but has limitations such as up to 12 hours of run time and shuts down after 90 minutes of idle time. It provides you with about 12GB RAM and 50GB disk space (although the disk is half full when started due to preinstalled packages). After disconnected, you can still reconnect with whatever you wrote in the notebook, but you will lost whatever is in the RAM and disk. For a home project, Colab is probably sufficient, the disk space is not a limitation since we can store training data in google storage. (Note that it is also possible to connect Google drive in Colab so that you don’t need to start from scratch every time.) On the other hand, AI platform notebook could be pricey if you want to keep it running (0.137 / hour and 99.89 / month for a non-gpu machine).

Before we move on, we also have to understand the differences between computation and storage on the Google AI platform. Unlike personal computers where disk space and computation are tightly integrated, they are separated on the cloud. There are machines that are responsible for computation and machines that are responsible for storage. Here, Google AI platform is responsible for the computation while the Google Cloud Storage takes care of the stored data and code. Therefore, before we start using the platform we will need to first create a storage space called bucket. This can be easily done through a one line command once you created a Google Cloud account.

If you are using Colab, you will also need to have the code for training your neural network downloaded to your Colab virtual machine. One common work flow would be to use software version control services such as Github for your code and just clone the files to Colab every time you start. It makes more sense to use Colab if you are collaborating with others or want to share how you train your model, otherwise doing everything locally might be simpler.

So the whole training process looks like this:

1. Create a Google Cloud Project.
2. Create a bucket where the Google AI platform can perform computations on.
3. With a single command, upload your code to the bucket and request the AI platform to perform training.
4. Can also perform hyper parameter tuning if needed.
5. If you want the trained model locally, you can simply download it from the bucket through a user interface or command.

A trained model is not very useful if not used. Google AI platform provides an easy way to deploy your model as a service in the cloud. Before continuing, we should clarify some Google terminology. At Google AI platform, a “model” means an interface that solves certain tasks and a trained model is named  a “version” of this “model” (reference). In the following, quotation marks will be put around Google specific terminologies to avoid confusion.

The deployment and prediction process is then the following:

1. Create a “model” at AI platform.
2. Create a “version” of the “model” by providing the trained model stored in the bucket.
3. Make predictions through one of the following approaches:
   • gcloud commands
   • Python interface
   • Java interface
   • REST API
     (the first three methods are just easier ways to generate a REST request)

And that’s all you need to grant your home made web application access to scalable deep learning prediction capability. You can run this whole process I described above through this official tutorial in Colab and more descriptions of this tutorial can be found here. I will be posting follow up posts on building specific applications on Google AI platform, so stay tuned if you are interested.

References:

• Introduction to AI platform: https://cloud.google.com/ml-engine/docs/technical-overview
• Buckets: https://cloud.google.com/storage/quotas
• Online-Predictions: https://cloud.google.com/ml-engine/docs/online-predict
• This Coursera course should contain a lot more details.

Talk the Talk: Optimization’s Untold Gift to Learning

by Li Yang Ku (Gooly)

In this post I am going to talk about a fascinating talk by Nati Srebro at ICML this June. Srebro have given similar talks at many places but I think he really nailed it this time. This talk is interesting not only because he provided a different view of the role of optimization in deep learning but also because he clearly explained why many researcher’s argument on the reason that deep learning works doesn’t make sense.

Srebro first look into what we know about deep learning (typical feed forward network) based on three questions. The first question is regarding the capacity of the network. How many samples do we need to learn certain network architecture? The short answer is that it should be proportional to the number of parameters in the network, which is the total number of edges. The second question is about the expressiveness of the network. What can we express with certain model class? What type of questions can we learn? Since a two layer neural network is a universal approximator, it can learn any continuous function, this is however not a very useful information since it may require an exponentially large network and exponential amount of samples to learn. So the more interesting question is what can we express with a reasonable sized network? Many recent research more or less focuses on this question. However, Srebro argues that since there is another theory that says any function that can be executed within a reasonable amount of time can be captured by a network of reasonable size (please comment below if you know what theory this is), all problems that we expect to be solvable can be expressed by a reasonable sized network.

The third question is about computation. How hard is it to find optimal parameters? The bad news is that finding the weights for even tiny networks is NP-Hard. Theories (link1 link2) show that even if the training data can be perfectly expressed by a small neural network there are no polynomial time algorithm to find such set of weights. This means that neural network’s expressiveness described in question 2 doesn’t really do much good since we aren’t capable of finding the optimal solution. But we all know that in reality neural network works pretty well, it seems that there are some magical property that allows us to learn neural networks. Srebro emphasizes that we still don’t know what is the magical property that makes neural networks learnable, but we do know it is not because we can represent the data well with the network. If you ask vision folks why neural networks work, they might say something like the lower layers of the network matches low level visual features and the higher layers match higher level visual features. However, this answer is about the expressiveness of the network described in question 2 which is not sufficient for explaining why neural networks work and provides zero evidence since we already know neural networks have the power to express any problem.

Srebro then talked about the observed behavior that neural networks usually don’t overfit to the training data. This is an unexpected property quite similar to the behavior of Adaboost, which was invented in 1997 and quite popular in the 2000s. It was only after the invention that people discovered that the reason Adaboost doesn’t overfit is because it is implicitly minimizing the L-1 norm that limits the complexity. So the question Srebro pointed out was whether the gradient decent algorithm for learning neural networks are also implicitly minimizing certain complexity measure that would be beneficial in reaching a solution that would generalize. Given a set of training data, a neural network can have multiple optimal solutions that are global minima (zero training error). However, some of these global minima perform better than the others on the test data. Srebro argues that the optimization algorithm might be doing something else other than just minimizing the training error. Therefore, by changing the optimization algorithm we might observe a difference in how well can a neural network generalize to test data, and this is exactly what Srebro’s group discovered. In one experiment they showed that even though using Adam optimization achieves lower training error then stochastic gradient decent, it actually performs worse on the test data. What this means is that we might not be putting enough emphasize on optimization in the deep learning community where a typical paper looks like the following:

The contributions are on the model and loss function, while the optimization is just a brief mention. So the main point Srebro is trying to convey is that different optimization algorithms would lead to different inductive biases, and different inductive biases would lead to different generalization properties. “We need to understand optimization algorithm not just as reaching some global optimum, but as reaching a specific optimum.”

Srebro further talked about a few more works based on these observations. If you are interested by now, you should probably watch the whole video (You would need to fast forward a bit to start.) I am however going to put in a little bit of my own thoughts here. Srebro emphasizes the importance of optimization a lot in this talk and said the deep models we use now can basically express any problem we have, therefore the model is not what makes deep learning work. However, we also know that the model does matter based on claims of many papers that invented new model architectures. So how could both of these claims be true? We have to remember that the model architecture is also part of the optimization process that shapes the geometry which the optimization algorithm is optimizing on. Hence, if the nerual network model provides a landscape that allows the optimization algorithm to reach a desired minimum more easily, it will also generalize better to the test data. In other words, the model and the optimization algorithm have to work together.