{"id":25188,"date":"2016-09-01T19:30:32","date_gmt":"2016-09-02T00:30:32","guid":{"rendered":"http:\/\/www.dannyadam.com\/blog\/?p=25188"},"modified":"2020-02-18T14:01:30","modified_gmt":"2020-02-18T19:01:30","slug":"matrix-factorization-with-theano","status":"publish","type":"post","link":"https:\/\/www.dannyadam.com\/blog\/2016\/09\/matrix-factorization-with-theano\/","title":{"rendered":"Matrix Factorization with Theano"},"content":{"rendered":"

Matrix factorization algorithms factorize a matrix D<\/em> into two matrices P<\/em> and Q<\/em>, such that D<\/em>\u00a0\u2248 PQ<\/em>. By limiting the dimensionality of P<\/em> and Q<\/em>, PQ<\/em> provides a low-rank approximation<\/a> of D<\/em>. While singular value decomposition<\/a> (SVD) can also be used for this same task, the matrix factorization algorithms considered in this post accommodate\u00a0missing data in matrix D<\/em>, unlike SVD.<\/p>\n

For an overview of matrix factorization, I recommend\u00a0Albert Au Yeung’s tutorial<\/a>. That post describes matrix factorization, motivates the problem with\u00a0a ratings prediction task, derives the gradients used by\u00a0stochastic gradient descent, and implements the algorithm in Python.<\/p>\n

For exploratory work, it would be great if the algorithm could be implemented\u00a0in such a way that the gradients could be automatically derived. With such an approach, gradients would not have to be\u00a0re-derived\u00a0when e.g., a change is made to the loss function (either the error term\u00a0and\/or the regularization term). In general, automatically derived gradients for machine learning problems facilitate increased exploration of ideas by removing a time-consuming step.<\/p>\n

Theano<\/a> is a Python library that allows users to specify their problem symbolically using NumPy-based syntax. The expressions are compiled to run efficiently on actual data. Theano’s webpage provides documentation and a tutorial<\/a>.<\/p>\n

The following code includes\u00a0a Theano-based implementation of matrix factorization using batch gradient descent. The parameters are similar to those in the quuxlabs<\/a>\u00a0matrix factorization implementation. D<\/em> is a second-order masked numpy.ndarray (e.g., a ratings matrix, where the mask indicates missing ratings), and P<\/em> and Q<\/em>\u00a0are the\u00a0initial matrix factors. The elements of P<\/em> and Q<\/em> are the parameters of the model, which are\u00a0initialized by the function’s caller. The rank of the factorization is specified by the dimensions\u00a0of P<\/em> and Q<\/em>. For a rank-k<\/em> factorization,\u00a0P<\/em> must be \"m and Q<\/em> must be \"k (where\u00a0D<\/em> is an \"m matrix). Additional parameters specify the number of iterations, the learning rate, and the regularization importance.<\/p>\n

The code\u00a0doesn’t contain any derived gradients. It specifies the loss function and the parameters that the loss function will be minimized with respect to. Theano figures out the rest!<\/p>\n