Deep learning: past, present and future


Deep learning is currently a subject of immense interest in many machine learning and AI applications. Its impressive performance in several real applications has lead to great expectations in other areas as well, and chemometry is no exception. In this talk, I will take a look at the roots of deep learning, its present state and make some predictions for its future. 


Ole Christian Lingjærde earned his PhD in applied mathematics at the Department of Informatics and Department of Mathematics at the University of Oslo (Norway). After a postdoc at the Department of Biology at University of Oslo, he became an Associate Professor and later a full Professor at the Department of Informatics at University of Oslo. His current research focuses on modeling of tumor evolution, molecular stratification in cancer, and development of cancer biomarkers for clinical purposes. Lingjærde is the author of ~100 publications in international peer-reviewed journals. E-mail: ole@mail.uio.no.

What is your relation to chemometrics?

My first encounter with chemometrics was probably around 1989 when I finished my master’s thesis on neural networks. The networks typically had a large number of parameters, and overfitting to the training data was a huge problem. Various strategies were put forth to find “generalizable” solutions, and one of the more popular one was called “early stopping”. It typically involved maximising a likelihood function by some form of gradient optimization and then terminating the algorithm before convergence was achieved. I wondered if similar methods had been used in other fields, and came across works by Rolf Manne, Inge Helland and others establishing that PLS could be seen as a quadratic optimization problem in a Krylov subspace. It seemed to imply that PLS was essentially an “early stopping” adaptation of conjugate gradient descent optimization in a linear model. I found this link intriguing and potentially helpful to understand why early stopping works in general, even in nonlinear models such as neural networks. This got me interested in the PLS literature and PLS applications, and I think this is the first time I came across the word ‘chemometrics’. A few years later, during my PhD, a course in chemometrics (it was probably called something else) was given by Kim Esbensen at the University of Oslo and it introduced me to the book Multivariate Calibration by Harald Martens and Tormod Næs. I also came across several people in the university environment in those years showing interest in chemometrics methods or applications of such methods, including Ulf Indahl, Lars Snipen, Narinder Singh Sahni and Tobias Dahl. In 1993, Ildiko Frank and Jerry Friedman published a very interesting paper on shrinkage properties of PLS, and this paper and the discussion in its aftermath made a lasting impression on me. A few years later, my colleague Nils Christophersen and I established mathematical proofs of many of the properties of PLS shown by simulations in Frank and Friedman’s paper – this was first presented at the weekly Dept of Statistics seminar at Stanford University in 1997, but it was not published until late 2000 at about the same time that another paper came out with similar analyses. I have sporadically encountered chemometrics methodology and applications in my later career, and I never lost my interest in the topic, but it is fair to say that I am not updated on all the new, exciting developments in the field.


What are your favourite methods of data analysis, and why?

I am strongly in favor of visualisation as a tool for data analysis. Finding the right way to express the data visually allows efficient use of our own in-built pattern recognition system. One of my favorite books of all times on this topic is Exploratory Data Analysis (EDA) by John Tukey. It advocates strongly for visual methods to explore data sets and to formulate hypotheses. In my current field of research – modeling and prediction in cancer biology – visual data exploration has been and remains a very important topic. The repertoire of methods available for data visualisation has expanded enormously since the birth of EDA, however. Many current methods perform heavy processing on the data to obtain a good visual representation of the information therein and sometimes these methods are not portrayed as visualisation tools at all, although I think many of them are just that. I would also like to mention the work by Edward Tufte on visual presentation of information – this is more focused on efficient and faithful visual presentations of information than the exploratory analysis per se, but the two are closely interlinked in my mind. I should mention that I of course also use other methods of data analysis in my work. A common approach is then to first establish some model for the system at hand and then look for algorithms allowing the model to be fitted to the data. Sometimes this leads us to methods for fitting linear models with many variables (such as PCR and PLSR) and sometimes it leads us in entirely different directions, for example toward prediction with SVMs, random forests, or deep neural networks.


How do you see the future of chemometrics?

I am probably not in the best position to have any strong opinion about the future of chemometrics. At the time when I had most contact with the field, which is a long time ago, I was often puzzled by what I perceived as a big divide between “mainstream statistics” and chemometrics. For those having doubts about this statement, I would recommend looking at the discussion in the aftermath of the Frank & Friedman paper in 1993. Clearly, some very clever people were working in chemometrics and they published good statistics papers in journals that were rarely read by people outside their scientific community. Back then (I would not know if it is still true) I got the impression that many statisticans found chemometrics papers hard to read, and to be too much focused on certain methods such as PLS and variants of PLS. PLS was considered to be a quite obscure method among many statisticians at the time – it was very different from the situation today. At any rate, I often thought back then that chemometrics and statistics should merge closer together, to allow more efficient information flow both ways. Probably, this has already happened! In which case I think chemometrics would have a bright future.


What do you look forward to at SSC16?

I look forward to get a thorough update on new developments in chemometrics, to meet old colleagues and become acquainted with new ones!