Bangda Sun

Bias and variance of Models.

A classification method with straightforward geometric intuition.

Bagging and Random Forest are one type of ensemble algorithms, they combine the weak learners (decision trees) by averaging or voting their predictions. Bagging is based on bootstrap, resamples data with replacement as new training data for individual trees. Random Forest is based on Bagging and feature subsample at each tree node split.

Method used to calculate the multivariate joint distribution.

There are multiple types of decision tree models for classification problem: CART, ID3, C4.5. The differences include learning algorithm (the way of splitting) and whether it can handle continuous features. In summary, CART can be applied to more general use cases.

Regression tree is one type of tree model, it applies to regression tasks, i.e. the regression version of CART (Classification And Regression Tree). It is trained by recursively splitting the sample and feature space, equivalent to fit the data with step functions, therefore it has no assumptions on data. It is also used as weak learner in Random Forest Regressor and GBDT (both classification and regression).

Linear Regression (LR) in practice includes regression and classification (apply sigmoid), in statistics it is purely for regression. With strong mathematical assumptions on data, linear regression is still a powerful tool and can be applied in every regression task.

Logistic Regression applies to almost all classification tasks in real world, as it can consume either discrete and continuous features, it can handle millions and billions of features, it can be trained real quick and it has strong interpretations.

Comments from a kaggle novice: my experiences and thoughts after two simple “Get-Started” competitions.

Choice of evaluation metrics in classification problem.