Regression

Regression predicts numeric values (e.g., house prices, temperatures).

Cf. Classification which predicts discrete values.

Regression algorithms include:

• Linear regression, GLM
• SVR, GPR
• Ensemble methods
• Decision trees
• Neural networks

Example: House Prices

House Size (sq ft)	Num. of Bedrooms	Num. of Bathrooms	Age of House (years)	House Price ($)
1360	4	2	27	251240
1794	1	2	42	262602
1630	4	2	28	282277
1595	4	1	16	266877
2138	4	1	15	346992
2669	3	2	47	405283
966	2	2	44	143916
1738	1	1	3	278097
830	2	1	37	113612
1982	4	1	7	342283

We have 4 features/descriptors: house size ($x_1$), number of bedrooms ($x_2$), number of bathrooms ($x_3$), and age of house ($x_4$). The house price ($y$) is the output/the variable of interest/the dependent variable, and is what we want to predict based on the 4 features.

The idea in regression is that we want to approximate $y$. We write $\hat{y}$ to indicate an approximation of $y$. In this case, we want to find some approximation $\hat{y}={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3+{\theta }_4{x}_4$. If we can find the $\theta$ parameters, then we would have an estimation of the house price as a linear combination of the input features.

Let’s say we obtain the following linear regression equation for the estimated house price ($\hat{y}$):

$\hat{y}=-8775.58+147.12\times \text{House Size}+9660.37\times \text{Number of Bedrooms}+25691.99\times \text{Number of Bathrooms}-1285.01\times \text{Age of House}$

By looking at the coefficients, we can understand the direction and magnitude of the effect from each of the features we have. For example, if we increase the age of the house (and keep everything else fixed), the house price will decrease.

Ideally, we would scale the house size, noting that its values are much larger than the other values. If we don’t, then we would end up with a much smaller coefficient (as can be seen in the equation). Since we want to interpret the importance of the 4 features based on the $\theta$ coefficients, we should rescale all the features (using a method like normalisation) so they have a similar scale. This would allow for a fair comparison.