{ "cells": [ { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "# Linear Least squares\n", "\n", "## Prediction of house sale prices\n", "\n", "Consider the following situation:\n", "We would like to sell our house (area $1000 m^2$), but we do not yet know what the market value of the house is or what price we can ask in relation to the current market situation and our neighborhood. To solve this problem we have thought of the following.\n", "1. from the sales of some surrounding houses in our neighborhood, we have created a dataset of tuples $(x,y)$, where $x_i$ represents the size of the living area in square meters of the house $i$ and $y_i$ represents the corresponding achieved sales price, for $1 \\leq i \\leq n$, $n\\in \\mathbb{N}$.\n", "2. we assume that the house price is linearly related to the living area, i.e., approx.\n", "$y_i = f(x_i) = \\Theta_1 + \\Theta_2 \\cdot x_i$\n", "for values $(\\Theta_1, \\Theta_2)^T \\in \\mathbb{R}^2$ unknown to us so far.\n", "\n", "We now try to determine the values $\\Theta_1, \\Theta_2$ based on the data we have in order to predict our selling price. This corresponds to the simplest form of a supervised learning algorithm.\n", "\n", "First, let's consider our dataset (this is from https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data and corresponds to actual data, for simplicity's sake we will only load the two cells price and area here, even though the data gives much more). We now want to load the data set:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" }, "scrolled": true }, "outputs": [], "source": [ "import numpy as np\n", "dataset_name = \"Houses_train.csv\"\n", "data = np.genfromtxt(dataset_name, delimiter=\",\")\n", "\n", "# calculate the area in square meter insted of square foot\n", "x_area = 1/10.764 * data[1:,4]\n", "\n", "# Euros instead of dollars\n", "y_price = 0.93 * data[1:,80]\n", "\n", "# clear four datapoints with way to high spreading\n", "for i in range(0, 4):\n", " for i in range(0, np.size(x_area)):\n", " if x_area[i] >= 10000:\n", " x_area = np.delete(x_area, i)\n", " y_price = np.delete(y_price, i)\n", " break\n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "For the sake of clarity, we now present this graphically:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "fig=plt.figure(figsize=(12, 12))\n", "plt.xlabel(\"area $[m^2]$\")\n", "plt.ylabel(\"Selling price $[€]$\")\n", "plt.title(\"Dataset sales\")\n", "# plot the data set\n", "plt.plot(x_area, y_price, \"b+\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Thus, using our approach in (2), we have the following minimization problem:\n", "\n", "\\begin{align*}\n", "\\min_{(\\Theta_1, \\Theta_2)^T \\in \\mathbb{R}^2} \\Vert A\\Theta - y \\Vert_2\n", "\\end{align*}\n", "Define the matrix $A$ by\n", "\\begin{align*}\n", "A := \\begin{pmatrix}\n", "1 & x_1\\\\\n", "1 & x_2\\\\\n", "\\vdots & \\vdots\\\\\n", "1 & x_n\\\\\n", "\\end{pmatrix}\n", "\\end{align*}\n", "and the vector $y$ as $y := (y_1, \\ldots , y_n)^T$ for the values from our data set. We now implement this in the code:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "\n", "# calculate size of the dataset\n", "n = np.size(x_area)\n", "# create column vectors of ones\n", "ones = np.ones(n)\n", "ones_column = ones[:, np.newaxis]\n", "# create column vectors of the dataset\n", "x_area_column = x_area[:, np.newaxis]\n", "# Define Matrix A\n", "A = np.hstack((ones_column, x_area_column))" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "We now know that a vector $(\\Theta_1, \\Theta_2)^T \\in \\mathbb{R}^2$ solves the linear compensation problem exactly when $(\\Theta_1, \\Theta_2)^T$ is a solution to the normal equation\n", "\\begin{align*}\n", "A^T A \\Theta = A^T y\n", "\\end{align*}\n", "We now want to solve these by means of QR-decomposition, thus $A = QR$ and thus it is sufficient to solve the system $R\\Theta =B$ for $R = Q^TA$ and $B = Q^T y$ by means of back substitution.\n", "This can be done by using oppy as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from oppy.leastSquares import linear_least_square\n", "from oppy.options import Options\n", "\n", "# define option, using QR\n", "opt = Options(disp=False, fargs='QR')\n", "# solve the least squares problem\n", "theta_sol = linear_least_square(A, y_price, opt)\n", "print(theta_sol)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "With ```theta_sol``` we now have a vector which represents the linear relationship in the data. With the function\n", "\\begin{align*}\n", "f: \\mathbb{R} \\to \\mathbb{R}, x \\mapsto \\Theta_1 + \\Theta_2 \\cdot x\n", "\\end{align*}\n", "we can now represent the straight line once graphically." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "print(\"The straight line has the form: f(x)={1:5.2f}x + {0:5.2f}\".format(theta_sol[0], theta_sol[1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The function $f$ is thus a model to predict, based on our data, the purchase price of a new, previously unknown house." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "x = np.linspace(0, 7500, 100000)\n", "f_predict = lambda x: theta_sol[0] + theta_sol[1]*x\n", "y = f_predict(x)\n", "\n", "fig=plt.figure(figsize=(12, 12))\n", "plt.xlabel(\"area $[m^2]$\")\n", "plt.ylabel(\"Selling price $[€]$\")\n", "plt.title(\"Dataset sales\")\n", "plt.plot(x_area, y_price, \"b+\", label=\"dataset\")\n", "plt.plot(x,y, \"r-\", label=\"linear model\")\n", "plt.legend(loc=\"upper right\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Back to our initial problem: We have a house with $1000 m^2$ area, whose purchase price we want to determine. Based on our model, we can expect the following sales price:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": { "is_executing": true } }, "outputs": [], "source": [ "print(\"Expected selling price: {:10.2f}€\".format(f_predict(1000)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This gives us a simple model that gives us the expected purchase price.\n", "\n", "Problems:\n", "1. the relationship is usually not linear.\n", "2. the target value often depends on more than one attribute (the data set has e.g. more than 80 attributes)\n", "3. the approach of a linear regression is very susceptible to individually widely scattered data points (here e.g. houses that were sold far below or above their actual price)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.0" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }