Blair branch

Blair's Work/1 - EDA_test.ipynb

@@ -0,0 +1,246 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# EDA"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "\n",
+    "- The os module has a perfect method to list files in a directory.\n",
+    "- Pandas json normalize could work here but is not necessary to convert the JSON data to a dataframe.\n",
+    "- You may need a nested for-loop to access each sale!\n",
+    "- We've put a lot of time into creating the structure of this repository, and it's a good example for future projects.  In the file functions_variables.py, there is an example function that you can import and use.  If you have any variables, functions or classes that you want to make, they can be put in the functions_variables.py file and imported into a notebook.  Note that only .py files can be imported into a notebook. If you want to import everything from a .py file, you can use the following:\n",
+    "```python\n",
+    "from functions_variables import *\n",
+    "```\n",
+    "If you just import functions_variables, then each object from the file will need to be prepended with \"functions_variables\"\\\n",
+    "Using this .py file will keep your notebooks very organized and make it easier to reuse code between notebooks."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# (this is not an exhaustive list of libraries)\n",
+    "import pandas as pd\n",
+    "import numpy as np\n",
+    "import os\n",
+    "import json\n",
+    "from pprint import pprint\n",
+    "from functions_variables import encode_tags"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Data Importing"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load one file first to see what type of data you're dealing with and what attributes it has"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# loop over all files and put them into a dataframe"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Data Cleaning and Wrangling"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "At this point, ensure that you have all sales in a dataframe.\n",
+    "- Take a quick look at your data (i.e. `.info()`, `.describe()`) - what do you see?\n",
+    "- Is each cell one value, or do some cells have lists?\n",
+    "- What are the data types of each column?\n",
+    "- Some sales may not actually include the sale price (target).  These rows should be dropped.\n",
+    "- There are a lot of NA/None values.  Should these be dropped or replaced with something?\n",
+    "    - You can drop rows or use various methods to fills NA's - use your best judgement for each column \n",
+    "    - i.e. for some columns (like Garage), NA probably just means no Garage, so 0\n",
+    "- Drop columns that aren't needed\n",
+    "    - Don't keep the list price because it will be too close to the sale price. Assume we want to predict the price of houses not yet listed"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# load and concatenate data here\n",
+    "# drop or replace values as necessary"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Dealing with Tags"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Consider the fact that with tags, there are a lot of categorical variables.\n",
+    "- How many columns would we have if we OHE tags, city and state?\n",
+    "- Perhaps we can get rid of tags that have a low frequency."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# OHE categorical variables/ tags here\n",
+    "# tags will have to be done manually"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Dealing with Cities"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "- Sales will vary drastically between cities and states.  Is there a way to keep information about which city it is without OHE?\n",
+    "- Could we label encode or ordinal encode?  Yes, but this may have undesirable effects, giving nominal data ordinal values.\n",
+    "- What we can do is use our training data to encode the mean sale price by city as a feature (a.k.a. Target Encoding)\n",
+    "    - We can do this as long as we ONLY use the training data - we're using the available data to give us a 'starting guess' of the price for each city, without needing to encode city explicitly\n",
+    "- If you replace cities or states with numerical values (like the mean price), make sure that the data is split so that we don't leak data into the training selection. This is a great time to train test split. Compute on the training data, and join these values to the test data\n",
+    "- Note that you *may* have cities in the test set that are not in the training set. You don't want these to be NA, so maybe you can fill them with the overall mean"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# perform train test split here\n",
+    "# do something with state and city"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Extra Data - STRETCH\n",
+    "\n",
+    "> This doesn't need to be part of your Minimum Viable Product (MVP). We recommend you write a functional, basic pipeline first, then circle back and join new data if you have time\n",
+    "\n",
+    "> If you do this, try to write your downstream steps in a way it will still work on a dataframe with different features!\n",
+    "\n",
+    "- You're not limited to just using the data provided to you. Think/ do some research about other features that might be useful to predict housing prices. \n",
+    "- Can you import and join this data? Make sure you do any necessary preprocessing and make sure it is joined correctly.\n",
+    "- Example suggestion: could mortgage interest rates in the year of the listing affect the price? "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# import, join and preprocess new data here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## EDA/ Visualization\n",
+    "\n",
+    "Remember all of the EDA that you've been learning about?  Now is a perfect time for it!\n",
+    "- Look at distributions of numerical variables to see the shape of the data and detect outliers.    \n",
+    "    - Consider transforming very skewed variables\n",
+    "- Scatterplots of a numerical variable and the target go a long way to show correlations.\n",
+    "- A heatmap will help detect highly correlated features, and we don't want these.\n",
+    "    - You may have too many features to do this, in which case you can simply compute the most correlated feature-pairs and list them\n",
+    "- Is there any overlap in any of the features? (redundant information, like number of this or that room...)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# perform EDA here"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Scaling and Finishing Up"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now is a great time to scale the data and save it once it's preprocessed.\n",
+    "- You can save it in your data folder, but you may want to make a new `processed/` subfolder to keep it organized"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}