Table of Contents

• 1  Introduction
• 2  Data Loading
  • 2.1  Installing Libraries
  • 2.2  Importing Libraries
  • 2.3  Reading File
  • 2.4  Viewing Shape of Data
  • 2.5  Viewing Top Data
• 3  Data Check
  • 3.1  Viewing Data Types
  • 3.2  Checking Missing Values
• 4  Data Transformation
  • 4.1  Datetime
  • 4.2  Device Name
• 5  Descriptive Analysis
  • 5.1  Frequency Distribution
    • 5.1.1  What is number of observations for each device?
      • 5.1.1.1  Insights
    • 5.1.2  What is the distribution of a field over a time?
      • 5.1.2.1  Insights
    • 5.1.3  What is the distribution of each Columns?
      • 5.1.3.1  CO Insights\
      • 5.1.3.2  All
      • 5.1.3.3  All Insights
  • 5.2  Central Tendency
    • 5.2.1  Overall Insights
    • 5.2.2  Insights Based on Device
    • 5.2.3  Finding Outliers
  • 5.3  Correlations
    • 5.3.1  Overall
    • 5.3.2  For Each Device
  • 5.4  Conclusion from Descriptive Analysis
• 6  Inferential Data Analysis
  • 6.1  Sampling
    • 6.1.1  Problems with Sampling
  • 6.2  Estimation
  • 6.3  Test
  • 6.4  Terms Widely Used in Testing
  • 6.5  Comparison Test
    • 6.5.1  Is the mean value of each fields same for each device's recorded data?
  • 6.6  Correlation Test
    • 6.6.1  Pearson's and Spearman's Test
      • 6.6.1.1  Pearson's Test For Linear Relationship Between Variables
      • 6.6.1.2  Spearman’s Correlation: Non-Linear Relationship between two variables.
    • 6.6.2  Insights
    • 6.6.3  Chi Square Test
    • 6.6.4  Collinearity vs Multicollinearity
  • 6.7  Regression Tests
• 7  Using Autoviz for Fast EDA

Introduction

Hello everyone, welcome back to another new blog where we will explore different ideas and concept one could perform while performing an EDA. In simple words, this blog is a simple walk-through of an average EDA process which might include (in top down order):

• Data Loading: From various sources (remote, local) and various formats (excel, csv, sql etc.)
• Data Check: This is very important task where we check the data types (numerical, categorical, binary etc) of a data. We often focus on number of missing values.
• Data Transformation: This includes filling up null values, or removing them from the table. We also do some data type conversions if required.
• Descriptive Analysis: This is the heart of any EDA because here, we do lots of statistical tasks like finding mean, median, quartiles, mode, distribution, relationships of fields. We also plot different plots to support the analysis. This is sometimes enough to give insights about the data and if the data is rich and we need to find more insights and make assumptions, we have to do Inferential Analysis.
• Inferential Analysis: This task sometimes is taken into the EDA part but most of the time we do inferential analysis along with model development. However, we do perform different tests (e.g Chi- Square Test) to calculate feature importance. Here we often do tests based on hypothesis and samples drawn from the population.

While walking through these major steps, one will try to answer different questions of analysis like how many times some categorical data has appeared, what is the distribution over a date, what is the performance over certain cases and so on.

Data Loading

Installing Libraries

!pip install autoviz
!pip install seaborn
!pip install plotly
!pip install cufflinks
!pip install pandas

• Autoviz is for auto visualization but it is heavy and power hungry.
• Seaborn is built on top of the matplotlib and is best for making rich static plots.
• Plotly is for interactive visualization.
• Cufflinks is for connecting pandas and plotly.
• Pandas is for data analysis.

Importing Libraries

If you do not have these libraries installed, please install them like below:

import autoviz
from autoviz.AutoViz_Class import AutoViz_Class
from pandas_profiling import ProfileReport
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import warnings
from plotly.offline import init_notebook_mode, iplot
import plotly.figure_factory as ff
import cufflinks
import plotly.io as pio 
cufflinks.go_offline()
cufflinks.set_config_file(world_readable=True, theme='pearl')
pio.renderers.default = "notebook" # should change by looking into pio.renderers

pd.options.display.max_columns = None
%matplotlib inline

Alert! from autoviz version 0.1.35, after importing, you must %matplotlib inline to display charts in Jupyter Notebooks.
    AV = AutoViz_Class()
    AV.AutoViz(filename, sep=',', depVar='', dfte=None, header=0, verbose=0, lowess=False,
               chart_format='svg',max_rows_analyzed=150000,max_cols_analyzed=30, save_plot_dir=None)
Note: verbose=0 or 1 generates charts and displays them in your local Jupyter notebook.
      verbose=2 does not display plots but saves them in AutoViz_Plots folder in local machine.
Updated: chart_format='bokeh' generates and displays charts in your local Jupyter notebook.
      chart_format='server' generates and displays charts in the browser - one tab for each chart.
      chart_format='html' silently saves charts HTML format - they are also interactive!