Pandas refreshers¶
Pandas is mostly used just to read data, fix missing values, mask and change it to numpy array.
It is also used to quickly visualize data from files
In [10]:
import pandas as pd
import numpy as np
pd.__version__
Out[10]:
Read CSV¶
Returns a padas.DataFrame object
In [3]:
pd.read_csv('files/data_csv.csv')
Out[3]:
Read JSON¶
In [36]:
#if the json is in format
#{column1 : [values_list], column2 : [values_list]}
pd.read_json('files/data_json.json')
Out[36]:
In [37]:
#Or in format [{col1:val,col2:val}, {col1:val,col2:val}]
pd.read_json('files/data_json_records.json',orient='records')
Out[37]:
Slice¶
In [38]:
df = pd.read_csv('files/data_csv.csv')
type(df)
Out[38]:
In [40]:
df["Salary"]
Out[40]:
In [42]:
df[["Country","Salary"]]
Out[42]:
View statistic¶
In [43]:
df.describe()
Out[43]:
Mask¶
Works just like numpy
In [44]:
#Masking just like numpy
df[df['Salary']>60000]
Out[44]:
Get numpy array from DataFrame¶
In [47]:
#Get the numpy array
df.values
Out[47]:
Sample data¶
In [3]:
import numpy as np
In [11]:
x = np.linspace(0,10,100000).reshape(-1,1) #otherwise we cant concat along axis 1 (use vstack instead?)
y = (np.linspace(0,10,100000) + np.random.normal(0,0.01,100000)).reshape(-1,1)
In [13]:
df = pd.DataFrame(np.concatenate((x,y),axis=1),columns=["X","Y"])
In [14]:
df.head() #shows the first five elements
Out[14]:
In [15]:
df.sample(n=10) #samples 10 random elements
Out[15]:
In [18]:
#More on data visualization refresher. It will be difficult to plot million points. so we plot a sample
%matplotlib inline
df.sample(n=25).plot(x='X',y='Y',kind='scatter')
Out[18]:
Apply function along column or rows¶
In [3]:
diabetes = pd.read_csv("files/pima-indians-diabetes.csv")
diabetes.head()
Out[3]:
Lets just normalize 'Number_pregnant' and 'Age' columns
In [3]:
diabetes.columns
Out[3]:
In [4]:
cols_to_normalize = ['Number_pregnant','Age']
In [6]:
diabetes[cols_to_normalize].head()
Out[6]:
In [7]:
#When axis=0(default), applies function to each column
#When axis=1, applies function to each row
fn = lambda x: (x-x.min())/(x.max()-x.min())
diabetes[cols_to_normalize] = diabetes[cols_to_normalize].apply(fn,axis=0)
In [9]:
diabetes.head()
Out[9]: