Missing data

Ways of displaying missing values

pd.NA

np.nan

np.nan == np.nan false

np.nan is np.nan true

Check for null

df.isnull() # returns df with values replaced with booleans 

df[df['col'].notnull()] 

Keep it

# dont do anything

Remove it

df.dropna() # drops all rows with missing values

df.dropna(thresh=2) # drops where they have atleast 2 values

df.dropna(axis=1) # drops cols with missing values

df.dropna(subset=['col']) # drops rows where 'col' has a missing value

Replace it

df.fillna('new value') # replaces null as 'new value'

df['col'] = df['col'].fillna(0) # replaces null on col

df['col'].fillna(df['col'].mean())

df.fillna(df.mean()) # note: only fills numeric cols, VERY RISKY