A short demo of how to use LOCOPath in Python¶
import our package varImp¶
In [1]:
import varImp
In [2]:
import numpy as np
from sklearn import linear_model
from sklearn import preprocessing
from scipy.interpolate import interp1d
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
Genereate some data¶
Let's do high-dimensional linear regression, with $n = 100$ records and $p = 1000$ features, where the active set is the 1st 10 features.¶
In [23]:
n = 100
p = 1000
s = 10
In [27]:
X = np.random.normal(size = [n, p])
beta = np.array([1] * 5 + [2] * 4 + [5] + [0] * (p - s))
Y = np.dot(X, beta) + np.random.normal(size = n)
Visualize the solution path -- we would first use LARS¶
If we want to measure importance of 10th feature, which is the most important one, so set $j = 9$ (start with 0)¶
In [51]:
j = 9
In [52]:
alphas, _, coefs = linear_model.lars_path(X, Y, Gram = None, method = 'lasso', return_path = True)
alphas_j, _, coefs_j = linear_model.lars_path(np.delete(X, j, axis = 1), Y, Gram=None, method = 'lasso', return_path=True)
Removing the most importance feature changed the path a lot ...¶
In [53]:
plt.subplot(121)
for i in range(coefs.shape[0]):
plt.plot(alphas, coefs[i,:])
plt.title("Before removal ")
plt.ylim(-1, 5)
plt.subplot(122)
for i in range(coefs_j.shape[0]):
plt.plot(alphas_j, coefs_j[i,:], )
plt.title("After removal")
plt.ylim(-1, 5)
Out[53]:
In [54]:
j = 7
alphas, _, coefs = linear_model.lars_path(X, Y, Gram = None, method = 'lasso', return_path = True)
alphas_j, _, coefs_j = linear_model.lars_path(np.delete(X, j, axis = 1), Y, Gram=None, method = 'lasso', return_path=True)
plt.subplot(121)
for i in range(coefs.shape[0]):
plt.plot(alphas, coefs[i,:])
plt.title("Before removal ")
plt.ylim(-1, 5)
plt.subplot(122)
for i in range(coefs_j.shape[0]):
plt.plot(alphas_j, coefs_j[i,:], )
plt.title("After removal")
plt.ylim(-1, 5)
Out[54]:
Removing un unimportant feature barely changed the path ...¶
In [31]:
j = 20
In [32]:
alphas_j, _, coefs_j = linear_model.lars_path(np.delete(X, j, axis = 1), Y, Gram=None, method = 'lasso', return_path=True)
In [33]:
plt.subplot(121)
for i in range(coefs.shape[0]):
plt.plot(alphas, coefs[i,:])
plt.title("Before removal ")
plt.subplot(122)
for i in range(coefs_j.shape[0]):
plt.plot(alphas_j, coefs_j[i,:])
plt.title("After removal")
Now you get the idea, how should we quantify the "change" ? -- The PARSE SCORE¶
$$LOCO\_PARSE\_SCORE_j = || \ \ ||\beta_i(\lambda) - \beta^{(j)}_i(\lambda)||_p \ ||_q$$
where the $L_p$ norm inside the $l_q$ norm is the $L_p$ norm of a function, defined via: $$ || f(\lambda) ||_p = \big (\int |f(\lambda)|^p d\lambda \big ) ^ {\frac{1}{p}} $$
For convinience, we use $p=q=2$ in this demo.
Feature importance of 1st 10 features:¶
In [34]:
for i in range(10):
print("Now compting {}th TS: {}".format(i, varImp.LOCO_TS(varImp.ExtractPath_LARS(X, Y, i, 0))))
Feature importance of unimportant features:¶
In [35]:
for i in range(10, 20):
print("Now compting {}th TS: {}".format(i, varImp.LOCO_TS(varImp.ExtractPath_LARS(X, Y, i, 0))))
In [36]:
imp_list = [ varImp.LOCO_TS(varImp.ExtractPath_LARS(X, Y, i, 0)) for i in range(0, 30) ]
In [37]:
parse_score = (imp_list - min(imp_list)) / (max(imp_list) - min(imp_list))
In [38]:
parse_score
Out[38]:
Clearly, the first 10 feature could be flagged as important features!¶
In [39]:
plt.bar(list(range(30)),parse_score, width = 0.2)
plt.scatter(list(range(30)), parse_score)
Out[39]:
Anything interesting in the 1st 100 features, other than the 1st 10? -- As we expected, no! Only the 1st 10 features!¶
In [40]:
imp_list = [ varImp.LOCO_TS(varImp.ExtractPath_LARS(X, Y, i, 0)) for i in range(0, 100) ]
In [48]:
plt.bar(list(range(100)),imp_list, width = 0.4)
plt.scatter(list(range(100)), imp_list)
plt.ylim(0, 6)
Out[48]:
In [49]:
imp_list
Out[49]: