Many high dimensional datasets with hundreds or thousands of covariates suffer from the presence of outliers. The problems of outlier detection in a high dimensional setting are fundamental in statistics and machine learning, and face huge challenges for state-of-the-art methods. In this paper, we consider the problem of linear regression in the presence of outliers. We propose a novel procedure combining Huber loss and convex regularization under the mean shift model to achieve better estimation and prediction. We derive the risk bounds and exact support recovery for outliers under some mild conditions. Then, we design a novel algorithm combining the alternating minimization method and accelerated proximal gradient method to solve the proposed model. Finally, extensive comparisons on simulation and the real dataset demonstrate the efficiency of the proposed procedure.