e-space
Manchester Metropolitan University's Research Repository

    On Regularisation Methods for Analysis of High Dimensional Data

    Sirimongkolkasem, Tanin and Drikvandi, Reza ORCID logoORCID: https://orcid.org/0000-0002-7245-9713 (2019) On Regularisation Methods for Analysis of High Dimensional Data. Annals of Data Science, 6 (4). pp. 737-763. ISSN 2198-5804

    [img]
    Preview
    Published Version
    Available under License Creative Commons Attribution.

    Download (1MB) | Preview

    Abstract

    High dimensional data are rapidly growing in many domains due to the development of technological advances which helps collect data with a large number of variables to better understand a given phenomenon of interest. Particular examples appear in genomics, fMRI data analysis, large-scale healthcare analytics, text/image analysis and astronomy. In the last two decades regularisation approaches have become the methods of choice for analysing such high dimensional data. This paper aims to study the performance of regularisation methods, including the recently proposed method called de-biased lasso, for the analysis of high dimensional data under different sparse and non-sparse situations. Our investigation concerns prediction, parameter estimation and variable selection. We particularly study the effects of correlated variables, covariate location and effect size which have not been well investigated. We find that correlated data when associated with important variables improve those common regularisation methods in all aspects, and that the level of sparsity can be reflected not only from the number of important variables but also from their overall effect size and locations. The latter may be seen under a non-sparse data structure. We demonstrate that the de-biased lasso performs well especially in low dimensional data, however it still suffers from issues, such as multicollinearity and multiple hypothesis testing, similar to the classical regression methods.

    Impact and Reach

    Statistics

    Activity Overview
    6 month trend
    342Downloads
    6 month trend
    299Hits

    Additional statistics for this dataset are available via IRStats2.

    Altmetric

    Repository staff only

    Edit record Edit record