[go: up one dir, main page]
IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i12p2729-d1172697.html
   My bibliography  Save this article

Enhancing Precision in Large-Scale Data Analysis: An Innovative Robust Imputation Algorithm for Managing Outliers and Missing Values

Author

Listed:
  • Matthias Templ

    (School of Business, University of Applied Sciences and Arts Northwestern Switzerland, 4600 Olten, Switzerland)

Abstract
Navigating the intricate world of data analytics, one method has emerged as a key tool in confronting missing data: multiple imputation. Its strength is further fortified by its powerful variant, robust imputation, which enhances the precision and reliability of its results. In the challenging landscape of data analysis, non-robust methods can be swayed by a few extreme outliers, leading to skewed imputations and biased estimates. This can apply to both representative outliers—those true yet unusual values of your population—and non-representative outliers, which are mere measurement errors. Detecting these outliers in large or high-dimensional data sets often becomes as complex as unraveling a Gordian knot. The solution? Turn to robust imputation methods. Robust (imputation) methods effectively manage outliers and exhibit remarkable resistance to their influence, providing a more reliable approach to dealing with missing data. Moreover, these robust methods offer flexibility, accommodating even if the imputation model used is not a perfect fit. They are akin to a well-designed buffer system, absorbing slight deviations without compromising overall stability. In the latest advancement of statistical methodology, a new robust imputation algorithm has been introduced. This innovative solution addresses three significant challenges with robustness. It utilizes robust bootstrapping to manage model uncertainty during the imputation of a random sample; it incorporates robust fitting to reinforce accuracy; and it takes into account imputation uncertainty in a resilient manner. Furthermore, any complex regression or classification model for any variable with missing data can be run through the algorithm. With this new algorithm, we move one step closer to optimizing the accuracy and reliability of handling missing data. Using a realistic data set and a simulation study including a sensitivity analysis, the new alogorithm imputeRobust shows excellent performance compared with other common methods. Effectiveness was demonstrated by measures of precision for the prediction error, the coverage rates, and the mean square errors of the estimators, as well as by visual comparisons.

Suggested Citation

  • Matthias Templ, 2023. "Enhancing Precision in Large-Scale Data Analysis: An Innovative Robust Imputation Algorithm for Managing Outliers and Missing Values," Mathematics, MDPI, vol. 11(12), pages 1-22, June.
    • Handle: RePEc:gam:jmathe:v:11:y:2023:i:12:p:2729-:d:1172697
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/12/2729/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/11/12/2729/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kowarik, Alexander & Templ, Matthias, 2016. "Imputation with the R Package VIM," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 74(i07).
    2. van Buuren, Stef & Groothuis-Oudshoorn, Karin, 2011. "mice: Multivariate Imputation by Chained Equations in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 45(i03).
    3. Matías Salibián-Barrera & Stefan Aelst & Gert Willems, 2008. "Fast and robust bootstrap," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 17(1), pages 41-71, February.
    4. Zeng, Chao & Ma, Changxi & Wang, Ke & Cui, Zihao, 2022. "Predicting vacant parking space availability: A DWT-Bi-LSTM model," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 599(C).
    5. Templ, Matthias & Kowarik, Alexander & Filzmoser, Peter, 2011. "Iterative stepwise regression imputation using standard and robust methods," Computational Statistics & Data Analysis, Elsevier, vol. 55(10), pages 2793-2806, October.
    6. Salibian-Barrera, Matias & Van Aelst, Stefan, 2008. "Robust model selection using fast and robust bootstrap," Computational Statistics & Data Analysis, Elsevier, vol. 52(12), pages 5121-5135, August.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gerko Vink & Laurence E. Frank & Jeroen Pannekoek & Stef Buuren, 2014. "Predictive mean matching imputation of semicontinuous variables," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 68(1), pages 61-90, February.
    2. La Vecchia, Davide & Camponovo, Lorenzo & Ferrari, Davide, 2015. "Robust heart rate variability analysis by generalized entropy minimization," Computational Statistics & Data Analysis, Elsevier, vol. 82(C), pages 137-151.
    3. Henry Webel & Lili Niu & Annelaura Bach Nielsen & Marie Locard-Paulet & Matthias Mann & Lars Juhl Jensen & Simon Rasmussen, 2024. "Imputation of label-free quantitative mass spectrometry-based proteomics data using self-supervised deep learning," Nature Communications, Nature, vol. 15(1), pages 1-15, December.
    4. Valéry Dongmo Jiongo & Pierre Nguimkeu, 2018. "Bootstrapping Mean Squared Errors of Robust Small-Area Estimators: Application to the Method-of-Payments Data," Staff Working Papers 18-28, Bank of Canada.
    5. Jiao, Xiyu & Pretis, Felix & Schwarz, Moritz, 2024. "Testing for coefficient distortion due to outliers with an application to the economic impacts of climate change," Journal of Econometrics, Elsevier, vol. 239(1).
    6. La Vecchia, Davide & Moor, Alban & Scaillet, Olivier, 2023. "A higher-order correct fast moving-average bootstrap for dependent data," Journal of Econometrics, Elsevier, vol. 235(1), pages 65-81.
    7. Alfons, Andreas & Croux, Christophe & Gelper, Sarah, 2016. "Robust groupwise least angle regression," Computational Statistics & Data Analysis, Elsevier, vol. 93(C), pages 421-435.
    8. Nicholas Tierney & Dianne Cook, 2018. "Expanding tidy data principles to facilitate missing data exploration, visualization and assessment of imputations," Monash Econometrics and Business Statistics Working Papers 14/18, Monash University, Department of Econometrics and Business Statistics.
    9. Angela Calvo & Christian Preti & Maria Caria & Roberto Deboli, 2019. "Vibration and Noise Transmitted by Agricultural Backpack Powered Machines Critically Examined Using the Current Standards," IJERPH, MDPI, vol. 16(12), pages 1-20, June.
    10. Salibian-Barrera, Matias & Van Aelst, Stefan & Yohai, Víctor J., 2016. "Robust tests for linear regression models based on τ-estimates," Computational Statistics & Data Analysis, Elsevier, vol. 93(C), pages 436-455.
    11. Nengsih Titin Agustin & Bertrand Frédéric & Maumy-Bertrand Myriam & Meyer Nicolas, 2019. "Determining the number of components in PLS regression on incomplete data set," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 18(6), pages 1-28, December.
    12. Khan, Jafar A. & Van Aelst, Stefan & Zamar, Ruben H., 2010. "Fast robust estimation of prediction error based on resampling," Computational Statistics & Data Analysis, Elsevier, vol. 54(12), pages 3121-3130, December.
    13. Kowarik, Alexander & Templ, Matthias, 2016. "Imputation with the R Package VIM," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 74(i07).
    14. Samanta, Mayukh & Welsh, A.H., 2013. "Bootstrapping for highly unbalanced clustered data," Computational Statistics & Data Analysis, Elsevier, vol. 59(C), pages 70-81.
    15. Adel Bosch & Steven F. Koch, 2021. "Individual and Household Debt: Does Imputation Choice Matter?," Working Papers 202141, University of Pretoria, Department of Economics.
    16. O’Shaughnessy, P.Y. & Welsh, A.H., 2018. "Bootstrapping longitudinal data with multiple levels of variation," Computational Statistics & Data Analysis, Elsevier, vol. 124(C), pages 117-131.
    17. Riani, Marco & Atkinson, Anthony C., 2010. "Robust model selection with flexible trimming," Computational Statistics & Data Analysis, Elsevier, vol. 54(12), pages 3300-3312, December.
    18. Stefan Van Aelst & Gert Willems, 2010. "Inference for robust canonical variate analysis," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 4(2), pages 181-197, September.
    19. Maria Lucia Parrella & Giuseppina Albano & Michele La Rocca & Cira Perna, 2019. "Reconstructing missing data sequences in multivariate time series: an application to environmental data," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 28(2), pages 359-383, June.
    20. Sanjoy K. Sinha, 2019. "Robust small area estimation in generalized linear mixed models," METRON, Springer;Sapienza Università di Roma, vol. 77(3), pages 201-225, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:12:p:2729-:d:1172697. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.