Computer Science > Machine Learning
[Submitted on 31 Jan 2022 (v1), revised 14 Feb 2022 (this version, v3), latest version 7 Jun 2024 (v8)]
Title:Out-of-distribution and in-distribution posterior calibration using Kernel Density Polytopes
View PDFAbstract:Any reasonable machine learning (ML) model should not only interpolate efficiently in between the training samples provided (in-distribution region), but also approach the extrapolative or out-of-distribution (OOD) region without being overconfident. Our experiment on human subjects justifies the aforementioned properties for human intelligence as well. Many state-of-the-art algorithms have tried to fix the overconfidence problem of ML models in the OOD region. However, in doing so, they have often impaired the in-distribution performance of the model. Our key insight is that ML models partition the feature space into polytopes and learn constant (random forests) or affine (ReLU networks) functions over those polytopes. This leads to the OOD overconfidence problem for the polytopes which lie in the training data boundary and extend to infinity. To resolve this issue, we propose kernel density methods that fit Gaussian kernel over the polytopes, which are learned using ML models. Specifically, we introduce two variants of kernel density polytopes: Kernel Density Forest (KDF) and Kernel Density Network (KDN) based on random forests and deep networks, respectively. Studies on various simulation settings show that both KDF and KDN achieve uniform confidence over the classes in the OOD region while maintaining good in-distribution accuracy compared to that of their respective parent models.
Submission history
From: Jayanta Dey [view email][v1] Mon, 31 Jan 2022 05:07:16 UTC (16,278 KB)
[v2] Sun, 6 Feb 2022 14:38:23 UTC (16,262 KB)
[v3] Mon, 14 Feb 2022 14:14:47 UTC (16,263 KB)
[v4] Fri, 25 Mar 2022 03:41:23 UTC (16,263 KB)
[v5] Fri, 19 May 2023 21:15:28 UTC (18,650 KB)
[v6] Thu, 19 Oct 2023 02:27:07 UTC (33,996 KB)
[v7] Tue, 12 Mar 2024 12:57:20 UTC (33,151 KB)
[v8] Fri, 7 Jun 2024 17:10:00 UTC (32,931 KB)
Current browse context:
cs.LG
References & Citations
Bibliographic and Citation Tools
Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)
Code, Data and Media Associated with this Article
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)
IArxiv Recommender
(What is IArxiv?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.