Title :
Understanding Where Your Classifier Does (Not) Work -- The SCaPE Model Class for EMM
Author :
Duivesteijn, Wouter ; Thaele, Julia
Author_Institution :
Lehrstuhl fur Kunstliche Intelligenz, Tech. Univ. Dortmund, Dortmund, Germany
Abstract :
FACT, the First G-APD Cherenkov Telescope, detects air showers induced by high-energetic cosmic particles. It is desirable to classify a shower as being induced by a gamma ray or a background particle. Generally, it is nontrivial to get any feedback on the real-life training task, but we can attempt to understand how our classifier works by investigating its performance on Monte Carlo simulated data. To this end, in this paper we develop the SCaPE (Soft Classifier Performance Evaluation) model class for Exceptional Model Mining, which is a Local Pattern Mining framework devoted to highlighting unusual interplay between multiple targets. In our Monte Carlo simulated data, we take as targets the computed classifier probabilities and the binary column containing the ground truth: which kind of particle induced the corresponding shower. Using a newly developed quality measure based on ranking loss, the SCaPE model class highlights subspaces of the search space where the classifier performs particularly well or poorly. These subspaces arrive in terms of conditions on attributes of the data, hence they come in a language a domain expert understands, which should aid him in understanding where his/her classifier does (not) work. Found subgroups highlight subspaces whose difficulty for classification is corroborated by astrophysical interpretation, as well as subspaces that warrant further investigation.
Keywords :
Monte Carlo methods; astronomical telescopes; astronomy computing; data mining; pattern classification; probability; search problems; EMM; FACT telescope; G-APD Cherenkov telescope; Monte Carlo simulated data; SCaPE; air shower detection; astrophysical interpretation; background particle; binary column; classifier probabilities; exceptional model mining; gamma ray; high-energetic cosmic particles; local pattern mining framework; ranking loss; search space; shower classification; soft classifier performance evaluation model class; Atmospheric modeling; Data mining; Loss measurement; Protons; Radio frequency; Telescopes; Terrestrial atmosphere; Astrophysics; Cherenkov radiation; Exceptional Model Mining; soft classifier;
Conference_Titel :
Data Mining (ICDM), 2014 IEEE International Conference on
Conference_Location :
Shenzhen
Print_ISBN :
978-1-4799-4303-6
DOI :
10.1109/ICDM.2014.10