Direction specific errors and granularity

In solar image segmentation, we identify many categories of structures on the Sun: coronal hole, filament, flare, active region, quiet sun, prominence. In our use case, some mistakes are more egregious than others. For example, mistaking a filament as a coronal hole is not too bad, no where close to as bad as calling it a flare. Assume we have a gold standard set for evaluation. (In reality, even this gold standard set may have errors, but we can ignore that for now.) It has a region labeled as filament. Ideally, we want our trained classifier to also call that filament. However, if it calls it quiet sun, we would be okay. Calling it coronal hole is also acceptable. Any other category is wrong, with the most egregious being if we call it outer space or flare. Now another portion of the Sun is labeled quiet sun in the gold standard. It is not okay for the classifier to then call it filament. In this way, it is acceptable to mistakenly label a filament as quiet sun but unacceptable to label quiet sun as anything else. The error depends on the direction of the mistake.

Similarly, in our current evaluation we evaluate errors on a pixel-by-pixel basis. In reality, we do not care about this granularity. We want coherent labeling. Small boundary disagreements are okay. We need a more robust evaluation metric.