Accuracy Assessment and the Confusion Matrix

Accuracy assessment and the confusion matrix: how right was my analysis?

Accuracy assessment with a confusion matrix is a validation method that compares your analysis result with the real answers in a table, showing where it was right and how it was wrong. For rare disasters, overall accuracy alone can hide missed events, so you need to look at misses and false alarms separately.

In short

  • Keep a validation sample that was not used in training.
  • Use a confusion matrix to see both what was right and what kind of errors occurred.
  • For rare disasters, look at misses and false alarms rather than accuracy.

Suppose you build a model that automatically finds flooded areas and the result says "accuracy 98%". It is a nice number for a report, but this number alone does not tell you whether the model pointed out the villages that were actually under water in the last heavy rain.

If you want to base evacuation plans or patrol areas on an analysis, you need to know what went wrong and how, more than how often it was right. This post covers, in turn, why you keep a separate validation sample, how to read a confusion matrix, and whether to reduce misses or false alarms first.

satellite imagery land cover map
Stock photo. Photo: WikiImages / Pixabay

Why keep the validation sample separate?

If you score a model with the same samples you used to build it, it is like taking an exam after seeing the questions in advance. The model gets the answers right almost by memory, so the score comes out higher than the real one. That is why you split the samples into training and validation sets, and the validation set must not be used at all during training. Spatial data needs one more precaution. Nearby points have similar values, so if you split at random, a validation point right next to a training point is effectively the same question. Where possible, it is safer to split by area or by event. For example, if you trained on the flood areas of one year, you should validate with flood traces from another year or from another river basin to get a score that holds up in the field.

How do I read a confusion matrix?

A confusion matrix is a small table that lines up the real answers against the predictions. For flood detection, the cell where real flooding was predicted as flooding is a true positive. The cell where real flooding was predicted as not flooded is a miss (false negative). The cell where a dry place was predicted as flooded is a false alarm (false positive). The cell where a dry place was predicted as dry is a true negative. Usually the rows are the real values and the columns are the predictions, but some tools put them the other way round, so check the axis names before reading the table. The diagonal cells are where the model was right and the other cells are where it was wrong. The position of the wrong cells shows which direction the errors go. For a landslide risk grade map, it shows which grade gets mixed up with which, which becomes the basis for deciding what data to improve.

Can a model with high accuracy still be useless?

Take a hypothetical example. Out of 1,000 grid cells, only 20 are actually flooded. A model that answers "not flooded" for every cell gets 980 right, so the overall accuracy is 98%. But it found none of the floods, so it is useless for disaster work. Rare events such as wildfires, landslides and floods are mostly normal areas, so overall accuracy is easily inflated. That is why you must always look at metrics for the event class alone next to the overall accuracy. If a report gives only one score, ask for the confusion matrix and the class ratio of the sample. Without that ratio you cannot interpret the number, and you may give decision makers a vague sense of safety.

In disaster detection, should I reduce misses or false alarms?

A miss is a real disaster that was overlooked, and a false alarm is a disaster reported where there is none. Which one to reduce more is decided by the operational need, not by the machine. For evacuation decisions, where a miss can lead to casualties, reducing misses first is the usual choice. However, too many false alarms lead to repeated false warnings, the alerts get ignored, and dispatch staff and evacuation costs are wasted. Lowering the decision threshold (the score boundary above which something counts as a disaster) reduces misses but increases false alarms. It is a trade-off, so you set the threshold by looking at both errors together. A realistic approach is to make the first screening loose so that it reduces misses, and then filter out false alarms in a second step where people check with satellite images or field reports.

What are recall and precision, and how do I check them myself?

Recall is the share of real disasters that were found, the opposite side of misses. Precision is the share of reported disasters that were real, the opposite side of false alarms. F1 combines the two into one value, but it hides which one is lacking, so look at them separately as well. Some things you learn only by trying. First, check that the validation answers come from real data, such as field surveys or reports. If the answers are the output of another model, it is not validation. Next, change the threshold and compare how the confusion matrix changes, and plot the wrong points on a map to see whether they cluster along rivers or on steep slopes. QGIS menu names can differ by version and language setting. Once you know the weak terrain, you have a basis for explaining the predicted flood areas.

Step by step on screen

The diagram shows the order of the steps and may use a Korean interface. Menu positions are the same in the English interface.

Overview of the steps

  1. Load the validation points
    Go to Layer > Add Layer > Add Vector Layer, select the validation point file that has an answer column and click Add.
  2. Attach the predicted values
    Type "sampling" in the search box of the Processing Toolbox, open Sample Raster Values and select the predicted result raster.
  3. Export to a table
    Right-click the validation point layer > Export > Save Features As, choose CSV as the format and save.
  4. Make a pivot table
    In a spreadsheet, open Insert > Pivot Table, and put the real value in rows, the prediction in columns and the count in values.
  5. Calculate the metrics
    In cells next to the table, enter recall as true positives / (true positives + misses) and precision as true positives / (true positives + false alarms).

Menu names may differ slightly between program versions. The figure above is a diagram that summarizes the order.

Prompts you can copy

Ask for your confusion matrix numbers to be interpreted from a disaster operations point of view

You are an assistant who helps validate GIS analysis in disaster safety. The reader is a practitioner who is new to spatial information.
Below is the confusion matrix of a [target disaster, for example heavy-rain flooding] detection result. Rows are real values and columns are predictions.
True positives [number], misses [number], false alarms [number], true negatives [number]
The validation sample was prepared by [source of the sample and how it was split].
1) Show the overall accuracy, recall and precision together with the formulas.
2) Explain which is the bigger problem in this result, misses or false alarms, based on [operational purpose, for example evacuation guidance priority].
3) Point out anything that could be misleading if you look only at overall accuracy.
Constraints: explain each technical term in one line the first time it appears, and do not invent numbers or criteria I did not provide. Organize the answer in a table and add a one-line conclusion at the end.
What a person must check: recalculate the results by hand, and confirm that the source of the validation sample is real survey data.

Plan how to split the validation sample

You are an adviser who helps design validation for spatial data. I want to evaluate a [target disaster] detection model at [organization or department name].
The data I have is [data type, approximate size, period and area]. Disaster points are rare in the whole set.
Suggest ways to split training and validation sets by area and by event, and compare their pros and cons in a table.
Explain in plain words how to avoid the problem of spatially close points ending up on both sides.
Constraints: do not state the features or versions of any specific software as fact, and write that you do not know where you do not know. Answer in 12 lines or fewer.
What a person must check: on a map, check that the proposed split fits the actual area and period of the data, and that the validation area does not overlap the training area.

Replace the bracketed parts for your situation. Do not use the answers as they are; check the facts first.

Try it yourself

  • Build a table in a spreadsheet like the hypothetical example, with 20 disasters out of 1,000, and calculate the accuracy and recall when everything is answered as normal.
  • Check whether the answers for your own analysis come from real survey data and whether they are samples not used in training, and write it down.
  • Change the threshold to two or three levels, compare how misses and false alarms change in a table, and pick the value that suits your work.

Common questions

How do I tell false alarms from misses in a confusion matrix?
A false alarm is a case reported as a disaster where there was none. A miss is a case where a real disaster was reported as not a disaster. If you first check which axis is real and which is predicted, you will not mix them up.
Is a model with high accuracy good enough for disaster detection?
Not necessarily. Disasters are rare events, so answering "normal" for everything can still give high accuracy. Check recall and precision as well.

How much of the whole data should I set aside for validation?
There is no fixed ratio. It depends on the amount of data and the number of disaster cases. Rather than the ratio, first check that the validation sample contains enough disaster cases and does not overlap the training area.

Accuracy assessment is not the last step of an analysis. It is the first step in deciding how far you can trust the result in the field. If you keep a separate validation sample, check how the model is wrong with a confusion matrix, and balance misses and false alarms for your purpose, you will not be fooled by a single number. Now that you know where your analysis is right and where it is wrong, the next question is this: when the errors cluster in certain areas, how can you show and explain them on a map? A later post in this series covers this.

This is an English adaptation of a post in the Korean GeoAI basics series. The original is here, and more learning materials are on GeoAI for Disaster Safety.

This post was drafted with the help of AI tools and reviewed by the author. Tool features and data portals change over time, so check the official documentation when you practice.

Comments

Popular posts from this blog

What Is GeoAI, and How Does It Help Disaster Safety?

Getting Started with QGIS: What to Do First

Whitebox QGIS Practice 26 — Elevation Statistics of Two Basins: Means of 2,000.1 m and 1,924.3 m, a 76 m Gap