Training Data for GeoAI: Labeling Imagery Without Fooling Yourself

A model can only learn what its labels teach it. If the labels are wrong, uneven or unexplained, the model will still produce a map. That map will just look more trustworthy than it deserves.

This post covers the part of GeoAI work that happens before any model runs: building training data. We will use flood and landslide mapping from satellite imagery as examples. The focus is on five decisions that decide whether your results mean anything.

A label is the answer you give the model for a piece of imagery, such as "flooded" or "not flooded". Training data is the set of images or pixels paired with those labels.

Points, polygons or masks?

You can record a label in three common ways. Each has a cost and a limit.

Points

A point label marks one location with a class. It is fast to collect. You can click through hundreds of locations in QGIS in an afternoon, or reuse field survey points.

Points work well for classification of single pixels or small patches. They do not tell the model where an object ends. A point inside a landslide scar says "landslide here", but not how large it is.

Polygons

A polygon outlines an area: a flooded field, a burned patch, a building footprint. Polygons keep shape and size information. They are also easy to store as vector data and to check against other layers. If you need a refresher, see our post on raster vs vector.

The trade-off is time. Drawing careful boundaries is slow. Boundaries are also fuzzy in reality. Where exactly does a flood edge sit when water thins out across a field? Two people will draw two lines.

Masks

A mask is a raster the same size as the image, where each pixel holds a class value. For example, 0 is background, 1 is water, 2 is building. Masks are what segmentation models, which assign a class to every pixel, usually need.

You can create a mask by rasterizing polygons. The mask then inherits every polygon error, pixel by pixel. If the polygons were drawn on a different image than the one you train on, the mask will be shifted. Always check alignment visually before training.

Which to choose

  • Use points for per-pixel or per-patch classification, or when time is short.
  • Use polygons when shape and area matter, or when you want to reuse labels across different imagery.
  • Use masks when the model predicts every pixel. Make them from polygons, and keep the polygons.

Match the label type to the question you want answered. "Is this village flooded?" needs less than "How many hectares flooded?"

Class imbalance

Class imbalance means some classes have far more examples than others. In disaster mapping this is the normal case. Most of any scene is not flooded, not burned and not collapsed.

The danger is simple. A model that predicts "no damage" everywhere can score high overall accuracy while finding nothing useful. We covered this in the post on accuracy assessment and the confusion matrix. Always look at per-class results, not just the overall number.

Some ways to handle imbalance:

  • Sample on purpose. Collect more examples of the rare class, and a fair range of hard negatives, such as dark roofs or shadows that look like water.
  • Weight the loss. Most training libraries let you give rare classes more weight, so mistakes on them cost more.
  • Resample. Repeat rare examples or drop some common ones in the training set.
  • Keep the test set realistic. Balance the training data if you must, but do not balance the test data. The real world is imbalanced, and your evaluation should show that.

That last point is easy to miss. If you rebalance before splitting, copies of the same example can land in both training and test sets. Split first, then rebalance only the training side.

Regional bias

A model learns the places it sees. If all your flood labels come from one river basin, the model learns that basin: its soil color, its crops, its building styles, its season.

Move to a mountain valley or a dense city and performance can drop sharply, even though the task name is the same. This is regional bias, a kind of dataset bias tied to geography.

Bias can come from several places:

  • Where labels are easy to get. Roads, cities and flat open land get labeled more than remote slopes.
  • Which events are documented. Well-reported disasters produce more labels than quiet ones.
  • Which imagery you pick. Cloud cover, season and sensor can differ between regions. Our satellite imagery basics post explains why those differences change what a pixel looks like.
  • Map sources. Data from OpenStreetMap is volunteer-made, and coverage varies from place to place. Using it as a label source carries that unevenness into your model.

Practical steps: map where your labels are, literally. Plot them in QGIS and look for empty areas. Compare label density with the area you plan to predict. If the gap is large, say so, or collect more labels there before trusting the output. Later in this series, a post on spatial cross-validation shows how to test whether your model works in places it has not seen.

Our earlier post on GeoAI ethics discusses what biased outputs can cost people. Here the point is narrower: bias starts in the labels.

Label noise

Label noise means labels that are wrong, vague or inconsistent. Every real dataset has some. The goal is to know where it comes from and keep it small.

Common sources in imagery labeling:

  • Time mismatch. You label from an image taken days after the event, but train on an image from another date. Water recedes. Debris is cleared.
  • Misalignment. Labels and imagery sit in different coordinate systems or are shifted by a few pixels. Check that both use the same CRS (coordinate reference system). Our posts on EPSG codes cover the usual mistakes.
  • Resolution limits. At 10 m pixels, as in Sentinel-2's main visible and near-infrared bands, a single house is smaller than one pixel. A "damaged building" label on such imagery is a guess.
  • Annotator disagreement. Two people label the same scene differently, especially for partial damage or mixed land cover.
  • Automatic labels. Labels from an index threshold or an older model are fast, but they carry that method's errors. A water mask from NDWI will miss what NDWI misses.

Ways to reduce it:

  1. Write short labeling rules with example images, including borderline cases.
  2. Have a second person label a small sample, then compare. Where you disagree, fix the rule, not just the label.
  3. Review labels on top of the imagery, at the same scale the model will see.
  4. Look at the model's most confident mistakes. They often point to bad labels, not bad models.
  5. Keep a "uncertain" class or flag instead of forcing a guess.

Document how labels were made

Labels are a product of decisions. If you do not write those decisions down, you will forget them, and nobody else can judge your results.

A short text file next to the data is enough. Record:

  • Source imagery: sensor, acquisition date, processing level, and where you got it (for example, Copernicus Data Space).
  • Label type and tool: points, polygons or masks, and the software used.
  • Class definitions: what exactly counts as "flooded" or "damaged", with edge cases.
  • Who labeled, and how: manual, field survey, automatic rule, or model-assisted. Include how many people and whether anyone reviewed the work.
  • Coordinate system: the EPSG code of the labels.
  • Coverage: the area and period the labels represent, and areas that are missing.
  • Class counts: how many examples per class, before and after any resampling.
  • Known problems: clouds, shadows, uncertain regions, and anything you skipped.
  • Version and date: so you can tell which labels trained which model.

Documentation also protects you from fooling yourself. When results look too good, the notes help you check: were the test labels made by the same person, on the same day, from the same scene as the training labels? If so, the score is optimistic.

What to try next

  • Pick one scene in QGIS and label the same area twice: once with points, once with polygons. Compare the time it took and the detail you kept.
  • Count your classes and write the numbers down. If one class is tiny, plan how you will sample more of it.
  • Plot your label locations on a map and mark the regions you want to predict but have no labels for.
  • Ask a colleague to label 20 examples using only your written rules. Note where you disagree.
  • Create a one-page label documentation file for your current project.

Next in the series: Spatial Cross-Validation: Why Random Splits Overstate Accuracy.

More materials (some in Korean) are on GeoAI for Disaster Safety.

This post was drafted with the help of AI tools and checked before publishing. Tool features and data portals change over time, so check the official documentation when you practice.

Comments

Popular posts from this blog

What Is GeoAI, and How Does It Help Disaster Safety?

Getting Started with QGIS: What to Do First

Whitebox QGIS Practice 26 — Elevation Statistics of Two Basins: Means of 2,000.1 m and 1,924.3 m, a 76 m Gap