Train YOLOv8 on Custom Dataset With Real Metrics and Lessons

To train YOLOv8 on custom dataset images, label them in YOLO format, describe the folders in a dataset YAML file, start from a pretrained nano or small model and call model.train(data="data.yaml", epochs=30, imgsz=640). Then read results.png and the precision, recall and mAP numbers to decide what to fix. In our own project the best run reached 0.84 precision but only 0.50 recall, so the next move was to collect more varied images.

How do you prepare a custom dataset for YOLOv8?

The pipeline is short: collect, label, split, describe, train, validate, repeat. We went through it in 2024 for a computer-vision research project that detects characters in video-game footage. Game footage is a handy test bed because frames are cheap to collect and scenes change constantly. We trained two detection models on personal datasets, and each dataset went through several versions, from under 100 images to several thousand, as error analysis showed what the model was missing. The best run used 719 training images and 70 validation images.

Ultralytics lists these dataset rules in its training tips:

• Use images that look like real deployment: different times of day, lighting, angles and sources.
• Label every instance of every class, with boxes that enclose each object tightly.
• Aim for at least 1,500 images and 10,000 labelled instances per class.
• Add about 0–10% background images with no objects, which helps reduce false positives.

For scale, 719 training images sits below that recommendation, and that matters later in this article. Labels use one text file per image, with one line per object in the form class x_center y_center width height, and all coordinates normalised to 0–1. Put images and labels in matching folders, then point a YAML file at them, as described in the detection dataset format docs:

dataset_root/
  images/train   images/val
  labels/train   labels/val

# data.yaml
path: dataset_root
train: images/train
val: images/val
names:
  0: character

Add one line under names for each further class.

Which YOLOv8 model size and training settings should you use?

YOLOv8 comes in five sizes, n, s, m, l and x. The two smallest are the usual starting points for real-time work:

Model	Pretrained weights	Parameters	COCO mAP (Ultralytics)
YOLOv8n (nano)	yolov8n.pt	3.2M	37.3
YOLOv8s (small)	yolov8s.pt	11.2M	44.9

Always start from pretrained weights. Ultralytics recommends it, and it is faster than training from scratch on a small dataset. We trained both nano and small variants, and our best run used the small model. This is the whole training call:

from ultralytics import YOLO

model = YOLO("yolov8s.pt")                         # pretrained start
model.train(data="data.yaml", epochs=30, imgsz=640)

# CLI equivalent
# yolo detect train data=data.yaml model=yolov8s.pt epochs=30 imgsz=640

The Ultralytics defaults are 100 epochs, 640 pixels, batch size 16 and early stopping after 100 epochs without improvement. Our best run used 30 epochs at 640 px, which is shorter than the default, so treat our numbers as a single run. Ultralytics notes that higher resolutions such as 1280 help datasets with small objects. Each run writes to runs/detect/train: weights/best.pt, weights/last.pt, results.png and results.csv. Current Ultralytics docs use newer model names, such as yolo26n.pt, in the same commands, so the workflow carries over when you swap the weights file.

How do you read results.png and the YOLOv8 metrics?

results.png plots training and validation losses plus precision, recall and mAP for every epoch. The pattern to catch is training loss that keeps falling while validation loss starts to rise, which signals overfitting. For the final numbers, use the validation metrics. Ultralytics defines them in its performance metrics guide, and here they are next to the best run of our project (YOLOv8s, 30 epochs, 640 px, 70 validation images):

Metric	What it measures	Our best run
Precision	How many detections were correct	0.84
Recall	How many of the real objects were found	0.50
mAP@50	Mean average precision at an IoU threshold of 0.50	0.59
mAP@50-95	Average of mAP over IoU thresholds from 0.50 to 0.95	0.36

IoU (intersection over union) is the overlap between a predicted box and the labelled box. At 0.50 a loose box still counts as a hit. At 0.95 only a near-perfect box does. Running validation also saves an F1 curve, a precision-recall curve and a confusion matrix, which show where the model confuses classes or misses objects.

What does low recall mean when precision is high?

Precision 0.84 means that when the model draws a box, it is usually right. Recall 0.50 means it finds about half of the labelled objects and misses the other half. The two are tied to the confidence threshold: raising it removes shaky detections, so precision goes up and recall goes down. Compare runs at the same threshold. As plain arithmetic, the harmonic mean of 0.84 and 0.50 is about 0.63. Ultralytics describes the pattern directly: when recall is the weak number, the model may be missing real objects, and more data can help.

The gap between mAP@50 (0.59) and mAP@50-95 (0.36) tells a second story. Boxes overlap the objects well enough at 0.50 IoU, but the score falls as the required overlap tightens, so localisation is looser than detection. One more caution about our numbers: 70 validation images is a small set, so a few hard images move recall visibly. Treat 0.50 as a rough position.

Why does more varied data beat a bigger model?

When recall is the limit, the usual cause is that the model never saw the situations it misses: a different distance, lighting, partial occlusion, another background. A bigger model has more capacity, but it learns from the same 719 images and can memorise them faster. Variety adds the missing situations. That was our conclusion from this run: precision was good, recall was the limit, and the next step was more varied training data before any larger model. It also matches the Ultralytics guidance above, which asks for varied images and at least 1,500 per class, more than double our 719 in total.

The loop we used, and the one we suggest:

1. Run validation and open the validation images where the model missed objects or drew wrong boxes.
2. Group the misses by cause: distance, light, blocked view, background, unusual pose.
3. Collect and label more images for the biggest group.
4. Retrain and compare on the same, untouched validation set.
5. Change model size only when added data stops moving the numbers.

Keep the validation set fixed and out of training, or every comparison becomes meaningless. It is the same habit as the fixed evaluation set in our LLM cost work. Read this as a priority order for experiments. We report one run here, and the advice to add data first is a cheap hypothesis to test on your own project.

How do you run YOLOv8 in real time on screen capture?

Real-time detection is a loop: grab a frame, run the model, draw the result. Ultralytics supports numpy frames (BGR, HWC, uint8) and a screen source, and the predict mode docs list the settings that matter here. conf defaults to 0.25 and sets the minimum confidence for a detection. imgsz defaults to 640. vid_stride skips frames in video inputs to speed up processing at the cost of temporal resolution.

Our loop applied the same two ideas by hand: skip frames and filter by confidence. It drew the results in a transparent overlay. A minimal version looks like this, where frames() yields BGR numpy arrays from your capture code and draw_overlay() is yours to write:

from ultralytics import YOLO

model = YOLO("best.pt")
SKIP = 3                 # run the model on every 3rd frame
CONF = 0.4               # example value, tune it on your validation images
boxes = []

for i, frame in enumerate(frames()):
    if i % SKIP == 0:
        result = model(frame, conf=CONF, imgsz=640, verbose=False)[0]
        boxes = result.boxes.xyxy.tolist()
    draw_overlay(boxes)  # reuse the last boxes on skipped frames

A higher CONF gives fewer false boxes and more misses, and a higher SKIP gives more speed and staler boxes. A note on use: this is a vision pipeline, and capturing the screen of third-party software can be restricted by that software's terms. Analyse footage and applications you are allowed to analyse.

How do you speed up YOLOv8 inference with TensorRT?

On an NVIDIA GPU, export the trained weights to a TensorRT engine and load that file instead of the .pt file. The Ultralytics TensorRT guide shows the call:

from ultralytics import YOLO

YOLO("best.pt").export(format="engine")      # needs a CUDA-capable GPU
model = YOLO("best.engine")                  # then use it like any other model

Ultralytics warns that TensorRT engines are hardware- and runtime-specific. The engine is tuned on its build GPU, so build it on the GPU you will deploy to and match the TensorRT and CUDA runtime. We wrote a conversion script for this step in our project. Measure the speedup on your own hardware, since it depends on the GPU and the model size, and run validation on the exported engine to confirm that the metrics still match the .pt file.

What does Danube Labs do with computer vision?

This project started as founder research at Danube Labs: two custom-trained detection models, real-time inference and a measured result, including the weak number. The same discipline applies to client work. An AI project should begin with a feasibility check and a fixed evaluation set, so you know the recall you can expect before you commit budget. If you have images and a detection problem, book a free intro call.