Knowledge Distillation and Structured Pruning at Matched Footprint for Sub-50 KB Cattle Behaviour Classification

Abstract

On-collar behaviour classification underpins automated oestrus and health monitoring in cattle, but accurate models exceed the memory a low-cost collar can afford. Knowledge distillation and structured pruning are both proposed as the remedy, but on-collar studies test them separately and without the control that makes both interpretable: a same-size network trained from hard labels. Using two public neck-accelerometer datasets of cattle, 6 Japanese Black beef cows (the beef dataset) and 30 pre-weaned calves (the calf dataset), a convolutional teacher is trained under leave-one-animal-out validation and compressed along three routes at a matched int8 footprint, with the frontier re-derived on the calf dataset. First, the released beef dataset contains cross-animal duplication: all 15 animal pairs share byte-identical blocks of varying signal, and leaving it in place inflates macro F1 by +0.076 even when each animal is scored only on the classes it shows (p = 0.0007). Second, the capacity deficit is almost nil: a 1.72 KB network trained from hard labels reaches 0.681 macro F1 against the 127.1 KB teacher’s 0.692, which caps what compression can be worth. Within that ceiling, neither route is reliably better than same-size training. Structured pruning gains +0.0130 at one seed (p = 0.0096, n = 42) but −0.0209 to +0.0145 over five seeds, collapsing for some animals at the smallest widths at three seeds. Fine-tuning it on the same schedule as the other routes removes most collapses and the margin with them (−0.0080 to +0.0038 over three seeds, never significant). Distillation gains are +0.0021 to +0.0086, significant at one seed. Third, on calves, where shrinking costs up to +0.077, pruning falls below same-size training at five of five seeds, significantly at two, so no ordering here is a property of the methods.

Authors 4

1. Bright Mutuma Gitari

   Dedan Kimathi University of Technology

   Affiliation as printed

   Department of Mechatronic Engineering, School of Engineering, Dedan Kimathi University of Technology, P.O. Box 657-10100, Nyeri, Kenya

2. Ying-Hsuan Chiang Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Templergraben 55, 52056 Aachen, Germany

3. Consolata Wanjiru Kiiru

   Jomo Kenyatta University of Agriculture and Technology

   Affiliation as printed

   Jomo Kenyatta University of Agriculture and Technology, Kenya

4. Samuel Macharia Kang’iri

   Dedan Kimathi University of Technology

   Affiliation as printed

   Department of Mechatronic Engineering, School of Engineering, Dedan Kimathi University of Technology, P.O. Box 657-10100, Nyeri, Kenya