Task overfitting and the memorisation problem

Task overfitting and the memorisation problem

For meta-learning to work, the sampled tasks must be mutually exclusive such that no single model can solve all tasks. In some cases where the task is identifiable (i.e. the classification labels are not shuffled), the meta-learning algorithm may simply discard the context set and memorise the predictions to minimise the error. In other cases, such as predicting the pose of an object from an image, if the task can be solved from the target inputs alone then the algorithm may minimise the loss while discarding the context set by memorising a function that solves all of the tasks. This results in a zero-shot model that makes average decisions for all tasks without adaptation, and thus poor performance on novel tasks.

Information theory to the rescue

The idea proposed by et al is to reduce the information flow between x* and y*, such that the algorithm is forced to use the information in D.

Widening the task distribution