Coming from a more algorithms background, all the ResNet variants seem to me as just "lego playing". There seems to not be much intuition behind choices, they are only shown to work in practice for some datasets and that's it. The core "residual" idea is the one that really stands out to me since 2015.

If you were to rank what really is the "breakthrough" here, what would it be?

Thanks