Yeah, I totally agreed. I guess this approach only suits some models. It’s just in the paper they achieved really great result on AmoebaNet, in terms of both memory usage and training time, which made me doubt my result.