Neural Architecture Search via Proxy Validation

This paper searches for the optimal neural architecture by minimizing a proxy of validation loss. Existing neural architecture search (NAS) methods used to discover the optimal neural architecture that best fits the validation examples given the up-to-date network weights. These intermediate validation results are invaluable but have not been fully explored. We propose to approximate the validation loss landscape by learning a mapping from neural architectures to their corresponding validate losses. The optimal neural architecture thus can be easily identified as the minimum of this proxy validation loss landscape. To improve the efficiency, a novel architecture sampling strategy is developed for the approximation of the proxy validation loss landscape. We also propose an operation importance weight (OIW) to balance the randomness and certainty of architecture sampling. The representation of neural architecture is learned through a graph autoencoder (GAE) over both architectures sampled during search and randomly generated architectures. We provide theoretical analyses on the validation loss estimator learned with our sampling strategy. Experimental results demonstrate that the proposed proxy validation loss landscape can be effective in both the differentiable NAS and the evolutionary-algorithm-based (EA-based) NAS.

Automated summary

This paper proposes a method to search for the optimal neural architecture by minimizing a proxy of validation loss. It approximates the validation loss landscape by learning a mapping from neural architectures to their corresponding validate losses, allowing for easy identification of the optimal neural architecture. A novel architecture sampling strategy and an operation importance weight are developed to improve efficiency and balance randomness and certainty in architecture sampling. Experimental results demonstrate the effectiveness of the proposed method in both differentiable NAS and EA-based NAS.