Journal
IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS
Volume 33, Issue 12, Pages 7172-7184Publisher
IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC
DOI: 10.1109/TNNLS.2021.3084319
Keywords
Training; Analytical models; Data models; Task analysis; Smoothing methods; Degradation; Aggregates; Deep graph neural networks; graph data mining; graph normalization; graph representation learning
Categories
Funding
- National Natural Science Foundation of China [62036006]
- Innovation Fund of Shanghai Aerospace Science and Technology [SAST2019-090]
- National Key Research and Development Program of China [2017YFB0802200]
Ask authors/readers for more resources
The paper introduces a deep hierarchical layer aggregation (DHLA) strategy and neighbor normalization (NeighborNorm) strategy for addressing the training difficulties in deep MPNNs. Experimental results demonstrate the necessity and effectiveness of these proposed strategies for graph message-passing neural networks.
As a unified framework for graph neural networks, message passing-based neural network (MPNN) has attracted a lot of research interest and has been shown successfully in a number of domains in recent years. However, because of over-smoothing and vanishing gradients, deep MPNNs are still difficult to train. To alleviate these issues, we first introduce a deep hierarchical layer aggregation (DHLA) strategy, which utilizes a block-based layer aggregation to aggregate representations from different layers and transfers the output of the previous block to the subsequent block, so that deeper MPNNs can be easily trained. Additionally, to stabilize the training process, we also develop a novel normalization strategy, neighbor normalization (NeighborNorm), which normalizes the neighbor of each node to further address the training issue in deep MPNNs. Our analysis reveals that NeighborNorm can smooth the gradient of the loss function, i.e., adding NeighborNorm makes the optimization landscape much easier to navigate. Experimental results on two typical graph pattern-recognition tasks, including node classification and graph classification, demonstrate the necessity and effectiveness of the proposed strategies for graph message-passing neural networks.
Authors
I am an author on this paper
Click your name to claim this paper and add it to your profile.
Reviews
Recommended
No Data Available