Green Deep Learning: Sustainable Neural Architecture Search and Vocabulary Optimization

In the past few years, pre‑trained models have become ubiquitous, leading to explosive computational demands; Green Deep Learning is proposed as a new direction to make AI more environmentally friendly.

1. What is Green Deep Learning? Its goal is to enable AI models to achieve better or comparable results with lower environmental impact, either by reducing computation or by improving efficiency without increasing compute, encouraging alternatives to simply scaling data and model size.

2. Benefits Smaller, greener models are easier to deploy in industry, allow more researchers with limited resources to participate, and can still deliver strong performance, fostering a healthier research ecosystem.

3. Green Neural Architecture Search (NAS) Traditional NAS requires costly training on downstream tasks to evaluate architectures. Training‑free Green NAS evaluates candidate structures without full training, using metrics such as the MGM kernel that capture gradient‑based relationships among samples.

Traditional NAS is computationally intensive, especially on large datasets. The proposed Training‑free Green NAS estimates performance by analyzing the MGM kernel, dramatically reducing evaluation cost.

The method includes three steps: (1) select architectures with top‑k MGM kernels, (2) evaluate them, and (3) choose the best trade‑off between efficiency and accuracy. Experiments show comparable accuracy to baselines with significant speed‑up.

4. Green Vocabulary Learning Vocabulary size and entropy (Information‑Per‑Char) critically affect model efficiency. Smaller vocabularies reduce parameters, while lower entropy reduces ambiguity, making models easier to train.

By defining a marginal benefit metric that balances vocabulary size (cost) against IPC (benefit), the optimal vocabulary can be found via discrete optimization such as Optimal Transport.

Experiments on bilingual and multilingual machine translation show that the proposed VOLT method reduces vocabulary size substantially while maintaining or improving BLEU scores, demonstrating the practicality of green vocabulary optimization.

In the Q&A session, the speaker clarified that VOLT is language‑agnostic and can be applied to Chinese tokenization, recommending SentencePiece/BPE as a practical implementation while using VOLT to determine optimal vocabulary size.

Overall, Green Deep Learning offers sustainable strategies for model architecture search and vocabulary design, enabling efficient AI development without sacrificing performance.