no code implementations • 3 Feb 2024 • Peijie Dong, Lujun Li, Xinglin Pan, Zimian Wei, Xiang Liu, Qiang Wang, Xiaowen Chu
Recent advancements in Zero-shot Neural Architecture Search (NAS) highlight the efficacy of zero-cost proxies in various NAS benchmarks.
1 code implementation • 14 Dec 2023 • Zimian Wei, Lujun Li, Peijie Dong, Zheng Hui, Anggeng Li, Menglong Lu, Hengyue Pan, Zhiliang Tian, Dongsheng Li
Based on the discovered zero-cost proxy, we conduct a ViT architecture search in a training-free manner.
no code implementations • 24 Nov 2023 • Zimian Wei, Hengyue Pan, Lujun Li, Peijie Dong, Zhiliang Tian, Xin Niu, Dongsheng Li
In this paper, for the first time, we investigate how to search in a training-free manner with the help of teacher models and devise an effective Training-free ViT (TVT) search framework.
1 code implementation • ICCV 2023 • Peijie Dong, Lujun Li, Zimian Wei, Xin Niu, Zhiliang Tian, Hengyue Pan
In particular, we devise an elaborate search space involving the existing proxies and perform an evolution search to discover the best correlated MQ proxy.
1 code implementation • 23 May 2023 • Xiaolong Liu, Lujun Li, Chao Li, Anbang Yao
By sequentially splitting the expanded student representation into N non-overlapping feature segments having the same number of feature channels as the teacher's, they can be readily forced to approximate the intact teacher representation simultaneously, formulating a novel many-to-one representation matching mechanism conditioned on a single teacher-student layer pair.
1 code implementation • 18 May 2023 • Shitong Shao, Xu Dai, Shouyi Yin, Lujun Li, Huanran Chen, Yang Hu
On CIFAR-10, we obtain a FID of 2. 80 by sampling in 15 steps under one-session training and the new state-of-the-art FID of 3. 37 by sampling in one step with additional training.
no code implementations • CVPR 2023 • Peijie Dong, Lujun Li, Zimian Wei
In this way, our student architecture search for Distillation WithOut Training (DisWOT) significantly improves the performance of the model in the distillation stage with at least 180$\times$ training acceleration.
no code implementations • 24 Jan 2023 • Peijie Dong, Xin Niu, Zhiliang Tian, Lujun Li, Xiaodong Wang, Zimian Wei, Hengyue Pan, Dongsheng Li
Practical networks for edge devices adopt shallow depth and small convolutional kernels to save memory and computational cost, which leads to a restricted receptive field.
1 code implementation • 24 Jan 2023 • Peijie Dong, Xin Niu, Lujun Li, Zhiliang Tian, Xiaodong Wang, Zimian Wei, Hengyue Pan, Dongsheng Li
In this paper, we propose Ranking Distillation one-shot NAS (RD-NAS) to enhance ranking consistency, which utilizes zero-cost proxies as the cheap teacher and adopts the margin ranking loss to distill the ranking knowledge.
no code implementations • 23 Jan 2023 • Kunlong Chen, Liu Yang, Yitian Chen, Kunjin Chen, Yidan Xu, Lujun Li
It is of great significance to estimate the performance of a given model architecture without training in the application of Neural Architecture Search (NAS) as it may take a lot of time to evaluate the performance of an architecture.
no code implementations • ICCV 2023 • Lujun Li, Peijie Dong, Zimian Wei, Ya Yang
In this paper, we present Auto-KD, the first automated search framework for optimal knowledge distillation design.
no code implementations • 16 Sep 2022 • Zimian Wei, Hengyue Pan, Lujun Li, Menglong Lu, Xin Niu, Peijie Dong, Dongsheng Li
Vision transformers have shown excellent performance in computer vision tasks.
1 code implementation • 27 Jun 2022 • Peijie Dong, Xin Niu, Lujun Li, Linzhen Xie, Wenbin Zou, Tian Ye, Zimian Wei, Hengyue Pan
In this paper, we present Prior-Guided One-shot NAS (PGONAS) to strengthen the ranking correlation of supernets.
no code implementations • 8 Mar 2022 • Zimian Wei, Hengyue Pan, Lujun Li, Menglong Lu, Xin Niu, Peijie Dong, Dongsheng Li
Neural architecture search (NAS) has brought significant progress in recent image recognition tasks.
1 code implementation • 16 Dec 2021 • Jie Qin, Jie Wu, Xuefeng Xiao, Lujun Li, Xingang Wang
Extensive experiments show that AMR establishes a new state-of-the-art performance on the PASCAL VOC 2012 dataset, surpassing not only current methods trained with the image-level of supervision but also some methods relying on stronger supervision, such as saliency label.
no code implementations • 3 Apr 2021 • Lujun Li, Yikai Kang, Yuchen Shi, Ludwig Kürzinger, Tobias Watzel, Gerhard Rigoll
Inspired by the extensive applications of the generative adversarial networks (GANs) in speech enhancement and ASR tasks, we propose an adversarial joint training framework with the self-attention mechanism to boost the noise robustness of the ASR system.
Automatic Speech Recognition Automatic Speech Recognition (ASR) +2
no code implementations • 1 Jan 2021 • Lujun Li, Yikai Wang, Anbang Yao, Yi Qian, Xiao Zhou, Ke He
In this paper, we present Explicit Connection Distillation (ECD), a new KD framework, which addresses the knowledge distillation problem in a novel perspective of bridging dense intermediate feature connections between a student network and its corresponding teacher generated automatically in the training, achieving knowledge transfer goal via direct cross-network layer-to-layer gradients propagation, without need to define complex distillation losses and assume a pre-trained teacher model to be available.
11 code implementations • 17 Jul 2020 • Ludwig Kürzinger, Dominik Winkelbauer, Lujun Li, Tobias Watzel, Gerhard Rigoll
In this work, we combine freely available corpora for German speech recognition, including yet unlabeled speech data, to a big dataset of over $1700$h of speech data.
Ranked #5 on Speech Recognition on TUDA (using extra training data)
Speech Recognition Audio and Speech Processing
no code implementations • 15 Jun 2020 • Tobias Watzel, Ludwig Kürzinger, Lujun Li, Gerhard Rigoll
Nowadays, attention models are one of the popular candidates for speech recognition.