Search Results for author:
Found 33 papers, 19 papers with code
A Recipe for Scaling up Text-to-Video Generation with Text-free Videos
Following such a pipeline, we study the effect of doubling the scale of training set (i. e., video-only WebVid10M) with some randomly collected text-free videos and are encouraged to observe the performance improvement (FID from 9. 67 to 8. 19 and FVD from 484 to 441), demonstrating the scalability of our approach.
Ranked #6 on
Text-to-Video Generation
on MSR-VTT
InstructVideo: Instructing Video Diffusion Models with Human Feedback
To tackle this problem, we propose InstructVideo to instruct text-to-video diffusion models with human feedback by reward fine-tuning.
AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis
Audio-driven talking head synthesis is a promising topic with wide applications in digital human, film making and virtual reality.
DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models
In this work, we propose a DreamTalk framework to fulfill this gap, which employs meticulous design to unlock the potential of diffusion models in generating expressive talking heads.
VideoLCM: Video Latent Consistency Model
Consistency models have demonstrated powerful capability in efficient image generation and allowed synthesis within a few sampling steps, alleviating the high computational cost in diffusion models.
Hierarchical Spatio-temporal Decoupling for Text-to-Video Generation
At the structure level, we decompose the T2V task into two steps, including spatial reasoning and temporal reasoning, using a unified denoiser.
Ranked #5 on
Text-to-Video Generation
on MSR-VTT
DreamVideo: Composing Your Dream Videos with Customized Subject and Motion
In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern.
I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models
By this means, I2VGen-XL can simultaneously enhance the semantic accuracy, continuity of details and clarity of generated videos.
Few-shot Action Recognition with Captioning Foundation Models
In this paper, we develop an effective plug-and-play framework called CapFSAR to exploit the knowledge of multimodal models without manually annotating text.
DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing
Based on DeltaSpace, we propose a novel framework called DeltaEdit, which maps the CLIP visual feature differences to the latent space directions of a generative model during the training phase, and predicts the latent space directions from the CLIP textual feature differences during the inference phase.
Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning
When pre-training on the large-scale Kinetics-710, we achieve 89. 7% on Kinetics-400 with a frozen ViT-L model, which verifies the scalability of DiST.
RLIPv2: Fast Scaling of Relational Language-Image Pre-training
In this paper, we propose RLIPv2, a fast converging model that enables the scaling of relational pre-training to large-scale pseudo-labelled scene graph data.
Ranked #1 on
Zero-Shot Human-Object Interaction Detection
on HICO-DET
(using extra training data)
ModelScope Text-to-Video Technical Report
This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i. e., Stable Diffusion).
Ranked #8 on
Text-to-Video Generation
on MSR-VTT
Temporally-Adaptive Models for Efficient Video Understanding
Spatial convolutions are extensively used in numerous deep video models.
Ranked #3 on
Action Recognition
on EPIC-KITCHENS-100
(using extra training data)
CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation
In this way, RGB images are not required during inference anymore since the 2D knowledge branch provides 2D information according to the 3D LIDAR input.
Freestyle 3D-Aware Portrait Synthesis Based on Compositional Generative Priors
Efficiently generating a freestyle 3D portrait with high quality and 3D-consistency is a promising yet challenging task.
VideoComposer: Compositional Video Synthesis with Motion Controllability
The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis.
Ranked #5 on
Text-to-Video Generation
on EvalCrafter Text-to-Video (ECTV) Dataset
(using extra training data)
MoLo: Motion-augmented Long-short Contrastive Learning for Few-shot Action Recognition
To address these issues, we develop a Motion-augmented Long-short Contrastive Learning (MoLo) method that contains two crucial components, including a long-short contrastive objective and a motion autodecoder.
Enlarging Instance-specific and Class-specific Information for Open-set Action Recognition
In this paper, we begin with analyzing the feature representation behavior in the open-set action recognition (OSAR) problem based on the information bottleneck (IB) theory, and propose to enlarge the instance-specific (IS) and class-specific (CS) information contained in the feature for better performance.
VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation
A diffusion probabilistic model (DPM), which constructs a forward diffusion process by gradually adding noise to data points and learns the reverse denoising process to generate new samples, has been shown to handle complex data distribution.
Ranked #7 on
Video Generation
on UCF-101
CLIP-guided Prototype Modulating for Few-shot Action Recognition
Learning from large-scale contrastive language-image pre-training like CLIP has shown remarkable success in a wide range of downstream tasks recently, but it is still under-explored on the challenging few-shot action recognition (FSAR) task.
The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognition
Recently, Unified Open-set Recognition (UOSR) has been proposed to reject not only unknown samples but also known but wrongly classified samples, which tends to be more practical in real-world applications.
LipFormer: High-Fidelity and Generalizable Talking Face Generation With a Pre-Learned Facial Codebook
Generating a talking face video from the input audio sequence is a practical yet challenging task.
Space-time Prompting for Video Class-incremental Learning
In this paper, we will fill this gap by learning multiple prompts based on a powerful image-language pre-trained model, i. e., CLIP, making it fit for video class-incremental learning (VCIL).
Revisiting Optimal Convergence Rate for Smooth and Non-convex Stochastic Decentralized Optimization
While (Lu and Sa, 2021) have recently provided an optimal rate for non-convex stochastic decentralized optimization with weight matrices defined over linear graphs, the optimal rate with general weight matrices remains unclear.
Communicate Then Adapt: An Effective Decentralized Adaptive Method for Deep Training
Decentralized adaptive gradient methods, in which each node averages only with its neighbors, are critical to save communication and wall-clock training time in deep learning tasks.
Communication Efficient SGD via Gradient Sampling With Bayes Prior
Different from all of them, we regard large and small gradients selection as the exploitation and exploration of gradient information, respectively.
Accelerating Gossip SGD with Periodic Global Averaging
Communication overhead hinders the scalability of large-scale distributed training.
DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training
Experimental results on a variety of computer vision tasks and models demonstrate that DecentLaM promises both efficient and high-quality training.
Visual Search at Alibaba
We hope visual search at Alibaba becomes more widely incorporated into today's commercial applications.
Large-Scale Training System for 100-Million Classification at Alibaba
In the last decades, extreme classification has become an essential topic for deep learning.
Large-Scale Visual Search with Binary Distributed Graph at Alibaba
For a deployed visual search system with several billions of online images in total, building a billion-scale offline graph in hours is essential, which is almost unachievable by most existing methods.
Distribution Adaptive INT8 Quantization for Training CNNs
Researches have demonstrated that low bit-width (e. g., INT8) quantization can be employed to accelerate the inference process.
