AI Rank
EN
← 开源
jeffreyyihuang

two-stream-action-recognition

Using two stream architecture to implement a classic action recognition method on UCF101 dataset

Model DevelopmentArchitecturePython
在 GitHub 打开
增长势头
+024 小时新增 Star0.0%
874
Star
246
Fork
+0
本周
2
贡献者
创建于 2017-07-26 · 更新于 2026-10-09 · 今日第 12998 名
主要开发者
jeffreyyihuangrajats643

jeffreyyihuang/two-stream-action-recognition 排名第 12,998，有 874 个星标。

同类别比较

jeffreyyihuang/two-stream-action-recognition 排名第 12,998，此处有 8 个同类条目；allenai/XNOR-Net 排名第 12,995。

同类别比较
名称排名指标
polarisZhao/PFLD-pytorch12,984
语言:
Python
星标数:
876
分叉数:
205
简介:
PFLD pytorch Implementation
lucidrains/mixture-of-experts12,989
语言:
Python
星标数:
875
分叉数:
71
简介:
A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models
sbi-dev/sbi12,990
语言:
Python
星标数:
875
分叉数:
272
简介:
sbi is a Python package for simulation-based inference, designed to meet the needs of both researchers and practitioners. Whether you need fine-grained control or an easy-to-use interface, sbi has you covered.
allenai/XNOR-Net12,995
语言:
Lua
星标数:
874
分叉数:
235
简介:
ImageNet classification using binary Convolutional Neural Networks
Newbeeer/Poisson_flow13,001
语言:
Python
星标数:
873
分叉数:
56
简介:
Code for NeurIPS 2022 Paper, "Poisson Flow Generative Models" (PFGM)
shubhomoydas/ad_examples13,009
语言:
Python
星标数:
873
分叉数:
180
简介:
A collection of anomaly detection methods (iid/point-based, graph and time series) including active learning for anomaly detection/discovery, bayesian rule-mining, description for diversity/explanation/interpretability. Analysis of incorporating label feedback with ensemble and tree-based detectors. Includes adversarial attacks with Graph Convolutional Network.
wangyuxinwhy/uniem13,011
语言:
Python
星标数:
873
分叉数:
72
简介:
unified embedding model
zhmiao/OpenLongTailRecognition-OLTR13,013
语言:
Python
星标数:
873
分叉数:
130
简介:
Pytorch implementation for "Large-Scale Long-Tailed Recognition in an Open World" (CVPR 2019 ORAL)
关键参数与规格
关键参数与规格
参数值
语言Python
类别Model Development
子类别Architecture
星标数874
分叉数246
贡献者2
创建日期2017-07-26
更新日期2026-10-09
1 天星标数0
7 天星标数0
排名12,998
上次排名12,891
排名变化-107
7 天星标排名13,713
上次 7 天星标排名17,455
7 天星标排名变化3,742
总星标排名12,394
上次总星标排名12,368
总星标排名变化-26
国家USA
README 摘录

two-stream-action-recognition We use a spatial and motion stream cnn with ResNet101 for modeling video information in UCF101 dataset. Reference Paper [[1] Two-stream convolutional networks for action recognition in videos]( [[2] Temporal Segment Networks: Towards Good Practices for Deep Action Recognition]( [[3] TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition]( 1. 1 Spatial input data -> rgb frames We extract RGB frames from each video in UCF101 dataset with sampling rate: 10 and save as

. 2 Motion input data -> stacked optical flow images In motion stream, we use two methods to get optical flow data. Download the preprocessed tvl1 optical flow dataset directly from 2. 0 method to generate 2-channel optical flow image and save its x, y channel as . jpg image in disk respectively, which cost about 56G. 3 (Alternative)Download the preprocessed data directly from feichtenhofer/twostreamfusion) RGB images Optical Flow 2. 1 Spatial cnn As mention before, we use ResNet101 first

pre-trained with ImageNet then fine-tuning on our UCF101 spatial rgb image dataset. 2 Motion cnn Input data of motion cnn is a stack of optical flow images which contained 10 x-channel and 10 y-channel images, So it's input shape is (20, 224, 224) which can be considered as a 20-channel image. In order to utilize ImageNet pre-trained weight on our model, we have to modify the weights of the first convolution layer pre-trained

with ImageNet from (64, 3, 7, 7) to (64, 20, 7, 7). In [2] Wang provide a method called **Cross modality pre- ** to do such weights shape transform. He first average the weight value across the RGB channels and replicate this average by the channel number of motion stream input( which is 20 is this case) 3. 1 Spatial cnn Here we utilize the techniques in Temporal Segment Network. For every videos in a mini-batch, we randomly select 3

frames from each video. Then a consensus among the frames will be derived as the video-level prediction for calculating loss. 2 Motion cnn In every mini-batch, we randomly select 64 (batch size) videos from 9537 training videos and futher randomly select 1 stacked optical flow in each video. 3 Data augmentation Both stream apply the same data augmentation technique such as random cropping. Testing method For every 3783 testing videos, we uniformly sample 19 frames in each video

and the video level prediction is the voting result of all 19 frame level predictions. The reason we choose the number 19 is that the minimun number of video frames in UCF101 is 28 and we have to make sure there are sufficient frames for testing in 10 stack motion stream. Pre-trained Model Spatial resnet101 Motion resnet101 7. Testing on Your Device Spatial stream Please modify this path and this funcition to fit the UCF101 dataset on your device

. Training and testing Only testing Motion stream Please modify this path and this funcition to fit the UCF101 dataset on

常见问题

记录了多少星标和分叉？

jeffreyyihuang/two-stream-action-recognition 有 874 个星标和 246 个分叉。

仓库排名是多少？

jeffreyyihuang/two-stream-action-recognition 排名第 12,998。

记录的语言和类别是什么？

jeffreyyihuang/two-stream-action-recognition 有 874 个星标，语言为 Python，类别为 Model Development。

哪个仓库排名最接近？

jeffreyyihuang/two-stream-action-recognition 与 allenai/XNOR-Net 比较，后者排名第 12,995，有 874 个星标。

README

two-stream-action-recognition

We use a spatial and motion stream cnn with ResNet101 for modeling video information in UCF101 dataset.

Reference Paper

1. Data

1.1 Spatial input data -> rgb frames

  • We extract RGB frames from each video in UCF101 dataset with sampling rate: 10 and save as .jpg image in disk which cost about 5.9G.

1.2 Motion input data -> stacked optical flow images

In motion stream, we use two methods to get optical flow data.

  1. Download the preprocessed tvl1 optical flow dataset directly from https://github.com/feichtenhofer/twostreamfusion.
  2. Using flownet2.0 method to generate 2-channel optical flow image and save its x, y channel as .jpg image in disk respectively, which cost about 56G.

1.3 (Alternative)Download the preprocessed data directly from feichtenhofer/twostreamfusion)

  • RGB images
wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.001
wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.002
wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_jpegs_256.zip.003

cat ucf101_jpegs_256.zip* > ucf101_jpegs_256.zip
unzip ucf101_jpegs_256.zip
  • Optical Flow
wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.001
wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.002
wget http://ftp.tugraz.at/pub/feichtenhofer/tsfusion/data/ucf101_tvl1_flow.zip.003

cat ucf101_tvl1_flow.zip* > ucf101_tvl1_flow.zip
unzip ucf101_tvl1_flow.zip

2. Model

2.1 Spatial cnn

  • As mention before, we use ResNet101 first pre-trained with ImageNet then fine-tuning on our UCF101 spatial rgb image dataset. 

2.2 Motion cnn

  • Input data of motion cnn is a stack of optical flow images which contained 10 x-channel and 10 y-channel images, So it's input shape is (20, 224, 224) which can be considered as a 20-channel image.
  • In order to utilize ImageNet pre-trained weight on our model, we have to modify the weights of the first convolution layer pre-trained with ImageNet from (64, 3, 7, 7) to (64, 20, 7, 7).
  • In [2] Wang provide a method called **Cross modality pre-

** to do such weights shape transform. He first average the weight value across the RGB channels and replicate this average by the channel number of motion stream input( which is 20 is this case)

3. Training strategies

3.1 Spatial cnn

  • Here we utilize the techniques in Temporal Segment Network. For every videos in a mini-batch, we randomly select 3 frames from each video. Then a consensus among the frames will be derived as the video-level prediction for calculating loss.

3.2 Motion cnn

  • In every mini-batch, we randomly select 64 (batch size) videos from 9537 training videos and futher randomly select 1 stacked optical flow in each video.

3.3 Data augmentation

  • Both stream apply the same data augmentation technique such as random cropping.

4. Testing method

  • For every 3783 testing videos, we uniformly sample 19 frames in each video and the video level prediction is the voting result of all 19 frame level predictions.
  • The reason we choose the number 19 is that the minimun number of video frames in UCF101 is 28 and we have to make sure there are sufficient frames for testing in 10 stack motion stream.

5. Performace

network top1
Spatial cnn 82.1%
Motion cnn 79.4%
Average fusion 88.5%

6. Pre-trained Model

7. Testing on Your Device

Spatial stream

  • Please modify this path and this funcition to fit the UCF101 dataset on your device.
  • Training and testing
python spatial_cnn.py --resume PATH_TO_PRETRAINED_MODEL
  • Only testing
python spatial_cnn.py --resume PATH_TO_PRETRAINED_MODEL --evaluate

Motion stream

  • Please modify this path and this funcition to fit the UCF101 dataset on your device.
  • Training and testing
python motion_cnn.py --resume PATH_TO_PRETRAINED_MODEL
  • Only testing
python motion_cnn.py --resume PATH_TO_PRETRAINED_MODEL --evaluate
排名相邻
同类项目