CNN Architectures

<CNN Architectures>

    1. AlexNet

    2. VGGNet

    3. GoogLeNet

    4. ResNet

---

AlexNet

 

LeNet-5 [LeCun et al., 1998]

LeNet-5

 

• 5x5 conv filter (stride = 1) 사용
• 2x2 pooling (stride = 2) 사용
• architecture: [CONV - POOL - CONV - POOL - FC -FC]  

 

AlexNet [krizhevsky et al. 2012]

AlexNet

[227x227x3] INPUT

[55x55x96] CONV1: 96 11x11 filters at stride 4, pad 0

[27x27x96] MAX POOL1: 3x3 filters at stride 2

[27x27x96] NORM1: Normalization layer

[27x27x256] CONV2: 256 5x5 filters at stride 1, pad 2

[13x13x256] MAX POOL2: 3x3 filters at stride 2

[13x13x256] NORM2: Normalization layer

[13x13x384] CONV3: 384 3x3 filters at stride 1, pad 1

[13x13x384] CONV4: 384 3x3 filters at stride 1, pad 1

[13x13x256] CONV5: 256 3x3 filters at stride 1, pad 1

[6x6x256] MAX POOL3: 3x3 filters at stride 2

[4096] FC6: 4096 neurons

[4096] FC7: 4096 neurons

[1000] FC8: 1000 neurons (class scores)

 

• 최초로 ReLU activation function 사용
• Norm layers 사용 → 지금은 잘 안 쓰임
• 많은 data augmentation 사용
• dropout(p = 0.5) 사용
• batch size = 128
• SGD Momentum (0.9) 사용
• Learning rate = 1e-2 (val accuracy가 plateaus할 때마다 10 감소)
• L2 weight decay = 5e-4 사용
• 7 CNN ensemble 사용 (18.2% → 15.4%)

 

---

VGGNet

 

VGGNet [Simonyan and Zisserman, 2014]

VGGNet

Small filters, Deeper networks

 

INPUT: [224x224x3] memory: 224*224*3=150K / params: 0

CONV3-64: [224x224x64] memory: 224*224*64=3.2M / params: (3*3*3)*64 = 1,728

CONV3-64: [224x224x64] memory: 224*224*64=3.2M / params: (3*3*64)*64 = 36,864

POOL2: [112x112x64] memory: 112*112*64=800K / params: 0

CONV3-128: [112x112x128] memory: 112*112*128=1.6M / params: (3*3*64)*128 = 73,728

CONV3-128: [112x112x128] memory: 112*112*128=1.6M / params: (3*3*128)*128 = 147,456

POOL2: [56x56x128] memory: 56*56*128=400K / params: 0

CONV3-256: [56x56x256] memory: 56*56*256=800K / params: (3*3*128)*256 = 294,912

CONV3-256: [56x56x256] memory: 56*56*256=800K / params: (3*3*256)*256 = 589,824

CONV3-256: [56x56x256] memory: 56*56*256=800K / params: (3*3*256)*256 = 589,824

POOL2: [28x28x256] memory: 28*28*256=200K / params: 0

CONV3-512: [28x28x512] memory: 28*28*512=400K / params: (3*3*256)*512 = 1,179,648

CONV3-512: [28x28x512] memory: 28*28*512=400K / params: (3*3*512)*512 = 2,359,296

CONV3-512: [28x28x512] memory: 28*28*512=400K / params: (3*3*512)*512 = 2,359,296

POOL2: [14x14x512] memory: 14*14*512=100K / params: 0

CONV3-512: [14x14x512] memory: 14*14*512=100K / params: (3*3*512)*512 = 2,359,296

CONV3-512: [14x14x512] memory: 14*14*512=100K / params: (3*3*512)*512 = 2,359,296

CONV3-512: [14x14x512] memory: 14*14*512=100K / params: (3*3*512)*512 = 2,359,296

POOL2: [7x7x512] memory: 7*7*512=25K / params: 0

FC: [1x1x4096] memory: 4096 / params: 7*7*512*4096 = 102,760,448

FC: [1x1x4096] memory: 4096 / params: 4096*4096 = 16,777,216

FC: [1x1x1000] memory: 1000 / params: 4096*1000 = 4,096,000

 

TOTAL memory: 24M * 4 bytes ~= 96MB / image (only forward! ~*2 for bwd)

TOTAL params: 138M parameters

 

• 16 - 19 layers (AlexNet: 8 layers)
• Only 3x3 conv filter (stride=1, pad=1) 사용
• 2x2 max pooling (stride=2) 사용

 

3x3 small filter (stride=1) 3번 사용 = 7x7 filter 한 번 사용

input channel = 64라면,

7x7 filter parameter = 64 x (7 x 7 x 64) = 200,704개

3x3 filter 3개 paramteter = 64 x (3 x 3 x 64) x 3 = 110,592개

→ parameter수 감소 효과

→ non-linearities 활용 증가

---

GoogLeNet

 

GoogLeNet

GoogLeNet

• 22 layers
• Inception module 사용
• No FC layers → Conv layer로 대체 (Activation map과 동일한 size의 1x1 conv layer 값 추출)
• Only 5 milion parameters (AlexNet보다 12배 감소)

 

 

Inception module

Inception module

3x3 max pooling (stride=1)로 size 유지 → 일종의 convolution 연산

Naive Inception module

→ parameter 수 감소

 

 

1x1 Convolutions

1x1 Convolution filter

size 유지한 채, depth를 lower dimension으로 감소 (일종의 dimension reduction 효과)

 

 

Bottleneck layer

bottleneck layer

• 1x1 conv 사용
• parameter = 1 x 1 x Input Channel x Output Channel → 총 paramter 수 감소
• pointwise convolution이라고도 불림

 

 

Auxiliary classification

Auxiliary classification

• low level의 feature들로 classification
• 중간 중간 Backpropagation 진행하여 gradient가 더 잘 전달되는 역할
• Vanishing gradient 해결
• Gradient 변질 감소

---

ResNet

 

Plain convolutional neural network를 deep하게 쌓았을 때,

→ 56-layer model이 20-layer model보다 error가 높게 나옴

why?

deeper layer model일수록 optimize하기가 어려움 

 

 

ResNet

ResNet

Very deep networks using residual connections

 

• 152 layer model for ImageNet

 

 

 

Residual Block

Residual block

• ResNet이 deep한 layer를 쌓을 수 있게 만들어 준 tool
• H(x) = F(x) + x

 

F(x)

• = H(x) - x
• 잔차 (residual)
• F(x)만 train→ 학습 easy

 

x

• skip connection (잔차 연결)
• 기존 train 정보를 보존하여 다음 input으로 넘겨줌
• train 시, 정보 손실 ↓

 

H(x) = F(x) + x

→ H´(x) = F´(x) + 1

• x의 도함수 1이 deep layer의 gradient vanishing 해결
• 덧셈 연산량이 증가할 뿐, 학습 parameter 수에 영향끼치지 않음

 

 

 

ResNet + bottleneck layer

• ResNet 50 이상일 시, bottleneck layer 사용하여 efficiency 증가

 

 

DenseNet

DenseNet

• 각 layer가 모든 다른 layer의 feedforward fashion으로 연결
• ResNet: add연산 vs. DenseNet: concat 연산