1.整体结构

在这里插入图片描述

1）输入一张RGB彩色图片

2）经过PatchEmbed层得到shape=[H/4, W/4, embed_dim]的特征矩阵

3）通过循环设置的BasicLayer（Swin Transformer Block + Patch Merging）结构得到shape=[H/32, W/32, embed_dim*8]的提取出来的特征图

4）通过最后的head进行分类，head包括全局平均池化，输出shape=[B, C, 1]，再通过全连接层，输出维度为类别数，得到分类结果。

2.各部分结构

2.1 PatchEmbed

这一部分其实就是将图片分为一个个的patches，只不过ViT是直接下采样16倍，而swin transformer是先下采样4倍，然后再逐层下采样两倍。

1）首先判断输入的图片大小是否是patch_size的整数倍，如果不是则需要对图片进行padding。

2）通过卷积对图片进行下采样，输出特征矩阵shape=[B, HW, C]

class PatchEmbed(nn.Module):
    """
    这个模块是将输入的图片切成一个个的patch，然后再对每个patch中的像素映射为embed dim维。
    2D Image to Patch Embedding
    """
    def __init__(self, patch_size=4, in_c=3, embed_dim=96, norm_layer=None):
        super().__init__()
        patch_size = (patch_size, patch_size)
        self.patch_size = patch_size
        self.in_chans = in_c
        self.embed_dim = embed_dim
        self.proj = nn.Conv2d(in_c, embed_dim, kernel_size=patch_size, stride=patch_size)
        self.norm = norm_layer(embed_dim) if norm_layer else nn.Identity()

    def forward(self, x):
        _, _, H, W = x.shape

        # padding
        # 如果输入图片的H，W不是patch_size的整数倍，需要进行padding
        pad_input = (H % self.patch_size[0] != 0) or (W % self.patch_size[1] != 0)
        if pad_input:
            # to pad the last 3 dimensions,
            # (W_left, W_right, H_top,H_bottom, C_front, C_back)
            # 在宽度方向的右侧padding，高度方向的底部padding
            x = F.pad(x, (0, self.patch_size[1] - W % self.patch_size[1],
                          0, self.patch_size[0] - H % self.patch_size[0],
                          0, 0))

        # 下采样patch_size倍
        x = self.proj(x)
        _, _, H, W = x.shape
        # flatten: [B, C, H, W] -> [B, C, HW]
        # transpose: [B, C, HW] -> [B, HW, C]
        x = x.flatten(2).transpose(1, 2)
        x = self.norm(x)
        return x, H, W

2.2 BasicLayer（Swin Transformer Block + Patch Merging）

注意：这里每一个BasicLayer包含的是下一个stage的Patch Merging。

1)因为每个swin transformer block中都含有SW-MSA模块，所以需要先得到attention mask

2）根据输入的数据搭建多个swin transformer block，swin transformer block模块的输入为特征矩阵和attention_mask。

3）通过patch_merging进行下采样，高和宽变为之前的一半，通道数翻倍。

2.2.1 create_mask

    def create_mask(self, x, H, W):
        # calculate attention mask for SW-MSA
        # 保证Hp和Wp是window_size的整数倍
        Hp = int(np.ceil(H / self.window_size)) * self.window_size
        Wp = int(np.ceil(W / self.window_size)) * self.window_size
        # 拥有和feature map一样的通道排列顺序，方便后续window_partition
        # 对于切好的windows进行shift，然后重新按照window size切，单个window若其中全是连续的，则直接计算MSA
        # 若有不连续的，则需要mask了
        img_mask = torch.zeros((1, Hp, Wp, 1), device=x.device)  # [1, Hp, Wp, 1]
        h_slices = (slice(0, -self.window_size),
                    slice(-self.window_size, -self.shift_size),
                    slice(-self.shift_size, None))
        w_slices = (slice(0, -self.window_size),
                    slice(-self.window_size, -self.shift_size),
                    slice(-self.shift_size, None))
        cnt = 0
        for h in h_slices:
            for w in w_slices:
                img_mask[:, h, w, :] = cnt
                cnt += 1

        mask_windows = window_partition(img_mask, self.window_size)  # [nW, Mh, Mw, 1]
        mask_windows = mask_windows.view(-1, self.window_size * self.window_size)  # [nW, Mh*Mw]
        # 通过相减的广播机制来进行扩充，[nW, 1, Mh*Mw]会将最后一个维度的数据复制Mh*Mw次得到[nW, Mh*Mw, Mh*Mw]
        attn_mask = mask_windows.unsqueeze(1) - mask_windows.unsqueeze(2)  # [nW, 1, Mh*Mw] - [nW, Mh*Mw, 1]
        # [nW, Mh*Mw, Mh*Mw]
        # 相减之后对于不等于0的区域写入-100，等于0的区域就是0
        # 这样就得到了该window进行attention时的mask了
        attn_mask = attn_mask.masked_fill(attn_mask != 0, float(-100.0)).masked_fill(attn_mask == 0, float(0.0))
        return attn_mask

1)因为W-MSA和SW-MSA都需要将特征矩阵划分为windows的形式，所以需要输入的矩阵是window_size的整数倍，输入进SwinTransformerBlock中的矩阵都是padding过的，所以在这要创建一个形状相同的img_mask，将矩阵按照shift之后的windows划分，img_mask中数字相同的位置为连续的。

下图为H=W=9，window_size=3时，shift之后的window_mask

2)通过window_partition将img_mask划为一个个不重叠的windows，输出的shape=[B*num_windows, Mh, Mw, C]

3）通过张量相减的广播机制，相减之后等于0的地方置0，不等于0的地方写入-100，这样的话计算attention时与mask相加之后，不连续的地方-100，softmax之后就可以看做是0了。

这步就得到了attn_mask了。

2.2.2.1 window_partition

将feature map按照window_size划分成一个个没有重叠的window

def window_partition(x, window_size: int):
    """
    将feature map按照window_size划分成一个个没有重叠的window
    Args:
        x: (B, H, W, C)
        window_size (int): window size(M)

    Returns:
        windows: (num_windows*B, window_size, window_size, C)
    """
    B, H, W, C = x.shape
    x = x.view(B, H // window_size, window_size, W // window_size, window_size, C)
    # permute: [B, H//Mh, Mh, W//Mw, Mw, C] -> [B, H//Mh, W//Mh, Mw, Mw, C]
    # view: [B, H//Mh, W//Mw, Mh, Mw, C] -> [B*num_windows, Mh, Mw, C]
    windows = x.permute(0, 1, 3, 2, 4, 5).contiguous().view(-1, window_size, window_size, C)
    return windows

2.2.2 SwinTransformerBlock

SwinTransformerBlock的输入为feature map和前面得到的attention mask。

1）先把feature map给pad到window size的整数倍，然后shift移位，shift_size=window_size // 2。W-MSA不用进行shift，只有SW-MSA需要。

2）shift_window后再通过window_partition进行切片。

3）然后对每个windows计算attention，再通过window_reverse从切片还原为feature map，再移位回去，然后再去掉pad。

2.2.2.1 WindowAttention

实现W-MSA全部功能和SW-MSA的部分功能。因为在SwinTransformerBlock中传入WindowAttention模块中的参数已经是shift过后的，所以这个WindowAttention模块实际上就是实现W-MSA功能的模块。

其中大部分内容和ViT中的MSA差不多，但是多了一个relative_position_bias，每个head上的relative_position_bias是不一样的。

$Attention(Q_{i},K_{i},V_{i}) = softmax(\frac{Q_{i}K_{i}^{T}}{\sqrt{d_{k}}}+B)V_{i}$

1）通过全连接层计算输入的每个window中的q, k, v向量。

2）q向量与scale相乘，结果再与k向量相乘（这里与ViT中不太一样，ViT是先q*k再与scale相乘）

3）加上相对位置偏移

4）对是否有mask进行判断，W-MSA没有mask，SW-MSA有。若包含mask，则将attention与mask相加，然后进行softmax。若不包含mask则直接进行softmax。

5）结果与v向量相乘

6）与通过全连接实现的Wo融合矩阵相乘，得到最后的结果。

class WindowAttention(nn.Module):
    r""" Window based multi-head self attention (W-MSA) module with relative position bias.
    It supports both of shifted and non-shifted window.

    Args:
        dim (int): Number of input channels.
        window_size (tuple[int]): The height and width of the window.
        num_heads (int): Number of attention heads.
        qkv_bias (bool, optional):  If True, add a learnable bias to query, key, value. Default: True
        attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0
        proj_drop (float, optional): Dropout ratio of output. Default: 0.0
    """

    def __init__(self, dim, window_size, num_heads, qkv_bias=True, attn_drop=0., proj_drop=0.):

        super().__init__()
        self.dim = dim
        self.window_size = window_size  # [Mh, Mw]
        self.num_heads = num_heads
        head_dim = dim // num_heads
        self.scale = head_dim ** -0.5

        # define a parameter table of relative position bias
        self.relative_position_bias_table = nn.Parameter(
            torch.zeros((2 * window_size[0] - 1) * (2 * window_size[1] - 1), num_heads))  # [2*Mh-1 * 2*Mw-1, nH]

        # get pair-wise relative position index for each token inside the window
        coords_h = torch.arange(self.window_size[0])
        coords_w = torch.arange(self.window_size[1])
        coords = torch.stack(torch.meshgrid([coords_h, coords_w], indexing="ij"))  # [2, Mh, Mw]
        coords_flatten = torch.flatten(coords, 1)  # [2, Mh*Mw]
        # [2, Mh*Mw, 1] - [2, 1, Mh*Mw]
        relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :]  # [2, Mh*Mw, Mh*Mw]
        relative_coords = relative_coords.permute(1, 2, 0).contiguous()  # [Mh*Mw, Mh*Mw, 2]
        relative_coords[:, :, 0] += self.window_size[0] - 1  # shift to start from 0
        relative_coords[:, :, 1] += self.window_size[1] - 1
        relative_coords[:, :, 0] *= 2 * self.window_size[1] - 1
        relative_position_index = relative_coords.sum(-1)  # [Mh*Mw, Mh*Mw]
        self.register_buffer("relative_position_index", relative_position_index)

        self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
        self.attn_drop = nn.Dropout(attn_drop)
        self.proj = nn.Linear(dim, dim)
        self.proj_drop = nn.Dropout(proj_drop)

        nn.init.trunc_normal_(self.relative_position_bias_table, std=.02)
        self.softmax = nn.Softmax(dim=-1)

    def forward(self, x, mask: Optional[torch.Tensor] = None):
        """
        Args:
            x: input features with shape of (num_windows*B, Mh*Mw, C)
            mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None
        """
        # [batch_size*num_windows, Mh*Mw, total_embed_dim]
        B_, N, C = x.shape
        # qkv(): -> [batch_size*num_windows, Mh*Mw, 3 * total_embed_dim]
        # reshape: -> [batch_size*num_windows, Mh*Mw, 3, num_heads, embed_dim_per_head]
        # permute: -> [3, batch_size*num_windows, num_heads, Mh*Mw, embed_dim_per_head]
        qkv = self.qkv(x).reshape(B_, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
        # [batch_size*num_windows, num_heads, Mh*Mw, embed_dim_per_head]
        q, k, v = qkv.unbind(0)  # make torchscript happy (cannot use tensor as tuple)

        # transpose: -> [batch_size*num_windows, num_heads, embed_dim_per_head, Mh*Mw]
        # @: multiply -> [batch_size*num_windows, num_heads, Mh*Mw, Mh*Mw]
        q = q * self.scale
        attn = (q @ k.transpose(-2, -1))

        # relative_position_bias_table.view: [Mh*Mw*Mh*Mw,nH] -> [Mh*Mw,Mh*Mw,nH]
        relative_position_bias = self.relative_position_bias_table[self.relative_position_index.view(-1)].view(
            self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1)
        relative_position_bias = relative_position_bias.permute(2, 0, 1).contiguous()  # [nH, Mh*Mw, Mh*Mw]
        attn = attn + relative_position_bias.unsqueeze(0)

        if mask is not None:
            # mask: [nW, Mh*Mw, Mh*Mw]
            nW = mask.shape[0]  # num_windows
            # attn.view: [batch_size, num_windows, num_heads, Mh*Mw, Mh*Mw]
            # mask.unsqueeze: [1, nW, 1, Mh*Mw, Mh*Mw]
            attn = attn.view(B_ // nW, nW, self.num_heads, N, N) + mask.unsqueeze(1).unsqueeze(0)
            attn = attn.view(-1, self.num_heads, N, N)
            attn = self.softmax(attn)
        else:
            attn = self.softmax(attn)

        attn = self.attn_drop(attn)

        # @: multiply -> [batch_size*num_windows, num_heads, Mh*Mw, embed_dim_per_head]
        # transpose: -> [batch_size*num_windows, Mh*Mw, num_heads, embed_dim_per_head]
        # reshape: -> [batch_size*num_windows, Mh*Mw, total_embed_dim]
        x = (attn @ v).transpose(1, 2).reshape(B_, N, C)
        x = self.proj(x)
        x = self.proj_drop(x)
        return x

2.2.3.2 Mlp

就是一个普通的全连接层

2.2.3 PatchMerging

就是实现特征矩阵的宽高减半，通道数翻倍的操作