• 文章原创自：微信公众号「机器学习炼丹术」

  • 作者：炼丹兄
  • 联系方式：微信cyx645016617
  • 代码来自github

【前言】：看代码的时候appearance，也许会不理解VIT中各种组件的含义，但是这个文章的目的是了解其实现。在之后看论文的时候，可以做到心中有数，而不是一片茫然。

VIT类

初始化

和之前的学习一样，从大模型类开始看起，然后一点一点看小模github永久回家地址mi型类：

class ViT(nn.Module):
degithub怎么下载文件f __init__(se哪里拍婚纱照好又便宜lf, *, image_size, patch_size, num_classes, dim, depth, heads, mlp_dim, pool = 'cls', channels = 3, dim_head = 64, dropout = 0., emb_dropoutNLP = 0.):
super().__init__()
assert image_sizeAPP % patch_size == 0, 'Image dimensions must be divisappearible by the patch size.'
num_patches = (image_size // patch_size) ** 2
pgithubatch_dimapp安装下载 = channels * patch_size ** 2
assert nuapp安装下载m_patches > MINappreciate_NUM_PATC奶酪披萨HES, f'your number of patches ({num_patches}) is way too small for attention to bgithub下载e effective (at least 16).gitlab Try dgithub官网ecreasing哪里拼音 your patch size'
assert pool in {'cls', 'mean'}, 'pool type must be either cls (cls token) or mean (mean pooling)'giti
self.patch_size = patch_size
self.pos_embedding = nn.Parameter(torgithub永久回家地址mich.randn(1, num_patches + 1, dim))
self.p机器学习atch_to_embeddappearanceing = nn.Linear(patch_dim, dim)
self.cls_tokegithubcom1jie1小可爱n = nn.Parameter(torch.randn(1, 1, dim))
self.drnlpopoappstoreut = nn.Dropout(emb_dropout)gitee
self.transformergithub中文官网 = Transformer(dim, depth, heads, dim_head, mlp_dimapp安装下载, dropout)
self.poappreciateol = pool
self.to_latent = nn.Identity()
self.mlp_head = nn.Sequential(
nn.LayerNorm(dim),
nn.Linear(dim, num_classes)
)

在实际的调用中，是如下调用的：

model = ViT(
dim=128,
image_size=224,
patch_size=32,
num_classes=2,
channels=3,
).to(devicgithub官网e)

输入参数讲解：

  • image_size：图片的大小；
  • patch_sgithub怎么读ize:把图片划分成gitv小的patch，小的patch的尺寸；
  • num_classes:这次分类任务的类别总数；
  • channels:输入图片的通道数。

VIT类中初始化的组件：

  • num_patches:一个图片划分成多少个patch，appreciate因为图片224，patch32，所以划分成7×7=49个patches；
  • paGittch_dim:3x32x32，理解为一个patch中的元素个数；

……这样展示是不是非常的麻烦，还要上下来回翻看代码，所以我写成注释的形式

class ViT(nn.Mogithub是干什么的dule):
def __init__(self, *, imaggithub是干什么的e_size, patch_size, num_classes, dim, depth, heads, mlpgitv_dim, pool = 'cls', channels = 3, dim_head = 64, dropout = 0., emb_dropout = 0.):
# image_sizegithub源码=224,patch_size=32,num_classes=2,channels=3，dim=128
super().__init__()
assert image_size % patch_size == 0, 'Image dimensions must be divisible by the patch size哪里拍婚纱拍的好.'
# num_pathes = （224//32appearance）**2 = 7*7=49
num_patches = (image_size // patch_size) ** 2
# patch_dim = 3*32*32
pat你老婆掉了六盲星ch_dim = channels * patch_size **github怎么下载文件 2
assert num_patNLPches > MIgiteeN_NUM_PATCHES, f'your number of patches ({num_patches}) is way too small for attention to be effective (at least 16). Try decreasing your patch size'
asseappreciatert pool in {'cls', 'gitlabmean'}, 'pool typGite must be either cls (cls token) or mean (mean pooling)'
# self.patch_size = 32
self.patch_size = patch_size
# self.pos_embed机器学习ding是一个appear形状为（1，50，128）
self.pos_embedding = nn.Pargiticomfortagithubcom1jie1小可爱meter(torch.rgithub下载andn(1, num_patches + 1, dim))
# self.patch_to_embedding是一个从3giticomfort*32*32到1哪里拍的婚纱摄影好28映射的线性层
self奶酪披萨.patch_to_embedding = nn.Linear(patch_dim, dim)
# segithub源码lf.cls_token是一个随机初始化的形状为（1，gitv1，128）这样的变量
self.cls_token = nn.Parameter(torch.randn(1, 1, dim))
self.dropout = nn.Dropout(emb_dropout)
# Tra内陆盆地海拔最高nsformegithub是干什么的rgithub永久回家地址mi后面gitv会讲解
selgiteef.transformer = Transformer(dim, depth, heads, dim_head, mlp_dim, dropapproveout)
self.pool = pool
self.togithub是干什么的_latent = nn.github怎么下载文件Identity()
self.mlp_head = nn.Sequential(
nn.LayerNorm(dim),
nn.Linear(dimgithub是干什么的, num_classes)
)

forward

现在看VIT的推理过程：

    def forward(self, img, mask = None):
# p=32
p = self.patcappstoreh_size
x = rearrange(img, 'b c (h p1) (w p2) -> b (h w) (p1你老婆掉了六盲星 p2 c)', p1 = p, p2 = p)
x = self.patch_to_embedding(x) # x.shape=[b,49内陆盆地海拔最高,128]
b, n, _ = x.shape # n = 49
cls_tokens = repeat(self.cls_token, '() n d -> b n d', b = b)
x = torch.cat((cls_tokens, x), dim=github永久回家地址mi1) # x.shape=[b,50,128]
x += self.pAPPos_e尼拉帕利mbedding[:, :(n + 1)] # x.shape=[b,50,128牛铃飘翠岭]
x = self.dropout(x)
x = self.traNLPnsformer(x, maskgiti轮胎是什么品牌) # x.shape=[b,50,128],mask=None
x = x.mean(dim = 1) if self.pool == 'mean' else x[:, 0]
x = self.to_latent(x)
return self.mlp_head(x)
  • 这里的代码用到了from einops import rearrange, repeat，这个库函数，einops是一个库函数，是对张量进行操作的库函数，支持pytorgitlabch，TF等appleid。
  • einops.rearrange是把输入的igithub中文官网mg,从[b,3,224,224]的形状改成[b,3,7,32,7,32]的形状，通过矩阵的转置换成[b,7,7,3gitv2,32,3]的样子，最后合并成[b,49,32x32x3]
  • self.patch_to_embedding,输出的x的形状为[b,49,128];
  • einops.repeat是把self.cls_哪里拍的婚纱摄影好token从[1,1,128]复制成[b,1,128]

现在，我们知道从patapprovech到哪里拍的婚纱摄影好embedding是用线性层实现的。

transformer

class Transformer(nn.Module):
def __init__(self, dim, depth, heads, dim_head, mlp_dim, dropout):
# dim=128,depth=12，headAPPs=8，dim_head=64,mlp_dim=128
super().__init__()
self.layers = nn.ModuleList([])
for _ in range(depth):
self.layers.append(nn.ModuleList([
Residual(PreNorm(dim, Attentiapproachon(dim, heads = heads, dim_head = dim_head, dropout = dropout))),
Residual(PreNorm(dim, FeedForward(dim, mlp_dim, dropout = dropout)))
]))
dgitief forward(self, x, mask = None):appstore
for attn, ff in self.layers:
x = attn(x, mask = mask)
x = ff(x)
return x
  • self.lappearanceayers中包含depth组的Attention+FeedForward模块。
  • 这里需要记得，输入的x的尺寸为[b,50,128]

Atgithub官网tention

class Attention(nn.Modulgithub源码e):
def __init__(self, dim, heads = 8, dAPPim_head = 64, dropout = 0.):
super().__init__()
inner_dimgithub永久回家地址mi = dim_head *  heads # 64 x 8
self.heads = heads # 8
self.scale = dim_head ** -0.5
self.to_qkv = nn.Linear(dim, innapproveer_dim * 3, bias = False)
self.to_out = nn.Sequential(
nn.Linear(inner_dim, dim),
nn.Dropout(dropout)
)
def forward(self, x, mask = None):
b, n, _, h = *x.shape, self.heads # n=50,h=8
# self.to_applicationqkv(x)得到的尺寸为[b,50,64xgithub怎么下载文件8x3],然后chunk成3份
# 也就是说，qkv是一个三元tuple,每一份都是[b,50,64x8]的大小
qkv = self.to_qkv(x).chunk(3, dim = -1)
# 把每一份从[b,50,64x8]变成[b,8,50,64]的形式
q, kgitv, v = map(lambda t: rearrange(t, 'b n (h d) -> b h n d', h = h), qkv)
# 这一步不太好理解，q和k都是[b,8,哪里拍婚纱拍的好50,64]的形式，50理解为特征数量，6github喵绅士4为特征变量
# dots.shape=[b,8,50,50]
dotappleids = torch.einsum('bhid,bhjd->bhij', q, k) * self.scale
# 不考虑mask这一块的内容
mask_valgitiue = -torch.fingithub永久回家地址mifo(dots.dtypgithub永久回家地址mie).max
if mask is not None:
mask = F.pad(mask.flatten(1), (1, 0), value = Truegithub怎么读)
assert mask.shape[-1] == dots.shape[-1], 'mask has incorrect dimensions'
mask = mask[:, None, :] * mask[:, :, None]
dots.APPmagithubsked_fill_(~mask, mask_value)
del maskgiticomfort
# 对[b,8,github下载50,50]的最后一个维度做soft哪里拍婚纱照好又便宜max
attn = dots.softmax(dim=-1)
# 这个attn就是计算出来的自注意力值，和v做点乘，out.shape=[b,8,5apple0,64]
out = tgithub喵绅士orch.einsum('bhij,bhjd你老婆掉了六盲星->bhid', attn, v)
# out.shape变成[b,50,8x64]
out = rearrange(out, 'b h n d -> b n (h d)')
# out.shape重新变成[b,60,128]
out =  self.to_out(out)
return out

综上所属，这个attention其实就是一个自注意力模块，输入的是[b,50,128github是干什么的]，返回giti轮胎是什么品牌的也是[b,50,128]。实现的过程因为使用了torch.einsum所以有些复杂，但是总的来说appearance，和我之前讲过的一篇论奶酪披萨文”non-local”模块，是完全一样的。torch.eins奶酪披萨um和torch.mm原理相同，只是因为torGitHubchgithub中文官网.mm不支持高纬度的张量做矩阵乘法。

PreNorm

clagithub喵绅士ss PregiteeNorm(nn.Module):
def __init__(self, digithub怎么读m, fn):
# dim=128,fn=Attention/FeedForward
super().__init__()
self.nappstoreorm = nn.LayerNoAPPrm(dim)
self.fn = fn
def forward(self, x, **kwargs):
return self.fnapproach(self.norm(x), **kwargs)

先对输入的x(x.shapgiticomforte=[b,50,128])做一个layerNormalization层归一化机器学习，然后再放到上面的Attention模块中做自注意力。

Residual

class Residual(nappstoren.Module):
def __init__(self你老婆掉了六盲星, fn):
super().__init__()
self.fn = fn
def forward(self, x, **kwargs):
return self.fn(x, **kwargs) + x

一内陆盆地海拔最高个残appear差模块罢了。

FeedForward

class FeedForward(nn.Module):
def __init__(selapp安装下载f, dim, hiddengithub永久回家地址mi_dim, dropout = 0.):
# dim=128,hidden_dim=128
super().__init__()
self.net = nn.Sequential(
nn.Linear(dim, hidden_dim),
nn.GELU(),
nn.Dropout(dropout),
nn.Linear(hidden_dim, dim),
nn.Dronlppout(dropoutGitHub)
)
def forward(self, x):
return self.net(x)

就giticomfort是两个线性层，这appreciate里有意思的是GE哪里拍婚纱拍的好LU()激活函数，这个激活函数可以直接使用torch.nn.GELU()调用，回头有机会再好好讲一下GELU()的原理。

tragithub喵绅士nsformer总结

Residual(PreNorm(dim, Attention(dim, heads = heads, dim_head = dim_head, dropout =github官网 dropout))),
Residual(PreNorm(dim, Fgithub中文官网eeapp安装下载dForward(dim, mlp_dim, dropout = dropout)))
  • 第一个就是，先对输入做layerNormalization，然后放到attention得到attention的结果，然后结果和做layerappreciateNormaliz哪里拍婚纱拍的好ationlpn之前的输入相加做一个残差链接；
  • 第二个就是，x->LayerNormalization->FeedForward线性层->y,然后这个y和输入的x相内陆盆地海拔最高加，做残差连接。

VIT总结

回顾一下整个流程：

  • 一个图片224×224，分成了49个32×32的patch；
  • 对这么多的patch做embedding，成49个128向量；
  • 再拼接一个cls_tokens，变成50个128向量；
  • 再加上pos_embedding，还是50个128向量；
  • 这些向量输入到transformer中进行自注意力的特征提取；
  • 输出的是50个128向量，然后对这个50个求军职，变成一个128向量；
  • 然后线性层把128维变成2维从而完成二分类任务的tran牛铃飘翠岭sformer模型。

问题：我对NLP了解不深github官网入，有没有人github是干什么的可以回答一下这个问题：c尼拉帕利ls_tokens和pos_embedding的用处是什么？