GPT1结构图如下所示：
GPT2结构图如下：
注意，GPT2的最后一个LayerNorm在24个transformers或是12个transformers结构之后添加的，
这里layernormalization放在前面类似于预激活函数的设定，在另外一篇文章Identity mappings in deep
residual networks.
这里在Idenity mapping in deep residual networks之中有推导过程，这里简单的写一下
原始残差单元计算公式如下：
y l = h ( x l ) + F ( x l , W l ) ( 1 ) y_{l} = h(x_{l})+F(x_{l},W_{l})(1) yl=h(xl)+F(xl,Wl)(1)
x l + 1 = f ( y l ) ( 2 ) x_{l+1} = f(y_{l}) (2) xl+1=f(yl)(2)
l为第l个单元输入的特征
其中F代表残差函数，我们假设函数h为恒等变换，则将(2)代入(1)中可以获得
x l + 1 = x l + ∑ i = 1 L − 1 F ( x l , W l ) ( 3 ) x_{l+1} = x_{l}+ \sum\limits_{i=1}^{L-1}F(x_{l},W_{l}) (3) xl+1=xl+i=1∑L−1F(xl,Wl)(3)
通过反向传播的链式法则进行求导，可以得到结果
∂ E ∂ x L = ∂ E ∂ x L ∗ ∂ x L ∂ x l = ∂ E ∂ x L ( 1 + ∂ ∂ x l ∑ i = 1 L − 1 F ( x i , W i ) ) \frac{\partial E}{\partial x_{L}} = \frac{\partial E}{\partial x_{L}} * \frac{\partial x_{L}}{\partial x_{l}} = \frac{\partial E}{\partial x_{L}}(1+\frac{\partial }{\partial x_{l}}\sum \limits_{i=1}^{L-1}F(x_{i},W_{i})) ∂xL∂E=∂xL∂E∗∂xl∂xL=∂xL∂E(1+∂xl∂i=1∑L−1F(xi,Wi))
如果说这里的 h ( x l ) h(x_{l}) h(xl)不为恒等函数的情况下
∂ E ∂ x L = ∂ E ∂ x L ∗ ∂ x L ∂ x l = ∂ E ∂ x L ( ∑ i = 1 L − 1 x i + ∂ ∂ x l ∑ i = 1 L − 1 F ( x i , W i ) ) \frac{\partial E}{\partial x_{L}} = \frac{\partial E}{\partial x_{L}} * \frac{\partial x_{L}}{\partial x_{l}} = \frac{\partial E}{\partial x_{L}}( \sum\limits_{i=1}^{L-1}x_{i}+\frac{\partial}{\partial x_{l}}\sum \limits_{i=1}^{L-1}F(x_{i},W_{i})) ∂xL∂E=∂xL∂E∗∂xl∂xL=∂xL∂E(i=1∑L−1xi+∂xl∂i=1∑L−1F(xi,Wi))