Transformer is a show

标签：topic   class   seve   rsa   cal   max   VID   dig   features   

• Performing bases on script, script inspires by story, story starts with ideas.
• Likely, at the beginning of Transformer, we have raw sequence, then we preprocess it to fit the shape of the model.

• Script Interpretation: x is the performing script, can be interpreted from 3 different aspects:　　
  • Q = plots, K = roles, V = characteristics
  • A script is consisted of a series of plots, each plot is performed by different roles, each role has his own characteristics.
  • In other words, different role performs different characteristic according to the plot, just like different key matches with different query.
  • Further, the same role may perform a different characteristic at different plot, just like the same word may have different meaning at different position of the sequence.
  • If Q=K=V=x, which is plots=roles=characteristics=script, it means s a solo performing, and the actor can perform according to his will to express the theme of the story. In keras Multi-Head Attention API, this is called "self-attention".
• Rehearsal: Multi-Head attention is interpreted as rehearsal, since it processes script interpretation (Q, K, V), just like actors have a rehearsal after understanding the script.
• Adjustment: Dropout -> ResAdd -> LayerNorm
  • Dropout: randomly cut some plot of the story, in case of highly depending on some actors’ personal performing or mainly betting on the climax
  • ResAdd: see the connection between each plot, not to isolate them
  • LayerNorm: normalize actors’ performing, in case of actors bringing in too much personal characteristics

• Crew Discussion: FullyConnected is interpreted as crew discussion to abstract the performing features and flaws.
• Adjustment: like above P2.

• After the repeatedly practices, the real_show (enc_output) will go live on stage.

• The assessment members will watch & record this stage show, just like enc_output will be passed to Decoder.

• P1: input_x -> preprocess(Embedding -> Scale -> Pos_encoding -> Dropout) -> x
• P2: Query=x, Value=x, Key=x, enc_padding_mask -> Multi_Head_Attention -> adjust(Dropout -> ResAdd -> LayerNorm) -> out1
• P3: out1 -> FC -> adjust(Dropout -> ResAdd -> LayerNorm) -> out2
• P4: loop(P2 - P3) -> update(out2)
• P5: enc_output = out2

• At the very beginning, the show is not on, bu rumors (<SOS>) already spread around, attracting people‘s attention, preparing for the premiere.
• When assessment members hear this rumors, they start to dig into the story, by reading introduction, comments, etc..
• Likely, we have no real input target but a sign of it <SOS> at the beginning of Decoder, then we do a preprocess for it.

• Ask Questions:
  • The first Multi-Head Attention outputs a Query, which is like people may think & ask some questions about what they heard & red.
  • So, in code, the input of the first Multi-Head Attention (MHA) is: target, target, target. Because when the curiosity is initially triggered, all you thinking is: more, more, more on the topic.
• Adjustment: see below P3.

• Watch the Show & Answer the Questions:
  • Connectedly, the second MHA is to answer the questions asked in the first MHA by watching the show.
  • So, in code, the Query from the first MHA and the enc_output from Encoder is passed to the second MHA as input. Since both Key and Value information are from the show, so we set Key=enc_output, Value=enc_output.
• Adjustment: adjustment in assessment is interpreted differently with adjustment in performing
  • Dropout: randomly delete some opinions of the jury, in case of the manipulation of authority
  • ResAdd: evaluate the show comprehensively with previous assessment , not to view it isolatly
  • LayerNorm: normalize the assessment to standard criterion

• Assessment Discussion: this FullyConnected is interpreted as a discussion about the evaluation & criterion stuff.
• Adjustment: like above P3.

• Assess several rounds, ensuring the show is well & fair understood and evaluated.

• Softmax is like voting, disclosing the final winner of the Oscar Award.

• P1: SOS -> preprocess(Embedding -> Pos_encoding) -> target
• P2: Query=target, Value=target, Key=target, look_ahead_mask -> Multi-Head Attention -> adjust(Dropout -> ResAdd -> LayerNorm) -> out1
• P3: Query=out1, Value=enc_output, Key=enc_output, dec_padding_mask -> Multi-Head Attention -> adjust(Dropout -> ResAdd -> LayerNorm) -> out2
• P4: out2 -> FC -> adjust(Dropout -> ResAdd -> LayerNorm) -> out3
• P5: loop(P2 - P4) -> update(out3) -> dec_output = out3
• P6: dec_output -> Dense(‘softmax‘) -> ?

Transformer is a show

标签：topic   class   seve   rsa   cal   max   VID   dig   features   

原文地址：https://www.cnblogs.com/DamonLai/p/14922243.html