码迷,mamicode.com
首页 > 其他好文 > 详细

Human-like Controllable Image Captioning with Verb-specific Semantic Roles(具有动词语义角色的类人可控图像字幕生成)

时间:2021-04-09 13:26:43      阅读:0      评论:0      收藏:0      [点我收藏+]

标签:role   Planner   isp   osal   oat   rem   between   mos   生成   

技术图片

 前人的缺陷:

CIC works mainly focus on (1)subjective control signals,(2)objective control signals  or (1) Content-controlled (2) Structure controlled

almost all existing objective control signals have overlooked two indispensable characteristics of an ideal control signal:

  1) Event-compatible:all visual contents referred to in a single sentence should be compatible with the describe activity.

  2) Sample-suitable: the control signals should be suitable for a specific image sample.

 

论文的创新点:

propose a new event-oriented objective control signal, Verb-specific Semantic Roles (VSR), to meet both event-compatible and sample-suitable requirements simultaneously。

VSR consists of a verb and some user-interested semantic roles。

Grounded Semantic Role Labeling: visual features of all grounded proposal sets。

Semantic Structure Plannerhierarchical semantic structure learning model, which aims to learn a reasonable sequence of sub-roles S。

Verb-specific Semantic RolesGrounded Semantic Role Labeling  υ  Semantic Structure Planner

技术图片

 

 

 

 

 



 

 step:we first use GSRL and SSP to obtain semantic structures and grounded regions features: (Sa; Ra) and (Sb; Rb).

Then,as shown in Figure above, we merge them by two steps。

  (a) find the sub-roles in both Sa and Sb which refer to the same visual regions 

  (b) insert all other sub-roles between the nearest two selected sub-roles


模型架构:

Faster R-CNN(ResNet-101) + Controllable LSTM + Controllable UpDn + SCT

原文: https://arxiv.org/abs/2103.12204

 

Human-like Controllable Image Captioning with Verb-specific Semantic Roles(具有动词语义角色的类人可控图像字幕生成)

标签:role   Planner   isp   osal   oat   rem   between   mos   生成   

原文地址:https://www.cnblogs.com/sfnz/p/14635500.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!