TL;DR: We can prompt chatGPT to generate an "attention map" for themselves (demo available at https://ywugwu.streamlit.app/).
Currently, we're working on getting better prompts via open-source LLMs like Llama3.
Introduction
We're interested in letting LLMs introspect. That is, let an LLM answer which part of the input contributes the most to the word we're interested in the output text (like an NLP-version Grad-Cam but by Prompting:
We want an NLP-version Grad-Cam (https://arxiv.org/pdf/1610.02391) but by Prompting
We have a demo at https://ywugwu.streamlit.app/ that can do this:
Method
An overview of our prompt: we merge the previous "input text" and "output text" into our prompt and ask LLMs to assign importance score in a word-to-word fashion.
We can also use different prompts like:
And we can compare the results of these prompts:
Future Work
A Future work (what we're doing now) is using Grad-Cam results as ground truth to optimize our prompt:
