Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What is meant by probe accuracy ? #2

Open
alxndrTL opened this issue Feb 1, 2024 · 0 comments
Open

What is meant by probe accuracy ? #2

alxndrTL opened this issue Feb 1, 2024 · 0 comments

Comments

@alxndrTL
Copy link

alxndrTL commented Feb 1, 2024

Hello,
I have read your paper Linear Latent World Models in Simple Transformers:
A Case Study on Othello-GPT
and I just wanted to be sure about something :

What is exactly meant by probe accuracy ? (which goes up to 99.5% at layer 6 and 7) ?
Is it the average cell error of the model ? Like it is able to predict the cell state correctly (empty, mine, yours) 99.5% of the time ?
Or is it the board accuracy (all 64 cells correctly predicted) ?

I think the answer is the former but I wanted to be sure.
Thank you very much your work !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant