Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GSK-4033] Fix correctness aggregation error #2090

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

henchaves
Copy link
Member

Description

Related Issue

Type of Change

  • 📚 Examples / docs / tutorials / dependencies update
  • 🔧 Bug fix (non-breaking change which fixes an issue)
  • 🥂 Improvement (non-breaking change which improves an existing feature)
  • 🚀 New feature (non-breaking change which adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to change)
  • 🔐 Security fix

Checklist

  • I've read the CODE_OF_CONDUCT.md document.
  • I've read the CONTRIBUTING.md guide.
  • I've written tests for all new methods and classes that I created.
  • I've written the docstring in Google format for all the methods and classes that I used.
  • I've updated the pdm.lock running pdm update-lock (only applicable when pyproject.toml has been
    modified)

Copy link

linear bot commented Dec 19, 2024

@henchaves henchaves self-assigned this Jan 7, 2025
@henchaves henchaves changed the title [WIP][GSK-4033] Fix correctness aggregation error [GSK-4033] Fix correctness aggregation error Jan 7, 2025
@henchaves henchaves marked this pull request as ready for review January 7, 2025 01:51
Copy link
Member

@kevinmessiaen kevinmessiaen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, however I'm thinking it would be better to add those check inside the parse_json_output since it has a builtin feature to ask the llm to correct the json output

out.content,
llm_client=llm_client,
keys=["correctness", "correctness_reason"],
caller_id=self.__class__.__name__,
)

if "correctness" in json_output and not isinstance(json_output["correctness"], bool):
raise LLMGenerationError(
f"Error in correctness evaluation: {json_output['correctness']}. Please make sure the agent answer is correctly formatted."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Switching to repr is better so we can differentiate "True" from True, also

Suggested change
f"Error in correctness evaluation: {json_output['correctness']}. Please make sure the agent answer is correctly formatted."
f"Error in correctness evaluation: {repr(json_output['correctness'])}. Please make sure the agent answer is correctly formatted. Correctness value should be of boolean type."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants