Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: added tests and redactPrivateRepoComments #24

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ To set up the `.dev.vars` file, you will need to provide the following variables
matchThreshold: 0.95
warningThreshold: 0.75
jobMatchingThreshold: 0.75
redactPrivateRepoComments: false
```


Expand Down
7 changes: 6 additions & 1 deletion manifest.json
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,17 @@
"jobMatchingThreshold": {
"default": 0.75,
"type": "number"
},
"redactPrivateRepoComments": {
"default": false,
"type": "boolean"
}
},
"required": [
"matchThreshold",
"warningThreshold",
"jobMatchingThreshold"
"jobMatchingThreshold",
"redactPrivateRepoComments"
]
}
}
2 changes: 1 addition & 1 deletion src/handlers/add-comments.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ export async function addComments(context: Context) {
const markdown = payload.comment.body;
const authorId = payload.comment.user?.id || -1;
const nodeId = payload.comment.node_id;
const isPrivate = payload.repository.private;
const isPrivate = context.config.redactPrivateRepoComments;
const issueId = payload.issue.node_id;

try {
Expand Down
2 changes: 1 addition & 1 deletion src/handlers/add-issue.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ export async function addIssue(context: Context) {
const markdown = payload.issue.body + " " + payload.issue.title || null;
const authorId = payload.issue.user?.id || -1;
const nodeId = payload.issue.node_id;
const isPrivate = payload.repository.private;
const isPrivate = context.config.redactPrivateRepoComments;

try {
if (!markdown) {
Expand Down
7 changes: 3 additions & 4 deletions src/handlers/update-comments.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,11 @@ export async function updateComment(context: Context) {
adapters: { supabase },
} = context;
const { payload } = context as { payload: CommentPayload };
const markdown = payload.comment.body;
const authorId = payload.comment.user?.id || -1;
const nodeId = payload.comment.node_id;
const isPrivate = payload.repository.private;
const issueId = payload.issue.node_id;

const issueId = payload.issue.node_id
const isPrivate = !context.config.redactPrivateRepoComments && payload.repository.private;
const markdown = payload.comment.body || null;
// Fetch the previous comment and update it in the db
try {
if (!markdown) {
Expand Down
2 changes: 1 addition & 1 deletion src/handlers/update-issue.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ export async function updateIssue(context: Context) {
const { payload } = context as { payload: IssuePayload };
const payloadObject = payload;
const nodeId = payload.issue.node_id;
const isPrivate = payload.repository.private;
const isPrivate = context.config.redactPrivateRepoComments;
const markdown = payload.issue.body + " " + payload.issue.title || null;
const authorId = payload.issue.user?.id || -1;
// Fetch the previous issue and update it in the db
Expand Down
1 change: 1 addition & 0 deletions src/types/plugin-inputs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export const pluginSettingsSchema = T.Object(
matchThreshold: T.Number({ default: 0.95 }),
warningThreshold: T.Number({ default: 0.75 }),
jobMatchingThreshold: T.Number({ default: 0.75 }),
redactPrivateRepoComments: T.Boolean({ default: false }),
},
{ default: {} }
);
Expand Down
37 changes: 37 additions & 0 deletions tests/__mocks__/adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,43 @@ export function createMockAdapters(context: Context) {
return commentMap.get(commentNodeId);
}),
} as unknown as Comment,

issue: {
findSimilarIssues: jest.fn(async (issueContent: string, threshold: number, currentIssueId: string) => {
// Return empty array for first issue in each test
if (currentIssueId === "warning1" || currentIssueId === "match1") {
return [];
}

// For warning threshold test (similarity around 0.8)
if (currentIssueId === "warning2") {
return [
{
issue_id: "warning1",
similarity: 0.8,
},
];
}

// For match threshold test (similarity above 0.95)
if (currentIssueId === "match2") {
return [
{
issue_id: "match1",
similarity: 0.96,
},
];
}

return [];
}),
createIssue: jest.fn(async () => {
// Implementation for createIssue
}),
},
fetchComments: jest.fn(async (issueId: string) => {
return Array.from(commentMap.values()).filter((comment) => comment.issue_id === issueId);
}),
},
voyage: {
embedding: {
Expand Down
4 changes: 3 additions & 1 deletion tests/__mocks__/db.ts
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ export const db = factory({
deployments_url: String,
},
issue: {
id: primaryKey(Number),
node_id: primaryKey(String),
number: Number,
title: String,
body: String,
Expand All @@ -78,6 +78,7 @@ export const db = factory({
comments: Number,
labels: Array,
state: String,
closed: Boolean,
locked: Boolean,
assignee: nullable({
login: String,
Expand Down Expand Up @@ -136,6 +137,7 @@ export const db = factory({
login: String,
id: Number,
},
issue_id: String,
author_association: String,
html_url: String,
issue_url: String,
Expand Down
17 changes: 17 additions & 0 deletions tests/__mocks__/handlers.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { http, HttpResponse } from "msw";
import { db } from "./db";

/**
* Intercepts the routes and returns a custom payload
*/
Expand Down Expand Up @@ -48,6 +49,21 @@ export const handlers = [
item.body = body;
return HttpResponse.json(item);
}),
//Update issue
http.patch("https://api.github.com/repos/:owner/:repo/issues/:issue_number", async ({ params: { issue_number: issueNumber }, request }) => {
const { body } = await getValue(request.body);
const item = db.issue.findFirst({ where: { number: { equals: Number(issueNumber) } } });
if (!item) {
return new HttpResponse(null, { status: 404 });
}
item.body = body;
return HttpResponse.json(item);
}),

//Fetch comments for the issue
http.get("https://api.github.com/repos/:owner/:repo/issues/:issue_number/comments", ({ params: { issue_id: issueId } }) =>
HttpResponse.json(db.issueComments.findMany({ where: { issue_id: { equals: String(issueId) } } }))
),
];

async function getValue(body: ReadableStream<Uint8Array> | null) {
Expand All @@ -63,4 +79,5 @@ async function getValue(body: ReadableStream<Uint8Array> | null) {
}
}
}
return {};
}
120 changes: 116 additions & 4 deletions tests/__mocks__/helpers.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,16 @@
import { db } from "./db";
import { STRINGS } from "./strings";
import usersGet from "./users-get.json";
import threshold95_1 from "../__sample__/match_threshold_95_1.json";
import threshold95_2 from "../__sample__/match_threshold_95_2.json";
import warning75_1 from "../__sample__/warning_threshold_75_1.json";
import warning75_2 from "../__sample__/warning_threshold_75_2.json";
import taskComplete from "../__sample__/task_complete.json";

interface SampleIssue {
title: string;
issue_body: string;
}

/**
* Helper function to setup tests.
Expand Down Expand Up @@ -34,14 +44,16 @@ export async function setupTests() {

// Insert issues
db.issue.create({
id: 1,
node_id: "1", //Node ID
number: 1,
title: "First Issue",
body: "This is the body of the first issue.",
user: {
login: STRINGS.USER_1,
id: 1,
},
owner: STRINGS.USER_1,
repo: STRINGS.TEST_REPO,
author_association: "OWNER",
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
Expand All @@ -65,14 +77,16 @@ export async function setupTests() {
});

db.issue.create({
id: 2,
node_id: "2", //Node ID
number: 2,
title: "Second Issue",
body: "This is the body of the second issue.",
user: {
login: STRINGS.USER_1,
id: 1,
},
owner: STRINGS.USER_1,
repo: STRINGS.TEST_REPO,
author_association: "OWNER",
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
Expand All @@ -96,7 +110,80 @@ export async function setupTests() {
});
}

export function createComment(comment: string, commentId: number, nodeId: string) {
export function createIssue(
issueBody: string,
issueNodeId: string,
issueTitle: string,
issueNumber: number,
issueUser: {
login: string;
id: number;
},
issueState: string,
issueCloseReason: string | null,
repo: string,
owner: string
) {
const existingIssue = db.issue.findFirst({
where: {
node_id: {
equals: issueNodeId,
},
},
});
if (existingIssue) {
db.issue.update({
where: {
node_id: {
equals: issueNodeId,
},
},
data: {
body: issueBody,
title: issueTitle,
user: issueUser,
updated_at: new Date().toISOString(),
owner: owner,
repo: repo,
state: issueState,
state_reason: issueCloseReason,
},
});
} else {
db.issue.create({
node_id: issueNodeId,
body: issueBody,
title: issueTitle,
user: issueUser,
number: issueNumber,
created_at: new Date().toISOString(),
updated_at: new Date().toISOString(),
author_association: "OWNER",
state: issueState,
state_reason: issueCloseReason,
owner: owner,
repo: repo,
reactions: {
url: "",
total_count: 0,
"+1": 0,
"-1": 0,
laugh: 0,
hooray: 0,
confused: 0,
heart: 0,
rocket: 0,
eyes: 0,
},
timeline_url: "",
comments: 0,
labels: [],
locked: false,
});
}
}

export function createComment(comment: string, commentId: number, nodeId: string, issueNumber?: number) {
const existingComment = db.issueComments.findFirst({
where: {
id: {
Expand All @@ -105,6 +192,17 @@ export function createComment(comment: string, commentId: number, nodeId: string
},
});

// Find the issue by nodeId to get its number
const issue = db.issue.findFirst({
where: {
node_id: {
equals: nodeId,
},
},
});

const targetIssueNumber = issueNumber || (issue ? issue.number : 1);

if (existingComment) {
db.issueComments.update({
where: {
Expand All @@ -115,13 +213,14 @@ export function createComment(comment: string, commentId: number, nodeId: string
data: {
body: comment,
updated_at: new Date().toISOString(),
issue_number: targetIssueNumber,
},
});
} else {
db.issueComments.create({
id: commentId,
body: comment,
issue_number: 1,
issue_number: targetIssueNumber,
node_id: nodeId,
user: {
login: STRINGS.USER_1,
Expand All @@ -133,3 +232,16 @@ export function createComment(comment: string, commentId: number, nodeId: string
});
}
}

export function fetchSimilarIssues(type?: string): SampleIssue[] {
switch (type) {
case "warning_threshold_75":
return [warning75_1, warning75_2];
case "match_threshold_95":
return [threshold95_1, threshold95_2];
case "task_complete":
return [taskComplete];
default:
return [threshold95_1, threshold95_2];
}
}
2 changes: 2 additions & 0 deletions tests/__mocks__/strings.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,6 @@ export const STRINGS = {
CONFIGURABLE_RESPONSE: "Hello, Code Reviewers!",
INVALID_COMMAND: "/Goodbye",
COMMENT_DOES_NOT_EXIST: "Comment does not exist",
SIMILAR_ISSUE: "Similar Issue",
SIMILAR_ISSUE_URL: "https://www.github.com/org/repo/issues",
};
4 changes: 4 additions & 0 deletions tests/__sample__/match_threshold_95_1.json
Copy link
Member

@0x4007 0x4007 Oct 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to test with our real specifications.

Tip

You can look through "closed as not complete" issues that contain comments like "duplicate." We generally link to the correct issue when closing as not complete.

You might need to use GraphQL because GitHub search UI does not clearly show that it is "closed as not complete" example.

I originally located the above using this comment.

Perhaps there is a way to "train" the system that all of our issues that exist open now are unique? We have been quite good about this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps there is a way to "train" the system that all of our issues that exist open now are unique?

ubiquity-os/plugins-wishlist#50 (comment)

I believe using this system would be effective. Docs would contain the right context anchors uniformly, then when we fine-tune/train our own model with these docs we train it around these context anchors specifically although they'll be pretty effective without it imo.

If we only have docs for closed tasks then this already effectively treats open tasks as unique and a spec contains the why and typically the how also, which our docs contain for each task, we'll have improved performance in my opinion.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we can utilize https://openpipe.ai/. Once we have our embeddings and text corpus ready, we can fine-tune an LLM for this use case. I can elaborate further, but I'm not quite clear on what you mean by 'all of our current issues are unique.' There should be some common themes in the tickets and workflow

Copy link
Member

@0x4007 0x4007 Oct 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unique means that the plugin should not flag them as redundant.

This is a tangible test with a simple boolean result that we can use to tweak our approach.

If the test says any are a match, then keep adjusting.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could tokenize the issues and use the Dice coefficient for comparison. If the coefficient falls below a certain threshold, we can permit the creation of that issue; otherwise, we won't.

Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"title": "Bug: NullPointerException in Python API Handler",
"issue_body": "Title: Bug: NullPointerException in Python API Handler\nDescription: A NullPointerException occurs in the get_user function of the Python API handler when a null value is passed for user_id. Implement a null check to handle this scenario.\nSteps to Reproduce: Pass a null user_id to get_user. Observe the exception in the API logs.\nExpected Behavior: Null values should be handled without causing an exception.\nCurrent Behavior: The function throws a NullPointerException.\nAttachments: Stack trace log."
}
4 changes: 4 additions & 0 deletions tests/__sample__/match_threshold_95_2.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"title": "Bug: NullPointerException in Python API Handler",
"issue_body": "Description: The get_user function in the Python API handler throws a NullPointerException when the user_id is not provided. This needs a null check to prevent the exception and handle invalid user IDs.\n\nSteps to reproduce: Call the get_user function with a null user_id. Check the exception in the API logs.\n\nExpected behavior: NullPointerException should be avoided with proper null handling.\n\nCurrent behavior: Exception NullPointerException is thrown.\n\nAttachments: Exception trace log."
}
4 changes: 4 additions & 0 deletions tests/__sample__/task_complete.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"title": "Review Cache Strategies for Java Service Layer to Handle High Load",
"issue_body": "Description: The Java service layer's performance deteriorates under high load, partly due to the absence of effective caching. Evaluate and implement advanced caching strategies to ensure the service layer performs optimally during peak usage.\nSteps to Reproduce:\nSimulate high load on the Java service layer.\nAnalyze performance metrics and identify bottlenecks.\nExpected Behavior: Service layer maintains stability and performance under high load with improved caching strategies.\nCurrent Behavior: Performance degradation and potential instability under high load.\nAttachments: N/A"
}
5 changes: 5 additions & 0 deletions tests/__sample__/warning_threshold_75_1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"title": "Feature Request: Add Caching to Java Service Layer",
"issue_body": "Description: The Java service layer does not currently utilize caching, resulting in performance issues with frequent requests. Implement caching to improve performance for repeated queries.\nSteps to Reproduce: Make frequent requests to the Java service layer. Monitor performance and response times.\nExpected Behavior: Improved performance with caching enabled.\nCurrent Behavior: Slower response times due to lack of caching.\nAttachments: N/A"
}

Loading