Relevance Augmentations

[0x4007]

There's a few high impact elements to determine the relevance of a comment. Which is why in the default formatting scoring we have high ($5) credit per instance.

image

Relevance currently doesn't understand image contents.

We can insert the image description inside of its "alt" attribute.

<img width="106" alt="@0x4007 profile pic" src="https://github.com/user-attachments/assets/a4109ba7-fdeb-49e0-b492-039cea79bcc7" />

link

It could also be really interesting to pull the text contents from links to also score their relevance.

We can insert a summary inside of its title attribute.

<a href="https://example.com" title="A summary of the link's contents">Example</a>

code

It would be interesting to discuss this in a separate proposal because this seems like it could be really complex to implement well.

Original context ubiquity-os-marketplace/command-wallet#28 (comment)

[ubiquity-os-beta]

Note

The following contributors may be suitable for this task:

sshivaditya2019

0% Match ubiquity-os-marketplace/text-vector-embeddings#36

[0x4007]

Saving the standardized metadata for the images can be tricky given that our metadata parser in the SDK seems unstable. @whilefoo can you add input on the state of this?

I think @sshivaditya should add input on what's best for the vector embeddings to understand image contents.

The problem I'm trying to solve here is that I don't want to ask LLMs to describe an image both in the conversation rewards and vector embeddings plugins. It's redundant work. So we can solve for this with my HTML comment metadata proposal above.

[sura1-0-1]

/help

[ubiquity-os-beta]

Available Commands

Command	Description	Example
/help	List all available commands.	/help
/allow	Allows the user to modify the given label type.	/allow @user1 label
/ask	Ask any question about the repository, issue or pull request	/ask
/query	Returns the user's wallet, access, and multiplier information.	/query @UbiquityOS
/start	Assign yourself and/or others to the issue/task.	/start
/stop	Unassign yourself from the issue/task.	/stop
/wallet	Register your wallet address for payments. Use '/wallet unset' to unlink your wallet.	/wallet ubq.eth

[ubiquity-os-beta]

+ Successfully set wallet

[sura1-0-1]

/wallet 0xB13260bfEe08DcA208F2ECc735171B21763EaaF6

[ubiquity-os-beta]

! Failed to register wallet because it is already associated with another user.

[sura1-0-1]

/wallet 0x331D1C984A43087427BBC224Cb4aD9f019336e75

[ubiquity-os-beta]

+ Successfully set wallet

[sura1-0-1]

/start

[ubiquity-os-beta]

Deadline	Sat, Dec 21, 9:51 AM UTC
Beneficiary	0x331D1C984A43087427BBC224Cb4aD9f019336e75

Tip

• Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
• Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
• Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

[sura1-0-1]

@0x4007 Can you provide more details on how the relevance score should be calculated? Are there specific weights or factors to consider for text, images, and links?

[0x4007]

@0x4007 Can you provide more details on how the relevance score should be calculated? Are there specific weights or factors to consider for text, images, and links?

That's already built and out of scope for this task. Refer to the existing prompts for inspiration on new prompts specific for these.

[ubiquity-os-beta]

Passed the disqualification threshold and no activity is detected, removing assignees: @sura1-0-1.

[3scava1i3r]

/start

[ubiquity-os-beta]

! You do not have the adequate role to start this task (your role is: member). Allowed roles are: collaborator, admin.

[3scava1i3r]

I finished my work should i PR then?

[3scava1i3r]

There seems to be this issue that if i have description and title the amount of words are increasing leading to threshhold reach for task

[0x4007]

@gentlementlegen you should figure out how to best solve this problem. We want to ensure that images with high relevance are scored higher. I'm assuming adding the description will help a lot but not sure if that's the best way for the conversation rewards to interpret and do the right thing.

@sshivaditya let us know what model is best and what settings for this use case.

[shiv810]

@sshivaditya let us know what model is best and what settings for this use case.

The usual models (such as 4o and Sonnet 3.5) would work, but a better choice would be Qwen VL, which excels in image understanding and handles OCR efficiently.

Since we are already using OpenRouter, the only change needed is to switch the model type to one that supports vision. And, the input structure should be updated as follows:

"messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Understand this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "<IMAGE_URL>"
            }
          }
        ]
      }
    ]

[3scava1i3r]

Got it thanks, just testing the changes and pushing it to PR today

[ubiquity-os-beta]

Important

• Be sure to link a pull-request before the first reminder to avoid disqualification.
• Reminders will be sent every 2 days and 8 hours if there is no activity.
• Assignees will be disqualified after 4 days and 16 hours of inactivity.

[3scava1i3r]

Thank @gentlementlegen

[3scava1i3r]

@sshivaditya let us know what model is best and what settings for this use case.

The usual models (such as 4o and Sonnet 3.5) would work, but a better choice would be Qwen VL, which excels in image understanding and handles OCR efficiently.

Since we are already using OpenRouter, the only change needed is to switch the model type to one that supports vision. And, the input structure should be updated as follows:

"messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "Understand this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "<IMAGE_URL>"
            }
          }
        ]
      }
    ]

I have updated the code as per the required logic using OpenRouter but am currently unable to test it. Could you kindly confirm if an OpenRouter API key is necessary for testing? Additionally, do I need an OpenAI Plus or Pro membership to proceed with the testing?

[0x4007]

Make an open router account and get an api key to test