bedrock amazon integration

[apsjohn]

Hey, you should consider adding amazon bedrock intergration to your plugin, it is the only 'enterprise' privacy free solution that is accessible by simple users... Only then such ai integration plugins have a point of existing with privacy in mind... Normal openai api's have no privacy considerations...

[aexshafii]

Hi @apsjohn Thanks for the suggestion. Could you explain how Amazon Bedrock is better than openAI for privacy? Did a quick search and didn't find a good explanation

[apsjohn]

Hey @aexshafii , as you will see here https://docs.aws.amazon.com/bedrock/latest/userguide/data-protection.html
it states "Amazon Bedrock doesn't store or log your prompts and completions. Amazon Bedrock doesn't use your prompts and completions to train any AWS models and doesn't distribute them to third parties."
It is generally accepted in related communities if you search that amazon bedrock is probably the safest choice, privacy wise, probably on par with the wanted 0 data retention enteprise plan that anthropic can directly give in very big customized plans....

[aexshafii]

@apsjohn That is also the case with openAI. They don't store prompts or completion when using the openAI API. See screenshot of our settings below. If you are self-hosting, you'll be able to make changes to the codebase and make it work with bedrock.

We'd like to make it compatible with any model, but that would require enormous amount of work on our end, as each new model comes with it's own peculiarities and requires extensive customization in the code. So we have decided to only focus on optimizing for openAI at the moment. It's the best all-round model for the range of tasks we need to perform with File Organizer 2000.

[apsjohn]

@aexshafii If i am not mistaken, despite supposedely not using your data, they have a 30 days 'log' retention period, as most api's like anthropic's claude etc...
The general concensus is that do you trust the company that those logs aren't being utilized in any way?
Deploying/using instances of those models in aws or azure services, at least in the general concesus, is that it is 'safer' because the model is cutoff and like i mentioned bedrock for example, doesnt keep any such logs...
I think the only model company that offers the same as deploying from azure or aws, is anthropic with their enterprise plans which the explicitly state that no logs are kept vs the normal 30 day log retention for 'safety reasons only'.

I did try implementing it, but it was a bit more hassle than i expected, TS isn't my primary language yet unfortunately.

P.S. if anything has changed and i have missed anything on OPENAI api privacy information, please do inform me, i would be very interested if we find something that will scratch off the BIG privacy concern...

[apsjohn]

I found a very good solution to make almost any llm or provider work with an openai api spec requirement, which i would be happy to share it my test is successful.
To test it i would need a way to specify an api endpoint for your plugin to send the requests there instead of openai api endpoint you are using right now.
Lastly i have tried to run the project locally by setting up a server, but to no avail. The provided docs on how to set it up to run locally aren't complete.

If you could help with the above, i would be happy to test.

[apsjohn]

@aexshafii Any input / update?

[benjaminshafii]

@apsjohn we're adding this today for self-hosters/lifetime members, but won't be adding it in our cloud version.

[apsjohn]

@aexshafii what are you adding direct bedrock integration? Or are you exposing the openai api style endpoint credentials so we can connect custom api's in the form of openai api's specs?

[apsjohn]

Also today i will try to rebuild your plugin and if i manage to get it working, and if indeed it works as described, i will be around as a contributor, so you will see me more around here! :)

[benjaminshafii]

we're making a few models available and a bit of documentation.

we won't be supporting this officially.

current official support is for: gpt-4o and latest gemini flasth

we use the minimal amount of model we can to make sure we don't spread ourselves too thin.

since we're open-source it's possible to experiment on their own.

as of today you can try our (untested) bedrock integration.

references:

• https://github.com/different-ai/file-organizer-2000?tab=readme-ov-file#b-self-hosting
• available bedrom models
• to configure custom model you need to do modify MODEL_NAME and VISION_MODEL_NAME in the .env

[apsjohn]

I successfully managed to build this locally, and use the bedrock integration!
I want to ask if it is normal that the plugin is making organizer requests everytime i:

1. open a new note
2. click in the organizer and back in the note
3. if i am in chat it still makes organizer requests
   ?

Maybe it is a bug with the bedrock implementation or is it normal?
Also i get ratelimit errors from bedrock cause of the simultaneous requests and cause it does requests so frequently when we move around in obsidian (unecessarily) that i am hitting rate limits super easily...

For example in bedrock we have 20 requests per minute rate limit...

[apsjohn]

Using litellm proxy (letting your program make usual requests to openai) and proxying those through litellm to bedrock seems to aleviate the rate limit issues i had with the direct bedrock integration you made.
Although i still see numerous requests done each time i open a note, even if it is the same note...
Can you confirm if this is intended? Cause there should be a mechanism as to when the organizer requests happen and not happen automatically on their own despite having the side panel open or not..