Build an alternative to cardano-db-sync with Ogmios

[CyberCyclone]

Hi,

I'm looking to see if there is a way to implement something like cardano-db-sync but with a different type of database (not Postgres). Could Ogmios be used to replicate db-sync to follow the chain and insert each block into a DB? Does it give enough data for this?

Thanks!

[KtorZ]

Yes. There are people actually working on similar tools, I don't know if they have plan to open-source them but yes it'd totally doable.

One project I had in mind was to actually make a connector to AWS DocumentDB which would be quite interesting.

One thing however that db-sync does (and is arguably the main source of complexity in db-sync...) that you won't get from ogmios is historical rewards history. Indeed, rewards aren't explicitly present in blocks but are computed from the ledger state and requires to (a) maintain a ledger state and (b) redo the reward calculation on each epoch to keep track of them. As such, the node currently offers an API to get the latest rewards values (and so does Ogmios), but does not provide easy access to historical data. Apart from that aspect, I don't think of anything else that wouldn't be already present in blocks.

[CyberCyclone]

Awesome, thanks for that @KtorZ!

I'll take a closer look into it. I've always preferred NoSQL DBs and I'm currently transitioning from MongoDB to RavenDB. I know it would be a big project but it would be pretty rewarding being able to have a lot of advantages of modern DBs, especially with RavenDB.

With regards to the Rewards, I take it that from what you're saying that it is possible to retrieve them with Ogmios as long as I grab them at each epoch ending? I'm aware that with db-sync that it's stored in-memory and if it's interrupted it needs to roll back to the start of the epoch to redo the rewards calculations.

So if I were to do this with Ogmios I would need to implement the same strategy?

[KtorZ]

as long as I grab them at each epoch ending?

If you had a way to force-pause the node synchronization, yes.. But this isn't possible unfortunately. So, at this stage, the only proper way to calculate rewards would require maintaining ledger state, or at least, the bits necessary for reward calculations. Also, to be clear, this is only regarding the historical details about rewards epoch by epoch, it is however always possible to fetch the current reward balance of any account in a single query.

[javiertc]

I would stay away from AWS DocumentDB for the following reasons:

-34% @mongodb API compatibility
-No Sharding support
-No change streams
-No serverless
-Poor documentation and developer experience.

If it's on AWS it should be DynamoDB.

[CyberCyclone]

as long as I grab them at each epoch ending?

If you had a way to force-pause the node synchronization, yes.. But this isn't possible unfortunately. So, at this stage, the only proper way to calculate rewards would require maintaining ledger state

I'll try and see if I can figure out how db-sync does it. I don't think they roll back the node to get the rewards as it can rebuild the whole DB while the node is in sync with the blockchain.

I'm an amateur at Haskell so I may not get anywhere with it, hence I like the concept of Ogmios, great work on it by the way!

[CyberCyclone]

If it's on AWS it should be DynamoDB.

I pretty much avoid AWS due to the insane costs. Even DocumentDB would cost a fortune based on their RU pricing.

[KtorZ]

Not sure what to do next with this issue? Shall we close it or move the discussion into a proper Github discussions (which feels more suited for this type of conversation)?