Make URLs shorter via encrypted cloud storage?

[borekb]

Hi, I've recently came across this amazing 😍 service, it's so nice that I don't need to install all the tools locally!

If I understand correctly, the hosted https://kroki.io/ doesn't persist any data – in only uses compute (VMs by Exoscale) but there's no long-term storage of any data. This means that the entire diagram has to fit into the URL, via deflate + base64 as described in the README.

I wonder if you would consider Excalidraw-like approach which uses a clever technique:

1. Encrypt a diagram in the browser.
2. Upload the unreadable blob of data to an S3 bucket or similar.
3. In the browser, construct a permalink that stores the decryption key in the anchor / hash part of the URL: excalidraw.com/#json=...

This means that Excalidraw URLs are fixed width regardless of diagram complexity:

Diagram	URL
Simple	https://excalidraw.com/#json=5645858175451136,8w-G0ZXiOfRYAn7VWpANxw
Complex	https://excalidraw.com/#json=4680957890134016,q6NWURUuPP_VThmbRQ89Jg

For comparison, here is the same scenes via Kroki:

Diagram	URL
Simple	https://kroki.io/excalidraw/svg/eNrtmltXm0gcwN_9FJ70tU7nP_fpm9fG7dZbqrHu6fEgIQlKgAIxSXv63XdADQkJirZaulseNMyFGWZ-_-vwbWV1tZFMQqfxdrXhjG3LczuRNWq8TsuvnSh2A99Ukew-DoaRnbXsJ0kYv33zJu-B7GBw08vxnIHjJ7Fp94-5X139lv2dGSdy7MTye56Tdciq8qEYV8XSvcDPhgUuMeGCEJi2cOMtM17idEx11_JiJ69Jixq0s03aTscbbhwdnMedjcnWmrTyYbuu57WSiXfzUpbdH0Yzk4qTKLhy2m4n6aejF8qn_eLALEHeKwqGvb7vxPFcnyC0bDeZpGUYT0tvVuHtal4yNneaIyoJlpgC4dOatC9RGGlJhFkEXpjOZuAFUTqdV-RCWXTmNS4s-6pnZuV3pm2SyPLj0IrMPuXtRncvKhliShE1O3rfcXv9JK2lCulibexkO6AFZpxgIqYV6bjhbieD4XNxAftWFN4uVCNOb2bmnE53-4aku-5Z3ffXy5lKnHGyDCcicSlOkjGupSKyMk7buxMdr1_j0-Hk-Cxo-e5Z73i37jgBBgRUC6YYlUWeJCKMAcPPCRSXixgtgYcyIeF50Ml3KPCTlvs1XSSC50p3rIHrTeYWOWPKDLDpuXPvdWHFjuf6GUNqrvW65_ZS6hqe053HMXGNppxWJ0HYeBLOgEkZzgQYo0CFqkzzid0_GTtDHnah3w_p35eyO6J1p5kqgjDDmOo5ZjOaJUflFNuaWMR6OsVSVaEYtKRaMyaghiDvOckoiK5elOR7jT0FKMOZagmEcKIr43y9teGpD14E-6MgiCYda63lq6fhTF4OZ6kR1qxo09POlGikQNyvm3F2_YCxxwzxojBN-aamknKYdzdu1TUwQQVltD62HspdR8GAUEVFdc8xOp4cfPLfb23qT7td7Z6u984N8XVXjpqaDSNkGU3pXiqq552An42TJFWUJBcgpRKE1lBH2qb6pa39vTrS7GepB0uI0pQC5pWxPqXRu4_jCIuL9yetVnh99JfYPqm9kuQUGXvABNZFD5YyhkBRrp-TatBGR3I1P0geERFEGS9Ea7ekE6DmMv61fnk1mUG7LMKmtFRNGheKcK1xZZ4GHTk43B73z8E6sj8enx0P2s3rEp7sKIjjtb6V2P06MCXNti03vIwiDPwHze4rJZSjxRKaCJJMYSbUUpoA4QJpd64l10bazYSf4FpmU6uoNj0rTjaDwcBNzLYfBK6fTssfet7MAFaUrEdRMOo7VqdY6_id0rowfVyeGkqv_NdqvkXZzfT359dLW5tAgEqY0wnptaYQAM0SFA89IRVdSuYkN3uCRmD0arYLFR6xsJtZOVKacDk_i5XZ_493b8Q9mTGhjTcmqqcyrtXBBfV70b43uTi6_NDfEZNLu_apDMWQMXl40bkhiD2UF3sZ34ZgRkzowmZs8v_bubFSXbA0-NO81A4Z3cmJIqwyzs6n3mEL7zWPqPh6uXnU4V_a7XHdcTY6CnGNFVnM9FLjc0haTLP-5JQG0RgJqZcbIoKIJvN1d4bIiJGZsLFHv5UhKpL4c23R4lJmliRfxUdagVKxEaTUDChtzBCRsnrOxP9wGlx13_t-YLdsQq1TvofDuosNF8L4_FgUUiOZ2EgTDgitCSj5jHKjKJKSsqXhAEYLM7uVGxMIaMkxFX_EJpePxaXMyvNVfKTc3B9LQ6nsMCJNGDkbqD0YSjMIx01XDr_Q8HBIN9u93UNVe9nRBAFeyAPdpc8xls9qcZ5-tgjcOLmcA65RwpGq0vQ1KADgxhOszlOLHm-qyOH7obfedE7Y4f45n9SdJ6GlkV_MC9zcHVYTE5Q8rw8jKnnlqTepOFe4hk55y4kMPDU6lGHlX2BQjI3jpx9xyAjtrR34mw6CVsvubRxFbqD2j-uecDTQIsHBxNNzfsRNqAmokLupzZkMYM6ZUTozXzT8-kOZ8iiPYCU1SHhE1gIbD_XSPttvsjNhnUTNvZ2v0Kq9jmQSYaXUUpo4UsAYp_DLMxdGOypqJqj_JC4qeJJCl5_KMCZV-v1WZaxFM3wXdneI7I8ugETJ4NKJ9muvJLFAmjMopMpvMugcPfRV0a87lAFCKDXxI6vTocw936kBKE4149V56p2AI955_sfT5PAMmm3_4Mvl6e9wKiMAI42XG17GECkeAdbiXEYrwgGEYn_OZf4T5zIrtw9tWGHYSqwkxfRGkM3Gup1b45WvX-PadUYbS0jrZldqZb6vfP8Xx186Rw==
Complex	Doesn't even render:

I understand that Kroki.io is simpler when it doesn't need to worry about data persistence but I hope that this example shows that the benefits of Excalidraw-like approach are substantial.

BTW a practical limit for URL length seems to be only around 2000 characters / 2kB, probably a bit more in modern scenarios but it's still way too little for powerful diagramming tools like Excalidraw or draw.io/diagrams.net (which is not currently implemented but would be super-nice if it was, and there's an issue for it: #405).

[ggrossetie]

Hi @borekb,

Thanks for your kind words abour Kroki! 🤗

If I understand correctly, the hosted kroki.io doesn't persist any data – in only uses compute (VMs by Exoscale) but there's no long-term storage of any data. This means that the entire diagram has to fit into the URL, via deflate + base64 as described in the README.

That's correct.

I wonder if you would consider Excalidraw-like approach which uses a clever technique:

This is a good idea and something I've considered but I'm relunctant to persist data (even if they are uncrypted).
And to be completely honest, if we end up providing this feature it will most likely require a subscription.
Why? Because we are now responsible for the data privacy/security and backup/recovery which come at a cost.

I understand that Kroki.io is simpler when it doesn't need to worry about data persistence but I hope that this example shows that the benefits of Excalidraw-like approach are substantial.

I'm fully aware of the benefits of such solution but also its constraints 😄
If we want to provide a good user experience, we would probably need a diagram editor where we would be able to "save" a diagram using a shorter URL. We could even use a user-defined name (i.e., https://img.krokio.io/borekb/client-server.svg). Lot of exciting possibilities but we are now talking about a fullblown product that requires funding.
The question is, are users willing to pay a subscription fee to get such features? So far I've been fortunate enough to get Exoscale support but it does not cover all expenses.

Having said that, anyone can implement/provide this feature on top of Kroki.

BTW a practical limit for URL length seems to be only around 2000 characters / 2kB, probably a bit more in modern scenarios but it's still way too little for powerful diagramming tools like Excalidraw or draw.io/diagrams.net (which is not currently implemented but would be super-nice if it was, and there's an issue for it: #405).

Yes, the max URL length is a well-know limitation (described in the documentation: https://docs.kroki.io/kroki/setup/configuration/#_max_uri_length).

[borekb]

Thanks for the response, I realize it would be quite a big change for the infrastructure / hosting and fair enough if this is too much for Kroki.

I had this idea in mind for the setup:

• Some serverless (pay only for the actual time consumed) hosting of Docker containers, like Google Cloud Run.
• Storage like S3 or R2 or Google Cloud Storage.
• Client-side encryption in the Kroki web app, similar to what Excalidraw does.
• A statement in Privacy Policy that the server never stores readable data.

You're right that anyone can build this on top of Kroki but I personally don't have the time right now (and in the near future), however, I'd be happy to sponsor a service that results from that, as in to cover the monthly costs. If this is something you'd like to discuss, feel free to reach me privately at borekb@gmail.com.

[ggrossetie]

You're right that anyone can build this on top of Kroki but I personally don't have the time right now (and in the near future)

Me neither, I don't have time in the near future to take on this effort.

however, I'd be happy to sponsor a service that results from that, as in to cover the monthly costs. If this is something you'd like to discuss, feel free to reach me privately at borekb@gmail.com.

Thanks, I will keep that in mind 👍🏻

I didn't mention it but another workaround is to use a POST request: https://docs.kroki.io/kroki/setup/usage/#_post_requests

[sturtison]

From a privacy and IP point of view, the use case of Kroki not persisting data is very strong too. Having Kroki persist anything at all would make it a problem for many users and in many organizations this would shut down the discussion about using it.

That Kroki is able to be deployed in containers locally, with no external traffic, is a great architectural decision and its implementation has made using diagrams-as-text exceeding useful.

That said, the idea of persistence in the way you described is appreciated.