Internal Cross Reference Simplifications using Markdown Links

[rowanc1]

Context

We plan to propose a cross-reference syntax that uses CommonMark links to support all use cases of cross-referencing both internally to a project and externally using intersphinx. The syntax aims to be familiar and work across different rendering platforms. Most internal content can be referenced using a hash-link, [](#my-id), which is the recommended replacement for the multiple role options that can do this in MyST currently (e.g. {ref}`my-id`, {eq}`my-id`, {numref}`my-id`). We also introduce a myst scheme/protocol, which allows for rich cross-project referencing and is compatible with intersphinx. For example, a [](myst:spec#crossReference) link will create a rich cross-reference to the MyST spec documentation. We provide options for increasing specificity for these links in all cases to deal with duplicate references across pages, domains, and projects.

Draft Content:

The draft of this MEP will be completed in HackMD, and then brought to a GitHub PR when ready for review.

• https://hackmd.io/us8ZmFHrTjaHBGkkdoz0CQ

Suggested Authorship:

• @chrisjsewell
• @rowanc1
• @fwkoch

Discussions and Ideas:
To improve threaded replies we will be using github discussions:

• Cross Reference Improvements #7

Suggested reviewers:

• Hopefully some power jupyter-book users for feedback!

Draft implementations, docs and prior-work:

• myst-parser (python)
  • ❗️ REFACTOR markdown links executablebooks/MyST-Parser#613
  • https://myst-parser--613.org.readthedocs.build/en/613/syntax/syntax.html#links-and-referencing
• mystjs (javascript)
  • curvenote/curvenote@5460169 (when it was curvenote)
  • https://www.myst.tools/docs/mystjs/external-references
  • https://www.myst.tools/docs/mystjs/cross-references

Some related links:

• Better link syntax for cross-references executablebooks/MyST-Parser#548
• Markdown [...](...) syntax does not play well with sphinx.ext.autosectionlabel executablebooks/MyST-Parser#569
• Support numref for rendered images jupyter-book#1383
• Reference an admonition? jupyter-book#1201
• Allow interactive visualizations to be "figures" (and referencable) jupyter-book#1191
• Canonical way to handle theorems / lemmas / proofs jupyter-book#748
• WARNING: 'myst' reference target not found jupyter-book#1657
• Intersphinx bug in our docs jupyter-book#1198

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[choldgraf]

Could I make a process suggestion? In order to make it clear what are the remaining questions or uncertainties that need to be resolved, I recommend two things:

1. Provide a list in this issue of any questions that you, as the author(s), think must be answered before we can move forward. This is where people should focus their reading and comments. All other comments in the doc are assumed to be non-essential or FYI.
2. Provide an expected timeline and next actions to take, so that others know when they need to give feedback or explicitly request extra time to discuss.

I think it's helpful to know whether / when we are converging so that we don't bog down discussions with uncertain paths towards resolution.

[rowanc1]

Thanks @choldgraf, I have also added an MEP: Draft label here, at this stage anyone who wants to have input can (we are working openly!) but there will also be a chance to more formal review once we set it to "Active". The current TODOs that I see on the draft before it gets wider "Active" status/review are:

• agree on using three "schemes"/protocols (project/path/myst). There is some redundancy here and we can possibly just simplify to myst, but we should make sure we don't loose anything. Alternatively we could go in the other direction, adding download instead of path?
• agree on syntax for interpolating text/title into a link template (currently suggested as %t, which is similar to Fig. %s). In sphinx this is {name}, which can also be supported, but maybe not mentioned in docs, etc.
• Some implications of the "scheme" choice are on downloads and how to download a markdown file easily, for example.
• I need a review from @chrisjsewell on the "search order specificity" and "implicit" targets in a document
• TODO: add some information on the crossReference spec AST changes.
• TODO: examples (I think these can be pretty light, as they are mostly laid out above)
• TODO: Look through a few more of the linked issues above to flag anything that users have raised (esp. the discussion in Better link syntax for cross-references executablebooks/MyST-Parser#548).

I have put these at the top of the draft doc, and will keep it up to date over there!

Re timeline: Ideally the authors (currently me, @chrisjsewell and @fwkoch) can get to some agreement in the next few weeks, turning this active in the next two weeks? Given holidays etc. I think we should have more time for review after it goes active.

[choldgraf]

I saw that a PR got merged in the python parser that implements some of this - I think that we should align on a pattern to use before people start implementing things. If folks want extra discussion or decisions on something, please just ping folks so that they know it's a good time to provide feedback.

Looking at the above unresolved points, I'll add suggestions for each below:

agree on using three "schemes"/protocols (project/path/myst). There is some redundancy here and we can possibly just simplify to myst, but we should make sure we don't loose anything. Alternatively we could go in the other direction, adding download instead of path?

I suggest we just use myst: for now, as it is in the proposal. I don't think there is a clear "right" answer here, and there are pros and cons on both sides, so my rationale is:

• A single myst: protocol is easier to remember than many protocols. It also means that under-the-hood it can be extended by developers without requiring users to remember new protocols.
• We might want to add more protocols in the future, but I think its better to start with the simplest / smallest UX change for people.
• We can advertise this functionality as experimental, and if it changes in the future, it won't be hard for people to do a find/replace for myst:// to something else.

agree on syntax for interpolating text/title into a link template (currently suggested as %t, which is similar to Fig. %s). In sphinx this is {name}, which can also be supported, but maybe not mentioned in docs, etc.

Let's go with %t for now. I think it might be useful to support more extensible patterns like {name} but that could be a future proposal to extend functionality. We don't have to get everything perfect the first time.

[chrisjsewell]

I suggest we just use myst: for now, as it is in the proposal.

IF, this protocol is being used solely to refer to Sphinx objects.inv formatted files, then I don't feel that it should be called myst.
It feels misleading and somewhat disingenuous to suggest that it's a myst format 😬

[choldgraf]

My understanding is that this would be used to refer to many other kinds of cross-project references as well, but I may be misunderstanding. Can you and @rowanc1 and @fwkoch align on and clarify the potential uses for these links so we can make a decision and move forward?

[chrisjsewell]

My understanding is that this would be used to refer to many other kinds of cross-project references as well, but I may be misunderstanding.

this was not the case in executablebooks/MyST-Parser#613, nor is it the case in executablebooks/MyST-Parser#673

[chrisjsewell]

A single myst: protocol is easier to remember than many protocols.
but I think its better to start with the simplest / smallest UX change for people.

I feel its a lot clearer, that inv: -> objects.inv, and a lot simpler, easier to implement and easier to understand (see https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html#cross-project-inventory-links)

At least in myst-parser, the rest of the stuff, with "local project links" etc, is a heck of a lot more difficult and complex to do

[choldgraf]

Can y'all get together and agree on a specific proposal to move forward the MEP? That is the source of truth for what the syntax will be, so we should focus discussion on the proposal there, get alignment, and make decisions so that we can move forward.

[rowanc1]

Hi @executablebooks/steering-council @executablebooks/core-team, we are setting this MEP to active!! 🎉

You can see the MEP here: https://myst-eps--10.org.readthedocs.build/en/10/meps/mep-0002/

Very exciting. :)

The documentation for the decision making is here -- excited about the process, thanks for your help!

We should add comments directly to the PR I think!

• Internal cross-references MEP #10

[rowanc1]

Thank you all for your work on this over the last several months - the MEP has been accepted and is available here:

https://mep.myst-tools.org/en/latest/meps/mep-0002/

As per the docs on the process, I am going to close this issue as Accepted!

Thanks everyone for participating in this process, very excited about this MEP. 🚀