Mapping sub-lattice and their electrons

[ArunPsiog]

Description

Current database, it doesn’t have a direct reference to sub-lattice (in lattices table) and their electrons (in electrons table).

Observation

The parent lattice ID of an electron or sub-lattice refers to the lattice in which each of those is invoked, but the database doesn't have any direct references to sub-lattice and their electrons. Hence, we are facing issues while referring sub lattice to their electron for Graph API.

Suggestion

We can introduce new column in electrons, which will refer to the immediate sub-lattice or lattice (in the absents of sub-lattice).

Change Implementation

Database changes,
1. covalent/_data_store/models.py, To modify for adding the new column
2. covalent_ui/api/v1/database/schema/electrons.py, To modify for adding the new column

Data layer changes,
1. insert_electrons_data in write_result_to_db.py to add new column data.
2. Electrons in electron_dal.py. To add dispatch id in electron id.

Queries

1. Can a sub-lattice be a child node of another sub lattice?
2. How to refer a sub-lattice which is child node an another sub-lattice?

cc: @mshkanth, @Prasy12

[cjao]

Hi @ArunPsiog, with the current schema, one can easily find a sublattice's parent lattice via the Lattices.electron_id column, but I agree that it is inconvenient to go the other way. If I understand correctly, you are proposing a sublattice_dispatch_id column in the electrons table which would be blank for ordinary electrons and populated with the sublattice dispatch id for sublattices, thereby making the relationship between a lattice and its sublattices bi-directional. I think such a change would be quite natural; it would greatly simplify querying all the sublattices of a lattice.

Edit: based on today's meeting I see that I misunderstood the initial question

[cjao]

Regarding your questions:

1. Sublattices can be nested, but every sublattice belongs to exactly one top-level lattice. The set of lattices forms a forest whose roots are the top-level lattices.
2. Once you have a procedure to determine the dispatch ids of the first level of sublattices within a transport graph, it can be applied iteratively to nested sublattices.

[ArunPsiog]

Hi @cjao , Thanks for your reply

We could have a new column named immediate_parent_lattice_id, which will refer to the immediate parent of the electron/sub-lattice.

• For an electron which is a direct child of a lattice, the new field can record immediate lattice id or dispatch id(it can be the same as parent lattice id).
• For a sub-lattice(sl2), which is a direct child of another sub-lattice(sl1). The new field can record immediate sub-lattice id or dispatch id(i.e sl1).

cc: @mshkanth, @Prasy12

[cjao]

From today's meeting, the main problem is that the parent_lattice_id of a sublattice electron always points to the root lattice, not to the immediate parent as intended. @Prasy12 would the new column still be needed after this implementation error is corrected?

[ArunPsiog]

Hi @cjao,

We might still need some reference to the root lattice in electron table, otherwise we need to iterate for every sub-lattice and then show the complete graph to the user, which is an inefficient way.

Lets assume we have multi level sub-lattice, as below image

L - Root Lattice
SL1 - Sub-lattice
SL2 - Sub-lattice (child of SL1)
d1,d2,d3 are the reference to immediate sub-lattice

Case 1: If we have Immediate parent lattice id,

• At first we will get immediate child of root lattice, then we need to identify electron's type, if its a sub-lattice, again we need to get its children.
• For this case we need to iterate twice as we have two sub- lattice.

Case 2: If we have immediate parent and root reference,
We can get all electrons from electron table with the help of root reference.

CC: @mshkanth, @Prasy12

[cjao]

Hi @ArunPsiog, I was under the impression that the user would expand one level of sublattices at a time:

• (L, e, SL1) (initial view)
• (L, e, (e, SL2)) (after clicking on SL1)
• (L, e, (e, (e, e))) (after clicking on SL2)

If this is the case, wouldn't the iterative approach suffice? Each sublattice electron could be expanded with two SELECTs (to get the nodes for the sublattice graph) and a JOIN (for the edges).

[ArunPsiog]

Hi @cjao,

My assumption is that once a user navigates to the graph page, the user can view only the first level children of the root lattice, but not the entire graph (without the children of the sub-lattice at that level). If a user needs to view the children nodes of a sub-lattice, the user can expand the sub-lattice, which will internally trigger an API call to get children of that sub-lattice.

CC: @Prasy12 , @mshkanth

[cjao]

I tried reproducing the issue with this script:

import covalent as ct

@ct.electron
def sub_sub_task(x):
    return x**2

@ct.electron
@ct.lattice
def sub_sub_workflow(x):
    # 1 -> 0
    return sub_sub_task(x)

@ct.electron
@ct.lattice
def sub_workflow(x):
    # 1 -> 0
    return sub_sub_workflow(x)

@ct.lattice
def workflow(x):
    # 1 -> 0
    return sub_workflow(x)

dispatch_id = ct.dispatch(workflow)(3)
print(dispatch_id)
print(ct.get_result(dispatch_id, wait=True))

Starting with an empty database, I get the following table entries:

Lattices (id|dispatch_id|electron_id|completed_electron_num|electron_num):

>>> sqlite3 ~/.local/share/covalent/dispatcher_db.sqlite "select id, dispatch_id, electron_id, completed_electron_num, electron_num from lattices"

1|f347eef4-650e-43e9-b938-3e342380ede6| |2|2
2|68dba942-48f0-401e-91a0-4e0bc1a3a034|1|2|2
3|93cf01bf-b12d-4ce2-b19f-576d41811415|3|2|2

Electrons (id|parent_lattice_id|type)

>>> sqlite3 ~/.local/share/covalent/dispatcher_db.sqlite "select id, parent_lattice_id, transport_graph_node_id, type from electrons"

1|1|0|sublattice
2|1|1|parameter
3|2|0|sublattice
4|2|1|parameter
5|3|0|function
6|3|1|parameter

>>> sqlite3 ~/.local/share/covalent/dispatcher_db.sqlite "select electron_id, parent_electron_id from electron_dependency"
1|2
3|4
5|6

The parent_lattice_id field is being populated as designed:

Starting with the top-level lattice (1), I see that electron 1 is a sublattice. If I want to expand the graph of that sublattice, I query the nodes with parent_lattice_id=2 and their corresponding edges. To expand the electron 3 representing a sub-sublattice, I query the nodes with parent_lattice_id=3.

@ArunPsiog clearly I'm missing something here. Is this example too simple?

[ArunPsiog]

@cjao, I ran your script, It worked fine. I think there are some issues with my script.

[mshkanth]

Hi @santoshkumarradha - Got a download of yesterday's between all of you. Want to quickly clarify expectation reg sub-lattices below -

1
Hope the expectation is that when a user loads the graph, all nodes will be collapsed first. If need be, the user will expand/collapse each sub-lattice and the sub-lattice's electrons will be loaded on-demand. ie Nowhere in the system are we expected a direct full load of the graph (including all sub-lattices & nested sub-lattices).

2

In the new sub-lattices tab where the list of sub-lattices are expected to be shown, again is the expectation only to show root level sub-lattices or are all nested sub-lattices expected to be shown? If it is the former, then there can be some drill down option for a user to drill into one sub-lattive and see the nested sub-lattices under it. If it is the later, then pls see the explanation below -

In the current database, the electrons.parent_lattice_id stores the immediate parent. Hence, it is easy to pick 1 sub-lattice and see all it's 1st level children. However take a nested sub-lattice scenario (Sub-lattice inside Sub-lattice) - To flatten out the entire hierarchy and also show all nested child sub-lattices, the query needs to loop between the lattices & electrons back & forth to retrive the whole hierarchy. To aid this process, we request that a root_lattice_id column also be maintained in the electrons table that will avoid the back & forth loop. This request however depends very much on the requirement 1 and 2 that I have pasted above. Pls let us know.

Cc @ArunPsiog @Prasy12

[Prasy12]

Thanks @cjao Can you let us know when the column addition can be done? It'll be easier for us as well to plan and estimate the whole feature end-end.
If we are in agreement that a new column will be add, for our development purpose we can create our own mock and work on that until you guys finish development and push the feature. Let me know if you have any different thoughts.

cc : @wjcunningham7 @santoshkumarradha @ArunPsiog

[wjcunningham7]

I agree this extra field makes sense. @cjao can you work with @AlejandroEsquivel to scope this and determine the timeline?

[cjao]

Working on this now. I have related suggestion -- since root_lattice_id is really a lattice-level property, would it fit better in the Lattices table?

[cjao]

@ArunPsiog if Lattices contains a root_lattice_id column, it would also be easy to discover all the sublattices under a top-level lattice.

[ArunPsiog]

@cjao,
Adding root_lattice_id in Lattices, would simplify the query to get sub-lattice, If electrons contain root_lattice_id that would help us to get sub-lattices (from electrons.type) and electrons associated with a particular workflow.

I would still suggest implementing root_lattice_id in Electrons, where it's simpler to query sub-lattices and electrons.. Please let us know if you have another opinion? Thanks.

CC: @Prasy12

[ArunPsiog]

Hi @cjao,

Links between,

• Lattice (L) -> Sub-lattice (SL)
• Sub-lattice (SL) -> Sub-sub-lattice (SSL)

Are missing in electron_dependency table. I have used this below script to create sub-lattices workflow,

import covalent as ct

@ct.electron
def sub_sub_task(x):
    return x**2

@ct.electron
@ct.lattice
def sub_sub_workflow(x):
    # 1 -> 0
    return sub_sub_task(x)

@ct.electron
@ct.lattice
def sub_workflow(x):
    # 1 -> 0
    return sub_sub_workflow(x)

@ct.lattice
def workflow(x):
    # 1 -> 0
    return sub_workflow(x)

dispatch_id = ct.dispatch(workflow)(3)
print(dispatch_id)
print(ct.get_result(dispatch_id, wait=True))

With a empty database. The workflow creates,

• 3 lattices in lattices table
• 6 electrons in electrons table
• 3 links in electron_dependency table.

Actual,

           L
             \
      SL      P
        \ 
  SSL    P
  /
P

Expected,

           L
         /   \
      SL      P
      /  \ 
  SSL    P
  /
P

/, \ - Represent links present in electron_dependency table
L - lattice
SL - sub-lattice of lattice (L)
SSL - sub-lattice of sub-lattice (SL)

Refer below screenshot for database entries,

Lattices table:

Electrons table:

Electron Dependency table

While adding the new field (root_lattice_id), Can you add this missing links in electron dependency table?

CC: @Prasy12 , @mshkanth

[Prasy12]

There is also one small problem with on demand expansion/collapse. The idea of the expansion and collapse is when a user clicks on a sublattices on the left side bar the expansion happens. Imagine if there are multi level sublattices.
L-SL1-SL2-SL3... In case if a user clicks SL3 on the left hand bar, we might have to first expand SL1 and then SL2 and then SL3 and put them all in a rounded box which might not be straighforward at all. @santoshkumarradha @cjao

[Prasy12]

Without the links that @ArunPsiog has mentioned, the concept of expanding/collapsing of a sublattice might not be possible as far as we have researched. Please do let us know if you think different and if there is a mechanism to do it.

Hi @Prasy12, in this example, would it be possible to enlarge electron 1 and render the graph of SL1 inside the enlarged electron? Something like

I will give this a bit of a thought and get back to you @cjao

[cjao]

If there is no relationship between 1 and 3 we would not be able to get this box and include 1,3 and 4 together. There has to be someway to know these are related to get that bounding box and resize electron 1.

Caution: The following is all naive thinking and assumes you have a great deal of control over the graph layout with your UI framework.

The size of electron 1 would depend only on the size of the bounding box for (3, 4) and not its origin. Could one first resize electron 1 and then adjust the origin of the bounding box?

In general, suppose one has a nested hierarchy L-SL1-SL2-...-SLn and the user clicks on SLn in the side bar. Would a two-pass approach like the following be feasible?

1. Determine the sizes of the bounding boxes for each sublattice in a bottom-up fashion: For each k starting at n: from the size of the bounding box for SL(k), compute the sizes of the expanded electron in SL(k-1) and the overall bounding box for SL(k-1).
2. Now that one knows the sizes of all the expanded electrons in the hierachy, composite the graphs top-down: for each k starting at 1, render the graph of SL(k), and align the origin of SL(k+1) with the expanded electron for SL(k).

[cjao]

@santoshkumarradha I'm out of my depth here. Please help when you get a chance.

[Prasy12]

@cjao Appreciate your valuable inputs and thoughts. Will go back and see if there is an easier way to do this as well.
@santoshkumarradha If you can confirm if this is the expectation, it would be great.

[Prasy12]

@cjao There is one expectation to have this below format where the whole graph along with the sublattices need to be shown.

With the current schema, do you think this might be a possible option to implement considering we don't have the links in the electron dependecny table as @ArunPsiog had pointed out
?

If not, @santoshkumarradha we might need to think of an alternative to show the sublattices in the graph view.

[ArunPsiog]

@cjao , If we consider the above workflow in the comment,

              [1 (sub_workflow)]
                 /          \
                /              2 [parameter]
           [3 (sub_sub_workflow)]

sub_workflow, is the sub-lattice of workflow and sub_sub_workflow is the sub-lattice of sub_workflow. As per our understanding, with the current schema, there are no direct references stating the relationship between electrons.id 1 and electrons.id 3. Without that reference, graphs look like,

              [1 (sub_workflow)]
                          \
                             2 [parameter]
           [3 (sub_sub_workflow)]

And if the user tries to expand the graph,

              [1 (sub_workflow)]
                          \
                             2 [parameter]
           [3 (sub_sub_workflow)]
               \
                4 [parameter]

Is this the expected behavior?

CC: @Prasy12

[cjao]

@ArunPsiog as I commented above, it would be misleading to draw an edge from 1 -> 3 because there is no input-output relationship between the those two nodes. A more accurate representation would look something like

[2 (parameter)]
       \
         \
-------------------------------
|      [4 (parameter)]          |
|               \               |
|       [3 (sub_sub_workflow)]  |      [1 (sub_workflow)]
|                               |
| ---------------------------   |

[ArunPsiog]

@cjao , In your example,

[2 (parameter)]
       \ 
         \
-------------------------------
|      [4 (parameter)]          |
|               \               |
|       [3 (sub_sub_workflow)]  |      [1 (sub_workflow)]
|                               |
| ---------------------------   |

How do you link electron 2 (Which is a parameter) -> electron 4 and 3( electrons from sub-lattice)? Is there any database link which we missed?

[cjao]

@ArunPsiog In this example, electron 2 is linked to electron 1, which is now represented by the enlarged box. The graph of the sublattice is intended to be superimposed over electron 1's box.

[cjao]

Is this possible without a major redesign of the front-end code? I would imagine you first render the sublattice graph and then compute the bounding box of the sublattice graph in order to resize electron 1 appropriately.

[santoshkumarradha]

Yes indeed that’s the ask, the edge will be connected to a graph (which when I expanded will be an electron of the sublattice.

On Fri, Aug 26, 2022 at 7:59 PM Casey Jao ***@***.***> wrote: @Prasy12 <https://github.com/Prasy12> It's possible if you are willing to enclose the expanded sublattice in a box or some other visual aid to indicate that the edge from L connects to the original sublattice *electron* and not to any node in the expanded sublattice graph. L / \ ----- P | SL | | __ | — Reply to this email directly, view it on GitHub <#1096 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG74SGBYE4VLLZMFPPNNQEDV3DIGHANCNFSM57APJ4YQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Santosh Kumar Radha Agnostiq Inc.