use cache.grains function to display inventory data

[erwindon]

Hi,

When doing some research on why i never get cached minion data on SaltGUI i discovered #538 which mentions there is a need for a database, however i found another possible solution:

Instead of using a Backend-Database to display some inventory/stored grains data of minions it should be possible to use the cache.grains function as documented in https://docs.saltproject.io/en/latest/topics/cache/index.html it looks like this cache also supports multiple backends/modules, see https://docs.saltproject.io/en/latest/ref/cache/all/index.html#all-salt-cache for a list so the database question is also answered.

It looks like implementing this could be really easy, actually a config-file option which exchanges the "normal grains command" with the "cached grains command" should do the job perfectly. After that only the documentation of salt for cached grains has to be followed.

If somebody wants to implement, just ensure to not hit saltstack/salt#48694

As we are just starting with salt, i would be willing to test.

Kind regards

Mike

[erwindon]

note: the cache API is in https://docs.saltproject.io/en/latest/ref/runners/all/salt.runners.cache.html

[erwindon]

this looks like a nice addition to SaltGUI as it can be implemented just using the salt-api. any decision on the implementation of the cache will not affect SaltGUI.

when made available in SaltGUI, it should be done for all 3 categories: grains, pillars and mine. (support for 'mine' is on a development branch only at this moment).

steps:

• investigate whether the proposed solution (using either regular data or cached data) is working fine, or that we need to use cached data only when regular data was not returned
• investigate further to see whether the solution can be applied everywhere. e.g. some screens use grains to show extra data.
• investigate gui design for cases where minion is offline, but cached data is available. how to display both the cached information and the offline status.
• add configuration to specify which categories (grains/pillars/mine) should use the cached data
• add new api functions
• (more steps depending on the investigations...)

[erwindon]

technical overview, current function use:

• grains
  JS function getLocalGrainsItems in Grains+GrainsMinion+Minions+Nodegroups[*]
• pillars
  JS function getLocalPillarItems in PillarsMinion
  JS function getLocalPillarObfuscate in Pillars
• mine
  JS function getLocalMineValid in Mine+MineMinion
  JS function getLocalMineGet in MineMinion

[*] Nodegroups screen is only reachable when at least one nodegroup is defined

[erwindon]

output for (salt) grains.items:

salt-minion-1:
    ----------
    biosreleasedate:
        12/01/2006
etc.

output for (salt-run) cache.grains:

salt-minion-1:
    ----------
    biosreleasedate:
        12/01/2006
etc.

--> OK to use as replacement

[erwindon]

output for (salt) pillar.items:

salt-minion-1:
    ----------
    en:
        top secret 1
    nl:
        zeer geheim 123

output for (salt-run) cache.pillars:

salt-minion-1:
    ----------
    en:
        top secret 1
    nl:
        zeer geheim 123

--> OK to use as replacement

[erwindon]

output for (salt) pillar.obfuscate:

salt-minion-1:
    ----------
    en:
        <str>
    nl:
        <str>

there is no strict equivalent that uses the cache.
but cache.pillars can still be used here.
pillar.obfuscate was only used to hide its output when only a count of the pillars is needed on the Pillars screen.

--> OK to use as replacement (i.e. use cache.pillars here too)

[erwindon]

changed this github issue to a PR

[erwindon]

decision: let's not bother with the mine function. that is on an experimental branch and still incomplete

[erwindon]

@mbgevers
The branch now contains a working version of SaltGUI that is capable of using cached grains and/or cached pillar information.
The current behavior remains the default.
But that can be overruled by adding one or both of these settings to the /etc/salt/master file:

saltgui_use_cache_for_grains: true
saltgui_use_cache_for_pillar: true

but you can also switch it on for the current session in the Options screen (ctrl-click on the SaltGUI logo).
On all screens where it applies, a warning message is added about the use of cached information.

Can you give it a try?
Let me know any problems or suggestions...

[mbgevers]

@mbgevers The branch now contains a working version of SaltGUI that is capable of using cached grains and/or cached pillar information. The current behavior remains the default. But that can be overruled by adding one or both of these settings to the /etc/salt/master file:

saltgui_use_cache_for_grains: true
saltgui_use_cache_for_pillar: true

but you can also switch it on for the current session in the Options screen (ctrl-click on the SaltGUI logo). On all screens where it applies, a warning message is added about the use of cached information.

Can you give it a try? Let me know any problems or suggestions...

Only seen this comment now, yeah for sure will give it a try

[mbgevers]

Hi, doing some testing at the moment, looks good so far, however found some drawbacks. There should be some symbol indicating if a minion is online (thus no cache used) or offline (cache used) what do you think about that? So cached should be a fallback. Thx much for your work already!

[erwindon]

[...] if a minion is online (thus no cache used) or offline (cache used) [...]
reminder: technically, SaltGUI is now either using cached information or live information, never a combination of that

There should be some symbol indicating if a minion is online [...] or offline [...]
previously that was visible by the indication "offline" in the status column.
in the no-cache case, that remained visible as no grains were returned from offline minions.

I'll investigate on how to display the offline status in a different way.

[mbgevers]

[...] if a minion is online (thus no cache used) or offline (cache used) [...]
reminder: technically, SaltGUI is now either using cached information or live information, never a combination of that

There should be some symbol indicating if a minion is online [...] or offline [...]
previously that was visible by the indication "offline" in the status column.
in the no-cache case, that remained visible as no grains were returned from offline minions.

I'll investigate on how to display the offline status in a different way.

I just had the idea, it should be sufficient if you just write the "cache timestamp" or "last cache update time" if that info is somehow available. Another maybe even easier method would be to only load cache if device is offline.

[erwindon]

[...] it should be sufficient if you just write the "cache timestamp" or "last cache update time" if that info is somehow available.

no such facility is described in https://docs.saltproject.io/en/latest/ref/runners/all/salt.runners.cache.html

Another maybe even easier method would be to only load cache if device is offline.

there should be no problem to always using cached information. it is used often in SaltStack. e.g. each time you the salt command with a grain-based target, the cache will be used to determine the actual target. see also https://docs.saltproject.io/en/latest/topics/targeting/grains.html.

The problem that we still need to solve is that regular salt-functions (those executed on minions) indirectly provide availability-indicator as extra information because they fail. In SaltGUI this was even optimized by using an extra call (wheel.minions.connected), so that the availability information is already visible while you are waiting. But with the use of cached information, that indicator is overwritten by the cached information. I'm now changing the presentation of that, so that it remains visible in all cases.

[erwindon]

@mbgevers
I've added a visual indication on the main screen when a minion is offline.
it is added only for minions for which there is information available, therefore the traditional case is unchanged.
does that work for you too?

[mbgevers]

@mbgevers I've added a visual indication on the main screen when a minion is offline. it is added only for minions for which there is information available, therefore the traditional case is unchanged. does that work for you too?

That looks nice, thank you.

[sonarqubecloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud