Add valgrind tests

[onurctirtir]

This pr aims to:

• migrate valgrind tests from tools repository to test-automation repository
• automate weekly valgrind tests on circle ci

Replaces #130

TODOs after this pr is merged:

• Remove tooling from tools repo (or deprecate it)
• Valgrind test results should be pushed to a single branch #170
• Putting branch name/sha of the HEAD and pg version to the valgrind test results
• Research github api to understand success of last valgrind test
• Research ways to automate custom tests (maybe pushing branches and removing them automatically ?)

[SaitTalhaNisanci]

I remember talking about the design of this PR a couple of months ago with you. We now have an improved structure in our test automation, with hammerdb tool I had the same too long to execute in one run problem for CI and I implemented that using screen command. Cleaning up resource groups from a VM on azure was also a problem but for that I added a job to test automation and we can delete a resource group by pushing to a branch with the resource group name, there is already a utility script for that https://github.com/citusdata/test-automation/blob/master/hammerdb/delete-resource-group.sh, so calling delete-resource-group.sh ${rg} will delete the resource group.

With those improvements I am wondering if that makes sense to use something similar for valgrind tests, I know that this might mean a big change in the current structure and I am not sure if it is worth that when we already have a working structure but it will make things simpler and we will only have one job instead of two. In the past there were some uncertain parts with the usage of detached session but now I already solved them so we can leverage that. Maybe we should think about this a bit, but the current PR is also working so we can use the current approach as well.

Another thing to make sure is that other jobs should still work correctly with this changeset, we can do this test right before merging

[SaitTalhaNisanci]

Should we mention for the periodic run?

[onurctirtir]

I guess I couldn't quite understand

[SaitTalhaNisanci]

I guess I meant should we mention that there is a periodic run for valgrind tests?

[onurctirtir]

oh now I understood, will apply that too

[onurctirtir]

+ that destroy job isn't actually executed automatically when user follows manual steps, so I think we should not mention this here ?

[onurctirtir]

adressed with last commit

[onurctirtir]

this doesn't seem to be the correct way of checking

[onurctirtir]

should use "==" ?

[onurctirtir]

no, they are very same when comparing strings

[SaitTalhaNisanci]

You can also create an issue to use a new structure for valgrind test. Something like:

"Valgrind job could use detached test structure so that we wouldn't need to have 2 jobs and things would be more clean. This wasn't trivial so we decided to merge the working PR as is"

[SaitTalhaNisanci]

Minor: This could be put right above "cron" field so that it is closer to where it is relevant.

[SaitTalhaNisanci]

We already have tmux in our servers by default now:
https://github.com/citusdata/test-automation/blob/master/azure/init.sh#L24

[SaitTalhaNisanci]

We kind of want to do the cleanup if it fails while creating the cluster I think, but we need the VALGRIND_TEST variable set before running ./create-cluster.sh

We can possibly do the following:

if [ "$rg" == "citusbot_valgrind_test_resource_group" ]; then
    # If running valgrind tests, export VALGRIND_TEST to be 1 to ensure
    # only coordinator instance is created in create-cluster script
    export VALGRIND_TEST=1
fi

./create-cluster.sh
if <VALGRIND_TEST variable = 1> ; then
    # If running valgrind tests, do not run cleanup function
    # This is because, as valgrind tests requires too much time to run,
    # we start valgrind tests via nohup in ci. Hence ssh session
    # will immediately be closed just after the fabric command is run
    trap - EXIT
fi
 

[onurctirtir]

makes a lot sense, thanks