Statically parse, rather than run, test files to extract @testitems

[nickrobinson251]

We could change the @testitem scanning from calling include to instead parsing the files (e.g. Meta.parse(read(file, String))) then walk the AST looking for top-level @testitems that we then eval.

This would have a few potential benefits:

• We could error on any non-top-level @testitems (/ @testsetups)
• We could @warn on @testitems that were not in _test.jl files
• We could error on any top-level code in _test.jl files that's not a @testitem (or @testsetup)
• We wouldn’t necessarily need the _test.jl suffix, since we could scan any file and just eval the @testitem

This is also more similar to what the VS Code extension does (#18)

[NHDaly]

We wouldn’t necessarily need the _test.jl suffix, since we could scan any file and just eval the @testitem

One possible policy that i think could be good for everyone would be any file that contains a @testitem or @testsetup must only contain testitems and testsetups. That could be a way to do away with the _test.jl suffix and still keep the other benefits of catching malformed testitems.

(But this aspect is much less relevant than the main thrust of what you're describing here, and i'd rather start with what you proposed 👍)

[NHDaly]

@nickrobinson251 oh haha sorry i forgot we'd already detailed these steps. sorry i spammed it on that slack thread!

[nickrobinson251]

i'm still gonna add you suggestion here 😁

literally replace include() with this simple function that does exactly the same thing:

  julia> function my_include(file)
          block = Meta.parse("begin\n" * read(file, String) * "\nend")
          exprs = block.args
          for expr in exprs
              eval(expr)
          end
      end

We can just start with that, which is a noop change, and then literally just add some super tiny error checks in there that each expression is either at @testitem or a @testsetup and boom we've done it, and now we've switched to the "static parsing" approach but everything else is unchanged! :)

We can even continue to run includes as well, if we want to, to make the transition easier

[NHDaly]

we didn't end up doing this yet, did we? Just trying to remember and clean up issues

[nickrobinson251]

In #78 we switched to first iterating over the parsed Exprs as part of the include, so we can assert that every Expr is a @testitem or @testsetup macro call. But we still only include files with the _test.jl or _testsetup.jl suffix.

[nickrobinson251]

After #170 (+ #179) we just parse the file to do all selection/filtering of @testitems (by name or tags)

I'm closing this issue as done and have opened an issue for the remaining idea from this thread: #181