Only run tests via commands

[razzeee]

No description provided.

[andys8]

Cross link discussion: #47 (comment)

[razzeee]

The real question is how do we achieve this?

[razzeee]

We should probably add a commands for running all tests. At the moment we only run the tests from the file your in on save.

[andys8]

We're back to "executing" tests? If so, I think there has to be an opt-out.

It takes 15 seconds for a toy project with fuzz tests. A well tested production project might be even worse.

There is and should be a big difference between making sure, the test is valid elm syntax, and execution the tests. I'm not sure, but it looks like both is done. Is this correct?

Can't really test because of issue file path. See slack :)

[andys8]

This is the current result:

[Trace - 3:23:19 PM] Received notification 'textDocument/publishDiagnostics'.
Params: {
    "uri": "file:///home/as/dev/repository/vim-emulation/tests/MainTest.elm",
    "diagnostics": [
        {
            "range": {
                "start": {
                    "line": 0,
                    "character": 0
                },
                "end": {
                    "line": 0,
                    "character": 0
                }
            },
            "message": "Equality - Comparison: Expect.equal\n Expected: Insert\n Actual: Normal\n",
            "severity": 1,
            "source": "Elm"
        }
    ]
}

There is currently no line number or test name. So it's hard to tell which test failed.

[razzeee]

Those are not provided by elm test unfortunately.

Best would be to add commands to run these. Until the language server spec implement's something test related.

[andys8]

It provides the test name as labels:

{
  "event": "testCompleted",
  "status": "fail",
  "labels": [
    "MainTest",
    "Main",
    "update",
    "Type a word"
  ],
  "failures": [
    {
      "given": null,
      "message": "Expect.equal",
      "reason": {
        "type": "Equality",
        "data": {
          "expected": "Buffer \"asd1f\"",
          "actual": "Buffer \"asdf\"",
          "comparison": "Expect.equal"
        }
      }
    }
  ],
  "duration": "5"
}

But I'm also very much in favor of using commands instead for now.

[andys8]

Command would be:

• All: elm-test in root project directory
• Specific file: elm-test tests/MyTest.elm

[razzeee]

so how would the labels need to be concated to form the name again?

[andys8]

It looks like the labels are:

• name of the test module
• n describe blocks
• name of the test function

[andys8]

Intersperse the list with > should probably look fine.

[razzeee]

Added

[andys8]

Nice. Do you plan to change it to a command instead of on save?

[razzeee]

Probably, but commands are at the moment, what I keep pushing back.

[andys8]

Yeah, I try to push back executing tests on save, because I won't be able to upgrade then :D

Edit: Configuration is another option, of course.

[razzeee]

Which is confusing to me, as this was already included in the last release

[andys8]

I didn't notice because I didn't write new tests in the last weeks ;) At one point I split up a large test file, because it became unusable.

Here is a demo with latest release. It shows, how long it takes to remove/fix a syntax error, because this'll trigger executing the test. Around 20 seconds. The file is already smaller, because I split up the tests, because of lagging at one point.

[razzeee]

I pushed this WIP branch. https://github.com/elm-tooling/elm-language-server/tree/test-commands

It has two problems:

1. Running all tests -> test runner doesn't say which file it's respond is for testCompleted event should contain file path rtfeldman/node-test-runner#371
2. Running tests from the current file -> Commands doesn't seem to be able to figure out the current file. Might have overlooked something.

So I would basically rip out test support for the moment.

[andys8]

Hey, didn't have a look at the branch, but regarding 1: Is the assumption holding that the first label is always the module name and can therefore deterministicly converted to file name? Is this the case or are there counter examples?

Update: rtfeldman/node-test-runner#371 (comment)