Create a basic integration test

[xeger]

Wow @kddnewton; when you said that writing tests for VS Code extensions was hard, you weren't kidding! I took it as a personal challenge -- and I have some other extension ideas, and wanted to delve into it a bit anyhow.

We now have a single test case, but it's a nominally useful one: prove that formatting works.

Not sure how actually-useful this is, especially if it isn't hooked into CI, but it's perhaps a bit easier than testing things by hand. Especially if we can use the APIs to inspect overlays, this should be sufficient to test basic functions, settings, and commands.

If you like the idea, I can look into adding a GitHub job for CI coverage. (I'm sure it will be pure joy getting Ruby, Node, Electron and all of the bells and whistles crammed into a container; at least Microsoft has some examples to get us started.)

[kddnewton]

Yeah I think this will work. Shouldn't be too bad to integrate it into GitHub actions. I had to set up something similar with prettier which is here: https://github.com/prettier/plugin-ruby/blob/main/.github/workflows/main.yml#L23-L32 should get you close to what you need.

[xeger]

Here's something that runs, but hangs. Based on official reference docs. Works locally on my Mac laptop.

The dbus warnings are probably harmless but the "socket got disposed" stuff is suspect; something's not running right.

[kddnewton]

@xeger looks like this is maybe working now? what's the status?

[xeger]

It's not quite working; I can get VS Code running in CI, but it hangs when we issue the command to format code. Need to SSH in and add tons of printf debugging. Thrilling!

The test suite is runnable in local dev; if you'd prefer, we could merge it now and hold off on the CI stuff. (Just merge everything except the .github changes in that case.)

[kddnewton]

Nah let’s keep the PR open. I’ve worked in code bases with tests that are local-only and they almost always go stale.

…

On Thu, Jul 7, 2022 at 8:33 PM Tony Spataro ***@***.***> wrote: It's not quite working; I can get VS Code running in CI, but it hangs when we issue the command to format code. Need to SSH in and add tons of printf debugging. Thrilling! The test suite is runnable in local dev; if you'd prefer, we could merge it now and hold off on the CI stuff. (Just merge everything except the .github changes in that case.) — Reply to this email directly, view it on GitHub <#15 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABG3P3QPWJFYYO27VG6V2Z3VS5ZN5ANCNFSM52OJ2O3Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[xeger]

Okay @kddnewton: I have something that is workable if not ideal. Some issues are chronicled below.

Too Slow

The biggest issues with testing are performance and visibility: stuff is just slow; in CI it's very slow, and extremely difficult to understand what has happened when things fail.

I spent Friday evening hacking Xvfb and ffmpeg to grab videos of the Code window during test execution, which was the key to understanding. I'd like to make this more elaborate in future (e.g. new video for every test case) but as-is, our extension is probably in the top 1% of usability among all extension test suites (of the few that even have tests!) I will turn the dbus/xvfb/video stuff into a GitHub action at some point because it's useful with other extensions, and potentially with other GUI software.

Too interdependent

Another significant issue with testing is circularity: ideally we need to inspect the output channel to verify the extension is doing what we want, but Code's API makes it impossible to interact with output channels unless you're the owner -- so we need to use the "Show Output Channel" command - thereby interacting with the system under test - and then rely on hacks (e.g. the channel is a TextDocument as long as it is visible on screen) to get the output. And, we have no way to clear the channel, so we can't distinguish new output from old!

I spent hours on this and just gave up in the end; instead I spy on the extension state by exporting its language client. Not quite as black-box as I'd like, but it works.

Too many language clients!

My hunch about the hang I was seeing with "Format Document" is: the extension is getting activated twice in a row! We had no guard against this from a cursory glance at the source, so we happily spawned two language servers and that explains why I saw two options for Syntax Tree when issuing the format-document command. (There is no API for interacting with those modal popups so once one is visible, it's game over for the test runner.) Not sure why this only happens in CI and not on my laptop.

Once I made start/stop 100% idempotent by checking for an existing languageClient before making a new one, the hang during tests went away. Succedd!

TL;DR

I am happy with this; it's a good start, and I'm very proud of the screen-recording artifacts. Merge away!

[kddnewton]

@xeger sorry but do you mind rebasing? Then I'll merge!

[xeger]

Done @kddnewton (& squashed to cleanup the commit history); good to merge once it goes green.