patch 9.0.1627: no generic mechanism to test syntax plugins
Problem: No generic mechanism to test syntax plugins.
Solution: Add a syntax plugin test mechanism, using screendumps. Add a
simple test for "c".
diff --git a/runtime/syntax/testdir/README.txt b/runtime/syntax/testdir/README.txt
new file mode 100644
index 0000000..22b6084
--- /dev/null
+++ b/runtime/syntax/testdir/README.txt
@@ -0,0 +1,97 @@
+Tests for syntax highlighting plugins
+=====================================
+
+Summary: Files in the "input" directory are edited by Vim with syntax
+highlighting enabled. Screendumps are generated and compared with the
+expected screendumps in the "dumps" directory. This will uncover any
+character attributes that differ.
+
+Without any further setup a screendump is made at the top of the file (using
+_00.dump) and another one at the end of the file (using _99.dump). The dumps
+are normally 20 screen lines tall.
+
+When the screendumps are OK an empty "done/{name}" file is created. This
+avoids running the test again until "make clean" is used. Thus you can run
+"make test", see one test fail, try to fix the problem, then run "make test"
+again to only repeat the failing test.
+
+When a screendump differs it is stored in the "failed" directory. This allows
+for comparing it with the expected screendump, using a command like:
+
+ let fname = '{name}_99.dump'
+ call term_dumpdiff('failed/' .. fname, 'dumps/' .. fname)
+
+
+Creating a syntax plugin test
+-----------------------------
+
+Create a source file in the language you want to test in the "input"
+directory. Make sure to include some interesting constructs with complicated
+highlighting.
+
+Use the filetype name as the base and a file name extension matching the
+filetype. Let's use Java as an example. The file would then be
+"input/java.java".
+
+If there is no further setup required, you can now run the tests:
+ make test
+
+The first time this will fail with an error for a missing screendump. The
+newly created screendumps will be "failed/java_00.dump",
+"failed/java_01.dump", etc. You can inspect each with:
+
+ call term_dumpload('failed/java_00.dump')
+ call term_dumpload('failed/java_01.dump')
+ ...
+ call term_dumpload('failed/java_99.dump')
+
+If they look OK, move them to the "dumps" directory:
+
+ :!mv failed/java_00.dump dumps
+ :!mv failed/java_01.dump dumps
+ ...
+ :!mv failed/java_99.dump dumps
+
+If you now run the test again, it will succeed.
+
+
+Adjusting a syntax plugin test
+------------------------------
+
+If you make changes to the syntax plugin, you should add code to the input
+file to see the effect of these changes. So that the effect of the changes
+are covered by the test. You can follow these steps:
+
+1. Edit the syntax plugin somewhere in your personal setup. Use a file
+ somewhere to try out the changes.
+2. Go to the directory where you have the Vim code checked out and replace the
+ syntax plugin. Run the tests: "make test". Usually the tests will still
+ pass, but if you fixed syntax highlighting that was already visible in the
+ input file, carefully check that the changes in the screendump are
+ intentional:
+ let fname = '{name}_99.dump'
+ call term_dumpdiff('failed/' .. fname, 'dumps/' .. fname)
+ Fix the syntax plugin until the result is good.
+2. Edit the input file for your language to add the items you have improved.
+ (TODO: how to add another screendump?).
+ Run the tests and you should get failures. Like with the previous step,
+ carefully check that the new screendumps in the "failed" directory are
+ good. Update the syntax plugin and the input file until the highlighting
+ is good and you can see the effect of the syntax plugin improvements. Then
+ move the screendumps from the "failed" to the "dumps" directory. Now "make
+ test" should succeed.
+3. Prepare a pull request with the modified files:
+ - syntax plugin: syntax/{name}.vim
+ - test input file: syntax/testdir/input/{name}.{ext}
+ - test dump files: syntax/testdir/dumps/{name}_99.dump
+
+As an extra check you can temporarily put back the old syntax plugin and
+verify that the tests fail. Then you know your changes are covered by the
+test.
+
+
+
+TODO: run test for one specific filetype
+
+TODO: testing with various option values
+TODO: test syncing by jumping around