expl3 syntax highlighting wrongly marks underscores as errors in csname-style arguments

[cfr42]

Description

Thanks for providing support for expl3 syntax highlighting (and for vimtex generally, obviously). Although the result is much better than my previous editor, there are a couple of irritating bugs and I'm not really good enough with vim (yet?!) to address them myself.

The main problem is that underscores are highlighted as errors when they should not be, to the point that the huge number of mistaken red flags is a significant distraction. At best, it makes it impossible to spot any genuine errors. At worst, it makes it very difficult to read the code at all (at least for me).

This is a noticeable problem in at least 3 situations:

1. when a c type argument specifier is used;
2. when a v type argument specifier is used;
3. when defining keys (l3keys).

Steps to reproduce

c type argument specifier

\documentclass{article}
\ExplSyntaxOn
\use:c { __mymod_somefn:Nnn } \l__mymod_mytl_tl { a } { b } 
\ExplSyntaxOff
\begin{document}
a
\end{document}

vimtex incorrectly highlights the underscores in __mymod_somefn:Nnn as errors, even though they are not. The c in \use:c means that the argument will first be turned into a control sequence. \use:c {<stuff>} is equivalent to \csname <stuff>\endcsname. This bug makes it really hard to see what is going on as I sometimes have dozens of 'erroneous' underscores in half a screen of code.

v type argument specifier

Similar to c, but gets the value of the control sequence constructed from the argument and passes that value to the base function rather than the control sequence itself.

\documentclass{article}
\ExplSyntaxOn
\cs_new_protected:Npn \__mymod_description:nnn #1#2#3
{
  \prop_new:c { g__mymod_#1_list_prop }
  \prop_gput:cve { g__mymod_#1_list_prop }
  { l__mymod_#2_tl }
  { \clist_use:c { l__mymod_#3_clist } }
}
\tl_new:N \l__mymod_tryme_tl
\tl_set:Nn \l__mymod_tryme_tl {token_list}
\clist_new:N \l__mymod_tryme_clist
\clist_set:Nn \l__mymod_tryme_clist { aardvark, bonobo, catalyst }
\__mymod_description:nnn { tryme } { tryme } { tryme }
\prop_show:N \g__mymod_tryme_list_prop
\ExplSyntaxOff
\begin{document}
a
\end{document}

The top lines of this trigger lots of red flags from the highlighter. The lower lines are just there to demonstrate the syntax is nonetheless valid by showing the contents of \g__mymod_tryme_list_prop on the console.

Key definitions

\documentclass{article}
\ExplSyntaxOn
\keys_define:nn { fixtounicode }
{
  default .tl_set:N = \l__fixtounicode_default_tl,
  default .initial:n = 2FFFF,
  default .default:V = \c_empty_tl,
  glyphs .clist_set:N = \l__fixtounicode_glyphs_clist,
  glyphs .value_required:n = true,
  pfb .tl_set:N = \l__fixtounicode_pfb_tl,
  pfb .initial:V = \c_empty_tl,
  tfm .tl_set:N = \l__fixtounicode_tfm_tl,
  tfm .initial:V = \c_empty_tl,
  unicodes .clist_set:N = \l__fixtounicode_unicodes_clist,
  unicodes .value_required:n = true,
}
\ExplSyntaxOff
\begin{document}
a
\end{document}

Here, the underscores before the = are all marked as erroneous, though those after the = are not. The markup on the left is standard syntax for defining keys in the l3keys module.

mymod (and fixtounicode) could also be @@ or @ etc. inside a .dtx.

minimal.vim

I'm actually mostly using nvim, but I assume a vim basis is better for reproduction, since the issue isn't nvim specific. I used the v example above to test reproducibiblity with minimal config.

set nocompatible
let &runtimepath  = '~/.vim/plugged/vimtex,' . &runtimepath
let &runtimepath .= ',~/.vim/plugged/vimtex/after'
filetype plugin indent on
syntax enable
" Add relevant options and VimTeX configuration below.

[Incidentally, VimtexInfo reports the use of packages even if \usepackage follows \end{document}. I don't know if this is a feature, a necessary evil or a bug? For example, it claimed my minimal example loaded tikz until I put the document in a clean file, even though tikz followed \end{document}.]

Expected behavior

Valid uses of underscores in these situations should not be highlighted as errors.

Actual behavior

Valid uses of underscores in these situations are highlighted as errors.

Do you use a latexmkrc file?

No

VimtexInfo

System info:
  OS: Arch Linux
  Vim version: VIM 9.1 (1-1236)
  Has clientserver: false

VimTeX project: p2
  base: p2.tex
  root: /home/dienw/profion
  tex: /home/dienw/profion/p2.tex
  main parser: current file verified
  document class: article
  compiler: latexmk
    engine: -pdf
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    callback: 1
    continuous: 1
    executable: latexmk
  viewer: General
  qf method: LaTeX logfile
~
~

[lervag]

Ok, I hoped to be able to get around to this faster, and now it's been a week. Sorry! I only skimmed the issue so far, but I hope to get around to it within this week.

[lervag]

Thanks for providing support for expl3 syntax highlighting (and for vimtex generally, obviously). Although the result is much better than my previous editor,

Glad to hear that!

And important: I really don't know the expl3 stuff. At all! So I probably need a lot of help here.

The main problem is that underscores are highlighted as errors when they should not be, to the point that the huge number of mistaken red flags is a significant distraction.

Would it help if I just avoid highlighting the underscores as errors in expl3 zones? At a first glance, it seems to be slightly difficult to spot one or a few simple patterns that would match your specified rules.

I'm pushing an update that simply ignores the texError group now, and I believe it should make things less red.

vimtex incorrectly highlights the underscores in __mymod_somefn:Nnn as errors, even though they are not. The c in \use:c means that the argument will first be turned into a control sequence. \use:c {<stuff>} is equivalent to \csname <stuff>\endcsname.

So, I could make a rule where the first group after \use:c allows underscores? I mean, the problem here is to understand how to highlight <stuff>.

Also, is use important? Could it be \foo:c?

I'm actually mostly using nvim, …

So am I :)

Incidentally, VimtexInfo reports the use of packages even if \usepackage follows \end{document}. I don't know if this is a feature, a necessary evil or a bug?

Not a feature, no. You might call it a bug, especially if you find it has any unexpected bad consequences. Feel free to open an issue if that's the case.

[cfr42]

Apologies for my delayed responses. I had eye surgery last week and reading is still rather a tall order.

And important: I really don't know the expl3 stuff. At all! So I probably need a lot of help here.

I am far from proficient, but I can use a chunk of it and should be able to help with the syntax, which is designed to be predictable/patterned.

The main problem is that underscores are highlighted as errors when they should not be, to the point that the huge number of mistaken red flags is a significant distraction.

Would it help if I just avoid highlighting the underscores as errors in expl3 zones? At a first glance, it seems to be slightly difficult to spot one or a few simple patterns that would match your specified rules.

I think this should help a lot. In expl3 zones, _ and : are letters - just as @ is a letter inside a .sty or \makeatletter zone, so it would make sense to me to switch off highlighting these as errors here. You cannot, for instance, write

$\theta_i$

and expect to get a subscript. You have to use a special macro instead. (If you really need this, though probably you should not have something like this in an expl3 zone at all.)

I'm pushing an update that simply ignores the texError group now, and I believe it should make things less red.

Thanks! I will try this as soon as my eyes can cope with editing code.

vimtex incorrectly highlights the underscores in __mymod_somefn:Nnn as errors, even though they are not. The c in \use:c means that the argument will first be turned into a control sequence. \use:c {<stuff>} is equivalent to \csname <stuff>\endcsname.

So, I could make a rule where the first group after \use:c allows underscores? I mean, the problem here is to understand how to highlight <stuff>.

Also, is use important? Could it be \foo:c?

It ought not be \foo:c, but it could be, say, \foo_bar:c. The part after the colon is the argument specification. If the base form of a function contains N, you can generate a variant with c in place of N and then that argument is made into a macro token before it is passed to the base function.

So if I have e.g.

\cs_new_protected:Npn \foo_bar:NnN #1#2#3 { #1 ; ~ #2 ; #3 }

then \foo_bar:NnN absorbs 3 arguments, where the first and third are control sequences and the second is a braced group of whatever. Then

\cs_generate_variant:Nn \foo_bar:NnN { c , Nnc , cnc }

generates \foo_bar:cnN, \foo_bar:Nnc and \foo_bar:cnc where the c is braced group which is made into a control sequence before being passed to \foo_bar:NnN. So the following would all be equivalent:

\foo_bar:NnN \l_foo_tokenlist_tl { some list of tokens here } \l_foo_count_int
\foo_bar:cNn { l_foo_tokenlist_tl }  { some list of tokens here } \l_foo_count_int
\foo_bar:Nnc \l_foo_tokenlist_tl { some list of tokens here }  { l_foo_count_int }
\foo_bar:cnc { l_foo_tokenlist_tl } { some list of tokens here }  { l_foo_count_int }

So, no, it doesn't need to be \use:c. Typically, it isn't \use:c. That was just the simplest example I could think of . Hence it may well be simplest to simply not raise errors for underscores in expl3 zones. Even though you could potentially match patterns here, there are going to be lots of other cases to catch and I'm not sure there is any point in catching them in a context where you know underscores and colons are letters.

If anything above looks like an error, please let me know. I can't reliably check what I've written at the moment.

[lervag]

Apologies for my delayed responses. I had eye surgery last week and reading is still rather a tall order.

No problem; and good luck on your recovery!

I'm pushing an update that simply ignores the texError group now, and I believe it should make things less red.

Thanks! I will try this as soon as my eyes can cope with editing code.

Looking forward to hearing what you think. Based on your input on the expl3 stuff, I believe it might solve your pain points and yield an acceptable level of support.