Handling of required and optional whitespaces in custom languages

IntelliJ Platform
1

Hi, I couldn’t find a good answer on that topic for my use-case.
I’m working (or trying to work) on a custom language which is quite close to CSS - basically, blocks for selectors with properties (key → term). In that case I don’t really care about whitespaces in places like MySelector<HERE>{} or color<HERE>:<HERE> value; - but they are fundamental for things like margin: 1 <HERE> 2 etc.

I looked at the PsiTree of CSS-Files in IntelliJ itself, and see that every whitespace is (seemingly?) tokenized. My current approach is, to ignore every whitespace except in lever states, where I want them (e.g. in values / ‘terms’).
Sadly the CSS BNF / JFlex is not open-source (if it is even made with grammar kit and not written with a custom lexer), so I can’t really see, if it’s just a bunch of WHITESPACE* in the BNF.

(Little extra question, though not that important: Is there any way to ‘prioritize’ what is matched in the lexer first? for example, a selector and a property key are both IDENTIFIERs, and selectors can be part of blocks, e.g.

Selector {
  color: red;
  OtherSelector {
  }
}

my current approach would be doing a kinda greedy lookahead (IDENTIFIER WHITESPACE* LCURLY) and pushing back the curly and all whitespaces, to differ between a new selector / block and a single property)

2

Okay, found a solution after finding https://intellij-support.jetbrains.com/hc/en-us/community/posts/9833396377362-Force-whitespaces-between-tokens

Quite simple, for every expected white space (which is required for my syntax) I match that in my Lexer and return my custom white space token type. for every other whitespace occurrence, i return the one from com.intellij.psi.TokenType

1 Like