I recently rewrote the PSI structure of a custom language plugin to use stub indexes.
Everything seemed to work fine and the performance improved a lot. i however had yet do discover that stub indexes are global and there is no way for me (AFAIK) to support conditional compilation with stubs.
After switching between a few projects i stared getting UpToDateStubIndexMismatch errors for files in the standard library, and soon realized that it was because the parser that only parses code branches enabled by CC flags was causing issues for the logic that creates stubs as the result after parsing where different between projects.
I Asked Claude for help and i got some suggestion looking at how its solved for Clion etc.
however for the language i am working on you can do conditional compilation stuff on a single token. It can be keywords, names, values etc.
#if shouldBePublic public #else private #end function someFunction (){...}
public function #if nameFlag someName #else someOtherName #end (){...}
since you can also nest these conditional compilation blocks its hopeless to try to parse all possible branches of code.
According to Claude there is no way to make the stubs local to the project and they have to be shared, that means any attempt at having 2 projects open at the same time is more or less guarantied to cause problems for my custom language plugin. There where some creative suggestions, like setting the version = a hash of the conditional compilation flags etc. but that just seems too hacky to me.
Anyone here got any suggestion / ideas on how to fix this, i really don’t want to have to revert everything as the stubs improved other aspects of the plugin a lot.
The language is Haxe, and my goal is to improve the performance of type checking annotating etc. the language supports implicit typing so to find the type of say a variable you might have to traverse several files, depending on whether or not methods called also are implicit typed. in my experience using stubs makes this a lot faster but my problem is as mentioned in my post above that the language also have conditional compilation in a way that makes shared stub indexes unfeasible (at least as far as i can tell).
The only current solution i can think of is filtering out everything that got conditional compilation when building stubs but that would mean that stub indexes are less useful as they will always be incomplete.
if i could keep my stubs local to my project this would probably be solvable, but as far as i can tell that is not possible ?
Treat stub-indexes as a skeleton of file that can help you resolve symbols without reading file contents. Following that logic you could implement them as a superset of all conditions and evaluate conditions depending on project during resolve.
For sure, you should not evaluate complex logic during stub indexes creation process.
I’ve been experimenting a bit, and at first i was worried that i could not use PSI stubs at all and that the serialized stubs was going to be an issue across projects, but it seems to only affect the indexes. or at least that is how i interpretate it after just disabling all indexes. so i guess ill try to see if i can achieve what i need with filebased indexes instead.
my goal with some of the indexes is to be able to show/find all available classes and/or members in the current state of the project (by state i mean how CC flags are configured and what is included when files are parsed). Since stubindexes breaks ill have to try filebased indexes and see if these can avoid these issues, worst case ill just have to make my own index solution.
As for the second reply, I am not building stubs for the method body, just the signature.
The point of my examples was to show that the method signature can be different depending on the CC flags and that includes the method name. I also wanted to illustrate that the conditional compilation can wrap a single keyword and its not required to be a complete expressions, meaning that its not (AFAIK) possible to parse and evaluate all conditions.
Then as I suggested, you could implement them as a superset and filter out unnecessary things on resolve/evaluations, reading the index which is the same for all projects
The problem is that i cant necessary evaluate all conditions. so a superset is not possible.
While this example is not something someone would normally write its technically possible.
#if CC_FLAG public function #end functionName(){...}
if CC_FLAG evaluates to true it would parse just find and become
public function functionName(){...}
however if CC_flag is false the result would be just
functionName (){...}
and this is not valid grammar and would not parse.
so i can not assume all possible outcomes of conditional compilation are valid grammar.
there might be multiple flags depending on architecture and platform that would work for all meaningful combinations but would break if you try to evaluate all combinations.
(for example enabling flags for both linux and windows at the same time)
This would not be a problem if i could keep stub-indexes local to a project, as changes to code, configuration or CC flags would trigger re-parsing and updates to the stub indexes. The issue in my case is that the indexes are shared. Since a superset is not possible and a subset without conditional compilation specific code does not cover what i need the indexes for, i can really use sub-indexes in its current form.
Did some more testing and i was wrong, i still get issues with indexes disabled, its the stub trees that are the problem
Is there any way to store these trees separately per project or module or something?
This is all the same if you move superset filteration to resolve instead of recomputing stubs, as resolve cache is reset on PSI changes. You can also reset it manually
The parser is parsing files based on what conditional compilation flags are enabled. anything that the conditional compilation flag would “turn off”/skip is treated as a dummy block / comment. this means that if you have 2 projects open and their conditional compilation flags are different, the parser would parse the same standard library files differently and both the stub tree and stub indexes will be different. this in turn causes intellij to throw errors as the size of the stub trees and indexes are different
This is the type of errors i get due to different parsing between projects and why i want to store the stub trees and indexes separatly per project.
java.lang.Throwable: psiElement is not instance of requiredClass.
psiElement=REFERENCE_EXPRESSION:sys.thread.Thread:HaxeReferenceExpressionImpl, psiElement.class=class com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceExpressionImpl, requiredClass=interface com.intellij.plugins.haxe.lang.psi.HaxeClass, operation=Looking for Timer in haxe.class.name, stubIdList=[4]@0.
ref: 20250127
no dumbMode
com.intellij.psi.stubs.UpToDateStubIndexMismatch: PSI and index do not match.
Please report the problem to JetBrains with the files attached
INDEXED VERSION IS THE CURRENT ONE file=Timer.hx, file.class=class com.intellij.plugins.haxe.lang.psi.HaxeFile, file.lang=Language: Haxe, modStamp=5, psi.length=5729
tree consistent
stub debugInfo=created in getStubTree(), with AST = false; with backReference
viewProvider=com.intellij.psi.SingleRootFileViewProvider{vFile=file://C:/HaxeToolkit/haxe/std/haxe/Timer.hx, vFileId=69838, content=VirtualFileContent{size=5926}, eventSystemEnabled=true}
viewProvider stamp: 0; file stamp: 0; file modCount: 1746802534000; file length: 5926
doc saved: true; doc stamp: 0; doc size: 5729; committed: true
indexing info: indexing timestamp = 1746802534000, binary = false, byte size = 5926, char size = 5729
ref: 20250127
latestIndexedStub=StubTree{myDebugInfo='created from index; with backReference', myRoot=HaxeFileStub}1184147106
same size=false
debugInfo=created from index; with backReference
at com.intellij.openapi.diagnostic.Logger.error(Logger.java:375)
at com.intellij.psi.stubs.StubProcessingHelperBase.inconsistencyDetected(StubProcessingHelperBase.java:226)
at com.intellij.psi.stubs.StubProcessingHelperBase.checkType(StubProcessingHelperBase.java:154)
at com.intellij.psi.stubs.StubProcessingHelperBase.processStubsInFile(StubProcessingHelperBase.java:84)
at com.intellij.psi.stubs.StubIndexEx.lambda$processElements$6(StubIndexEx.java:187)
at com.intellij.psi.stubs.StubIndexEx.processElements(StubIndexEx.java:238)
at com.intellij.psi.stubs.StubIndex.getElements(StubIndex.java:109)
at com.intellij.psi.stubs.StubIndex.getElements(StubIndex.java:98)
at com.intellij.plugins.haxe.lang.psi.stubs.index.HaxeClassNameStubIndex.getByNameFiltered(HaxeClassNameStubIndex.java:53)
at com.intellij.plugins.haxe.ide.documentation.HaxeDocumentationCodeVisitor.findUniqueClassFromIndex(HaxeDocumentationCodeVisitor.java:134)
at com.intellij.plugins.haxe.ide.documentation.HaxeDocumentationCodeVisitor.replaceIndexedClassName(HaxeDocumentationCodeVisitor.java:125)
at com.intellij.plugins.haxe.ide.documentation.HaxeDocumentationCodeVisitor.visit(HaxeDocumentationCodeVisitor.java:41)
at org.commonmark.node.Code.accept(Code.java:16)
at org.commonmark.node.AbstractVisitor.visitChildren(AbstractVisitor.java:137)
at org.commonmark.node.AbstractVisitor.visit(AbstractVisitor.java:93)
at org.commonmark.node.Paragraph.accept(Paragraph.java:10)
at org.commonmark.node.AbstractVisitor.visitChildren(AbstractVisitor.java:137)
at org.commonmark.node.AbstractVisitor.visit(AbstractVisitor.java:28)
at org.commonmark.node.Document.accept(Document.java:7)
at com.intellij.plugins.haxe.ide.documentation.HaxeDocumentationRenderer.parseAndRenderDocs(HaxeDocumentationRenderer.java:64)
at com.intellij.plugins.haxe.ide.documentation.providers.HaxeDocumentationProvider.generateRenderedDoc(HaxeDocumentationProvider.java:210)
at com.intellij.lang.documentation.CompositeDocumentationProvider.generateRenderedDoc(CompositeDocumentationProvider.java:163)
at com.intellij.codeInsight.documentation.render.PsiCommentInlineDocumentation.renderText(PsiCommentInlineDocumentation.java:45)
at com.intellij.codeInsight.documentation.render.DocRenderPassFactory.calcText(DocRenderPassFactory.java:126)
at com.intellij.codeInsight.documentation.render.DocRenderPassFactory.calculateItemsToRender(DocRenderPassFactory.java:98)
at com.intellij.codeInsight.documentation.render.DocRenderPassFactory.calculateItemsToRender(DocRenderPassFactory.java:90)
at com.intellij.codeInsight.documentation.render.DocRenderPassFactory$DocRenderPass.doCollectInformation(DocRenderPassFactory.java:77)
at com.intellij.codeHighlighting.TextEditorHighlightingPass.collectInformation(TextEditorHighlightingPass.java:76)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$doRun$2(PassExecutorService.java:449)
at com.intellij.platform.diagnostic.telemetry.helpers.TraceKt.use(trace.kt:29)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$doRun$3(PassExecutorService.java:445)
at com.intellij.openapi.application.impl.AppImplKt$runnableUnitFunction$1.invoke(appImpl.kt:124)
at com.intellij.openapi.application.impl.AppImplKt$runnableUnitFunction$1.invoke(appImpl.kt:124)
at com.intellij.platform.locking.impl.NestedLocksThreadingSupport.tryRunReadAction(NestedLocksThreadingSupport.kt:900)
at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1266)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$doRun$4(PassExecutorService.java:435)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$14(CoreProgressManager.java:744)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:819)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:775)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:743)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:87)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.doRun(PassExecutorService.java:434)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$run$0(PassExecutorService.java:409)
at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.cacheFileTypesInside(FileTypeManagerImpl.java:904)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$run$1(PassExecutorService.java:409)
at com.intellij.openapi.application.impl.ApplicationImpl.executeByImpatientReader(ApplicationImpl.java:306)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.run(PassExecutorService.java:407)
at com.intellij.concurrency.JobLauncherImpl$VoidForkJoinTask$1.exec(JobLauncherImpl.java:289)
at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:511)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
com.intellij.psi.stubs.UpToDateStubIndexMismatch: PSI and index do not match.
Please report the problem to JetBrains with the files attached
INDEXED VERSION IS THE CURRENT ONE file=Http.hx, file.class=class com.intellij.plugins.haxe.lang.psi.HaxeFile, file.lang=Language: Haxe, modStamp=1, psi.length=1315
tree consistent
stub debugInfo=created in getStubTree(), with AST = false; with backReference
viewProvider=com.intellij.psi.SingleRootFileViewProvider{vFile=file://C:/HaxeToolkit/haxe/std/haxe/Http.hx, vFileId=69817, content=VirtualFileContent{size=1348}, eventSystemEnabled=true}
viewProvider stamp: 0; file stamp: 0; file modCount: 1746802534000; file length: 1348
doc saved: true; doc stamp: 0; doc size: 1315; committed: true
indexing info: indexing timestamp = 1746802534000, binary = false, byte size = 1348, char size = 1315
ref: 20250127
latestIndexedStub=StubTree{myDebugInfo='created from index; with backReference', myRoot=HaxeFileStub}1744883551
same size=true
debugInfo=created from index; with backReference
at com.intellij.psi.stubs.StubTreeLoader.handleUpToDateMismatch(StubTreeLoader.java:231)
at com.intellij.psi.stubs.StubTreeLoader.access$100(StubTreeLoader.java:32)
at com.intellij.psi.stubs.StubTreeLoader$StubTreeAndIndexUnmatchCoarseException.doCreateCompleteException(StubTreeLoader.java:223)
at com.intellij.psi.stubs.StubTreeLoader$StubTreeAndIndexUnmatchCoarseException.lambda$createCompleteException$0(StubTreeLoader.java:199)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:819)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:775)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$computeInNonCancelableSection$4(CoreProgressManager.java:360)
at com.intellij.openapi.progress.Cancellation.computeInNonCancelableSection(Cancellation.java:156)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeInNonCancelableSection(CoreProgressManager.java:360)
at com.intellij.psi.stubs.StubTreeLoader$StubTreeAndIndexUnmatchCoarseException.createCompleteException(StubTreeLoader.java:198)
at com.intellij.psi.impl.source.PsiFileImpl.loadTreeElement(PsiFileImpl.java:298)
at com.intellij.psi.impl.source.PsiFileImpl.calcTreeElement(PsiFileImpl.java:897)
at com.intellij.extapi.psi.StubBasedPsiElementBase.getNode(StubBasedPsiElementBase.java:137)
at com.intellij.extapi.psi.ASTDelegatePsiElement.getFirstChild(ASTDelegatePsiElement.java:95)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceUtil.canBeQname(HaxeReferenceUtil.java:169)
at com.intellij.plugins.haxe.util.HaxeResolveUtil.tryResolveFullyQualifiedHaxeReferenceExpression(HaxeResolveUtil.java:959)
at com.intellij.plugins.haxe.util.HaxeResolveUtil.tryResolveClassByQNameWhenGetQNameFail(HaxeResolveUtil.java:1085)
at com.intellij.plugins.haxe.util.HaxeResolveUtil.tryResolveClassByQName(HaxeResolveUtil.java:938)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.checkIsType(HaxeResolver.java:2249)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.doResolveInner(HaxeResolver.java:182)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.lambda$doResolve$0(HaxeResolver.java:149)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:124)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.doResolve(HaxeResolver.java:149)
at com.intellij.psi.impl.source.resolve.ResolveCache.lambda$resolveWithCaching$5(ResolveCache.java:284)
at com.intellij.openapi.util.Computable.get(Computable.java:16)
at com.intellij.psi.impl.source.resolve.ResolveCache.lambda$loggingResolver$4(ResolveCache.java:250)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:124)
at com.intellij.openapi.util.RecursionGuard.doPreventingRecursion(RecursionGuard.java:28)
at com.intellij.openapi.util.RecursionManager.doPreventingRecursion(RecursionManager.java:79)
at com.intellij.psi.impl.source.resolve.ResolveCache.resolve(ResolveCache.java:228)
at com.intellij.psi.impl.source.resolve.ResolveCache.resolveWithCaching(ResolveCache.java:284)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.resolve(HaxeResolver.java:114)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.doResolve(HaxeReferenceImpl.java:257)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.multiResolve(HaxeReferenceImpl.java:248)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.multiResolve(HaxeReferenceImpl.java:290)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.resolve(HaxeReferenceImpl.java:304)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.resolve(HaxeReferenceImpl.java:183)
at com.intellij.plugins.haxe.model.type.HaxeTypeResolver.resolveTypeFromType(HaxeTypeResolver.java:670)
at com.intellij.plugins.haxe.model.type.HaxeTypeResolver.getTypeFromType(HaxeTypeResolver.java:596)
at com.intellij.plugins.haxe.model.type.HaxeTypeResolver.getTypeFromTypeOrAnonymous(HaxeTypeResolver.java:696)
at com.intellij.plugins.haxe.model.type.HaxeTypeResolver.getTypeFromTypeOrAnonymous(HaxeTypeResolver.java:687)
at com.intellij.plugins.haxe.model.HaxeClassModel.getUnderlyingType(HaxeClassModel.java:314)
at com.intellij.plugins.haxe.model.HaxeClassModel.getUnderlyingType(HaxeClassModel.java:308)
at com.intellij.plugins.haxe.util.HaxeResolveUtil.tryResolveTypeDefClass(HaxeResolveUtil.java:1293)
at com.intellij.plugins.haxe.util.HaxeResolveUtil.lambda$searchInSamePackage$10(HaxeResolveUtil.java:1265)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:124)
at com.intellij.openapi.util.RecursionGuard.doPreventingRecursion(RecursionGuard.java:28)
at com.intellij.plugins.haxe.util.HaxeResolveUtil.searchInSamePackage(HaxeResolveUtil.java:1265)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.doResolveInner(HaxeResolver.java:263)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.lambda$doResolve$0(HaxeResolver.java:149)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:124)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.doResolve(HaxeResolver.java:149)
at com.intellij.psi.impl.source.resolve.ResolveCache.lambda$resolveWithCaching$5(ResolveCache.java:284)
at com.intellij.openapi.util.Computable.get(Computable.java:16)
at com.intellij.psi.impl.source.resolve.ResolveCache.lambda$loggingResolver$4(ResolveCache.java:250)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:124)
at com.intellij.openapi.util.RecursionGuard.doPreventingRecursion(RecursionGuard.java:28)
at com.intellij.openapi.util.RecursionManager.doPreventingRecursion(RecursionManager.java:79)
at com.intellij.psi.impl.source.resolve.ResolveCache.resolve(ResolveCache.java:228)
at com.intellij.psi.impl.source.resolve.ResolveCache.resolveWithCaching(ResolveCache.java:284)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.resolve(HaxeResolver.java:114)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.doResolve(HaxeReferenceImpl.java:257)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.multiResolve(HaxeReferenceImpl.java:248)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.multiResolve(HaxeReferenceImpl.java:290)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.resolve(HaxeReferenceImpl.java:304)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.resolve(HaxeReferenceImpl.java:183)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.resolveChain(HaxeResolver.java:2378)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.checkIsChain(HaxeResolver.java:2148)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.doResolveInner(HaxeResolver.java:183)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.lambda$doResolve$0(HaxeResolver.java:149)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:124)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.doResolve(HaxeResolver.java:149)
at com.intellij.psi.impl.source.resolve.ResolveCache.lambda$resolveWithCaching$5(ResolveCache.java:284)
at com.intellij.openapi.util.Computable.get(Computable.java:16)
at com.intellij.psi.impl.source.resolve.ResolveCache.lambda$loggingResolver$4(ResolveCache.java:250)
at com.intellij.openapi.util.RecursionManager$1.computePreventingRecursion(RecursionManager.java:124)
at com.intellij.openapi.util.RecursionGuard.doPreventingRecursion(RecursionGuard.java:28)
at com.intellij.openapi.util.RecursionManager.doPreventingRecursion(RecursionManager.java:79)
at com.intellij.psi.impl.source.resolve.ResolveCache.resolve(ResolveCache.java:228)
at com.intellij.psi.impl.source.resolve.ResolveCache.resolveWithCaching(ResolveCache.java:284)
at com.intellij.plugins.haxe.lang.psi.HaxeResolver.resolve(HaxeResolver.java:114)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.doResolve(HaxeReferenceImpl.java:257)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.multiResolve(HaxeReferenceImpl.java:248)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.multiResolve(HaxeReferenceImpl.java:290)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.resolve(HaxeReferenceImpl.java:304)
at com.intellij.plugins.haxe.lang.psi.impl.HaxeReferenceImpl.resolve(HaxeReferenceImpl.java:183)
at com.intellij.plugins.haxe.ide.HaxeLineMarkerProviderNS.collectRecursionMarkers(HaxeLineMarkerProviderNS.java:72)
at com.intellij.plugins.haxe.ide.HaxeLineMarkerProviderNS.collectSlowLineMarkersWorker(HaxeLineMarkerProviderNS.java:63)
at com.intellij.plugins.haxe.ide.HaxeLineMarkerProvider.collectSlowLineMarkers(HaxeLineMarkerProvider.java:45)
at com.intellij.codeInsight.daemon.impl.LineMarkersPass.queryProviders(LineMarkersPass.java:238)
at com.intellij.codeInsight.daemon.impl.LineMarkersPass.lambda$doCollectMarkers$2(LineMarkersPass.java:122)
at com.intellij.codeInsight.daemon.impl.Divider.divideInsideAndOutsideInOneRoot(Divider.java:93)
at com.intellij.codeInsight.daemon.impl.LineMarkersPass.doCollectMarkers(LineMarkersPass.java:118)
at com.intellij.codeInsight.daemon.impl.LineMarkersPass.doCollectInformation(LineMarkersPass.java:90)
at com.intellij.codeHighlighting.TextEditorHighlightingPass.collectInformation(TextEditorHighlightingPass.java:76)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$doRun$2(PassExecutorService.java:449)
at com.intellij.platform.diagnostic.telemetry.helpers.TraceKt.use(trace.kt:29)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$doRun$3(PassExecutorService.java:445)
at com.intellij.openapi.application.impl.AppImplKt$runnableUnitFunction$1.invoke(appImpl.kt:124)
at com.intellij.openapi.application.impl.AppImplKt$runnableUnitFunction$1.invoke(appImpl.kt:124)
at com.intellij.platform.locking.impl.NestedLocksThreadingSupport.tryRunReadAction(NestedLocksThreadingSupport.kt:900)
at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1266)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$doRun$4(PassExecutorService.java:435)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$14(CoreProgressManager.java:744)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:819)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:775)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:743)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:87)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.doRun(PassExecutorService.java:434)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$run$0(PassExecutorService.java:409)
at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.cacheFileTypesInside(FileTypeManagerImpl.java:904)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.lambda$run$1(PassExecutorService.java:409)
at com.intellij.openapi.application.impl.ApplicationImpl.executeByImpatientReader(ApplicationImpl.java:306)
at com.intellij.codeInsight.daemon.impl.PassExecutorService$ScheduledPass.run(PassExecutorService.java:407)
at com.intellij.concurrency.JobLauncherImpl$VoidForkJoinTask$1.exec(JobLauncherImpl.java:289)
at java.base/java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:511)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1450)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2019)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)
Caused by: java.lang.AssertionError: Stub count (10) doesn't match stubbed node length (11)
at com.intellij.psi.impl.source.FileTrees.lambda$reconcilePsi$4(FileTrees.java:192)
at com.intellij.psi.impl.DebugUtil.performPsiModification(DebugUtil.java:602)
at com.intellij.psi.impl.source.FileTrees.reconcilePsi(FileTrees.java:180)
at com.intellij.psi.impl.source.FileTrees.withAst(FileTrees.java:146)
at com.intellij.psi.impl.source.PsiFileImpl.loadTreeElement(PsiFileImpl.java:284)
... 109 more
```
Unfortunately, per-project indexes are not supported. There is a ticket about that:
But we are not going to work on it anytime soon.
The problem you are having with stubs is that the stub tree produced with one set of flags does not match the psi tree produced with another set of flags. This is expected behavior.
There are several possibilities:
Leave the stub empty if there is conditional compilation involved.
In this case, the stubs will be available only for files where the conditional compilation is missing.
Always parse all branches of a given file and try to do your best when one of the branches contains broken code. Probably the std lib does not contain broken code, so it might be a good alternative.
In this case, the stubs will always be present at the cost of much more complicated parsing.
Build your own machinery for storing persistent multiverse stub trees that can be reused across all projects. IntelliJ VFS allows storing arbitrary binary data associated with all indexable virtual files with the help of com.intellij.openapi.vfs.newvfs.FileAttribute. You will need to use it for resolving declarations instead of PSI.
That’s the most complex solution but also the most powerful and flexible one.
The more i look into how to work around this the worse it looks.
I thought maybe i could just not make stubs for elements that are in conditional compilation blocks, but this seems to break any logic that relies on stub children/parents in PsiTreeUtil and it looks more and more likely that the only solution is to revert all the stub logic
is there no other way to get around this limitation. it would really suck to have to revert days of work and strip out all the use of stubs.
I’m really sorry. Unfortunately, IntelliJ is not designed to deal with conditionally compiled languages, and fixing it is a huge task.
My advice for you is to switch to lazy but persistent simplified “stub” caches with the help of FileAttributes.
You already have the stub hierarchy extracted, so you need to write serialization (though it should be rather easy with the help of kotlinx.serialization) and start using it for resolve.
Let me know if you have any particular questions, will be glad to help.
The plugin uses Grammarkit with and a lot of declarations that have stubclass defined. This generates a lot of code that expect there to be a stub and avoids having to (re-)parse the file.
Would it be feasible to just override getStub() in my Psi base class, and try to handle things from there ? (or maybe do the entire tree overriding getStubTreeOrFileElement() in the File class)
The getStub() overload would do Something like
Check if stub should be created
Create stub if necessary using my existing StubElementFactory classes
Use existing StubSerializer classes to serialize and then store stub data somewhere (FileAttributes ?)
Return newly created or desierialized stub
The base PSI class or a Project level service:
Some kind of logic to invalidate/update stubs when file changes
Btw. i tried to use LanguageStubDefinition#shouldBuildStubFor to skip files with CC content (i am just reading content as a string and do a contains(“#if”) ), but i still get stub tree issues, not sure if i am just misunderstanding how this is intended to work or i am missing something when trying to ignore files with CC.
