I have tried a bunch of things now and asked AI for help testing different approaches and angles of attack, but it always ends up failing unless i accept the use of “@internal” APIs or use of reflection. as far as i can tell the API is just to strict and limiting for me to make something useful out of it.
I have made some stub and index stuff work as completely separate logic, but this just slows down the load time as intellij ends up having to do work twice, and too me the slow down is not worth it, the stubs and indexes where suppose to speed things up and this just slows things down.
it always ends up failing unless i accept the use of “@internal” APIs or use of reflection. as far as i can tell the API is just to strict and limiting for me to make something useful out of it.
I don’t promise anything, but we might consider revising internal API if you let us know.
I have made some stub and index stuff work as completely separate logic
Yes, that’s probably the best approach 
FTR when you compare stub vs no-stub, there is usually a significant difference in performance because the no-stub version usually does not parse the sources at all. So a slowdown is expected. What do you mean by “doing the work twice”?
I don’t remember all the different problems that stopped me from doing things, but i could probably mention some some road blocks i met.
I tried to replace StubBasedPsiElementBase with my own baseclass and also implemented StubBasedPsiElement in the hopes i could control and set my own substrate / stubs and node references. That failed due to com.intellij.psi.impl.source.FileTrees expects a StubBasedPsiElementBase and not the interface (StubBasedPsiElement).
Simply extending StubBasedPsiElementBase wont work as the methods i would need to override are final.
Its also a bit disappointing that StubTreeSerializer and all its implementations are internal.
It would be nice to use this and not having to make my own logic collecting serializers when i have already registered my factories and serializers with StubRegistryExtension
when it comes to doing things twice, i mean actually reading the files and create stubs and build indexes. there’s one pass done by intellij, and then there’s my pass done by my code.
I did try some experiments trying to copy the stubs made by intellij, but due to the deferred stub creation getStub would usually return null if i recall correctly. this might be due to the fact that i cant let intellij store stub trees as those would get the incorrect stub count when loaded by other projects.
i am currently considering rewriting all my stub-indexes to filebased-indexes as they can (as far as i know) separate the indexed values between projects even if the service is application wide. and then do another try filtering files and only allow shared stubs for files without conditional compilation.
random thought i got while digging around in intellij code, its probably not going to work but going to ask anyway.
Would it be possible as part of preprocessing a file before parsing it to create a “hidden for the user” preferably non-physical or temporary virtualFile (filename: original filename + conditional compilation flags) that is used when parsing and storing stub trees ?
The idea is that if i can create these hidden files and the name contains the CC flags used for parsing, it would mean that any attempt to at a later point loading stubs for it would work as based on the file name lookup we know that its using the same CC values.
Any other CC flag combination would refer to a different hidden/temp filename
Not sure if this even makes sense for you guys, but its just a random thought on how i could work around the shared stub tree limitations