Long read action due to FileBasedIndexImpl.ensureUpToDate() while calling FileBasedIndexEx.getAllKeys()

IntelliJ Platform
1

I have one real nuisance of an intermittent long read action attributable to the following stack trace (with unnecessary frames removed to keep it semi-compact):

Long read action in com.illuminatedcloud.intellij.lwc.typescript.env.AuraEnabledApexTypeDefinitionsEnvironmentUpdater.lambda$getMetadataFiles$1

======= Stack Trace: ========= 
"pool-5-thread-3" prio=0 tid=0x0 nid=0x0 waiting on condition
     java.lang.Thread.State: TIMED_WAITING
	at java.base@21.0.9/jdk.internal.misc.Unsafe.park(Native Method)
	at java.base@21.0.9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:410)
	at com.intellij.openapi.progress.impl.CoreProgressManager.sleepIfNeededToGivePriorityToAnotherThread(CoreProgressManager.java:953)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.runCheckCanceledHooks(ProgressManagerImpl.java:291)
	at com.intellij.openapi.progress.impl.CoreProgressManager.doCheckCanceled(CoreProgressManager.java:192)
	at com.intellij.openapi.progress.ProgressManager.checkCanceled(ProgressManager.java:336)
	at com.intellij.openapi.progress.ProgressIndicatorProvider.checkCanceled(ProgressIndicatorProvider.java:41)
	at com.intellij.psi.PsiElementVisitor.visitElement(PsiElementVisitor.java:43)
	at com.intellij.lang.javascript.psi.JSRecursiveWalkingElementVisitor.visitElement(JSRecursiveWalkingElementVisitor.java:23)
	at com.intellij.lang.javascript.psi.impl.JSFileBaseImpl$1.visitElement(JSFileBaseImpl.java:78)
    ...
	at com.intellij.lang.javascript.stubs.factories.TypeScriptInterfaceStubFactory.createStub(TypeScriptInterfaceStubFactory.kt:10)
	at com.intellij.psi.stubs.DefaultStubBuilder$StubBuildingWalkingVisitor.createStub(DefaultStubBuilder.java:87)
	at com.intellij.lang.javascript.psi.stubs.impl.JSFileStubBuilder$JSStubBuildingWalkingVisitor.createStub(JSFileStubBuilder.java:133)
	at com.intellij.psi.stubs.DefaultStubBuilder$StubBuildingWalkingVisitor.visitNode(DefaultStubBuilder.java:58)
	at com.intellij.lang.javascript.psi.stubs.impl.JSFileStubBuilder$1.visitNode(JSFileStubBuilder.java:89)
	at com.intellij.psi.stubs.DefaultStubBuilder$StubBuildingWalkingVisitor.buildStubTree(DefaultStubBuilder.java:53)
	at com.intellij.lang.javascript.psi.stubs.impl.JSFileStubBuilder$JSStubBuildingWalkingVisitor.buildStubTreeAndUpdateLinks(JSFileStubBuilder.java:127)
	at com.intellij.lang.javascript.psi.stubs.impl.JSFileStubBuilder.doBuildStubTree(JSFileStubBuilder.java:104)
	at com.intellij.lang.javascript.psi.stubs.impl.JSFileStubBuilder.buildStubTree(JSFileStubBuilder.java:44)
	at com.intellij.psi.stubs.StubTreeBuilder.lambda$buildStubTree$1(StubTreeBuilder.java:145)
    ...
	at com.intellij.psi.stubs.StubUpdatingIndex$2.computeValue(StubUpdatingIndex.java:156)
	at com.intellij.util.indexing.SingleEntryIndexer.map(SingleEntryIndexer.java:29)
	at com.intellij.util.indexing.SingleEntryIndexer.map(SingleEntryIndexer.java:19)
	at com.intellij.util.indexing.impl.MapReduceIndex.mapByIndexer(MapReduceIndex.java:363)
	at com.intellij.util.indexing.impl.MapReduceIndex.mapInput(MapReduceIndex.java:354)
	at com.intellij.util.indexing.impl.MapReduceIndex.mapInputAndPrepareUpdate(MapReduceIndex.java:279)
	at com.intellij.psi.stubs.StubUpdatingIndexStorage.mapInputAndPrepareUpdate(StubUpdatingIndexStorage.java:62)
	at com.intellij.psi.stubs.StubUpdatingIndexStorage.mapInputAndPrepareUpdate(StubUpdatingIndexStorage.java:21)
	at com.intellij.indexing.composite.CompositeInvertedIndexBase.updateBaseIndex(CompositeInvertedIndexBase.java:278)
	at com.intellij.indexing.composite.CompositeInvertedIndexBase.mapInputAndPrepareUpdate(CompositeInvertedIndexBase.java:59)
	at com.intellij.indexing.composite.CompositeInvertedIndexBase.mapInputAndPrepareUpdate(CompositeInvertedIndexBase.java:25)
	at com.intellij.util.indexing.FileBasedIndexImpl.createSingleIndexValueApplier(FileBasedIndexImpl.java:1674)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$doIndexFileContent$21(FileBasedIndexImpl.java:1544)
	at com.intellij.util.indexing.FileBasedIndexImpl$$Lambda/0x0000000802165e20.run(Unknown Source)
	at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.lambda$freezeFileTypeTemporarilyWithProvidedValueIn$15(FileTypeManagerImpl.java:775)
	at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl$$Lambda/0x0000000801e97bb0.run(Unknown Source)
	at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.cacheFileTypesInside(FileTypeManagerImpl.java:846)
	at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.freezeFileTypeTemporarilyWithProvidedValueIn(FileTypeManagerImpl.java:771)
	at com.intellij.util.indexing.FileBasedIndexImpl.doIndexFileContent(FileBasedIndexImpl.java:1497)
	at com.intellij.util.indexing.FileBasedIndexImpl.indexFileContent(FileBasedIndexImpl.java:1472)
	at com.intellij.util.indexing.FileBasedIndexImpl.processRefreshedFile(FileBasedIndexImpl.java:1412)
	at com.intellij.util.indexing.FileBasedIndexImpl$VirtualFileUpdateTask.doProcess(FileBasedIndexImpl.java:1798)
	at com.intellij.util.indexing.FileBasedIndexImpl$VirtualFileUpdateTask.doProcess(FileBasedIndexImpl.java:1792)
	at com.intellij.util.indexing.UpdateTask.process(UpdateTask.java:65)
	at com.intellij.util.indexing.UpdateTask.processAll(UpdateTask.java:34)
	at com.intellij.util.indexing.FileBasedIndexImpl.forceUpdate(FileBasedIndexImpl.java:1818)
	at com.intellij.util.indexing.FileBasedIndexImpl.ensureUpToDate(FileBasedIndexImpl.java:878)
	at com.intellij.util.indexing.FileBasedIndexEx.processAllKeys(FileBasedIndexEx.java:181)
	at com.intellij.util.indexing.FileBasedIndexImpl.processAllKeys(FileBasedIndexImpl.java:906)
	at com.intellij.util.indexing.FileBasedIndexEx.processAllKeys(FileBasedIndexEx.java:167)
	at com.intellij.util.indexing.FileBasedIndexEx.getAllKeys(FileBasedIndexEx.java:161)
	at com.illuminatedcloud.intellij.apex.index.AbstractApexDeclarationNameIndex.getAllNames(AbstractApexDeclarationNameIndex.java:160)
	at com.illuminatedcloud.intellij.lwc.typescript.env.AuraEnabledApexTypeDefinitionsEnvironmentUpdater.lambda$getMetadataFiles$1(AuraEnabledApexTypeDefinitionsEnvironmentUpdater.java:146)
	at com.illuminatedcloud.intellij.lwc.typescript.env.AuraEnabledApexTypeDefinitionsEnvironmentUpdater$$Lambda/0x000000080295ef40.compute(Unknown Source)
	at com.intellij.openapi.application.impl.AppImplKt$rethrowCheckedExceptions$2.invoke(appImpl.kt:126)
	at com.intellij.platform.locking.impl.NestedLocksThreadingSupport.runReadAction(NestedLocksThreadingSupport.kt:856)
	at com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(ApplicationImpl.java:1069)
	at com.intellij.openapi.application.ReadAction.compute(ReadAction.java:66)
	at com.illuminatedcloud.intellij.lwc.typescript.env.AuraEnabledApexTypeDefinitionsEnvironmentUpdater.getMetadataFiles(AuraEnabledApexTypeDefinitionsEnvironmentUpdater.java:146)

I have this logic running on a background/worker thread, and it checks quite diligently for indexing/scanning, aborting and rescheduling the work if that’s the case. Nonetheless, the IDE intermittently reports that my plugin may be causing the IDE to run slowly with the thread dump pointing at the stack trace above as the culprit.

As you can see from the stack trace, it’s synchronously building indexes that I never need for this purpose, e.g., the JavaScript stub index.

Note that I’ve tried various ways of enumerating index keys including processAllKeys() instead of getAllKeys() (even though the latter calls the former), but the result is the same.

Is there any way that I can perhaps determine proactively whether or not querying my own index will result in a forced index rebuild like this, or is there something else I should be doing here when enumerating keys from this index that won’t result in this same potential long read action while doing so? I’ve found FileBasedIndexEx.diableUpToDateCheckIn(), but it’s marked as @ApiStatus.Internal so I’m obviously not planning to use it.

I feel like I’m doing (or at least trying to do) everything right here, but I’m still running into the much-feared “Long read action due to ”.

Thanks in advance for any insights/suggestions.

2

This is a real problem here. Can you replace it with ReadAction.nonBlocking API or coroutines readAction { } / smartReadAction { } that would cancel and retry when users try to interact with IDE or WriteAction occurs?

ReadAction.compute is non-cancellable code which cannot be interrupted and retried.

3

If you need a blocking retryable variant of the ReadAction in a background thread you can use:

ReadAction.nonBlocking {

    // do not forget validity check here for VFS / PSI files and Module/Project
}
    .inSmartMode(project) // if access to indexes needed
    .executeSynchronously()
1 Like
4

Thanks. I’ll give that a shot and see how it goes.

5

Also the rest of the stacktrace could help to understand what kind of processing you do. Is it manual/custom scanning of files? Would it be possible to share some pseudo-code of that?

6

That specific custom file-based index contains the qualified names of classes in my plugin’s Java-like language that have a very specific annotation denoting that they can be accessed via a JavaScript/TypeScript remoting feature.

In this specific usage of the index, I need all keys because my plugin generates TypeScript type definitions (*.d.ts files) when a project is opened for that project’s classes that can be imported into JavaScript/TypeScript source files in the same project to facilitate the client-side API of that remoting feature.

Let me know if that doesn’t make sense.

7

Do you use our indexing framework for it and plug into the platform scanning process? Or you enumerate VFS files / all keys manually in a background thread? It seems the latter.

8

I use the platform’s file-based index feature, specifically the fileBasedIndex EP with a subclass of ScalarIndexExtension that also implements DataIndexer, i.e.:

public class AuraEnabledApexDeclarationShortNameIndex 
    extends ScalarIndexExtension<String> 
    implements DataIndexer<String, Void, FileContent>

And of course I query that for the keys using:

FileBasedIndex.getInstance().getAllKeys(indexId, project)

All pretty boiler-plate indexing of frequently-needed content information from source files.

9

To be even more specific, the code for the long-running stack trace above is effectively the following executed from a pooled background thread:

Set<String> classNames = ReadAction.compute(() -> FileBasedIndex.getInstance()
    .getAllKeys(indexId, project)
    .stream()
    .filter(key -> !key.startsWith(CASE_INSENSITIVE_INDEX_KEY_PREFIX))
    .collect(Collectors.toCollection(LinkedHashSet::new)));

That call is resulting in a forced index update via FileBasedIndexImpl.forceUpdate() which is the true reason that this read action ends up being long-running.

10

Oh, bingo! Then non-blocking ReadAction fits, use it for long-running processing in background, this code must also take into account dumb mode - inSmartMode(), check isDisposed of project inside read action before first access to it (once).

FileBasedIndexImpl.forceUpdate()

It may happen if there are less than 50 unindexed files and we are scanning in smart mode, then an access attempt to index under read action results in forced in-place indexing. Because IDE promised smart mode and it must then provide index access.

1 Like
11

Just to be 100% clear, you’re suggesting that I change:

Set<String> classNames = ReadAction.compute(() → FileBasedIndex.getInstance().getAllKeys(indexId, project));

to:

Set<String> classNames = ReadAction.nonBlocking(() -> {
    return project.isDisposed() ?
        Collections.emptySet() :
        FileBasedIndex.getInstance().getAllKeys(indexId, project);
}).inSmartMode(project).executeSynchonously();

Is that a correct interpretation of the suggestions you’ve made? And thanks again for the assistance!

12

Yes, in Kotlin it would be just smartReadAction with lambda so one can understand why we write async and background code with coroutines.

Or you can use the callback style with .submit + app executor service and do not block an extra thread.

1 Like