How to diagnose and fix `Range for element: ... is out of file range ...`

IntelliJ Platform
1

For BashSupport Pro, I received this exception:

java.lang.Throwable: Range for element: 'FUNCTION_DEFINITION' = (5246,8273) is out of file 'ShFile: file.sh)' range: (0,7855); file contents length: 7855
 file provider: com.intellij.psi.SingleRootFileViewProvider{vFile=file://path/file.sh, vFileId=1428361, content=VirtualFileContent{size=8309}, eventSystemEnabled=true}

The code, which is throwing it is in an inspection, which detects if a variable could be made local (i.e. it does not have any global uses outside the current scope). It’s running a ReferenceSearch for the inspection. This is probably not the best, but I can’t think of a better solution right now.

The code usually works, but appears to be causing the exception on some setups.

What’s causing these range is out of range … errors?

Code:

 ReferencesSearch.search(referenceTarget, effectiveUseScope).anyMatch {
  // ...
 }

Stacktrace starting from the reference search:

java.lang.Throwable: Range for element: 'FUNCTION_DEFINITION' = (5246,8273) is out of file 'ShFile: file.sh)' range: (0,7855); file contents length: 7855
 file provider: com.intellij.psi.SingleRootFileViewProvider{vFile=file:///path/file.sh, vFileId=1428361, content=VirtualFileContent{size=8309}, eventSystemEnabled=true}
 committed=true
 sharedSourceSupport:true, cachedViewProviders:1
 root Language: BashSupport Pro Shell Script length=7855; contentsLoaded=true
	at com.intellij.openapi.diagnostic.Logger.error(Logger.java:375)
	at com.intellij.psi.impl.search.LowLevelSearchUtil.diagnoseInvalidRange(LowLevelSearchUtil.java:253)
	at com.intellij.psi.impl.search.LowLevelSearchUtil.getTextOccurrencesInScope(LowLevelSearchUtil.java:159)
	at com.intellij.psi.impl.search.PsiSearchHelperImpl$1.processInReadAction(PsiSearchHelperImpl.java:315)
	at com.intellij.psi.impl.search.PsiSearchHelperImpl$1.processInReadAction(PsiSearchHelperImpl.java:299)
	at com.intellij.openapi.application.ReadActionProcessor.lambda$process$0(ReadActionProcessor.java:11)
	at com.intellij.openapi.application.impl.AppImplKt$rethrowCheckedExceptions$2.invoke(appImpl.kt:126)
	at com.intellij.platform.locking.impl.NestedLocksThreadingSupport.runReadAction(NestedLocksThreadingSupport.kt:854)
	at com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(ApplicationImpl.java:1109)
	at com.intellij.openapi.application.ReadAction.computeBlocking(ReadAction.java:88)
	at com.intellij.openapi.application.ReadAction.compute(ReadAction.java:70)
	at com.intellij.openapi.application.ReadActionProcessor.process(ReadActionProcessor.java:11)
	at com.intellij.concurrency.JobLauncherImpl.lambda$processImmediatelyIfTooFew$8(JobLauncherImpl.java:245)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$14(CoreProgressManager.java:744)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:819)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:775)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:743)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:87)
	at com.intellij.concurrency.JobLauncherImpl.lambda$processImmediatelyIfTooFew$9(JobLauncherImpl.java:242)
	at com.intellij.openapi.application.impl.AppImplKt$runnableUnitFunction$1.invoke(appImpl.kt:124)
	at com.intellij.openapi.application.impl.AppImplKt$runnableUnitFunction$1.invoke(appImpl.kt:124)
	at com.intellij.platform.locking.impl.NestedLocksThreadingSupport.runReadAction(NestedLocksThreadingSupport.kt:854)
	at com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(ApplicationImpl.java:1099)
	at com.intellij.concurrency.JobLauncherImpl.processImmediatelyIfTooFew(JobLauncherImpl.java:252)
	at com.intellij.concurrency.JobLauncherImpl.invokeConcurrentlyUnderProgressAsync(JobLauncherImpl.java:78)
	at com.intellij.concurrency.JobLauncherImpl.invokeConcurrentlyUnderProgress(JobLauncherImpl.java:65)
	at com.intellij.concurrency.JobLauncher.lambda$invokeConcurrentlyUnderContextProgress$0(JobLauncher.java:65)
	at com.intellij.concurrency.ConcurrencyUtils.runWithIndicatorOrContextCancellation(ConcurrencyUtils.java:38)
	at com.intellij.concurrency.ConcurrencyUtils.runWithIndicatorOrContextCancellation(ConcurrencyUtils.java:20)
	at com.intellij.concurrency.JobLauncher.invokeConcurrentlyUnderContextProgress(JobLauncher.java:63)
	at com.intellij.psi.impl.search.PsiSearchHelperImpl.bulkProcessElementsWithWord(PsiSearchHelperImpl.java:323)
	at com.intellij.psi.impl.search.PsiSearchHelperImpl.processSingleRequest(PsiSearchHelperImpl.java:1344)
	at com.intellij.psi.impl.search.PsiSearchHelperImpl.lambda$processRequests$22(PsiSearchHelperImpl.java:903)
	at com.intellij.concurrency.ConcurrencyUtils.runWithIndicatorOrContextCancellation(ConcurrencyUtils.java:38)
	at com.intellij.concurrency.ConcurrencyUtils.runWithIndicatorOrContextCancellation(ConcurrencyUtils.java:20)
	at com.intellij.psi.impl.search.PsiSearchHelperImpl.processRequests(PsiSearchHelperImpl.java:887)
	at com.intellij.psi.search.SearchRequestQuery.processResults(SearchRequestQuery.java:21)
	at com.intellij.util.AbstractQuery.doProcessResults(AbstractQuery.java:88)
	at com.intellij.util.AbstractQuery.delegateProcessResults(AbstractQuery.java:105)
	at com.intellij.util.MergeQuery.processResults(MergeQuery.java:22)
	at com.intellij.util.AbstractQuery.doProcessResults(AbstractQuery.java:88)
	at com.intellij.util.AbstractQuery.delegateProcessResults(AbstractQuery.java:105)
	at com.intellij.util.UniqueResultsQuery.processResults(UniqueResultsQuery.java:37)
	at com.intellij.util.AbstractQuery.doProcessResults(AbstractQuery.java:88)
	at com.intellij.util.AbstractQuery.forEach(AbstractQuery.java:80)
	at com.intellij.util.Query.anyMatch(Query.kt:80)
	at pro.bashsupport.editor.inspections.ShOnlyLocallyUsedGlobalVariableInspection.M(ShOnlyLocallyUsedGlobalVariableInspection.kt:111)
3

Hi Joachim,

What IJ version is this?

4

Hi Maksim,

that’s PS-261.22158.283 on Windows 11 (a user’s system, not mine).

Thanks!

5

The exception is rather mysterious.

The following is happening:

PsiElement ‘FUNCTION_DEFINITION’ is processed inside a read action.

It is checked that it’s valid, then its containing file is taken, and it turns out that the content of the file is shorter than the range of the psiElement.

The very-very suspicious thing is that I see content=VirtualFileContent{size=8309} and BashSupport Pro Shell Script length=7855.

I see two options at the moment:

  1. we have a serious bug in PSI infrastructure, and PSI can run into an inconsistent state with its file.
  2. ‘FUNCTION_DEFINITION’ violates PSI contracts and returns an inconsistent range or returns an unrelated PsiFile as its containing file.

Could you comment on the second possibility?

6

Thank you for your reply.

Right now, the 2nd possibility is more likely, I think.

I checked the implementation of my FUNCTION_DEFINITION PSI element.

It’s not returning a different file. It’s older code and is using caching for getTextRange and a few related methods. The resolving is using the ranges a lot and it was an optimization to reduce the overhead.

But apparently it’s somehow broken or not always reliable, I suppose.

Would you recommend to drop this entirely? Or is there an obvious problem with the code?

I’m currently unsure about the use of greenStub (there seems to be newer API now, which I just noticed) and I’m not sure if subtreeChanged is guaranteed to be called when the PSI element or a parent is changed.

The code is like this:

class ShFunctionDefinitionMixin : StubBasedPsiElementBase<ShFunctionDefinitionStub>, ShFunctionDefinition {
    // For unknown reasons, ShExternalRenameTest failed after adding caching of textRange.
    // Apparently, subtreeChanged() was not called on (clone of?) the modified file.
    // [...]
    // We're now dropping the cache also when the modified stamp doesn't match anymore.
    @Volatile
    private var cachedStamp = -1L

    @Volatile
    private var cachedTextRange: TextRange? = null

    override fun subtreeChanged() {
        dropCaches()
        super.subtreeChanged()
    }

    override fun clone(): Any {
        val clone = super.clone() as ShFunctionDefinitionMixin
        clone.dropCaches()
        return clone
    }

    private fun dropCaches() {
        cachedTextRange = null
        cachedStamp = -1L
    }

    private fun refreshCacheIfNeeded() {
        val currentStamp = containingFile.modificationStamp
        val currentCachedStamp = this.cachedStamp
        if (currentCachedStamp == -1L || currentCachedStamp != currentStamp) {
            val cachedTextRange = greenStub?.range ?: super.getTextRange()

            this.cachedTextRange = cachedTextRange
            this.cachedStamp = currentStamp
        }
    }

    override fun getTextRange(): TextRange {
        refreshCacheIfNeeded()
        return cachedTextRange
            ?: throw IllegalStateException("cached values must be non-null after refreshCacheIfNeeded")
    }
}
7

subtreeChanged is definitely not enough in this case. If another psi element is added before the given element, the given element’s text range will shift right, but its subtreeChanged() won’t be called leading to a stale cached value.

I’d recommend dropping caching for text ranges completely. If you see performance issues because of that, it might be a good idea to discuss them with us and potentially fix them on the platform side.

8

Thank you!
I’ll drop the caching and profile again.