Make python .pyi project libraries visible to LightVirtualFile instances

IntelliJ Platform
1

I’m using LightVirtualFile objects to represents files located on an embedded microcontroller device. I download them into memory from the device, let the user edit them via the LightVirtualFile, and then upload them back to the device.

I also add project level .pyi typehint packages for MicroPython specific modules.

However, the LightVirtualFiles aren’t children of the project directory and thus don’t recognize these typehint packages.

I tried creating the LightVirtualFile overriding its parent forcefully to be the project dir - that made stub packages work, but understandably the IDE complained about the LightVirtualFile actually not being found amongst the project directory’s children.

Is there anyway to make the LightVirtualFile see the project libraries? Or is there a better way to go about this than using a LightVirtualFile? Below is how I add the python library.

    private fun addMpyLibrary(newStubPackageName: String) {
        DumbService.getInstance(project).smartInvokeLater {
            ApplicationManager.getApplication().runWriteAction {
                // Try to find the stub package
                val stubPackage = getStubPackages().first.find { "${it.name}_${it.mpyVersion}" == newStubPackageName }

                if (!settings.state.areStubsEnabled || stubPackage == null || !stubPackage.isInstalled) {
                    return@runWriteAction
                }

                val modelsProvider = ModifiableModelsProvider.getInstance()
                val projectLibraryModel = modelsProvider.getLibraryTableModifiableModel(project)

                // Create library
                val newLibrary = projectLibraryModel.createLibrary(LIBRARY_NAME, PythonLibraryType.getInstance().kind)
                val libraryModel = newLibrary.modifiableModel

                // Add roots
                val rootUrl = "${MpyPaths.stubBaseDir}/$newStubPackageName"

                val stdlibUrl = "$rootUrl/stdlib"

                val rootFile = LocalFileSystem.getInstance().findFileByPath(rootUrl)
                val stdlibFile = LocalFileSystem.getInstance().findFileByPath(stdlibUrl)

                if (rootFile != null) {
                    libraryModel.addRoot(rootFile, OrderRootType.CLASSES)
                }
                if (stdlibFile != null) {
                    libraryModel.addRoot(stdlibFile, OrderRootType.CLASSES)
                }

                // Commit library changes first
                libraryModel.commit()
                projectLibraryModel.commit()

                // Then add to modules
                for (module in ModuleManager.getInstance(project).modules) {
                    val moduleModel = modelsProvider.getModuleModifiableModel(module)
                    moduleModel.addLibraryEntry(newLibrary)
                    modelsProvider.commitModuleModifiableModel(moduleModel)
                }
            }
        }
    }
2

It is considered in-memory file and treated like that by many IDE subsystems. I’d suggest implementing you own VirtualFileSystem and do not subclass LightVirtualFile for its files.

Also wanted to highlight that modifying Libraries with push semantics is kind of unreliable move. Plugins such as GraphQL do this correctly by contributing libraries to workspace model via com.intellij.ide.projectView.impl.nodes.ExternalLibrariesWorkspaceModelNodesProvider. This way libraries model is always consistent with configuration files, indexes and will not face concurrent modification issues.

3

Thanks for the suggestion.

I’ve considered implementing my own VirtualFileSystem before. However, due to the remote file system’s nature (a single core microcontroller) - refresh, saving etc. are operations that can only be done under certain conditions.

I only use the LightVirtualFile to download a selected number of files and show them to the user/let them modify them in the editor and then save them/refresh them (ensuring that only one operation happens at a time).

It seems like implementing a whole VirtualFileSystem would be too much effort just for the sake of facilitating on-device file editing. Would there be some other API or workaround that would let me stay closer to the model I have right now?

I solely need to make the existing LightVirtualFiles see the project’s Python Libraries / have an option to manually add python libraries to each LightVirtualFile when instantiating it.

Thanks for the suggestion regarding the appropriate way to add stubs. I’ve struggled with figuring out the right way before not seeing any examples, but I’ll look a GraphQL and try to replicate what it does.

4

Please take a look also at com.intellij.openapi.roots.AdditionalLibraryRootsProvider and com.intellij.openapi.roots.SyntheticLibrary, it might be an easier way

5

Will SyntheticLibrary work for Python type hints?

class MpyStubsSyntheticLibrary(
    private val stubRoots: Collection<VirtualFile>
) : SyntheticLibrary() {

    override fun getSourceRoots(): Collection<VirtualFile> {
        return stubRoots
    }

    override fun getBinaryRoots(): Collection<VirtualFile> {
        return stubRoots
    }

    override fun equals(other: Any?): Boolean {
        if (this === other) return true
        if (other !is MpyStubsSyntheticLibrary) return false
        return stubRoots == other.stubRoots
    }

    override fun hashCode(): Int = stubRoots.hashCode()
}

I’m trying to add the above, but the python files don’t recognize the stubs. I have debug printing in the roots provider confirming the roots are found on device, and that a list of the SyntheticLibraries containing the appropriate .pyi roots which work with my project library approach.

Do I need to somehow mark these roots so that the type hint system of PyCharm will recognize them? I tried using both source and binary roots of the library to no avail.

6

@yuriy.artamonov

Could you please point me in the right direction here?