Adding `Go to definition` actions to Jinja within .sql and .yml files

Hello Community,

The past year I’ve been working on and off on a dbt-core plugin for Jetbrains IDEs, called dbtToolkit. Currently I am in a phase where I would like to add Jinja support to my plugin.

My next goal is to add Jinja support for all the dbt .sql and .yml files. Additionally I would like to add Go to Declaration functionality for {{ ref('some_model') }} and {{ source('some_source') }} function calls within the Jinja. For example to open the referencing .sql model or .yml source definition file, e.g. user can ctrl + click on a source or ref definition.

Currently I am a bit stuck on how I should move forward to implement this. Are there for example existing yaml and SQL parsers within the Jetbrains ecosystem, that I can extend with extra functionality for jinja and go to decleration support? I assume there must be for the existing syntax highlighting within the IDE. Or do I need to implement my own parser and/or lexer? Are there any good sample projects (ideally in Kotlin) which I can take a look at when implementing functionality like this? Would be great to get some advice and or high-level guidance (e.g. pushing me in the right direction) from someone who has done similar implementations in the past.

Greetings,

Ramon

I think you are looking for Language Injection and References and Resolve.

The Django plugin by JetBrains has Jinja 2 support (or so it claims; I have never tried it). You can thus reuse its parser for free by declaring a dependency on it. One thing to consider is that it only supports paid IDEs.

After that, making Go to Declaration work is as simple as implementing PsiReferenceContributor.

Thanks a lot for your suggestions!

Reading through the language injection docs I think this is indeed what I am looking for. Because the original source file is already either yaml or sql, and I would like to add some functionality on top of that, e.g. the jinja support + go to definition actions.

Will definitely take a look at the Django Jinja2 implementation to get a bit of a feeling for what the implementation would look like. Ideally I want to support community edition IDEs as well, to serve more users. However, if it is impossible to offer this functionality to community IDEs I will probably switch to ultimate only.

Jinja2 is a template language. We don’t provide Template Language APIs in community edition IDEs.

It is rather different from what Language Injection does and cannot be replaced by it anyhow.

Hmm I see, and let’s say I only want to support references and no syntax highlighting or full JInja support? Purely the ability to click on a reference in the SQL / YML file e.g. {{ ref('target') }} and then jump to the definition?

I tried to implement this in a small PoC, and it works fine for Community Editions. However, I actually run into the problem where it doesn’t work for Ultimate Edition.

Community edition:

Ultimate edition:
… new members can only put in one media item per post, see next post …

Any idea what can be causing this problem, and if it is solvable? I tried quite some debugging already in Ultimate Edition.

The main problem seems to be around the JinjaReferenceContributor. In the Ultimate Edition (I think because of the built-in SQL support) there are no PsiElements being pushed to the registrar.registerReferenceProvider. For example when I remove the regex pattern and purely use **PlatformPatterns.psiElement()** as filter (most broad filter I can think of), I only get the PsiElements for the comment blocks in the SQL, but no PsiElement that actually contains the SQL itself so that it can be pushed down to the JinjaReferenceProvider.

My implementation:

Or via Github: Comparing 47be2da215f3260ea1b0c85bef8d68c304210f48...4bbcf781c3b4f5db5017051badecf29d5e452a6b · ramonvermeulen/dbt-toolkit · GitHub

Entry in plugin.xml

<psi.referenceContributor implementation="com.github.ramonvermeulen.dbtToolkit.jinja.JinjaReferenceContributor" order="last"/>

JinjaPatterns.kt

package com.github.ramonvermeulen.dbtToolkit.jinja

object JinjaPatterns {
    val REF_PATTERN = Regex("""\{\{.*?ref\s*\(\s*['"](.*?)['"]\s*\).*?\}\}""")
}

JinjaReferenceContributor.kt

package com.github.ramonvermeulen.dbtToolkit.jinja

import com.intellij.patterns.PatternCondition
import com.intellij.patterns.PlatformPatterns
import com.intellij.psi.PsiElement
import com.intellij.psi.PsiReferenceContributor
import com.intellij.psi.PsiReferenceRegistrar
import com.intellij.util.ProcessingContext

class JinjaReferenceContributor : PsiReferenceContributor() {
    override fun registerReferenceProviders(registrar: PsiReferenceRegistrar) {
        registrar.registerReferenceProvider(
            PlatformPatterns.psiElement().with(object : PatternCondition<PsiElement>("jinjaRefPattern") {
                override fun accepts(element: PsiElement, context: ProcessingContext?): Boolean {
                    return JinjaPatterns.REF_PATTERN.containsMatchIn(element.text)
                }
            }),
            JinjaReferenceProvider(),
        )
    }
}

JinjaReferenceProvider.kt

package com.github.ramonvermeulen.dbtToolkit.jinja

import com.intellij.openapi.util.TextRange
import com.intellij.psi.PsiElement
import com.intellij.psi.PsiReference
import com.intellij.psi.PsiReferenceProvider
import com.intellij.util.ProcessingContext

class JinjaReferenceProvider : PsiReferenceProvider() {
    override fun getReferencesByElement(element: PsiElement, context: ProcessingContext): Array<PsiReference> {
        val fileContent = element.containingFile.text
        val matches = JinjaPatterns.REF_PATTERN.findAll(fileContent)

        return matches.map { match ->
            val refValue = match.groupValues[1]
            JinjaReference(element, TextRange(match.range.first, match.range.last + 1), refValue)
        }.toList().toTypedArray()
    }
}

JinjaReference.kt

package com.github.ramonvermeulen.dbtToolkit.jinja

import com.intellij.openapi.util.TextRange
import com.intellij.psi.PsiElement
import com.intellij.psi.PsiManager
import com.intellij.psi.PsiReferenceBase
import com.intellij.psi.search.FilenameIndex
import com.intellij.psi.search.GlobalSearchScope

class JinjaReference(
    element: PsiElement,
    range: TextRange,
    private val refValue: String,
) : PsiReferenceBase<PsiElement>(element, range) {
    override fun resolve(): PsiElement? {
        return findSqlFile(refValue)
    }

    private fun findSqlFile(refValue: String): PsiElement? {
        val project = element.project
        val sqlFileName = "$refValue.sql"

        val files = FilenameIndex.getVirtualFilesByName(sqlFileName, GlobalSearchScope.allScope(project))
        val psiFile = PsiManager.getInstance(project).findFile(files.first())

        return psiFile
    }
}

As addition on my last post:

Ultimate Edition:

It should be possible to create a “multiHostInjector” and set a prefix (so it knows in which “context” it will render this part). There you can define a range within tokens, which are languageInjectable (YAMLScalarValue in general for YAML for example). Then it should also find references automatically.

Docs: Language Injection | IntelliJ Platform Plugin SDK