Non-Static Imports?

[rohan-datar]

I'm trying to use pkl to solve a problem where I need to render similar data in different formats. The basic idea is that I have a common template and I need to emit it as xml or json depending on the context. This is how I've been thinking about it so far:
I start with defining a class for the structure of the data:
objdef.pkl

class myObj {
   local self = this
   name: String
   value: Int
   jsonField: String?
   xmlField: String?
   
   function asJSON() = new {
      name = self.name
      value = self.value
      specificField = self.jsonField ?? default
   }
   function asXML() = (xml.Element(self.name)) {
      value = self.value
      specificField = self.xmlField ?? default
   }
}

Then I have a few files that actually define the data
foo.pkl

import "objdef.pkl"

objects = Listing<myObj> {
   new myObj {
     name = "foo"
     value = 10
     jsonField = "foo-json"
     xmlField = "foo-xml"
   }
...
}

bar.pkl
foo.pkl

import "objdef.pkl"

objects = Listing<myObj> {
   new myObj {
     name = "bar"
     value = 20
     jsonField = "bar-json"
     xmlField = "bar-xml"
   }
...
}

I then have a common render module for each type
renderJSON.pkl

data = import("foo,pkl") <- Want to be able to change this based on what file I'm rendering

output = {
   for (_obj i objects) {
     _obj.asJSON()
   }
}

I tried passing in the filepath as a prop and reading it in, but putting anything other than a string literal into an import expression seems to throw an error. Is there any way around this?

[bioball]

One way to work around this is to use a glob import, and then select the actual import from the resulting mapping.

local allModules = import*("**.pkl")

local myModule = read("prop:my-module")

data = allModules[myModule]

By the way, sidenote: zero-argument functions are kind of a smell in Pkl, because their result is un-cached (calling the same function multiple times will execute the function body multiple times; Pkl does not memoize function calls).

Instead, use a hidden fixed property, because properties get cached after they are accessed. And, Pkl properties are very similar to zero-argument functions (see late binding).

-   function asJSON() = new {
+   hidden fixed asJSON = new {
      name = self.name
      value = self.value
      specificField = self.jsonField ?? default
   }

[bioball]

Another note: consider putting this logic inside renderer converters, rather than inside your class definition.

For example:

local jsonRenderer: JsonRenderer = new {
  [myObj] = (it) -> new Dynamic {
    name = it.name
    value = it.value
    specificField = it.jsonField
  }
}

local xmlRenderer: XmlRenderer = new {
  [myObj] = (it) -> (xml.Element(it.name)) {
    value = it.value
    specificField = it.xmlField
  }
}

output {
  renderer = // whatever logic you need here
}

[rohan-datar]

The actual functions aren't generally zero argument, there are usually some extra properties I need to pass in from the command line. I was just trying to get a minimal example. But your point is noted.

The globbed imports are interesting, but that still means that the data module files have to live at a fixed path relative to the renderer? I was hoping to keep them separately since the data is likely to be changed fairly often, and I would like to keep those files in a separate git repo. It's workable, but frustrating nonetheless.

Alternatively, would it be possible to make pkl use a custom renderer, either via the command line or pkl-go? Extending the json and xml renderers sounds doable, but I'm not sure how to make that work.

[HT154]

Can you clarify what you mean by "custom renderer"? I'm confused by what a custom renderer would do here that the built-in ones (with properly defined converters) wouldn't. You can certainly implement your own renderers in Pkl, eg. https://github.com/apple/pkl-pantry/blob/main/packages/pkl.toml/toml.pkl

[HT154]

The globbed imports are interesting, but that still means that the data module files have to live at a fixed path relative to the renderer? I was hoping to keep them separately since the data is likely to be changed fairly often, and I would like to keep those files in a separate git repo. It's workable, but frustrating nonetheless.

I've run into this before as well. You have some "aggregation" logic in a module that you want to ship as part of your package, but you don't want to copy it around into each repo that needs it. Here's how I've addressed that:

aggregate.pkl (in package)

include: Mapping<String, Any>

// do stuff with `include`, eg. rendering logic

aggregate.pkl (usage site)

amends "package://...#/aggregate.pkl"
include = import("<glob expression>")

So you do still need a little bit of boilerplate, but it's very minimal.

[rohan-datar]

Can you clarify what you mean by "custom renderer"? I'm confused by what a custom renderer would do here that the built-in ones (with properly defined converters) wouldn't. You can certainly implement your own renderers in Pkl, eg. https://github.com/apple/pkl-pantry/blob/main/packages/pkl.toml/toml.pkl

I meant converter, I hadn't really encountered the concept before, and I wasn't familiar with the difference sorry

[rohan-datar]

The globbed imports are interesting, but that still means that the data module files have to live at a fixed path relative to the renderer? I was hoping to keep them separately since the data is likely to be changed fairly often, and I would like to keep those files in a separate git repo. It's workable, but frustrating nonetheless.

I've run into this before as well. You have some "aggregation" logic in a module that you want to ship as part of your package, but you don't want to copy it around into each repo that needs it. Here's how I've addressed that:

aggregate.pkl (in package)

include: Mapping<String, Any>

// do stuff with `include`, eg. rendering logic

aggregate.pkl (usage site)

amends "package://...#/aggregate.pkl"
include = import("<glob expression>")

So you do still need a little bit of boilerplate, but it's very minimal.

I think I can work with this, adding a file which has the import expression in it would be fairly easy. Thanks! I'm still curious about doing it via a converter though. I can think of a few places where that might be a better solution overall

[rohan-datar]

So I think I might be able to accomplish this with a renderer converter. The one thing I would like to clarify is if I can define the renderer at a template level. For example, if I could do something like
template.pkl

objects = Listing<myObj> 

output {
  files {
    ["out.json"] {
       renderer = CustomJsonRenderer{}
    }
    ["out.xml"] {
      renderer = CustomXmlRenderere{}
    } 
  }
}

And then have
foo.pkl

amends "template.pkl"
import "objdef.pkl"

objects = Listing<myObj> {
   new myObj {
     name = "bar"
     value = 20
     jsonField = "bar-json"
     xmlField = "bar-xml"
   }
...
}

Would evaluating foo.pkl produce an out.json and out.xml? Could I configure it to only produce one of those?

[HT154]

I'm a unsure of the context and goals here, so take this with a grain of salt.

Here's one way you can go about this:

local outputFormats = read("prop:outputFormats").split(",").toSet()
objects = Listing<myObj> 

output {
  files {
    when (outputFormats.contains("json")) {
      ["out.json"] {
        value = objects
        renderer = ...
      }
    }
    when (outputFormats.contains("xml")) {
      ["out.xml"] {
        value = objects
        renderer = ...
      } 
    }
  }
}

You'd then evaluate this with pkl eval -m . -p outputFormats=json,xml <module.pkl>

[rohan-datar]

I think I understand that, but my question was more if amends will change the output format of the file that is amending the template module. In my example above would running pkl eval foo.pkl cause it to produce out.json and/or out.xml?

[HT154]

Ah, you're looking to change the base name based on the input filename? I think you may be able to derive that using the pkl:reflect library by parsing the module URI. You might also look into batch evaluation via the CLI which can do some basic templating, but I'm not sure it works for this particular case.

[rohan-datar]

I'll give it a shot. Thanks!