Forum/email list to discuss DSL for different languages

[justjacksonn]

Hi,

I am currently working on building a Java SDK generator from RAML and Json schema. I find myself writing a lot of StringBuilder code to build up the Java class file string, and then output it to a file. This is all well and good, but this will only work for Java. I am not too familiar with DSL, but it seems like there is at least some attempt by Mulesoft and/or the community to have the ability to build client side SDKs for different languages.

Currently I believe APImatic can already do something like this, importing a RAML file and spitting out a zip file with different language SDKs using a common cross-language/platform Http client for handling the rest API calls. However, no doubt many places want to in-house this process, maybe customize it, and/or not depend on an outside party for this process. It seems fitting then that there is this project, but from what I can tell it lacks quite a bit of documentation, examples, etc on how to get going with adding in new languages.

Therefore, I am requesting some sort of (expedient if possible) process, be it an email list, forum, wiki, etc to get the process rolling so that those of us needing SDK generation in other languages can possibly contribute to this project. I myself need java, .net, ruby and python SDKs generated.

I am also interested in exactly what sort of SDK is generated. I assume it is purely a one to one mapping of resources in the RAML file. I also assume the Json Schema would be converted to some form of POJO to allow the consumer of the generated SDK to use the POJOs + the resource calls to perform the API operations.

Lastly, it would be ideal if generated SDKs did not mandate or force upon the consumer of it to use specific libraries. By this I mean, on the java side there is typically a set of dependencies on libraries like Gson, HttpClient, and so forth. If you already have an existing application with dependent libraries, you don't want to add bloat to your application just to use an SDK. Well, maybe some don't care, but, and especially in the mobile development arena, I'd assume most do not want to have to add more libraries. Therefore whatever SDK generation occurs, there would need to be some sort of transport interface that could then be adapted by consumers to use their own frameworks of choice. I did something similar in Java where I had a simple interface for Request/Response objects, and then implemented wrapper classes for using HttpClient's versions of those classes. It may not have been the prettiest bit of code, but it worked.

Anyway, where can we continue this sort of discussion? Can a list be started, or a forum? Is this the right project (being javascript/node specific at this point) to start this sort of larger SDK generation process under? I am personally not well versed in node, therefore not sure how capable it is to use for something like this.

Thanks.

[ddossot]

Don't generate Java by appending Strings: use a code generation library.

Until this morning, I would have told you: use CodeModel, but today Square has released JavaPoet... so be a poet 😉

[blakeembrey]

@justjacksonn Let's definitely get a discussion going and we can keep it in the issues here. Adding a new language to this generator should be relatively straightforward - but it'll take a little bit of JavaScript knowledge. You just need to add a new folder under the languages folder and export it in index.js. Since it's all templates you should able to write a template for whichever language you need, sorry there aren't better documentation on that yet. They are Handlebars templates though, so just look at https://github.com/mulesoft/raml-client-generator/blob/master/languages/javascript/index.js to get started. It references all the files and functions used to generate the JavaScript client.

As for the dependency injection question, that's more of a library implementation detail. So you can have the generated code expose methods for setting the request/response handlers.

Feel free to make a list of specific questions and we can update the readme as we work through them. I'd love to help you build this for Java 👍

[sichvoge]

Hi @justjacksonn, you're at the right project. It is actually a multi client generator one, but needs to be extended as it only supports Javascript by now. Can I ask you to go through the Implementation Guide for hints how you can implement your own DSL. If there is something missing or you still don't know how to start, let me know and I'll try to get you as much help as possible.

With regards to a forum. I would say that you can still use the raml.org forum, or do you see any necessarily to open a new one? As @blakeembrey already said - we should keep it here so nothing get's lost ;)

[sichvoge]

@justjacksonn - with regards to specific libraries - maybe we could try to not use any specific one using plain Java and what the JDK gives to us. Therefore we would not depend on something like that or trying to force someone. What do you think?

If you still want to give the consumer more choice about what library will be used - I think that can be solved with handlebars as well. Right @blakeembrey?

[justjacksonn]

Hi all,

Thank you for the replies. I'd definitely like to get a discussion going for a few languages. In particular my first choice is Java, then .net, python and ruby. I am curious though, this project has been around for a little while, is there just no awareness that it exists? It seems to me one of the big lures to using RAML is the ability to create SDKs, server side code and documentation from a single source of truth. Why is there no work (that I am aware of) in producing client side SDK generators for other languages?

Regardless, let's start it up. @sichvoge I am fine with any forum, just wasn't sure if this was the appropriate place or not. Should I start a new topic on the forum?

@blakeembrey I have cloned this project and starting to mess around with it for a java language generator. I am in unfamiliar territory with regards to DSLs and what to do next. @ddossot mentioned this new JavaPoet.. would that be the right way to go possibly? It does look a little cumbersome as opposed to some sort of template setup that would use token replacement with attributes from the RAML file... what's your thoughts?

[sichvoge]

Hi @justjacksonn,

the only thing I can say about it is that there is not a clear awareness that we've got this project yet. There are many people, including me, who evangelise it when ever there is any possibility. The project itself was release some month ago as it was private before to test it. We still have to go a long way with that, but we have to start somewhere, right? In the past (and still), we mostly concentrated on the API Designer/Console and other tooling. That is why there was no work before. As this project got better, we can start to create new templates now.

If you want to create templates - you should read something about http://handlebarsjs.com/ first. I think that makes it more clear how the javascript one is doing it. I am in the middle of creating something for Python to test the documentation for any gaps. Let you know guys how that went :)

[justjacksonn]

Ok.. well maybe I and some others can help evangelize it and get more people involved. APImatic, APIgee, etc have client side generators for several languages, so it's only fitting that RAML can support several languages for SDK generation. :)

Is it required to use javascript + handlebars to generate each language? Is that a capable enough solution? I've read about some of this before, but haven't actually dabbled in handlebars and such before. IF that is all is needed, where does DSL fit? Do we basically build a DSL from javascript + handlebars for each language we want to support?

[sichvoge]

Handlebars actually been used by swagger and their sdk as well (they use mustache actually). I would recommend to deep dive into it as this makes creating templates in the future way easier. I have no idea of it as well, but testing it right now :) If we can get a great documentation out of it - even better. I would start with an easy language to get behind it, or even not a programming language, but an HTML page listing information about your API :) Just to get a feel for it ;)

If you feel comfortable with that, create templates for your language and let the client generator written in nodejs doing the magic injecting the information and creating the sdk for you. That means, your job is only creating the template. Have a look on the guide I posted before.

[justjacksonn]

@sichvoge Indeed I am reading up on DSL, which to me seems not a fit for this.. at least in that we're generating code in a specific language, not a new specific-built language. Maybe my understanding of DSL is too narrow.

Suggestion, as generating SDKs or server side code is a pretty big topic when you add in all the language possibilities, perhaps we can add a new top level forum topic, with sub topics on client/server languages? Maybe even 3 levels.. client side/server side, and below each languages we'd like to discuss specifically? I fear a single forum post might get difficult to follow when posts about specific languages start getting added to the mix. What do you think?

[sichvoge]

Completely agree. Let's here what the other guys are thinking. cc @usarid @blakeembrey

[aldonline]

@justjacksonn . A DSL ( Domain Specific Language ) is a generic term denoting a language or maybe a convention ( over a host language ) used to express the semantics of a Specific Domain.

In this particular case I believe that @blakeembrey is using the term DSL in the latter form. If you look closely at the Javascript client, there is an elegant mapping between the constructed runtime object and the RAML abstract model. In particular, Resource paths are mapped to something like this: path.to.some("id").get()

So. In this case. The DSL would be this style of mapping resources/paths to objects and functions, and the host language is Javascript.

The DSL is mentioned and described so others can implement a similar DSL or "style" when building clients for other host languages.

[usarid]

I'm just impressed by how far this discussion has progressed just within github comments on an issue :-)

By all means, let's have it on the forums. Ideally someone should recreate this content on there, though, so people can look at the discussion in one place.

[justjacksonn]

Agreed @usarid. Can you guys add a new topic to the drop down list..perhaps something like API/SDK/CLI or Generating SDKs, CLIs, etc.. not sure the best name.. but basically the ability to generate output from RAML input that fits the general topic. It doesn't seem that it's possible to go further than one topic level on the forum so I suspect we'll have a few different forum posts under the one category that can at least start off discussions.

[justjacksonn]

In the meantime.. I added a java/ folder to languages/ and added

exports.java = require('./java');

to the languages/index.js file. I also copied the javascript/ to java/. When I run the command line tool with -l javascript it works. When I try -l java, it tells me language not supported in raml-client.js. I see it is looking at the languages variable, which is using require('../languages'); I admit I am not entirely sure how the variable assigned to a require() works, but I assume it reads the index.js output.. and should see my export statement and work. I definitely have more learning to do. Any pointers on making this work are appreciated.

[sichvoge]

I was able to actually develop a little HTML example. This is what I have in my index.js

/**
 * Export all implemented languages.
 */
exports.javascript = require('./javascript');
exports.python = require('./python');

What I am missing is a basic explanation of the RAML explicit Handlebars objects, type, description, etc. cc @blakeembrey

What I have right now is:

allResources: {
    type: array,
    description: contains a set of available resources starting with '/',
    items: {
         type: "resource"
    }
}

resource: {
     type: object,
     description: details of a specific resource,
     properties: {
         children: { ... },
         key: {
             type: string,
             description: name of the resource,
             ...
         }

         ??? 
     }
}

Would be good if we have such an overview as I said - doesn't necessarily to be JSON schema ;)

[sichvoge]

BTW, @justjacksonn did you run npm install -g again after you changed it?

[justjacksonn]

Well that may have been the problem @sichvoge! I did try the original sudo npm install raml-client-generator -g but that didn't seem to make it work. Just did the sudo npm install -g in the same directory and then it worked! So thank you for that. Now what's a good way to learn/understand the helpers/ partials/ and templates/ folders and how to modify them for java? Can you just write it for me? ;) I am looking up Handlebars, and I get the gist of it.. just not sure yet how I would work with the listed folders to take RAML and generate SDK java classes. Fun stuff though!

[sichvoge]

You actually don't need to use the same structure as the js template uses (helpers/, partials/, templates/). Start within the templates/ folder -> index.js.hbs! It is the main javascript file. In there you sometimes will find something like {{> resources}}. {{> ...}} means injection. resources for example you'll find inside the partials/resources.js.hbs. And so on, dot dot dot :D

What I have done is something very simple:

1. create a new language folder html

2. change content of the languages/index.js -> stated above

3. create new file languages/html/index.js with following content

   var generator = require('../../lib/generator');
   
   /**
   * Export a client generator instance.
   *
   * @type {Function}
   */
   module.exports = generator({
    templates: {
       'index.html': require('./templates/index.html.hbs'),
    },
    format: {
       variable: require('camel-case')
    }
   });
   

   In templates you put everything from your templates folder in.

4. create template inside languages/html/templates/index.html.hbs

   <!DOCTYPE html>
   <html>
   <body>
   
   <h1>My First Heading</h1>
   
   <ul>
   {{#each allResources}}
   {{#if key}}
   <li>{{key}}</li>
   {{/if}}
   {{/each}}
   </ul>
   </body>
   </html>
   

This is just really a simple example.

For all the handlebars object available for you inside your templates, like allResources, we definitely need a better overview. As I said in my comment above. I already started and will communicate with @blakeembrey to get that finished.

[justjacksonn]

I actually got almost that far.. I figured that I'd add a new .hbs file.. however I tried to be crafty.. I want an output of src/file.java so I created a src/file.java.hbs file in the templates folder. I then added:

'src/file.java': require('./templates/src/file.java.hbs') to the index.js, redid the npm install -g thing (kinda sux you gotta do that every time!). When I ran it, I got Error: ENOENT, open '/path/file.java'. Is there something special to creating sub-folders with files in them?

Anyway, this is no doubt not the right place to use to learn from nor do I think you're going to teach me handlebars and this app's process in a continuing discussion. Obviously a much more detailed and better guide and possibly more "robust" sort of example would be very helpful. That said, I am also on a timeline to show something very soon, even if its basic. So @blakeembrey how do we proceed perhaps as a forum topic with learning to use this module for others to have for reference.. maybe we can chat out some of this and from that conclude on how best to present a tutorial on the subject for others while we build a Java SDK generator? I do appreciate your help.

[sichvoge]

@justjacksonn - good idea. we definitely need some more clarification around this. Teach us @blakeembrey :D I think most of it - I got already. Maybe we could organise a webcast and after that we could create a blog article explaining how you can create a template with a simple example.

[blakeembrey]

@justjacksonn Sorry, just woke up 😄 With npm you can use npm link to link something globally once, you shouldn't need to do it every time.

Edit: As for the Handlebars context object, we should definitely get some definition in. The object looks something like:

{
  id: string,
  title: string,
  version: (string|number),
  baseUri: string,
  security: object,
  resources: object,
  baseUriParameters: object
}

addResources and allMethods are the resources nested structure flattened into a single array since working with recursion in a template is difficult. Every resource and method should have a unique id which is formatted using the format you specify in the configuration. They also have a key which is the language-usable display name. The resources have a reference to the current resource methods and children (for knowing what/where to chain). I'll try to get a better schema out ASAP.

[sichvoge]

QQ: How do you get all parameter to a specific resource? (including query, uri, etc.) Let's assume we've got the following RAML:

#%RAML 0.8
title: NHA
version: v1.0

/sample:
  post:
    description: Send a digital request for a sample to be tested
    queryParameters:
      labReference:
        displayName: Lab Reference
        type: string
        description: The Lab that the sample came from
        required: true
        example: LAB123
    body:
      application/json:
        example: |
          {
            "test": "sample"
          }
    responses:
      200:
        body:
          application/json:
            example: |
              {
                 "data": {
                   "sample_id": "samp1234",
                   "customer_id": "ABC123",
                   "digitalRequestReceived": "true",
                   "datetime": 1341533193,
                 },
                 "status": "in progress"
              }
  /{test}/{test2}:
    get:
  /{sampleID}:
    /status:  
      get:
        description: Find out if a physical sample has been received at the lab
        responses:
          200:
            body:
              application/json:
                example: |
                  {
                    "status": 200
                  } 
    /results:
      get:
        description: Get test results for a customer
        responses:
          200:
            body:
              application/json:
                example: |
                  {
                    "data": {
                      "sample_id": "samp1234",
                      "customer_id": "ABC123",
                      "test_result": "pass",
                      "test_result_reason": "Clean result",
                      "datetime": 1341533193,
                     },
                     "success": "completed"
                  }

Assuming that one operation is getStatus we would need the sampleId as a parameter for the operation. How do we get that in a scenario where you build your operation name based on void <method><resourceName>(<params>).

I could do the following:

{{#each allResources}}
{{#each methods}}
void {{key}}{{../key}}(???)
{{/each}}
{{/each}}

But know I want to replace the ??? with the params where one of it is the URI parameter. So far I wasn't really successful with the current way of how the client generator structures the handlebars objects. Any tips?

[blakeembrey]

@sichvoge This snippet might be useful to you: https://github.com/mulesoft/raml-client-generator/blob/master/languages/javascript/partials/resources.js.hbs#L26-L58

Edit: For what you are trying to do, you could make a join helper {{join (pluck uriParameters 'displayName') ', '}} or something similar.

[justjacksonn]

Interesting.. so if I understand the project, it could be used to template
output to any language? It's just a matter of knowing Handlebars templates
well enough and the target language? On that note, how do we start getting
.NET, python, ruby and other language fans to contribute to this to add new
templates? Do you guys have something internal to build these out (sooner
than later)... or is this going to be a pure community driven exercise? I
am fine with helping build the java SDK, but I don't know the other
languages well enough to build those as well. I could probably figure out
some of the python stuff. Ruby and .NET I've not worked with at this time.
What other languages? Go seems pretty popular these days. I assume your
javascript one is a pure nodejs sdk implementation?

On Wed, Feb 4, 2015 at 9:07 PM, Blake Embrey notifications@github.com
wrote:

@sichvoge https://github.com/sichvoge This snippet might be useful to
you:
https://github.com/mulesoft/raml-client-generator/blob/master/languages/javascript/partials/resources.js.hbs#L26-L58

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[blakeembrey]

@justjacksonn Definitely. There's a small discussion happening around the method name generation, but it's easy to generate for any language. At the moment this is community driven, but I'm stretched pretty thin at times 😄 If you're looking at building the Java SDK, shoot me an email (on my profile) and we can connect to chat about it. The JavaScript generation works on both node and the browser (which was a requirement for us).

[justjacksonn]

Ok. Can you guys recommend any good tutorials on the net or otherwise for
Handlebars? I see some books over a year or two ago on amazon, not sure if
those are up to date. Speaking of, I see a 2.0.0 release of Handlebars is
available. What's the chances this codegen module can be updated (if it's
not already) to take advantage of any additions/improvements... and for
that matter.. is there anything 2.0.0 offers that would make it the version
to learn now? I am afraid a book of 2+ years ago may be outdated given how
fast things change.

On Thu, Feb 5, 2015 at 8:46 AM, Blake Embrey notifications@github.com
wrote:

@justjacksonn https://github.com/justjacksonn Definitely. There's a
small discussion happening around the method name generation, but it's easy
to generate for any language. At the moment this is community driven, but
I'm stretched pretty thin at times [image: 😄] If you're looking at
building the Java SDK, shoot me an email (on my profile) and we can connect
to chat about it. The JavaScript generation works on both node and the
browser (which was a requirement for us).

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[sichvoge]

@justjacksonn FYI - #9 - That doesn't answer your question, but this is something Blake can help you with ;)

[justjacksonn]

Thanks.. just read the thread..this is a great start.. and a difficult one (the method naming thing) to solve. The more parts to the path, the more difficult it's going to be.

[sichvoge]

Exactly - that's why we started this conversation to gather ideas around it :) FYI - I am also creating the categories for the forum. I just need final approval for the categories :)

Basically I want to create the following:

Code Generator
 General
 Client
 Server

Discussions about specific languages or issues I would leave here on github. The forum should be there to generally discuss approaches or wish for a language.

What do you think?

[blakeembrey]

@justjacksonn It's already using 2.0 😄 It should be simple enough to look at the Handlebars homepage and then hack on the JS or another one and see how the output changes. For the most part, you only need to know {{value}}, {{helper arg}}, {{#helper}}block{{/helper}} and {{helper (nestedHelper arg)}}. Most of the logic is in creating usable code and a seamless DSL in your language.

Edit: Oh, an {{{unescaped}}}.

[justjacksonn]

Looks good Christian.

On Thu, Feb 5, 2015 at 9:35 AM, Blake Embrey notifications@github.com
wrote:

@justjacksonn https://github.com/justjacksonn It's already using 2.0 [image:
😄] It should be simple enough to look at the Handlebars homepage
and then hack on the JS or another one and see how the output changes. For
the most part, you only need to know {{value}}, {{#helper arg}},
{{#helper}}block{{/helper}} and {{#helper (nested arg)}}.

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[sichvoge]

Done!

[justjacksonn]

A question I still need answered if possible... in my templates folder, I
have /src/package-name/file.java.hbs. How do I get that to appear in the
output path in the same directory structure? Can I use sub-folders in the
templates folder to separate/organize files? Or is it just a flat
structure? I assume there is a solution to using paths here.

On Thu, Feb 5, 2015 at 11:33 AM, Christian Vogel notifications@github.com
wrote:

Done!

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[blakeembrey]

@justjacksonn Right now it's support as nested objects. For example:

templates: {
  src: {
    'file.java': require('../templates/src/file.java.hbs')
  }
}

[ponprathip]

I am looking for Java language support .
Looks like there are already some work done for this support .
Can you share the templates for java language support ?

[petrochenko-pavel-a]

As for me It actually really depends from what they need pretty much. RAML for JAX-RS does good job in converting schemas to DTOs and is capable to generate stub of server side code and clients (not actually well tested) which conforms javax.rs spec. So when we are speaking about generating something simple but very custom it is simpler to stick to templates, if they need a normal stack with few or no customisations use extension interfaces, if you need to build something complex and custom start from javax-rs and customise the code itself. But it is harder to customise then templates.

Regards,
Pavel

[justjacksonn]

Hi all,

I just rigged up raml to JAX-RS for a round-trip cycle and it works quite
well. Not too sure what else needs to be done, but I can assure you that I
have built json schema + raml, ran it through, implemented the generated
interfaces, deployed and made API requests and it all works great. Very
impressed. I'd like to see it wrapped up in a 1.0 version at some point,
maybe when the 1.0 spec comes out.

As for client side, there isn't much out there in the way of Java yet, but
I can assure you there is a suite of SDK generators coming in the near
future with RAML support. I hope sooner than later.

You didn't specify if you were looking for server side or client side
support?

Besides raml to pure SDKs, Postman (the tool) can import RAML (work in
progress still) giving you a quick way to turn RAML into API requests. I
don't know where it stands right now, but I know they plan on full 1.0
support with full JSON Schema support for request/response entities as
well, so in the not too distant future you should be able to go that route.
As well, Postman is adding code generators for their own internal format,
so it's possible at some point you could do RAML -> Postman -> SDK/Client
code.

If you are needing something right this minute, you could write your own
pretty fast. I would use raml-generator project instead of
raml-client-generator, and stick to something like HttpClient 4.x generated
code. You can then use jsonschema2pojo (with raml to jax-rs uses) to turn
JSON Schema to java pojos which is very good, and write a Handlebars.js
template to turn RAML into Java resource files. I actually started on doing
just this and had a very very simple SDK generation that did make API
calls. But I am no longer working on that at this time.

On Wed, Jun 3, 2015 at 3:05 PM, petrochenko-pavel-a <
notifications@github.com> wrote:

As for me It actually really depends from what they need pretty much. RAML
for JAX-RS does good job in converting schemas to DTOs and is capable to
generate stub of server side code and clients (not actually well tested)
which conforms javax.rs spec. So when we are speaking about generating
something simple but very custom it is simpler to stick to templates, if
they need a normal stack with few or no customisations use extension
interfaces, if you need to build something complex and custom start from
javax-rs and customise the code itself. But it is harder to customise then
templates.

Regards,
Pavel

—
Reply to this email directly or view it on GitHub
#8 (comment)
.

[sichvoge]

You can also use https://apimatic.io/ to generate code based on a RAML file as long as you need something for now.