| N-to-N code generation [message #1859358] |
Fri, 02 June 2023 09:42  |
 Simon Cockx
Messages: 69
Registered: October 2021 |
Member |
|
|
The `IGenerator2` seems to assume 1-to-N code generation, that is, for one DSL file there may be any number of generated output files. This makes sense for supporting incremental builds.
However, we have use cases where this relation differs, and I'm not sure about the best approach to handle these.
=== Use case 1: generate a package structure per namespace. ===
Our DSL files start with the declaration of a namespace, much like in Java, e.g. `namespace com.foo.bar`. Multiple DSL files might have the same namespace.
One of the languages we generate to is Python, which uses a convention of having one `__init__.py` file per namespace. How does this fit into Xtext's generation infrastructure?
Example: suppose we have three DSL files.
- `a.dsl`, with namespace `com.foo.bar`
- `b.dsl`, with namespace `com.foo.bar`
- `c.dsl` with namespace `com.foo.util`
I would like to generate the following layout:
com
- foo
- bar
- __init__.py
- a.py
- b.py
- util
- __init__.py
- c.py
This is definitely our most important use case, for which we currently don't have a workaround.
=== Use case 2: generating project files. ===
It would be useful to have a way of generating project files such as a file for dependency management.
To take the example of Python again (but this is relevant for almost all languages), we would like to automatically generate a `pyproject.toml` file or a `requirements.txt` file for defining a list of dependencies, and potential metadata such as the project name and version.
Questions:
- How does this fit into Xtext's generation infrastructure? Are there hooks that fire once before all generation takes place and after all generation has finished?
- Is it possible to access context of the project such as the name and the version during generation? (e.g., the name/version of the Maven project when generating using the `xtext-maven-plugin`, or the name of the `ProjectDescription` when generating using the Xtext language server)
Right now we have a workaround of providing these files statically per project/target language.
=== Use case 3: generating patches. ===
Our language is constantly evolving, and being time pressured we find it sometimes convenient to temporarily generate patches for our dependencies. Right now we do so in the `afterGenerate` hook, which is called for every input DSL file, meaning we regenerate these patches many many times although a single generation would suffice.
Context
The use cases we want to support is generation via Maven (e.g., the `xtext-maven-plugin`) and generation via Xtext's language server.
What I've tried so far is extending the `IGenerator2` interface and providing a `beforeAllGenerate` and `afterAllGenerate` hook, which have access to the entire resource set. I've experimented with an extension of the `xtext-maven-plugin` which will actually use these calls, but which will will probably not fit into incremental builds. Additionally, I've noticed that the `IGenerator2` instance is not reused by the builder, but a new instance is created for each DSL file, meaning it is not possible to maintain state in the code generator.
These are difficult problems that might be solvable, but I just wonder:
- is there out-of-the-box Xtext support for this?
- are there alternative approaches for these use cases?
- have people encountered similar problems? What were their solutions?
I did see this StackOverflow post, but there wasn't really a solution: https://stackoverflow.com/questions/43904786/xtext-generation-from-multiple-model-files
[Updated on: Fri, 09 June 2023 12:45] Report message to a moderator |
|
|
|
| Re: N-to-N code generation [message #1859362 is a reply to message #1859359] |
Fri, 02 June 2023 10:20  |
Simon Cockx
Messages: 69
Registered: October 2021 |
Member |
|
|
Yeah, I guess that could work for implementing a `beforeAllGenerate` hook. Is it possible to disable incremental deletes in the language server as well?
EDIT: I guess this EAdaptor marker would have to be cleared between changes as well (for the language server use case). Not sure how that would work.
[Updated on: Fri, 02 June 2023 10:24] Report message to a moderator |
|
|
|
| Re: N-to-N code generation [message #1859365 is a reply to message #1859363] |
Fri, 02 June 2023 12:23 |
Simon Cockx
Messages: 69
Registered: October 2021 |
Member |
|
|
Quote:org.eclipse.xtext.ide.server.ProjectManager.newBuildRequest(List<URI>, List<URI>, List<Delta>, CancelIndicator) seems to create new
Interesting, I did not realize that. That would indeed make the EAdaptor marker option feasible.
Quote:org.eclipse.xtext.generator.OutputConfiguration.isCleanUpDerivedResources()
Having glanced at the code, setting this option to `false` would disable incremental deletes, is that right?
I think I'll use that workaround for now. In the future, I might look into proper support for a `IGenerator2::forAllGenerate` hook in Xtext's incremental builders (`IncrementalBuilder.InternalStatefulIncrementalBuilder` for Xtext's language server; `StandaloneBuilder` for the `xtext-maven-plugin`). I don't see this feature conflicting with incremental builds in a fundamental way. Conceptually, I could see it working like this:
- add a method `forAllGenerate(ResourceSet input, IFileSystemAccess2 fsa, IGeneratorContext context)` to the `IGenerator2` interface.
- in the incremental builders, associate files generated with this method with a synthetic URI `synthetic:/forAllGenerate` representing the whole resource set.
- if any resource is modified, added or deleted, consider the resource set to be modified. I.e., clean all generated files associated to `synthetic:/forAllGenerate` and rerun `forAllGenerate`.
- the resource set is considered "added" once initially. This triggers the first run of `forAllGenerate`.
- the resource set is never considered "deleted".
@ChristianDietrich, do you think Xtext would be open to this kind of contribution?
[Updated on: Fri, 02 June 2023 12:24] Report message to a moderator |
|
|
|
|
|
|
|
|
Powered by
FUDForum. Page generated in 0.04971 seconds