[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datasource caching #2214

Open
BenjyWiener opened this issue Aug 21, 2024 · 6 comments
Open

Datasource caching #2214

BenjyWiener opened this issue Aug 21, 2024 · 6 comments
Labels

Comments

@BenjyWiener
Copy link

I have a large JSON datasource that's used in a nested template, which is executed many times in a loop.

I currently load the datasource at the top level and use dict to pass the value along with the actual context to the nested template.

I would like to be able to load the datasource from the nested template where it's used, but the runtime impact of reading and deserializing the JSON file each iteration is huge (50-100x slower in my case). An option to cache datasource reads would solve this.

Possible mechanisms:

  1. cache option in config file and/or as query parameter

  2. A new function that caches datasource reads (datasourceCached)

  3. A more general set of functions that allow writing to and reading from some global "state".

@hairyhenderson
Copy link
Owner

An option to cache datasource reads

datasource reads are already cached, though I don't think it's documented...

If you use the --context flag to load datasources, they're read once at startup and always held in memory. If you use --datasource then reference with the datasource/ds function, the data is cached at first read.

@hairyhenderson hairyhenderson closed this as not planned Won't fix, can't repro, duplicate, stale Sep 17, 2024
@BenjyWiener
Copy link
Author
BenjyWiener commented Sep 18, 2024

Thanks for the reply.

Looking at the source code, I see that the string content of the datasource is cached, but not the parsed result.

I'm working with a large JSON file, so the time it takes to parse each time results in a ~100x slowdown compared to using context.

@hairyhenderson
Copy link
Owner

@BenjyWiener ah. is using context an option? Usually I recommend that anyway, given the simpler syntax.

Reopening since there may be a possible feature request here...

@BenjyWiener
Copy link
Author

I'm currently using context, but I use a lot of nested/recursive templates, which requires a lot of {{ template "..." (dict "current_item" . "top_level" $.top_level) }}.

@hairyhenderson
Copy link
Owner

I'm currently using context

ok, so then the data and parsed objects are already cached.

but I use a lot of nested/recursive templates,

I'm not sure I follow why this affects caching - could you elaborate? A concrete example would help...

Copy link

This issue is stale because it has been open for 60 days with no
activity. If it is no longer relevant or necessary, please close it.
Given no action, it will be closed in 14 days.

If it's still relevant, one of the following will remove the stale
marking:

  • A maintainer can add this issue to a milestone to indicate that
    it's been accepted and will be worked on
  • A maintainer can remove the stale label
  • Anyone can post an update or other comment

@github-actions github-actions bot added the Stale label Nov 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants