[go: up one dir, main page]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: Record batch time #278

Open
wants to merge 22 commits into
base: main
Choose a base branch
from
Open

feature: Record batch time #278

wants to merge 22 commits into from

Conversation

isamu-isozaki
Copy link
Contributor
@isamu-isozaki isamu-isozaki commented Feb 10, 2024 •
edited
Loading

This piece of code helped me identify how much time is spent in the data processing stage, the forward pass, and backward pass so that I know where to optimize.
The main steps blocking this pr will be

  • Figure out how to log with trainer since I think the def log function is not present currently because of refactoring

@isamu-isozaki isamu-isozaki marked this pull request as draft February 10, 2024 20:25
piercus added a commit to piercus/refiners that referenced this pull request Feb 13, 2024
@piercus
monitor time, inspired from finegrain-ai#278
a32b023
@isamu-isozaki isamu-isozaki marked this pull request as ready for review February 16, 2024 17:20
@isamu-isozaki
Copy link
Contributor Author

I'm thinking if this can become a callback

limiteinductive
limiteinductive requested changes Feb 20, 2024
View reviewed changes
Copy link
Member
@limiteinductive limiteinductive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks this is gonna be useful! Maybe you can take inspiration on how the TrainingClock Callback is implemented? Let's try to pollute as little as possible the core of the Trainer

@isamu-isozaki isamu-isozaki marked this pull request as draft February 20, 2024 18:23
@isamu-isozaki isamu-isozaki marked this pull request as ready for review February 21, 2024 03:33
@isamu-isozaki
Copy link
Contributor Author

@limiteinductive do you think I should make a timer config or merge timer with clock?

@limiteinductive
Copy link
Member

@limiteinductive do you think I should make a timer config or merge timer with clock?

I think it makes sense to merge with Clock and extend its config.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@limiteinductive limiteinductive limiteinductive requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@isamu-isozaki @limiteinductive