Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add backup for metrics.csv #345

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

AntonioMirarchi
Copy link
Contributor

We currently use CSVLogger from torch.lightning to write metrics to a CSV file in the log directory. However, when using load_model (i.e., loading from a checkpoint), or more in general to append data to an existing CSV, we overwrite the current metrics.csv, losing the original data.
This PR introduces a simple function that checks if the metrics file already exists and renames it using date and time information, preserving previous data.
Ideally, we would prefer to append to the existing file, but AFAIK this is not currently supported by CSVLogger.

@AntonioMirarchi
Copy link
Contributor Author

It seems there is a problem with csvlogger.experiment.metrics_file_path when multiple GPUs are used for training. So I would just keep it simple and retrieve the path name as opj(log_dir, metrics.csv)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant