on the textual inversion training colab, a way to output .bin file before the training is fully completed

[ciaobnsr]

since the training usually takes few hours, and for different datasets, objects or styles, you can adapt with different training parameters, is there a way to output the .bin -let's say- every 1000 steps or so?

that would be very useful to check if the training is going in a good direction and maybe you can economize your gpu/usage if you are happy with the results.
it's very frustrating when you are 1000steps to complete a 10000steps training and runtime disconnects ending with hours of hours of training gone without a checkpoint.

also the ability to somehow save and resume the training would be a dream.

[patrickvonplaten]

cc @patil-suraj

[isamu-isozaki]

HI! I implemented this for my project and I'm happy to make a pr here if you want! It can also log images at fixed intervals

[isamu-isozaki]

I think I did it here #645

[patil-suraj]

Thanks a lot for the PR @isamu-isozaki, taking a look now!

[isamu-isozaki]

@patil-suraj Np! Glad to help.

[patrickvonplaten]

@patil-suraj - re-pinging you here

[isamu-isozaki]

@patrickvonplaten @patil-suraj hullo. I made a pr and I think it already got merged.

[patrickvonplaten]

Super, closing this one then!