mobilevit usage.

[abaqusstudent]

Hi there, I am new to timm (thanks to everyone for this awesome lib), and I am trying to use mobilevit as stated here: https://github.com/rwightman/pytorch-image-models/releases/tag/v0.1-mvit-weights

However, after installing the master branch, I am not able to find this model. So, I tried:

>>> timm.list_models('*mobile*', pretrained=True)
['mobilenetv2_050', 'mobilenetv2_100', 'mobilenetv2_110d', 'mobilenetv2_120d', 'mobilenetv2_140', 'mobilenetv3_large_100', 'mobilenetv3_large_100_miil', 'mobilenetv3_large_100_miil_in21k', 'mobilenetv3_rw', 'mobilenetv3_small_050', 'mobilenetv3_small_075', 'mobilenetv3_small_100', 'tf_mobilenetv3_large_075', 'tf_mobilenetv3_large_100', 'tf_mobilenetv3_large_minimal_100', 'tf_mobilenetv3_small_075', 'tf_mobilenetv3_small_100', 'tf_mobilenetv3_small_minimal_100']

>>> timm.list_models('*vit*', pretrained=True)
['convit_base', 'convit_small', 'convit_tiny', 'crossvit_9_240', 'crossvit_9_dagger_240', 'crossvit_15_240', 'crossvit_15_dagger_240', 'crossvit_15_dagger_408', 'crossvit_18_240', 'crossvit_18_dagger_240', 'crossvit_18_dagger_408', 'crossvit_base_240', 'crossvit_small_240', 'crossvit_tiny_240', 'levit_128', 'levit_128s', 'levit_192', 'levit_256', 'levit_384', 'vit_base_patch8_224', 'vit_base_patch8_224_dino', 'vit_base_patch8_224_in21k', 'vit_base_patch16_224', 'vit_base_patch16_224_dino', 'vit_base_patch16_224_in21k', 'vit_base_patch16_224_miil', 'vit_base_patch16_224_miil_in21k', 'vit_base_patch16_224_sam', 'vit_base_patch16_384', 'vit_base_patch32_224', 'vit_base_patch32_224_in21k', 'vit_base_patch32_224_sam', 'vit_base_patch32_384', 'vit_base_r50_s16_224_in21k', 'vit_base_r50_s16_384', 'vit_huge_patch14_224_in21k', 'vit_large_patch16_224', 'vit_large_patch16_224_in21k', 'vit_large_patch16_384', 'vit_large_patch32_224_in21k', 'vit_large_patch32_384', 'vit_large_r50_s32_224', 'vit_large_r50_s32_224_in21k', 'vit_large_r50_s32_384', 'vit_small_patch8_224_dino', 'vit_small_patch16_224', 'vit_small_patch16_224_dino', 'vit_small_patch16_224_in21k', 'vit_small_patch16_384', 'vit_small_patch32_224', 'vit_small_patch32_224_in21k', 'vit_small_patch32_384', 'vit_small_r26_s32_224', 'vit_small_r26_s32_224_in21k', 'vit_small_r26_s32_384', 'vit_tiny_patch16_224', 'vit_tiny_patch16_224_in21k', 'vit_tiny_patch16_384', 'vit_tiny_r_s16_p8_224', 'vit_tiny_r_s16_p8_224_in21k', 'vit_tiny_r_s16_p8_384']

I cant see mobilevit in them :(

Is there any template code I can use to load the arch and weights? or is it named something else?

Thank you.

Rachel

[rwightman]

@abaqusstudent it's on the norm_norm_norm (same train scripts as master) and bits_and_tpu (new train scripts, XLA support) branches right now, not merged to master as there are quite a few significnant changes lumped with it that need some further testing.

[abaqusstudent]

Thank you @rwightman that solves it. I have two other follow up quetsions and I hope it is ok to ask within the same thread:

[1] when I do model tracing on the mobilevit_xxs, I get:

model = timm.create_model('mobilevit_xxs', pretrained=True, num_classes=0, exportable=True)
traced_model = torch.jit.trace(model, torch.rand(8, 3, 224, 224))   # why batchsize 8?
TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  new_h, new_w = math.ceil(H / patch_h) * patch_h, math.ceil(W / patch_w) * patch_w
TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!

I presume I can safely ignore this warning and that the mobilevit models are retrainable and traceable?

Finally, when I look at your suggested tutorial here:
https://towardsdatascience.com/getting-started-with-pytorch-image-models-timm-a-practitioners-guide-4e77b4bf9055#c193

I see that the author uses a batch size of 8 for tracing.Is this something specific to timm?
Normally I would pass one image and trace. I could not find an explanation for this.

Thank you for all your help and the beautiful package.

[rwightman]

@abaqusstudent tracing fixes the model for the given tensors used as the input, the model will work but as it says those values are locked for the input size you use during trace.

Also you want to trace with the batch size you're using if you're using the traced model in pytorch. If you're tracing for onnx export, there are exceptions that allow keeping specified dims dynamic (like batch size or spatial dim), but changing the spatial dim will break due to those width height calculations being turned into constants....

scripting is generally more flexible than tracing, really depends on your use case

[abaqusstudent]

@rwightman thank you again for your explanation. It makes sense. Your answer leads me to one other generic question for me:

What if I trace some pytorch model with some size, say 224x224 and and the trace is successful with no warnings. Does this mean that this traced model can now be used with any input?

The other question is have is, by using:

model = timm.create_model('mobilevit_xxs', pretrained=True, num_classes=0, exportable=True)

then saving the model state dict, loading the state dict and tracing (the standard approach) - is this legit to do with timm models?

Thank you for all your explanations.