Pytorch lightning 1.4 issues

[adamgayoso]

1. Data splitting

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/core/datamodule.py:424: LightningDeprecationWarning: DataModule.setup has already been called, so it will not be called again. In v1.6 this behavior will change to always call DataModule.setup.
  f"DataModule.{name} has already been called, so it will not be called again. "

It's getting called once in our train runner so that we have access to the indices of train/test/val, but then it gets called again during PyTorch Lightning fit

2. Pyro training plan -- it's unhappy that loss is a float and not a torch Tensor.

3. Adversarial training plan has this new issue

Lightning-AI/pytorch-lightning#8603

4. Now we need to set log_every_n_steps

/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/data_loading.py:323: UserWarning: The number of training samples (36) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
  f"The number of training samples ({self.num_training_batches}) is smaller than the logging interval"

[adamgayoso]

Let's also check that #1112 has been fixed. It could also be related to the log_every_n_steps?