Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAMEM dataset evaluation fails due to cv_split 5 in eval #239

Closed
Div12345 opened this issue Oct 5, 2021 · 3 comments · Fixed by #256
Closed

MAMEM dataset evaluation fails due to cv_split 5 in eval #239

Div12345 opened this issue Oct 5, 2021 · 3 comments · Fixed by #256
Labels

Comments

@Div12345
Copy link
Collaborator

Div12345 commented Oct 5, 2021

Within Session Evaluation fails while running for MAMEM dataset with the following error -

/usr/local/lib/python3.7/dist-packages/sklearn/model_selection/_split.py:672: UserWarning: The least populated class in y has only 4 members, which is less than n_splits=5.
% (min_groups, self.n_splits)), UserWarning)

The CV has been hard coded as 5 in the Evaluation -

cv = StratifiedKFold(5, shuffle=True, random_state=self.random_state)

@Div12345 Div12345 added the bug label Oct 6, 2021
@Div12345
Copy link
Collaborator Author

As per discussion with @sylvchev, a possible cause of this might be from the Rest condition being included in the data. Need to check the data in further detail.

@sylvchev
Copy link
Member

It seems that this is not the reason: there is a resting period but is it is not extracted in the actual dataset class. The only classes available are the flash frequencies : 6.66, 7.5, 8.57, 10 and 12 Hz.
For MAMEM1, there are between 12 and 20 trials per class, for MAMEM2 and MAMEM3, between 20 to 30 trials per class. I'm re-checking the trials ID, there are different (and conflicting) indications in the documentation on the website, in the data description (PDF) and in the matlab codebase.

@sylvchev
Copy link
Member

After some investigation, the problem is an incorrect description: MAMEM sessions are instead 'runs'. The errors come from the low number of stimulus in each runs. PR #256 correct this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants