confidence intervals of aggregated resampling measures

[mb706]

(inspired by this question on stackoverflow)

There should probably be a way in which to represent confidence intervals / standard errors for aggregates. On the one hand there are packages that provide measures with CIs, such as cvAUC, on the other there are probably good things that we can do for this in a measure-independent way; for example indicate which folds are statistically independent from each other so that holdout resampling and repeated CV calculate default standard errors but CV does not. (Maybe there is a way to use corrective factors for CV?)

I don't know that much about uncertainty estimation for CV results, but I always thought this was missing.