a Twitter thread from @tiagopeixoto

1.

New blog post: "Descriptive vs. inferential community detection"

A mini-thread for those too lazy to click the link! 1/7

(Based on recent pre-print: https://arxiv.org/abs/2112.00183)

https://skewed.de/tiago/blog/descriptive-inferential

2.

There are essentially two objectives when doing community detection (or any data analysis): To "describe" or to "infer".

Description involves finding patterns, inference involves finding explanations. 2/7

3.

Most community detection methods out there (modularity, infomap, walktrap, etc.) are descriptive. The communities they find are there, but they cannot explain. In a very concrete sense, they *overfit* your data, confusing actual structure with random fluctuations. 3/7

4.

Inferential approaches do not do this. They find the most parsimonious explanation for the data — according to Occam's razor — and do not confuse randomness with structure. 4/7

https://arxiv.org/abs/1705.10225

5.

So, when should we "infer" or "describe"? Here's a good litmus test:

Q: "Would the usefulness of our conclusions change if we learn, after obtaining the communities, that the network being analyzed is completely random?"

5/7

6.

If the answer is "yes", then an inferential approach is needed.

If the answer is "no", then an inferential approach is not required. 6/7

7.

It's arguable that in most scientific contexts the answer would be "yes". I this case we need inferential approaches, and descriptive ones are just not up to the task!

There is lots more to say about this. Read it here: https://arxiv.org/abs/2112.00183 7/7

Read this thread on Twitter