Missing commits info

[dyangUCI]

We found many cases of missing commits info for projects (about 10% for C++ and Python projects, 5% for Java projects). That is, we can actually see commits for a project on Github, but GHTorrent doesn't have the info included.
For example,
project id=694420, we can see the commits here at https://github.com/zdevito/vector_benchmarks/commits/master, but neither "commits" table nor "project_commits" table contain info about this project;
project id=712376, https://github.com/schamp/Schazamp/commits/master
project id=842516, https://github.com/daviddoria/SmartNearestNeighbors/commits/master

[MatthewThirkettle]

I also find that there are commits missing from the GHtorrent database. For example the commits from users joerixaop and cbuben are missing from project id=1:

https://github.com/tosch/ruote-kit/graphs/contributors

[MatthewThirkettle]

I looked at my particular case of a missing commit for user cbuben (committer_id=232212). The user committed to project_id=1 on 2011-01-25 and the commit is missing from the GHtorrent database. The user also committed to project_id=1376474 on the same day (this commit is recorded in the GHtorrent database). Both of these projects are forks from project_id=2, and the forks occurred after 2011-01-25. Consequently this commit appears on all forks of project_id=2 and has the same commit id:

b144943753a45ccbccfa8b0072848b3273458620,

GHtorrent only saved one instance of the commit. This may explain why many projects are missing commits in the GHtorrent commit tables. It is strange that the commit is recorded on a forked project rather than the master project where the commit originally occurred. Were commits with duplicate commit IDs randomly dropped?