Download Google Scholar Citation Papers

These scripts aim to download all the citation papers of one paper via selenium and Sci-Hub

Tutorial on the macOS

NOTICE1: Due to the reCAPTCHA (Robot Check <- If you are a human) of Google Scholar backend, to avoid the CAPTCHA, we have to log in our Google Account first and then crawl the citation papers' links and titles.

NOTICE2: You may still need to check the reCAPTCHA for the first time!

Un-comment this line at first to check the reCAPTCHA, and then comment it.

Open Terminal and cd to the Google Chrome directory

$ cd ../../Applications/Google\ Chrome.app/Contents/MacOS

Open Google Chrome and use a remote port, and don't close this Terminal

$ ./Google\ Chrome --remote-debugging-port=9222 --user-data-dir="~/ChromeProfile"

Log in your own Google Account

Select a paper where you want to download citations

Copy the link of all the citations papers

Edit Codes -> crawel_citations.py

  # The path of the Chrome Driver
  driver_path = '/Users/shuyuej/.wdm/drivers/chromedriver/mac64/96.0.4664.45/chromedriver'

  # Citation Path
  citation_url_start = 'https://scholar.google.com/scholar?start='
  citation_url_end = '&hl=en&as_sdt=2005&sciodt=2006&cites=17910156571874886383&scipsc='
  num_citation = 208

1. Change the Chrome Driver path to your own path

Un-comment this line to download the Chrome Driver

2. Change the citation URL
3. Change the number of citations your paper currently earned

You can see that the papers are crawled.

NOTICE3: Although some paper links are overlapped, it's okay as all the citation papers are downloaded, if they are available at Sci-Hub.

Print the paper title with its webpage link -> citation_title_with_link.py

Change the Google Account if u encounter this:

References

Google Scholar Public API