Add list of proxies to lines 12, 13, etc.This system will rotate randomly among however many proxies you put in Google seems to be pretty good at detecting proxy usage so only use good proxies. Just in case, there is a full list of Googlebot user agents here: You should be able to change it to whichever user agent you are looking for. Right now it is set at the standard Chrome agent. On line 32 of google-rank-checker.py you will see an option called 'user-agent=.'. As long as you leave the previously generated HTML files in the html folder, your keyword_rankings.csv file will still contain all the keywords.

Remove all the keywords that have already been run from keywords.csvĭoing this should restart the process from where you left off.

Figure out the last keyword that was run by checking either the terminal output or look at the html folder and sort by Date Modified.

I got stopped! Do I need to rerun the script for all keywords?.

Output a CSV of the URL ranks in the html folder.

Use a HTML parser (BeautifulSoup + lxml) to search through all the HTML in the folder 'html' and find the exact info of hrefs.

Download all the source code from the first page and put it into a folder called html.

is all rendering normally because it acts like a real browser)

For each keyword, use Selenium to open a new incognito Chrome browser and google search the keyword (HTML, CSS, JS, etc.

Import the CSV of keywords to check the rank for.

It isn't perfect, but it will give you a good enough measure of approximately what is ranking on the first page.įor the curious at a high level, the script works in 5 basic parts:

This could be limited to personalized and/or local search since it is just opening a normal Google browser.

Again, I only needed the URLs for this specific project

This only gives you the URLs of the first page of search results.

I might consider writing something that goes to the other pages, but it isn't built right now. This was built for a specific project and I just needed to look at the first page.

This only looks at the 1st page of search results.

I'm sure you can build out a large scale scraping operation with hundreds of AWS EC2 instances and thousands of proxies, but if you know how to do that, you probably don't need this script.

This is only for small scale rank checking, it goes at around ~6 terms a minute to be a good citizen of Google.

Go to the HTML folder and open keywords_rankings.csv.

Search engine rank checker