A simple Python script that uses the Google Custom Search API to fetch search results and export them into a CSV file.
- Python 3.8+
- A Google API key
- A Google Custom Search Engine (CX) ID
- Install dependencies:
pip install requests pip install beautifulsoup4 pip install python-csv pip install argparse
-
Get a Google API key from Google Cloud Console
-
Create a Custom Search Engine (CX) at Google CSE
-
Run the script with your API key to create setting.json:
python main.py -sq [SEARCH_QUERY] --add_api_key [YOUR_API_KEY]
Search with query:
python scraper.py -sq "github"Fetch multiple pages (10 results per page):
python scraper.py -sq "github" --pages 3- Results are saved in output.csv in the following columns:
# , Title , Link
[!NOTE]
Free quota: 100 queries/day (10 results per query).
Ifsetting.jsonis missing or doesn’t have an API key, use--add_api_key.