diff --git a/drive/gdrive_api_tutorial.ipynb b/drive/gdrive_api_tutorial.ipynb new file mode 100644 index 00000000..7ff324b2 --- /dev/null +++ b/drive/gdrive_api_tutorial.ipynb @@ -0,0 +1,770 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Google Drive Python API Tutorial" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The purpose of this notebook is to demonstrate the different capabilities of the Google Drive Python API. Note that the reader is encouraged to first check if [PyDrive](https://pythonhosted.org/PyDrive/) can do the job. PyDrive is mainly used for simple requests, such as creating folders and files. However, in the case that a more complex task is required, then the Google Drive Python API is more appropriate. This guide may be complemented with the official [documentation](http://googleapis.github.io/google-api-python-client/docs/dyn/drive_v3.html).\n", + "\n", + "A very helpful quickstart guide can be found [here](https://www.thepythoncode.com/article/using-google-drive--api-in-python).\n", + "\n", + "Note that the Google Drive API is a REST API. This means that its architecture is based on a known blueprint. This blueprint is detailed [here](https://restfulapi.net/).\n", + "\n", + "To get started, complete step 1 and step 2 on this [page](https://developers.google.com/drive/api/v3/quickstart/python) to download a `credentials.json` file into your working directory and to install the Google Python client library.\n", + "\n", + "Once completed, the next step is gain access to your Google Drive using the API. First, define what kind of read and write permissions you will have:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "SCOPES = ['https://www.googleapis.com/auth/drive.file']" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In this case, you will have the ability to create and upload files using the Google Drive API. However, other types of permissions are available. See [here](https://developers.google.com/drive/api/v3/about-auth) for details.\n", + "\n", + "Next, check if the user's access and refresh tokens already exist:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import pickle\n", + "import os\n", + "\n", + "if os.path.exists('token.pickle'):\n", + " with open('token.pickle','rb') as f:\n", + " creds = pickle.load(f)\n", + "else:\n", + " creds = None" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "If the user's access and refresh tokens don't exist, or they are no longer valid, then create them and save them. This will **not** work if the `credentials.json` file is not in your working directory." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from google_auth_oauthlib.flow import InstalledAppFlow\n", + "from google.auth.transport.requests import Request\n", + "\n", + "if not creds or not creds.valid:\n", + " if creds and creds.expired and creds.refresh_token:\n", + " creds.refresh(Request())\n", + " else:\n", + " flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES)\n", + " creds = flow.run_local_server(port=0)\n", + " \n", + " with open('token.pickle', 'wb') as f:\n", + " pickle.dump(creds,f)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Now check that the authentication flow worked:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "creds.client_id" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This should be the same as the string in the `\"client_id\"` field in the `credentials.json` file. Additionally, if the access and refresh tokens did not already exist, then a `token.pickle` file should now be in your working directory.\n", + "\n", + "Finally, a `drive` object is created to directly interface with your Google Drive:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from googleapiclient.discovery import build\n", + "\n", + "drive = build('drive', 'v3',credentials=creds)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `drive` object has several associated public methods, including `files`, `drives`, and `comments`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "dir(drive)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "These public methods have different purposes. Details on their purposes can be found [here](https://developers.google.com/drive/api/v3/reference).\n", + "\n", + "For the purpose of this tutorial, however, only the `files` method will be used. The associated public methods of `drive.files()` are detailed [here](https://developers.google.com/drive/api/v3/reference/files)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "dir(drive.files())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create a folder" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "In the Google Drive API, files and folders are treated almost the same way. Both require the definition of their metadata before creation. More information about the metadata for a file/folder can be found [here](https://developers.google.com/drive/api/v3/reference/files#resource-representations). This metadata is defined using a Python dictionary.\n", + "\n", + "For example, to create a folder, its metadata could be:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "folder_metadata = {\n", + " \n", + " 'name': 'TestFolder',\n", + " 'mimeType': 'application/vnd.google-apps.folder'\n", + " \n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `'mimeType'` key refers to the type of media (file, folder, text file, PDF,...) to be created. More information about this can be found [here](https://stackoverflow.com/questions/3828352/what-is-a-mime-type). A list of **some** of the MIME types that are identifiable by the Google Drive API can be found [here](https://developers.google.com/drive/api/v3/mime-types).\n", + "\n", + "Note that that the `body` parameter is available to all `drive.files()` methods that have a request body, including `create()` and `update()`. Here is an [example](https://developers.google.com/drive/api/v3/reference/files/update#request-body).\n", + "\n", + "Next, a folder is created using the `create()` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "create_folder_req = drive.files().create(body = folder_metadata,fields = 'id,name')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `body` parameter takes the metadata dictionary as input. The `fields` parameter is used to choose specific metadata to be returned in the response body of the HTTP request. More details on the format of the `fields` parameter can be found [here](https://developers.google.com/drive/api/v3/fields-parameter#formatting_rules_for_the_fields_parameter). More information about the available parameters for the `create()` method can be found [here](http://googleapis.github.io/google-api-python-client/docs/dyn/drive_v3.files.html#create).\n", + "\n", + "More precisely, since `create_folder_req` is a HTTP request:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "create_folder_req" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Then it has an associated response body. Also, since `create_folder_req` is a HTTP request, then it needs to be executed:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "folder = create_folder_req.execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The associated metadata fields that were requested in `create_folder_req` are then returned in the response body:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "folder" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Every file and folder in Google Drive has a unique `id`. This means that two files or folders can have the same `name` with a different `id`." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Upload a file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Since the `TestFolder` folder has been created, a file can be uploaded to it. First, create a sample `HelloWorld.txt` file to upload:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with open('HelloWorld.txt','w') as f:\n", + " f.write('Hello World!')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, create metadata for this file:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file_metadata = {\n", + " 'name': 'HelloWorld.txt',\n", + " 'parents': [folder['id']]\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `'parents'` field indicates the folders that should contain this file. Note that in Google Drive, several folders can contain the same file. In this case, the `id` of the `TestFolder` folder is specified as the parent folder.\n", + "\n", + "Next, a `MediaFileUpload` object is created to complete the upload:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from googleapiclient.http import MediaFileUpload\n", + "\n", + "media = MediaFileUpload('HelloWorld.txt',chunksize = -1,resumable = False)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `chunksize` parameter indicates the size of each piece of the file to be consecutively uploaded. A `chunksize` of `-1` indicates that the whole file should be uploaded as one piece, which is only possible if the file is less than 5 MB in size. The `resumable` parameter indicates whether the upload should be done over multiple requests or not. In this case, the upload is done using a single request. More details on the parameters for the `MediaFileUpload` object can be found [here](https://github.com/googleapis/google-api-python-client/blob/21af37b11ea2d6a89b3df484e1b2fa1d12849510/googleapiclient/http.py#L534).\n", + "\n", + "Finally, upload the file:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file = drive.files().create(body = file_metadata,media_body = media,fields = 'id').execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `id` of the uploaded `HelloWorld.txt` file is:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file['id']" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Delete a file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The uploaded `HelloWorld.txt` file can be deleted using the `delete()` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "drive.files().delete(fileId = file['id']).execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that if the `fileId` is the `id` of a folder, then the folder and the files within it are deleted. See [here](https://developers.google.com/drive/api/v3/reference/files/delete) for details." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Make a file indexable" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "There are three ways to make a file indexable (searchable):\n", + "\n", + "1. Using the `contentHints.indexableText` metadata field.\n", + "2. Using the `description` metadata field.\n", + "3. Using the `properties` metadata field." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `contentHints.indexableText` metadata field" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For example, suppose that the uploaded `HelloWorld.txt` file was associated with a specific day of the week, such as Monday. Then its metadata can be:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file_metadata = {\n", + " 'name': 'HelloWorld.txt',\n", + " 'parents': [folder['id']],\n", + " 'contentHints': {'indexableText':'day of week: Monday'}\n", + "}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The `HelloWorld.txt` file can then be re-uploaded with the new metadata:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "media = MediaFileUpload('HelloWorld.txt',chunksize = -1,resumable = False)\n", + "file = drive.files().create(body = file_metadata,media_body = media,fields = '*').execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The sentence `day of week: Monday` can then be entered into the search bar in Google Drive to return this file.\n", + "\n", + "Notice that the `contentHints` field does not show up in the response body:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Also, there is an inherent problem with this approach. The `contentHints.indexableText` field is included in the body of text that is searched by Google Drive when searching file contents. This means that if there are PDF, text, or any other content-indexable files in the drive, and any of them happen to contain `day of week: Monday`, then they will also be returned with `HelloWorld.txt` when performing a search." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `description` metadata field" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Alternatively, the `description` metadata field can be used. However, instead of deleting the original `HelloWorld.txt` file then re-uploading it with new metadata, its metadata can be updated while it is in the drive using the `update()` method:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "new_file_metadata = {\n", + " 'contentHints': '', # remove previous metadata\n", + " 'description': 'day of week: Monday'\n", + "}\n", + "\n", + "file = drive.files().update(fileId = file['id'],body = new_file_metadata,fields = '*').execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Notice that the `description` field is visible in the response body:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The only advantage that the `description` field offers over the `contentHints.indexableText` field is that it is visible when right-clicking a file and selecting the 'View details' option. However, it suffers from the same problem, since searching for `day of week: Monday` in the search bar also yields other files that contain this sentence. This is because it is not possible to restrict the search to only use the `description` field. See [here](https://support.google.com/drive/answer/2375114?hl=en&authuser=1) for the types of filters that can be used while searching using the Google Drive search bar." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### `properties` metadata field" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The solution to this problem is to use the `properties` metadata field instead, which can be searched using the Google Drive API. See [here](https://developers.google.com/drive/api/v3/properties) and [here](https://developers.google.com/drive/api/v3/search-files) for details." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "properties = {'day_of_week' : 'Monday'}\n", + "\n", + "new_file_metadata = {\n", + " 'description': '', # remove previous metadata\n", + " 'properties':properties\n", + "}\n", + "\n", + "file1 = drive.files().update(fileId = file['id'],\n", + " body = new_file_metadata,\n", + " fields = 'name,id,properties').execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Note that the keys in the `properties` dict cannot contain spaces. The `properties` field has a standard format that must be followed, which is `properties = {'key1' : 'value1', 'key2' : 'value2', ...}`. Here is the response body for the file:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "file1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, the `files().list()` method can be used to search for the `HelloWorld.txt` file. Note that in this case, the parameters in the `fields` parameter need to be enclosed with `files()`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"properties has {key = 'day_of_week' and value = 'Monday'}\"\n", + "\n", + "drive.files().list(q = query,fields = 'files(name,properties)').execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Search using multiple properties" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Additionally, multiple properties can be defined to filter search results. To demonstrate this, the properties of the original `HelloWorld.txt` file will be updated and two other files `HelloWorld2.txt` and `HelloWorld3.txt` with different properties will be uploaded:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "properties = {\n", + " 'day_of_week' : 'Monday',\n", + " 'month' : 'June',\n", + " 'color' : 'blue'\n", + "}\n", + "\n", + "new_file_metadata = {'properties':properties}\n", + "\n", + "file1 = drive.files().update(fileId = file['id'],\n", + " body = new_file_metadata,\n", + " fields = 'name,id,properties').execute()\n", + "\n", + "file1" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, create and upload `HelloWorld2.txt` and `HelloWorld3.txt` with different properties:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with open('HelloWorld2.txt','w') as f:\n", + " f.write('Hello World!')\n", + "\n", + "properties = {\n", + " 'day_of_week' : 'Monday',\n", + " 'month' : 'August',\n", + " 'color' : 'green'\n", + "}\n", + "\n", + "file_metadata = {\n", + " 'name': 'HelloWorld2.txt',\n", + " 'parents': [folder['id']],\n", + " 'properties': properties\n", + "}\n", + "\n", + "media = MediaFileUpload('HelloWorld2.txt',chunksize = -1,resumable = False)\n", + "file2 = drive.files().create(body = file_metadata,media_body = media,fields = 'id,name,properties').execute()\n", + "\n", + "file2" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "with open('HelloWorld3.txt','w') as f:\n", + " f.write('Hello World!')\n", + "\n", + "properties = {\n", + " 'day_of_week' : 'Tuesday',\n", + " 'month' : 'September',\n", + " 'color' : 'blue'\n", + "}\n", + "\n", + "file_metadata = {\n", + " 'name': 'HelloWorld3.txt',\n", + " 'parents': [folder['id']],\n", + " 'properties': properties\n", + "}\n", + "\n", + "media = MediaFileUpload('HelloWorld3.txt',chunksize = -1,resumable = False)\n", + "file3 = drive.files().create(body = file_metadata,media_body = media,fields = 'id,name,properties').execute()\n", + "\n", + "file3" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Finally, `files().list()` can be used to search for these files. First, to return `HelloWorld.txt` and `HelloWorld2.txt`:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"properties has {key = 'day_of_week' and value = 'Monday'}\"\n", + "\n", + "drive.files().list(q = query,fields = 'files(name,properties)').execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, to return `HelloWorld2.txt` only:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"properties has {key = 'day_of_week' and value = 'Monday'} and properties has {key = 'color' and value = 'green'}\"\n", + "\n", + "drive.files().list(q = query,fields = 'files(name,properties)').execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Notice that the general format to query multiple properties at the same time is:\n", + "\n", + "`\"properties has {key = 'key1' and value = 'value1'} and properties has {key = 'key2' and value = 'value2'} and properties has {key = 'key3' and value = 'value3'} and ...\"`\n", + "\n", + "As another example, to return `HelloWorld3.txt` only:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "query = \"properties has {key = 'month' and value = 'September'} and properties has {key = 'day_of_week' and value = 'Tuesday'}\"\n", + "\n", + "drive.files().list(q = query,fields = 'files(name,properties)').execute()" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.7" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}