6
11 Comments

How to create an automated Twitter bookmark organizer in Notion

If you're anything like me, you're constantly bookmarking tweets that you want to come back to later. But when the time comes to actually find that one particular tweet, it's nearly impossible to do so. Wouldn't it be great if there was a way to automatically organize all of your Twitter bookmarks?

Well, good news! There is a way, and this guide will walk you through the entire process, from getting an auth token to fetching your bookmarks to adding them to Notion with their API. Plus, you can even set up a periodic SaaS to save all of your future Twitter bookmarks. So what are you waiting for?

Read on to learn how to get your very own Twitter bookmark organizer up and running!

Here's how you can create your own automatic Twitter bookmark organizer in Notion with the Twitter API, GPT-3 API, and Notion API.


Steps

1. Get Twitter developer API credentials

2. Get OAuth 2.0 Twitter API token

3. Fetch Twitter bookmarks

4. Get bookmark keywords with GPT-3 API (optional)

5. Get Notion API keys

6. Create a Notion table and connect with API

7. Add bookmarks to Notion

8. Refresh your auth token


Here's what we'll be using:

1. Python 🐍

2. Twitter API 🐦

3. Notion API 📝

4. GPT-3 API 🤖 (optional)

Let's get going.


1. Get Twitter developer API credentials

The first step is to get your app's client id and client secret from your Twitter developer account.

Go to the dashboard of your Twitter developer account, click + Add App, choose Create new, pick an environment and then choose a name.

Save the bearer token revealed at the last stage. You won't need it in this guide, but having it for other purposes is good. Like if you want to support fellow indie makers you're following that are launching on Product Hunt.

Twitter developer portal

Now that you have an app, it's time to set up user authentication. You'll need this to call certain endpoints, allowing your app to make specific requests for authenticated users.

Click on Set up below the newly created app, keep Read and choose Web App, Automated App or Bot because you'll need the confidential client:

Set up user authentication

Add the callback URL and website URL for your app. The callback/redirect URL is the destination that OAuth is allowed to redirect after the authentication process. You'll need this URL in the coming steps.

Set up user authentication

Make sure to save the client id and the client secret revealed. We'll use them in the coming steps when making API calls to get your bookmarks later.

Now your user authentication settings should be set up:

User authentication set up

Let's use these credentials in the next step, where we'll get the auth token for your Twitter app.

2. Get OAuth 2.0 Twitter API token

In this part, where we'll generate a token using OAuth 2.0 Authorization Code Flow with PKCE, I'm using code from this great repo: OAuth 2.0 Authorization Code Flow with PKCE

Start by importing all the libraries:

import base64
import datetime
import hashlib
import json
import os
import re
import requests
from time import sleep

from requests.auth import AuthBase, HTTPBasicAuth
from requests_oauthlib import OAuth2Session

Then add your Twitter credentials, and use the same redirect URL as you added in the previous step:

client_id = "YOUR_CLIENT_ID"
client_secret = "YOUR_CLIENT_SECRET"
redirect_uri = "YOUR_REDIRECT_URL"

Set up permission scope:

# Set the scopes
# offline.access makes it possible to fetch 
# a new refresh token when the access token have expired
scopes = ["bookmark.read", "tweet.read", "users.read", "offline.access"]

The access token we're trying to generate is only valid for 2 hours.
So if you want to fetch your Twitter bookmarks regularly, you'll need to obtain a new access token every time you want to call the endpoint.

offline.access makes it possible to fetch a new refresh token without prompting a new login session. When the access token has expired, you can fetch a new one with the refresh token. The refresh token is valid for 6 months.

Then create a code verifier:

code_verifier = base64.urlsafe_b64encode(os.urandom(30)).decode("utf-8")
code_verifier = re.sub("[^a-zA-Z0-9]+", "", code_verifier)

Create a code challenge:

code_challenge = hashlib.sha256(code_verifier.encode("utf-8")).digest()
code_challenge = base64.urlsafe_b64encode(code_challenge).decode("utf-8")
code_challenge = code_challenge.replace("=", "")

Finally, start the OAuth 2.0 session:

oauth = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scopes)

Now it's time to create an authorize URL that you'll visit to allow your app to make requests on behalf of yourself:

auth_url = "https://twitter.com/i/oauth2/authorize"
authorization_url, state = oauth.authorization_url(
    auth_url, code_challenge=code_challenge, code_challenge_method="S256"
)

Now visit the URL you received from the variable authorization_url to authorize your app to make requests on your behalf.

Authorize your app

You'll be redirected to a new URL when you have authorized your app. Copy that URL in a new variable:

authorization_response = "THE_URL_YOU_GOT_REDIRECTED_TO_AFTER_AUTHORIZATION"

Now we can fetch our access token:

token_url = "https://api.twitter.com/2/oauth2/token"
auth = HTTPBasicAuth(client_id, client_secret)

token = oauth.fetch_token(
    token_url=token_url,
    authorization_response=authorization_response,
    auth=auth,
    client_id=client_id,
    include_client_id=True,
    code_verifier=code_verifier,
)

If we inspect the token dict, you'll see both the access token you need for the API call to get your bookmarks but also the refresh token you'll need to generate a new access token once it is expired:

Access and refresh tokens

The next step is to fetch your bookmarks.

3. Fetch Twitter bookmarks

Before we call the bookmark API endpoint and fetch your bookmarks, we'll need your numerical user id. Fetch your numerical user id with this endpoint:

user_me = requests.get(
    "https://api.twitter.com/2/users/me",
    headers={"Authorization": f"Bearer {access_token}"},
).json()

user_id = user_me["data"]["id"]

Now we're ready to call the bookmark API endpoint and fetch your bookmarks:

url = f"https://api.twitter.com/2/users/{user_id}/bookmarks"
headers = {
    "Authorization": f"Bearer {access_token}",
}
response = requests.get(url, headers=headers, params={
    'tweet.fields':'author_id,created_at',
    'expansions':'author_id',
    'user.fields':'username',
})

This API call will fetch 100 bookmarks. To to paginate to the next 100 ones, you'll need to include the pagination_token that we received in the response. This guide will cover how to store the 100 latest ones.

The last part of this section is to include the username and name of each Twitter user with each tweet. The Twitter API response returns the tweets and the user data in two different keys:

Bookmark API response

Create a mapping dict for user data:

user_mapping = {user['id']:user for user in response.json()['includes']['users']}

Then combine bookmarks with user data:

for tweet in tweets:
    tweet.update({
        'name':user_mapping[tweet['author_id']]['name'], 
        'username': user_mapping[tweet['author_id']]['username']
    })

So far, we have generated an access token, refresh token and your 100 latest bookmarks combined with user data.

The following section is optional; we'll use GPT-3 to get 5 keywords for each bookmark.

4. Get bookmark keywords with GPT-3 API (optional)

This section is optional and requires you to have access to Open AI GPT-3 API keys. We'll let GPT-3 give us 5 keywords for each Twitter bookmark. That way, it'll be a breeze once you're searching for a specific tweet you bookmarked, and you'll be able to organize the tweets according to keywords.

First, get your API key, go to https://beta.openai.com/, log in and click on your avatar and View API keys:

Open AI API keys

Then create a new secret key and save it for the request:

Create Open AI API key

For the endpoint text-davinci-002, we'll need a prompt that asks the endpoint to give us 5 keywords related to the tweet. Feel free to play around with this prompt. Here's the one I'm using:

prompt = f"Here is a tweet, give me 5 keywords, each keyword on a new line, that describe what the tweet is about \n\n --- tweet start ---- \n\n  {tweet['text']} \n\n  --- tweet end ---:"

Now we can make a call to the Davinci endpoint for each tweet in our list and expand each dict with a new key; Keywords:

import openai
api_key ="YOUR_OPEN_AI_GPT3_API_KEY"
for i, tweet in enumerate(tweets, start=1):

    print(f"Processing {i}/{len(tweets)} tweets", end='\r')
    
    # Create a prompt for the completion endpoint
    prompt = f"Here is a tweet, give me 5 keywords, each keyword on a new line, that describe what the tweet is about \n\n --- tweet start ---- \n\n  {tweet['text']} \n\n  --- tweet end ---:"
    
    response_gpt3 = openai.Completion.create(
      model="text-davinci-002",
      prompt=prompt,
      temperature=0.7,
      max_tokens=100,
      top_p=1,
      frequency_penalty=0,
      presence_penalty=0
    )
    
    # Update tweet with keywords
    tweet.update({'keywords':response_gpt3['choices'][0]['text'].strip()})

This takes a little while so I've added a little counter print(f"Processing {i}/{len(tweets)} tweets", end='\r') just to make sure it's making progress.

Great, now each of your bookmarks should be expanded with 5 keywords, all decided by GPT-3:

Tweet keywords

Each keyword has a trailing \n to add a new line in your Notion database.

The next step is to get Notion API keys to be able to add your bookmarks to a Notion table.

5. Get Notion API keys

Start creating an integration by logging into your Notion account, going to https://www.notion.so/my-integrations, and clicking on the + New integration button.

Give your integration a name, select the workspace where you want to install this integration and the capabilities that the integration will have, then click Submit to create the integration:

Create integration

Copy the Internal Integration Token on the next page and save it somewhere. We'll use it together with the bookmarks from Twitter later:

Save internal integration token

The next step is to create a page with a database.

6. Create a Notion table and connect with API

Continue by creating a new page and a new table with the columns Username, Name, Keywords, Tweet, URL, and Tweeted at:

Access and refresh tokens

Integrations don't have access to any pages or databases in the workspace at first. We'll have to share each database with the newly created integration. Click Share and pick the integration you created in the previous step and click Invite:

Share database with integration

Your integration now has the requested permissions on the new database; the last step is to get the id of the database. Ensure you're viewing the database as a full page if you use an inline database.

If you have a workspace, the database id is the part of the URL after your workspace name and the slash (myworkspace/) and before the question mark (?).

The id is 32 characters long, containing numbers and letters. If you're using the Notion desktop app, click the Share button and select Copy link.

Copy and save the id for your database for the next step:

https://www.notion.so/y9btc56182y429inm1180e9e42w2r177?v=...
                     |-------- Your database id ------|

7. Add bookmarks to Notion

Now that we have the API keys for your Notion integration let's create a payload with all your bookmarked

Finally, let's make a call and add your bookmarked tweets to your database:

# Get the API key from the environment variable
notion_key = "YOUR_NOTION_INTEGRATION_KEY"

# Get the database ID from the environment variable
notion_database_id = "YOUR_NOTION_DATABASE_ID"

# Set the headers
headers = {
    'Authorization': 'Bearer ' + notion_key,
    'Content-Type': 'application/json',
    'Notion-Version': '2021-08-16'
}

# Create the payload and make request
for i,tweet in enumerate(tweets, start=1):

    payload = {
        'parent': { 'database_id': notion_database_id },
        'properties': {
            'title': {
                'title': [
                    {
                        'text': {
                            'content':  tweet['username']
                        }
                    }
                ]
            },
           "Keywords": {"rich_text": [{ "type": "text", "text": { "content": tweet['keywords'] } }]},
            "Name": {"rich_text": [{ "type": "text", "text": { "content": tweet['name'] } }]},
            "Tweet": {"rich_text": [{ "type": "text", "text": { "content": tweet['text'] } }]},
            "URL": {'url': f"https://twitter.com/twitter/status/{tweet['id']}"},
            "Tweeted at": {"date": {"start": tweet['created_at'] }}

        }
    }

    # Make the request
    r = requests.post('https://api.notion.com/v1/pages', headers=headers, data=json.dumps(payload))

    # Print the response
    print(f"Tweet {i}/{len(tweets)} Response: {r.json()['object']}", end='\r')

You should now have 100 of your bookmarks in your Notion database:

Populated Notion database

We're all set! You can now filter and sort your bookmarked tweets in your Notion database.

In the last section below, we'll see how you can fetch a new access token after it expired.

8. Refresh your auth token

As mentioned earlier, the access token is only valid for 2 hours. So if you're creating a service that is automatically and regularly fetching your new Twitter bookmarks, you'll need to request a new access token every time.

You can use your refresh token, which is valid for 6 months, to fetch a new access token like this:

token = "YOUR_PREVIOUSLY_SAVED_TOKEN"

refreshed_token = oauth.refresh_token(
          client_id=client_id,
          client_secret=client_secret,
          token_url=token_url,
          auth=auth,
          refresh_token=token["refresh_token"],
        )

Summary

1. Get Twitter developer API credentials

2. Get OAuth 2.0 Twitter API token

3. Fetch Twitter bookmarks

4. Get bookmark keywords with GPT-3 API (optional)

5. Get Notion API keys

6. Create a Notion table and connect with API

7. Add bookmarks to Notion

8. Refresh your auth token


Next steps

1. Repo with source code

Here is the repo with a Jupyter notebook with all the source code if you'd like to implement this on your own ⬇️
https://github.com/norahsakal/automatically-organize-twitter-bookmarks-in-notion

2. Don't feel like coding?

Join the waitlist to get your Twitter bookmarks automatically added to your Notion ⬇️
https://bookmark-organizer.carrd.co


Originally published at https://norahsakal.com/blog/automatically-organize-twitter-bookmarks-in-notion

  1. 2

    Hello, thank you for writing this guide!

    I'm having some trouble on step 3; specifically the "combine bookmarks with user data" step. Every time I try to run that block in Jupyter
    (cell 33, the following line: " for tweet in tweets: "),
    I get the following error:

    NameError: name 'tweets' is not defined

    Is there a way to fix the issue? Maybe there's a library I'm missing?

    1. 2

      Hey! Thank you for bringing this to my attention, great catch!

      I have added a line in the notebook where I am defining the missing variable like this:

      tweets = response.json()['data']

      Like this:

      Add variable definition

      Please add this right after the section called Make a request to the bookmarks url.

      I have also updated the repo so you have it in this link: https://github.com/norahsakal/automatically-organize-twitter-bookmarks-in-notion/blob/main/twitter_bookmark_organizer_guide.ipynb

      1. 1

        Thank you so much for the quick response! This fixed the "tweets" error that was occurring! However, I am experiencing one final error in Step 7, on the following line:
        print(f"Tweet {i}/{len(tweets)} Response: {r.json()['object']}", end='\r')

        The block runs without errors, but r.json()['object'] returns "error" for every single tweet (example output: Tweet 100/100 Response: error). Nothing is added in the Notion database, either. Do you know what might be causing this error?

        1. 2

          Hey! I just reran the code and it is supposed to work.

          Here are some pointers for troubleshooting:

          • Do you have the right notion key? notion_key = "YOUR_NOTION_INTEGRATION_KEY"

          • Do you have the right database id? notion_database_id = "YOUR_NOTION_DATABASE_ID"

          • Did you create a table with the exact right columns? Username, Name, Keywords, Tweet, URL, and Tweeted at

          • Did you correctly authorize your integration to access your Notion database?

          What does the error message from Notion say? Just run 1 loop like this:

          # Get the API key from the environment variable
          notion_key = "YOUR_NOTION_INTEGRATION_KEY"
          
          # Get the database ID from the environment variable
          notion_database_id = "YOUR_NOTION_DATABASE_ID"
          
          # Set the headers
          headers = {
              'Authorization': 'Bearer ' + notion_key,
              'Content-Type': 'application/json',
              'Notion-Version': '2021-08-16'
          }
          
          # Create the payload and make request
          for i,tweet in enumerate(tweets, start=1):
          
              if i > 2:
                  break
          
              payload = {
                  'parent': { 'database_id': notion_database_id },
                  'properties': {
                      'title': {
                          'title': [
                              {
                                  'text': {
                                      'content':  tweet['username']
                                  }
                              }
                          ]
                      },
                     "Keywords": {"rich_text": [{ "type": "text", "text": { "content": tweet['keywords'] } }]},
                      "Name": {"rich_text": [{ "type": "text", "text": { "content": tweet['name'] } }]},
                      "Tweet": {"rich_text": [{ "type": "text", "text": { "content": tweet['text'] } }]},
                      "URL": {'url': f"https://twitter.com/twitter/status/{tweet['id']}"},
                      "Tweeted at": {"date": {"start": tweet['created_at'] }}
          
                  }
              }
          
              # Make the request
              r = requests.post('https://api.notion.com/v1/pages', headers=headers, data=json.dumps(payload))
              print(r.json())
          

          Try to print out r.json() and see what the error message says, this will give you a hint about what the problem might be.

          1. 1

            Hello, so sorry for the late reply! I was receiving this error:
            {'object': 'error', 'status': 400, 'code': 'validation_error', 'message': 'URL is expected to be rich_text. Tweeted at is expected to be rich_text.'}

            But I solved it! I had the URL and Date Tweeted columns as text types, instead of as URL and Date types respectively. Thank you so much for your help!

            1. 1

              Thanks for the update! I'm glad you were able to make it work 🙌

  2. 2

    Thanks for this awesome detailed guide!

    1. 1

      Thanks for checking it out Mariam! 🙌

  3. 1

    Thank you so much for sharing this, Norah. I'm really looking forward to trying it out on the weekend!

    1. 1

      Thanks for checking it out Sofia! Let me know how it goes 🙌

  4. 1

    Hackers can gain access to cryptocurrency owners' cryptocurrency wallets and exchange accounts to steal crypto protect your account with the help of innocent, Contact via : { [email protected] } IF You Have Lost Your Passwords & Wallets Contact You Are Safe.
    One of the best ways to protect your investment is to secure a wallet. There are two primary types of cryptocurrency wallets. Of the two, "cold storage" or "cold wallet" hardware devices are the safer option. These wallets look like USB drives and act as a physical store for tokens or coins.

Trending on Indie Hackers
How I grew a side project to 100k Unique Visitors in 7 days with 0 audience 47 comments Competing with Product Hunt: a month later 33 comments Why do you hate marketing? 27 comments $15k revenues in <4 months as a solopreneur 14 comments Use Your Product 13 comments How I Launched FrontendEase 13 comments