TwitterRESTHandler offers a proxy for the Twitter 1.1 REST API. Here is an example:
url:
twitter:
pattern: /twitter/(.*)
handler: TwitterRESTHandler
kwargs:
# Visit https://apps.twitter.com/ to get these keys
key: "..."
secret: "..."
redirect:
header: Referer
url: /$YAMLURL/
Follow the steps for Twitter auth to get the keys above.
Now, follow these steps:
If you don’t want the user to log in, and want to use a pre-authorised login, add
the following to the kwargs: section:
url:
twitter-open:
pattern: /twitter-open/(.*)
handler: TwitterRESTHandler
kwargs:
# Visit https://apps.twitter.com/ to get these keys
key: "..."
secret: "..."
access_key: "..."
access_secret: "..."
Now /twitter-open/search/tweets.json?q=beer even without you logging into Twitter. It runs on behalf of the developer with their access token.
To use this via jQuery, use this snippet:
fetch("twitter-open/statuses/home_timeline.json?count=1");
// OUTPUT
To hard-code a specific REST API, use the path: parameter. For example:
url:
twitter:
pattern: /twitter/search # Maps this URL
handler: TwitterRESTHandler
kwargs:
path: search/tweets.json # specifically to the API
...
… maps /twitter/search to https://api.twitter.com/1.1/search/tweets.json
with the relevant authentication.
A typical Twitter app page will have the following flow:
For example:
fetch("twitter/statuses/home_timeline.json")
.done(function (data) {
display(data);
})
.fail(function (xhr, status, msg) {
if (msg == "access token missing")
location.href = "twitter/"; // Redirect the user to log in
else alert(msg); // Alert if it's some other error
});
After the login, users can be redirected via the redirect: config
documented the redirection configuration.
If your app needs to persist the user’s access token, add access_key: persist
and access_secret: persist to the kwargs. The first time, the user is asked to
log in. Thereafter, the user’s credentials are available for all future requests.
This is typically used to show authenticated information on behalf of a user to the public. Typically, such requests are cached as well. Here is a sample configuration:
url:
twitter-persist:
pattern: /persist/(.*)
handler: TwitterRESTHandler
kwargs:
key: "..."
secret: "..."
access_key: persist # Persist the access token after first login
access_secret: persist # Persist the access token after first login
cache:
duration: 300 # Cache requests for 5 seconds
Here is a sample response:
fetch("persist/statuses/home_timeline.json?count=1"); // OUTPUT
The first time, you get an access_key error. Visit /persist/ to log in. Thereafter, your access_key and access_secret will be stored and used for future requests until it expires, or a user logs in again at /persist/.
The following request searches for mentions of Gramener and fetches the first response:
fetch("twitter-open/search/tweets.json?q=gramener&count=1"); // OUTPUT
The endpoint /search/tweets.json is the same as that in the Twitter API, which internally acts as an input to the api Gramex endpoint.
Search results are paginated. Fetch .search_metadata.next_results to get the next page. For example:
function twitter_search(query) {
fetch("twitter-open/search/tweets.json" + query)
.then((r) => r.json())
.then((result) => {
// Show the result
console.log(result.statuses);
// Fetch next results if any
if (result.search_metadata.next_results)
twitter_search(result.search_metadata.next_results);
});
}
// Keep searching for tweets by Gramener
twitter_search("?q=gramener");
This script fetches the list of followers for Gramener:
fetch("twitter-open/followers/list.json?screen_name=gramener&count=1"); // OUTPUT
You can use the transform: configuration to modify the response in any way. It can be any
expression or pipeline. Here is a transform that adds the sentiment to each tweet:
twittersentiment:
pattern: /$YAMLURL/sentiment
handler: TwitterRESTHandler
kwargs:
...
path: search/tweets.json
transform:
sentiment:
function: twitterutils.add_sentiment
Here’s what twitterutils.add_sentiment looks for the last about Gramener:
from textblob import TextBlob
def add_sentiment(result, handler):
for tweet in result['statuses']:
blob = TextBlob(tweet['text'])
tweet['sentiment'] = blob.sentiment.polarity
return result
This transforms the tweets to add a sentiment: key measuring its sentiment.
fetch("sentiment?q=gramener&count=1"); // OUTPUT
The transform should either return a JSON-encodable object, or a string.
Transforms are executed in a separate thread. This makes the application more responsive. But you need to ensure that your code is thread-safe.
To append all tweets into a JSON-Line file, use a function like this:
def save_tweet_transform(result, handler):
with open('tweets.jsonl', 'a') as out:
for status in result['statuses']:
json.dump(status, out + '\n')
return result
You can then include this in the TwitterHandler transform section as follows:
...
kwargs:
transform:
sentiment:
function: module.save_tweet_transform
The TwitterRESTHandler processes results asynchronously. So when one request
is being processed, it can process another as well.
Here, we send two requests. The time taken for both requests is almost the same as the time taken for each individual request. They execute in parallel.
We use Promise.all to wait for all requests.
// Latest tweet for Gramener
var q1 = fetch("twitter-open/search/tweets.json?q=gramener&count=1");
// Latest tweet for Richard Dawkins
var q2 = fetch("twitter-open/search/tweets.json?q=RichardDawkins&count=1");
Promise.all(q1, q2); // OUTPUT
The methods: parameter specifies which methods to use to access the API. The
default is [GET, POST]. You can replace it with [POST] to just use POST. This
prevents external sites from requesting the page. Note that you need to handle
XSRF for POST requests.
url:
twitter:
pattern: /twitter/search # Maps this URL
handler: TwitterRESTHandler
kwargs:
...
methods: [POST] # Allow only POST requests
The Twitter streaming API provides a live source of tweets. This can be set up as a schedule:
schedule:
twitter-stream:
function: TwitterStream
kwargs:
track: beer,wine # Track these keywords
follow: Starbucks,Microsoft # OR follow these users' tweets
path: tweets.{:%Y-%m-%d}.jsonl # Save the results in this file
# Visit https://apps.twitter.com/ to get these keys
key: ...
secret: ...
access_key: ...
access_secret: ...
flush: 60 # Flush data every 60 seconds
startup: true # Run on startup
thread: true # in a separate thread (REQUIRED)
This runs the TwitterStream class on startup in a separate thread.
TwitterStream opens a permanent connection to Twitter and receives all tweets
matching either beer or wine. It also receives any tweets from @Starbucks
and @Microsoft.
The results are saved in tweets.xxx.jsonl. (The extension .jsonl indicates
the JSON Lines format.) You can use standard Python
date formatting. For example, {:%b-%Y}.jsonl will
create files like Jan-2016.jsonl, Feb-2016.jsonl, etc. while {:%H-%M}.jsonl
will save the tweets into an hour-minute files. (It will append to the file, so
you won’t lose data.)
Note: You can run multiple Twitter streams, but you need different access keys for each.