Академический Документы
Профессиональный Документы
Культура Документы
To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including
cookie policy.
This is your last free story this month. Sign up and get an extra one for free.
Telegram has a lot of potential as a messaging app. The Bot APIs can act as assistants to
perform many functions in chats and channels. At the same time, Twitter provides
easy-to-use APIs for developers to publish, manage, and analyse tweets. With the
flexibility the APIs offer, the potential for Twitter and Telegram in data science is
endless.
During the coronavirus, I wanted to keep myself up to date with the daily coronavirus
figures. In the UK, the figures are released on a daily basis but at a different time
throughout the day.
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 1/8
6/7/2020 Customise your own Twitter notification with Telegram Bot
What
To makeIMedium
think would belog
work, we nice,
useris to get
data. a notification
By using Medium, youwhen the
agree to ourfigures
Privacy are released,
Policy, including
cookie policy.
together with the data that I am interested in.
In this tutorial, I will walk through on how to create your own Twitter news alert. This
will include
1. Scrape tweets using Tweepy
3. Create a telegram bot, and have the bot sending message to channel/chat
using Python
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 2/8
6/7/2020 Customise your own Twitter notification with Telegram Bot
To 2. Click
make ‘Apps’,
Medium then
work, we ‘Create an app’.
log user data. By using Medium, you agree to our Privacy Policy, including
cookie policy.
3. Complete the application
4. You can see the app that you just created, and the consumer API key and consumer
secret key.
5. At ‘Create my access token’, you can get the access token and the access token
secret.
You will need the script below to mine the tweets, save the auth keys as a dictionary.
import tweepy
import datetime
class TweetMiner(object):
result_limit = 20
data = []
api = False
self.twitter_keys = keys_dict
auth = tweepy.OAuthHandler(keys_dict['consumer_key'],
keys_dict['consumer_secret'])
auth.set_access_token(keys_dict['access_token_key'],
keys_dict['access_token_secret'])
self.api = tweepy.API(auth)
self.twitter_keys = keys_dict
self.result_limit = result_limit
data = []
last_tweet_id = False
page = 1
statuses =
To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including
self.api.user_timeline(screen_name=user,
cookie policy.
count=self.result_limit,
max_id=last_tweet_id - 1,
tweet_mode =
'extended',
include_retweets=True
)
else:
statuses =
self.api.user_timeline(screen_name=user,
count=self.result_limit,
tweet_mode =
'extended',
include_retweets=True)
mined = {
'tweet_id': item.id,
'name': item.user.name,
'screen_name': item.user.screen_name,
'retweet_count': item.retweet_count,
'text': item.full_text,
'mined_at': datetime.datetime.now(),
'created_at': item.created_at,
'favourite_count': item.favorite_count,
'hashtags': item.entities['hashtags'],
'status_count': item.user.statuses_count,
'location': item.place,
'source_device': item.source
}
try:
mined['retweet_text'] =
item.retweeted_status.full_text
except:
mined['retweet_text'] = 'None'
try:
mined['quote_text'] =
item.quoted_status.full_text
mined['quote_screen_name'] =
status.quoted_status.user.screen_name
except:
mined['quote_text'] = 'None'
mined['quote_screen_name'] = 'None'
last_tweet_id = item.id
data.append(mined)
page += 1
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 4/8
6/7/2020 Customise your own Twitter notification with Telegram Bot
To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including
return data
cookie policy.
With the above TweetMiner function, we can start collecting tweets with just two lines
of codes.
tweets_df = pd.DataFrame(uk_tweets)
The dataframe will contain all the tweets that we scrape, but we are only interested in
the tweets that includes the daily figures. So we need to do some filtering. I chose the
word ‘died’ as a unique string to help collect all the tweets.
tweets_df = tweets_df[tweets_df['text'].str.contains("died")]
To better visualise data, I have created a date column and changed the dataframe to
numeric.
tweets_df['date'] = tweets_df.created_at.dt.strftime('%Y-%m-%d')
tweets_df = tweets_df[['screen_name',
'date','text']].reset_index(drop=True)
tweets_df = tweets_df.apply(pd.to_numeric, errors='ignore')
From the tweet, there is a lot of figures we can extract. Let’s try to get the death rate as
an example. The way that I calculate the death rate is (cumulative death) / (the
number of positive cases).
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 5/8
6/7/2020 Customise your own Twitter notification with Telegram Bot
You
To can
make find awork,
Medium lot ofweregex pattern
log user data. Byonline. I usedyou
using Medium, theagree
following pattern
to our Privacy to match
Policy, all the
including
cookie
numberpolicy.
within the tweet to extract the figures as two new columns, ‘death total’ and
‘positive total’.
number_regex = '\d+(?:,\d+)*'
tweets_df['death_total'] =
tweets_df.text.str.findall(number_regex).str[8]
tweets_df['positive_total'] =
tweets_df.text.str.findall(number_regex).str[5]
We can now create a column called ‘death_rate’ by putting the figures into the formula.
We can use the similar ways to extract other figures. But for now, let’s add data
transparency by extracting the url with another regex.
url_regex = '(?:(?:https?|ftp):\/\/)?[\w/\-?=%.]+\.[\w/\-?=%.]+'
tweets_df['url'] =
tweets_df.text.str.findall(url_regex).apply(''.join)
To draft up the telegram message, we need to extract the numbers from the dataframe.
msg = 'On {0}, the death toll has reached {1}, the death rate thus
far is {2}. {3}'.format(latest_date, \
death_total, death_rate, url)
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 6/8
6/7/2020 Customise your own Twitter notification with Telegram Bot
2. To create new bot, Send another ‘/newbot’ message, then follow the instructions to
configure a name and a username. Your user name will be the bot_chatID.
3. send ‘/token’ to get the token that access to HTTP API and then save it as
bot_token.
What we are doing here is to use bot to send message to a target channel. So the
bot_chatID should be the channel name you create, starting with ‘@’. Remember
to make sure you bot is the admin of the channel. Otherwise, the bot allowed to
publish content to the channel.
import requests
Run the function, and voila! You will see the message is being sent to telegram!
00 21 * * * /path/to/script/.sh
For example, the above crontab is set to automatically run the script every day at 9pm.
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 7/8
6/7/2020 Customise your own Twitter notification with Telegram Bot
Alternatively,
To you can
make Medium work, also
we log run
user theByscript
data. every five
using Medium, you minutes and
agree to our sendPolicy,
Privacy out aincluding
message
cookie
once policy.
the daily figures are released. That would need another tutorial. But you can
check out how I do it on my github.
That’s it — now you have written a python script that scrapes tweets, analyses data,
and automatically sent message to a telegram channel.
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 8/8