Вы находитесь на странице: 1из 8

6/7/2020 Customise your own Twitter notification with Telegram Bot

To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including
cookie policy.

This is your last free story this month. Sign up and get an extra one for free.

Customise your own Twitter noti cation with


Telegram Bot
Linus Ng
May 27 · 5 min read

Telegram has a lot of potential as a messaging app. The Bot APIs can act as assistants to
perform many functions in chats and channels. At the same time, Twitter provides
easy-to-use APIs for developers to publish, manage, and analyse tweets. With the
flexibility the APIs offer, the potential for Twitter and Telegram in data science is
endless.
During the coronavirus, I wanted to keep myself up to date with the daily coronavirus
figures. In the UK, the figures are released on a daily basis but at a different time
throughout the day.

1. The o cial daily tweet on coronavirus in UK

https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 1/8
6/7/2020 Customise your own Twitter notification with Telegram Bot

What
To makeIMedium
think would belog
work, we nice,
useris to get
data. a notification
By using Medium, youwhen the
agree to ourfigures
Privacy are released,
Policy, including
cookie policy.
together with the data that I am interested in.

2. The automatic telegram noti cation

In this tutorial, I will walk through on how to create your own Twitter news alert. This
will include
1. Scrape tweets using Tweepy

2. Extract numeric data from tweets

3. Create a telegram bot, and have the bot sending message to channel/chat
using Python

4. Create a crontab job that enables the script recurringly

1. Scrape tweets using tweepy


Before we start collecting data for twitter, we need to become Twitter developers.

1. Login to twitter developer section.

https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 2/8
6/7/2020 Customise your own Twitter notification with Telegram Bot

To 2. Click
make ‘Apps’,
Medium then
work, we ‘Create an app’.
log user data. By using Medium, you agree to our Privacy Policy, including
cookie policy.
3. Complete the application

4. You can see the app that you just created, and the consumer API key and consumer
secret key.

5. At ‘Create my access token’, you can get the access token and the access token
secret.

You will need the script below to mine the tweets, save the auth keys as a dictionary.

import tweepy
import datetime

auth = {'consumer_key': 'XXXXXX',


'consumer_secret':'XXXXXX',
'access_token_key':'XXXXXX',
'access_token_secret': 'XXXXXX'
}

class TweetMiner(object):

result_limit = 20
data = []
api = False

def __init__(self, keys_dict=auth, api=api, result_limit = 20):

self.twitter_keys = keys_dict

auth = tweepy.OAuthHandler(keys_dict['consumer_key'],
keys_dict['consumer_secret'])
auth.set_access_token(keys_dict['access_token_key'],
keys_dict['access_token_secret'])

self.api = tweepy.API(auth)
self.twitter_keys = keys_dict

self.result_limit = result_limit

def mine_user_tweets(self, user="linusnhh",


mine_rewteets=False,
max_pages=5):

data = []
last_tweet_id = False
page = 1

while page <= max_pages:


if last_tweet_id:
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 3/8
6/7/2020 Customise your own Twitter notification with Telegram Bot

statuses =
To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including
self.api.user_timeline(screen_name=user,
cookie policy.
count=self.result_limit,

max_id=last_tweet_id - 1,
tweet_mode =
'extended',

include_retweets=True
)
else:
statuses =
self.api.user_timeline(screen_name=user,

count=self.result_limit,
tweet_mode =
'extended',

include_retweets=True)

for item in statuses:

mined = {
'tweet_id': item.id,
'name': item.user.name,
'screen_name': item.user.screen_name,
'retweet_count': item.retweet_count,
'text': item.full_text,
'mined_at': datetime.datetime.now(),
'created_at': item.created_at,
'favourite_count': item.favorite_count,
'hashtags': item.entities['hashtags'],
'status_count': item.user.statuses_count,
'location': item.place,
'source_device': item.source
}

try:
mined['retweet_text'] =
item.retweeted_status.full_text
except:
mined['retweet_text'] = 'None'
try:
mined['quote_text'] =
item.quoted_status.full_text
mined['quote_screen_name'] =
status.quoted_status.user.screen_name
except:
mined['quote_text'] = 'None'
mined['quote_screen_name'] = 'None'

last_tweet_id = item.id
data.append(mined)

page += 1

https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 4/8
6/7/2020 Customise your own Twitter notification with Telegram Bot

To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including
return data
cookie policy.

With the above TweetMiner function, we can start collecting tweets with just two lines
of codes.

Tweet = TweetMiner(result_limit = 10)


uk_tweets = Tweet.mine_user_tweets(user=’DHSCgovuk’, max_pages=1)

2. Extract numeric data from tweets


The collected tweets are in the form of the list. To make it easier for visualisation, we
need to convert the list to a dataframe.

tweets_df = pd.DataFrame(uk_tweets)

The dataframe will contain all the tweets that we scrape, but we are only interested in
the tweets that includes the daily figures. So we need to do some filtering. I chose the
word ‘died’ as a unique string to help collect all the tweets.

tweets_df = tweets_df[tweets_df['text'].str.contains("died")]

To better visualise data, I have created a date column and changed the dataframe to
numeric.

tweets_df['date'] = tweets_df.created_at.dt.strftime('%Y-%m-%d')
tweets_df = tweets_df[['screen_name',
'date','text']].reset_index(drop=True)
tweets_df = tweets_df.apply(pd.to_numeric, errors='ignore')

From the tweet, there is a lot of figures we can extract. Let’s try to get the death rate as
an example. The way that I calculate the death rate is (cumulative death) / (the
number of positive cases).

https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 5/8
6/7/2020 Customise your own Twitter notification with Telegram Bot

You
To can
make find awork,
Medium lot ofweregex pattern
log user data. Byonline. I usedyou
using Medium, theagree
following pattern
to our Privacy to match
Policy, all the
including
cookie
numberpolicy.
within the tweet to extract the figures as two new columns, ‘death total’ and
‘positive total’.

number_regex = '\d+(?:,\d+)*'
tweets_df['death_total'] =
tweets_df.text.str.findall(number_regex).str[8]
tweets_df['positive_total'] =
tweets_df.text.str.findall(number_regex).str[5]

We can now create a column called ‘death_rate’ by putting the figures into the formula.
We can use the similar ways to extract other figures. But for now, let’s add data
transparency by extracting the url with another regex.

url_regex = '(?:(?:https?|ftp):\/\/)?[\w/\-?=%.]+\.[\w/\-?=%.]+'
tweets_df['url'] =
tweets_df.text.str.findall(url_regex).apply(''.join)

Your dataframe shouold look like this.

To draft up the telegram message, we need to extract the numbers from the dataframe.

death_total = format(tweets_df.death_total.iloc[0], ',')


death_rate = round(tweets_df.death_rate.iloc[0]*100, 1)
url = tweets_df['url'].iloc[0]

With the variables, we can start putting them into a template.

msg = 'On {0}, the death toll has reached {1}, the death rate thus
far is {2}. {3}'.format(latest_date, \
death_total, death_rate, url)

https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 6/8
6/7/2020 Customise your own Twitter notification with Telegram Bot

3. Send telegram message via Python


To make Medium work, we log user data. By using Medium, you agree to our Privacy Policy, including
cookie policy.
We have the message now! We need to figure out how we can send that msg to
telegram with Python. To start, we need to create a telegram bot.

1. On Telegram, search for ‘BotFather’ and send a ‘/start’ message.

2. To create new bot, Send another ‘/newbot’ message, then follow the instructions to
configure a name and a username. Your user name will be the bot_chatID.

3. send ‘/token’ to get the token that access to HTTP API and then save it as
bot_token.

What we are doing here is to use bot to send message to a target channel. So the
bot_chatID should be the channel name you create, starting with ‘@’. Remember
to make sure you bot is the admin of the channel. Otherwise, the bot allowed to
publish content to the channel.

import requests

def telegram_bot_sendtext(bot_token, bot_chatID, bot_message):


send_text = 'https://api.telegram.org/bot' + bot_token +
'/sendMessage?chat_id=' + bot_chatID + '&parse_mode=Markdown&text='
+ bot_message
response = requests.get(send_text)

Run the function, and voila! You will see the message is being sent to telegram!

4. Running the script recurringly


You can try to run the script on a daily basis by setting up a crontab. You can customise
the schedule of your choice, there is a very good tutorial on how to do that here.

00 21 * * * /path/to/script/.sh

For example, the above crontab is set to automatically run the script every day at 9pm.
https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 7/8
6/7/2020 Customise your own Twitter notification with Telegram Bot

Alternatively,
To you can
make Medium work, also
we log run
user theByscript
data. every five
using Medium, you minutes and
agree to our sendPolicy,
Privacy out aincluding
message
cookie
once policy.
the daily figures are released. That would need another tutorial. But you can
check out how I do it on my github.

That’s it — now you have written a python script that scrapes tweets, analyses data,
and automatically sent message to a telegram channel.

Python Telegram Twitter Web Scraping Cron

About Help Legal

Get the Medium app

https://medium.com/swlh/customise-your-own-twitter-notification-with-telegram-bot-f7bf95b6f9f7 8/8

Вам также может понравиться