Hi all,
Today i am uploading one of the new fun project i was working on.
As i am big fan of Manga, and its hard to keep track of them, when new ones are available.
So, i decided to create a RSS feeder.
What You'll learn,
- How to create a simple web app using FLASK.
- Using multi-threading for parallel processing.
- DB operations on MY-SQL DB, it's what i have used.
Code Snippet :-
Below is the snippet of the project file structure.
HTML :-
1) layout.html :- This is the backbone of the application design.
2) home.html :- Home page of the application.
3) article.html :- this one the list of the rss feeds.
4) includes folder consist app navigation bar and messages(to send flash notification).
Python :-
1) add_rss.py is the one where mostly everything is happening.
Importing functions from FLASK and SQLAlchemy.
#for flask and its functionality
from flask import Flask, render_template, flash, redirect, url_for, session, logging, request
#for mysql connection
from flask_sqlalchemy import SQLAlchemy
#for reading rss feeds
import feedparser
#for parallel processing
import threading
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['SQLALCHEMY_ECHO'] = True
#do set your My-SQL username and password, also make sure to create the DB as given in DB.png.
db = SQLAlchemy(app)
url_data = []
#Function i am using to get the data from feed.
def get_source(feed):
return {
'title' : feed['title'],
'link' : feed['link'],
'published' : feed['published']
}
#function to get feed from URL.
def get_feed(db_data):
rss = db_data.url
feed = feedparser.parse(rss)
feed = feed["entries"][0]
new_feed = get_source(feed)
new_feed['name'] = db_data.name
if db_data.title != new_feed.get('title'):
db_data.title = new_feed['title']
db_data.link = new_feed['link']
db_data.published = new_feed['published']
new_feed['is_read'] = 1
#update the row
db.session.commit()
url_data.append(new_feed)
return
#DB model of my DB
class rss_model(db.Model):
__tablename__ = "rss"
id = db.Column(db.Integer, primary_key=True)
url = db.Column(db.Unicode)
name = db.Column(db.Unicode)
title = db.Column(db.Unicode)
link = db.Column(db.Unicode)
published = db.Column(db.Unicode)
def __init__(self,url,name, is_read=0):
self.url = url
self.name = name
self.is_read = is_read
#Route of home page. this is how to tell application, which page to load.
@app.route('/')
def show_all():
return render_template('home.html')
#Function to get the feed, read the db and run the threads, so mostly everything is happening.
@app.route('/article')
def articles():
global url_data
url_data = []
#Query to get all data from DB.
data = rss_model.query.all()
#empty list of same size as the number of rows fetched from DB.
t = [None] * len(data)
for i in range(len(data)):
# starting the same number of thread as the rows, so the processing will be faster.
t[i] = threading.Thread(target=get_feed, args=(data[i],))
print ("\n\n", "data:",data[i], "\n\n", file=sys.stdout)
try:
t[i] = threading.Thread(target=get_feed, args=(data[i],))
t[i].start()
except:
print ("Error: unable to start thread", file=sys.stdout)
for i in range(len(data)):
#here we are waiting for all the threads to complete.
t[i].join()
return render_template('article.html', chapters = url_data)
#function to add new RSS feed.
@app.route('/add_rss', methods=['GET','POST'])
def add_rss():
if request.method == 'POST':
name = request.form['name']
url = request.form['url']
new_rss = rss_model(name = name, url = url)
db.session.add(new_rss)
db.session.commit()
db.session.flush()
msg="New Manga Added"
return render_template('home.html',msg=msg)
return render_template('add_rss.html')
#Setting debug to true so, the changes reflect ASAP in application, and manually setting the port to 5876.
if __name__ == '__main__':
app.run(debug=True, port=5876)
#end of the rss_app.py
rss_python.sh
Just a script to start the application. i am using this in my bashrc to start it with alias.
ps_out=`ps -ef | grep $1 | grep -v 'grep' | grep -v $0`
result=$(echo $ps_out | grep "$1")
if [ "$result" != "" ];then
echo "Server already Running"
else
~/<path to virtualenv python>/python3 ~/<path to file>/rss_app.py &>/dev/null &
fi
FYI :- you can clone the repo from
https://github.com/Shapboyz13/rssFeeder
Today i am uploading one of the new fun project i was working on.
As i am big fan of Manga, and its hard to keep track of them, when new ones are available.
So, i decided to create a RSS feeder.
What You'll learn,
- How to create a simple web app using FLASK.
- Using multi-threading for parallel processing.
- DB operations on MY-SQL DB, it's what i have used.
Code Snippet :-
Below is the snippet of the project file structure.
HTML :-
1) layout.html :- This is the backbone of the application design.
2) home.html :- Home page of the application.
3) article.html :- this one the list of the rss feeds.
4) includes folder consist app navigation bar and messages(to send flash notification).
Python :-
1) add_rss.py is the one where mostly everything is happening.
Importing functions from FLASK and SQLAlchemy.
#for flask and its functionality
from flask import Flask, render_template, flash, redirect, url_for, session, logging, request
#for mysql connection
from flask_sqlalchemy import SQLAlchemy
#for reading rss feeds
import feedparser
#for parallel processing
import threading
#DB configuration
app.config['SQLALCHEMY_DATABASE_URI'] = "mysql://<UserName>:<Password>@localhost/RSS_python"app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['SQLALCHEMY_ECHO'] = True
#do set your My-SQL username and password, also make sure to create the DB as given in DB.png.
db = SQLAlchemy(app)
url_data = []
#Function i am using to get the data from feed.
def get_source(feed):
return {
'title' : feed['title'],
'link' : feed['link'],
'published' : feed['published']
}
#function to get feed from URL.
def get_feed(db_data):
rss = db_data.url
feed = feedparser.parse(rss)
feed = feed["entries"][0]
new_feed = get_source(feed)
new_feed['name'] = db_data.name
if db_data.title != new_feed.get('title'):
db_data.title = new_feed['title']
db_data.link = new_feed['link']
db_data.published = new_feed['published']
new_feed['is_read'] = 1
#update the row
db.session.commit()
url_data.append(new_feed)
return
#DB model of my DB
class rss_model(db.Model):
__tablename__ = "rss"
id = db.Column(db.Integer, primary_key=True)
url = db.Column(db.Unicode)
name = db.Column(db.Unicode)
title = db.Column(db.Unicode)
link = db.Column(db.Unicode)
published = db.Column(db.Unicode)
def __init__(self,url,name, is_read=0):
self.url = url
self.name = name
self.is_read = is_read
#Route of home page. this is how to tell application, which page to load.
@app.route('/')
def show_all():
return render_template('home.html')
#Function to get the feed, read the db and run the threads, so mostly everything is happening.
@app.route('/article')
def articles():
global url_data
url_data = []
#Query to get all data from DB.
data = rss_model.query.all()
#empty list of same size as the number of rows fetched from DB.
t = [None] * len(data)
for i in range(len(data)):
# starting the same number of thread as the rows, so the processing will be faster.
t[i] = threading.Thread(target=get_feed, args=(data[i],))
print ("\n\n", "data:",data[i], "\n\n", file=sys.stdout)
try:
t[i] = threading.Thread(target=get_feed, args=(data[i],))
t[i].start()
except:
print ("Error: unable to start thread", file=sys.stdout)
for i in range(len(data)):
#here we are waiting for all the threads to complete.
t[i].join()
return render_template('article.html', chapters = url_data)
#function to add new RSS feed.
@app.route('/add_rss', methods=['GET','POST'])
def add_rss():
if request.method == 'POST':
name = request.form['name']
url = request.form['url']
new_rss = rss_model(name = name, url = url)
db.session.add(new_rss)
db.session.commit()
db.session.flush()
msg="New Manga Added"
return render_template('home.html',msg=msg)
return render_template('add_rss.html')
#Setting debug to true so, the changes reflect ASAP in application, and manually setting the port to 5876.
if __name__ == '__main__':
app.run(debug=True, port=5876)
#end of the rss_app.py
rss_python.sh
Just a script to start the application. i am using this in my bashrc to start it with alias.
ps_out=`ps -ef | grep $1 | grep -v 'grep' | grep -v $0`
result=$(echo $ps_out | grep "$1")
if [ "$result" != "" ];then
echo "Server already Running"
else
~/<path to virtualenv python>/python3 ~/<path to file>/rss_app.py &>/dev/null &
fi
FYI :- you can clone the repo from
https://github.com/Shapboyz13/rssFeeder
Hope you'll enjoy it,
Comments, review, questions and suggestion are always welcome,
you can also connect me here. shashank17nov@gmail.com
And at last, as always thanks to all the stack overflow sources for the help.
