Flask app 4 - word counts with AngularJS polling the back-end
In the previous section (Flask app 3 : word counts with Redis task queue), we setup the Redis task queue. In this tutorial, we'll use AngularJS to poll the back-end to see if the task is complete and the data is made available. Then, update the DOM.
Github source : akadrone-flask
To use Angular, we need to add the following to templates/index.html:
<script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script>
We also need to add the following directives as well:
- ng-app:
- ng-controller:
- ng-submit:
So, we bootstrapped Angular. In other words, we're telling Angular to treat this HTML document as an Angular application. We'll add a controller, and then a function called getResults() which will be triggered on the form submission.
Here is the code:
<!DOCTYPE html> <html ng-app="WordcountApp"> <head> <title>Wordcount</title> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen"> <style> .container { max-width: 1000px; } </style> </head> <body ng-controller="WordcountController"> <div class="container"> <div class="row"> <div class="col-sm-5 col-sm-offset-1"> <h1>Wordcount</h1> <br> <form role="form" ng-submit="getResults()"> <div class="form-group"> <input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;"> </div> <button type="submit" class="btn btn-default">Submit</button> </form> <br> {% for error in errors %} <h4>{{ error }}</h4> {% endfor %} <br> </div> <div class="col-sm-5 col-sm-offset-1"> {% if results %} <h2>Frequencies</h2> <br> <div id="results"> <table class="table table-striped" style="max-width: 300px;"> <thead> <tr> <th>Word</th> <th>Count</th> </tr> </thead> {% for result in results%} <tr> <td>{{ result[0] }}</td> <td>{{ result[1] }}</td> </tr> {% endfor %} </table> </div> {% endif %} </div> </div> </div> <br><br> <script src="//code.jquery.com/jquery-1.11.0.min.js"></script> <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script> <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script> <script src="{{ url_for('static', filename='main.js') }}"></script> </body> </html>
Create a static directory, and then add a file main.js to that directory.
Here is the basic code:
(function () { 'use strict'; angular.module('WordcountApp', []) .controller('WordcountController', ['$scope', '$log', function($scope, $log) { $scope.getResults = function() { $log.log("test"); }; } ]); }());
When the form is submitted, getResults() is called. This function simply logs the text "test" to the JavaScript console in the browser as shown in the picture below:
In the code, we're using Dependency Injection (DI) to "inject" the $scope object and $log service.
The injected $scope is a way of communicating between the View and Controller.
Angular takes this approach because it allows us to lessen the worries of dependencies and to focus on unit-testing. We can, for example, configure the DI framework to use mock-objects for underlying components instead of real services during our unit tests.
So, when we change a variable attached to $scope in Controller, the variable in View will automatically updated vice versa (data binding).
We'll use the Angular $http service to handle "POST" request asynchronously. With this, the form submission no longer sends a POST request to the back end.
Here is the updated static/main.js:
(function () { 'use strict'; angular.module('WordcountApp', []) .controller('WordcountController', ['$scope', '$log', '$http', function($scope, $log, $http) { $scope.getResults = function() { $log.log("test"); // get the URL from the input var userInput = $scope.url; // fire the API request $http.post('/start', {"url": userInput}). success(function(results) { $log.log(results); }). error(function(error) { $log.log(error); }); }; } ]); }());
We need to update the input element in index.html.
From:
<input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;">
To:
<input type="text" ng-model="url" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;">
With the injected $http service, we grabbed the URL from the input box (via ng-model="url"), and then issued a POST request to the back-end. The success and error callbacks handle the response. In the case of a 200 response, it will be handled by the success handler, and logs the response to the console.
Before we test our app again, we need to refactor the back-end to put the /start endpoint.
We want to refactor the Redis job creation from the index view function, and then add it to a new view function called get_counts() in aka.py:
:import os import requests import operator import re import nltk import json from flask import Flask, render_template, request from flask.ext.sqlalchemy import SQLAlchemy from stop_words import stops from collections import Counter from bs4 import BeautifulSoup import rq from rq import Queue from rq.job import Job from worker import conn from flask import jsonify app = Flask(__name__) app.config.from_object('config') app_settings = app.config['APP_SETTINGS'] app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://localhost/wordcount_dev' db = SQLAlchemy(app) q = Queue(connection=conn) from models import * def count_and_save_words(url): errors = [] try: r = requests.get(url) except: errors.append( "Unable to get URL. Please make sure it's valid and try again." ) return {"error": errors} # text processing raw = BeautifulSoup(r.text).get_text() nltk.data.path.append('./nltk_data/') # set the path tokens = nltk.word_tokenize(raw) text = nltk.Text(tokens) # remove punctuation, count raw words nonPunct = re.compile('.*[A-Za-z].*') raw_words = [w for w in text if nonPunct.match(w)] raw_word_count = Counter(raw_words) # stop words no_stop_words = [w for w in raw_words if w.lower() not in stops] no_stop_words_count = Counter(no_stop_words) # save the results try: result = Result( url=url, result_all=raw_word_count, result_no_stop_words=no_stop_words_count ) db.session.add(result) db.session.commit() return result.id except: errors.append("Unable to add item to database.") return {"error": errors} @app.route('/', methods=['GET', 'POST']) def index(): return render_template('index.html') @app.route('/start', methods=['POST']) def get_counts(): # get url data = json.loads(request.data.decode()) url = data["url"] if 'http://' not in url[:7]: url = 'http://' + url # start job job = q.enqueue_call( func=count_and_save_words, args=(url,), result_ttl=5000 ) # return created job id return job.get_id() @app.route("/results/<job_key>", methods=['GET']) def get_results(job_key): job = Job.fetch(job_key, connection=conn) print 'job=',job if job.is_finished: result = Result.query.filter_by(id=job.result).first() results = sorted( result.result_no_stop_words.items(), key=operator.itemgetter(1), reverse=True )[:10] return jsonify(results) else: return "Nay!", 202 if __name__ == '__main__': app.run()
If we submit a new URL, we can see the job id in our JavaScript console:
Here is the main.js where we updated the controller:
(function () { 'use strict'; angular.module('WordcountApp', []) .controller('WordcountController', ['$scope', '$log', '$http', '$timeout', function($scope, $log, $http, $timeout) { $scope.getResults = function() { $log.log('test'); // get the URL from the input var userInput = $scope.url; // fire the API request $http.post('/start', {'url': userInput}). success(function(results) { $log.log(results); getWordCount(results); }). error(function(error) { $log.log(error); }); }; function getWordCount(jobID) { var timeout = ''; var poller = function() { // fire another request $http.get('/results/'+jobID). success(function(data, status, headers, config) { if(status === 202) { $log.log(data, status); } else if (status === 200){ $log.log(data); $timeout.cancel(timeout); return false; } // continue to call the poller() function every 2 seconds // until the timeout is cancelled timeout = $timeout(poller, 2000); }); }; poller(); } }]); }());
Note that we injected the $timeout service into the controller.
If HTTP request is successful, the code will call the getWordCount() function.
Within the poller() function, we called the /results/job_id endpoint.
timeout = $timeout(poller, 2000);
Let's test the update. We may see something like this in Javascript console:
Via the $timeout service, the poller() function continues to fire every 2 seconds until the timeout is cancelled when a 200 response is returned along with the word counts:
In our case, we get 202 response once and then got 200 and word counts.
We'll add a required attribute to the input tag, which indicate that the input box must be filled out before the form can be submitted.
Our new polling will be handled completely on the client side using Angular tags utilizing double curly braces, {{}}. So, we don't need the the server side Jinja2 template tags.
<!DOCTYPE html> <html ng-app="WordcountApp"> <head> <title>Wordcount</title> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen"> <style> .container { max-width: 1000px; } </style> </head> <body ng-controller="WordcountController"> <div class="container"> <div class="row"> <div class="col-sm-5 col-sm-offset-1"> <h1>Wordcount</h1> <br> <form role="form" ng-submit="getResults()"> <div class="form-group"> <input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;" ng-model="url" required> </div> <button type="submit" class="btn btn-default">Submit</button> </form> <br> {% for error in errors %} <h4>{{ error }}</h4> {% endfor %} <br> </div> <div class="col-sm-5 col-sm-offset-1"> <h2>Frequencies</h2> <br> {% raw %} <div id="results"> {{wordcounts}} </div> {% endraw %} </div> </div> </div> <br><br> <script src="//code.jquery.com/jquery-1.11.0.min.js"></script> <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script> <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script> <script src="{{ url_for('static', filename='main.js') }}"></script> </body> </html>
As we can see from the code, we escaped the Jinja2 tags using {% raw %} and {% endraw %}.
Let's update the success handler in the poller() function in static/main.js:
(function () { 'use strict'; angular.module('WordcountApp', []) .controller('WordcountController', ['$scope', '$log', '$http', '$timeout', function($scope, $log, $http, $timeout) { $scope.getResults = function() { $log.log('test'); // get the URL from the input var userInput = $scope.url; // fire the API request $http.post('/start', {'url': userInput}). success(function(results) { $log.log(results); getWordCount(results); }). error(function(error) { $log.log(error); }); }; function getWordCount(jobID) { var timeout = ''; var poller = function() { // fire another request $http.get('/results/'+jobID). success(function(data, status, headers, config) { if(status === 202) { $log.log(data, status); } else if (status === 200){ $log.log(data); $scope.wordcounts = data; $timeout.cancel(timeout); return false; } // continue to call the poller() function every 2 seconds // until the timeout is cancelled timeout = $timeout(poller, 2000); }); }; poller(); } }]); }());
In the code, we attached the results to the $scope object so that it's available in the View.
Now we can see the $scope object (frequency data)in our view.
We need to reconstruct the return value like this:
['color',13] to {'color':13}
aka.py:
import os import requests import operator import re import nltk import json from flask import Flask, render_template, request from flask.ext.sqlalchemy import SQLAlchemy from stop_words import stops from collections import Counter from bs4 import BeautifulSoup import rq from rq import Queue from rq.job import Job from worker import conn from flask import jsonify from flask import Response app = Flask(__name__) app.config.from_object('config') app_settings = app.config['APP_SETTINGS'] app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = True app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql://localhost/wordcount_dev' db = SQLAlchemy(app) q = Queue(connection=conn) from models import * def count_and_save_words(url): errors = [] try: r = requests.get(url) except: errors.append( "Unable to get URL. Please make sure it's valid and try again." ) return {"error": errors} # text processing raw = BeautifulSoup(r.text).get_text() nltk.data.path.append('./nltk_data/') # set the path tokens = nltk.word_tokenize(raw) text = nltk.Text(tokens) # remove punctuation, count raw words nonPunct = re.compile('.*[A-Za-z].*') raw_words = [w for w in text if nonPunct.match(w)] raw_word_count = Counter(raw_words) # stop words no_stop_words = [w for w in raw_words if w.lower() not in stops] no_stop_words_count = Counter(no_stop_words) # save the results try: result = Result( url=url, result_all=raw_word_count, result_no_stop_words=no_stop_words_count ) db.session.add(result) db.session.commit() return result.id except: errors.append("Unable to add item to database.") return {"error": errors} @app.route('/', methods=['GET', 'POST']) def index(): return render_template('index.html') @app.route('/start', methods=['POST']) def get_counts(): # get url data = json.loads(request.data.decode()) url = data["url"] if 'http://' not in url[:7]: url = 'http://' + url # start job job = q.enqueue_call( func=count_and_save_words, args=(url,), result_ttl=5000 ) # return created job id return job.get_id() @app.route("/results/", methods=['GET']) def get_results(job_key): job = Job.fetch(job_key, connection=conn) if job.is_finished: result = Result.query.filter_by(id=job.result).first() results = sorted( result.result_no_stop_words.items(), key=operator.itemgetter(1), reverse=True )[:10] results_dict = {} for t in results: results_dict[t[0]] = t[1] return jsonify(results_dict) else: return "Nay!", 202 if __name__ == '__main__': app.run()
To display the dictionary properly on our view, we want to modify templates/index.html:
<!DOCTYPE html> <html ng-app="WordcountApp"> <head> <title>Wordcount</title> <meta name="viewport" content="width=device-width, initial-scale=1.0"> <link href="//netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet" media="screen"> <style> .container { max-width: 1000px; } </style> </head> <body ng-controller="WordcountController"> <div class="container"> <div class="row"> <div class="col-sm-5 col-sm-offset-1"> <h1>Wordcount</h1> <br> <form role="form" ng-submit="getResults()"> <div class="form-group"> <input type="text" name="url" class="form-control" id="url-box" placeholder="Enter URL..." style="max-width: 300px;" ng-model="url" required> </div> <button type="submit" class="btn btn-default">Submit</button> </form> <br> {% for error in errors %} <h4>{{ error }}</h4> {% endfor %} <br> </div> <div class="col-sm-5 col-sm-offset-1"> <h2>Frequencies</h2> <br> <div id="results"> <table class="table table-striped"> <thead> <tr> <th>Word</th> <th>Count</th> </tr> </thead> <tbody> {% raw %} <tr ng-repeat="(key, val) in wordcounts"> <td>{{key}}</td> <td>{{val}}</td> </tr> {% endraw %} </tbody> </table> </div> </div> </div> </div> <br><br> <script src="//code.jquery.com/jquery-1.11.0.min.js"></script> <script src="//netdna.bootstrapcdn.com/bootstrap/3.1.1/js/bootstrap.min.js"></script> <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.4.9/angular.min.js"></script> <script src="{{ url_for('static', filename='main.js') }}"></script> </body> </html>
In the next chapter, we'll update the front-end to make our page more user-friendly.
Ph.D. / Golden Gate Ave, San Francisco / Seoul National Univ / Carnegie Mellon / UC Berkeley / DevOps / Deep Learning / Visualization