A journey through the creation of an app with Node, MongoDB, React, Gulp

> node test.js 5
8.206 - starting task: 0
8.206 - starting task: 1
8.310 - starting task: 2
8.373 - starting task: 3
8.944 - starting task: 4
[ { task: 0, time: 927 },
  { task: 1, time: 110 },
  { task: 2, time: 61 },
  { task: 3, time: 556 },
  { task: 4, time: 520 } ]

Few things here:

• I’m using lodash to have some nice functions available such as _.range.
• I’m reading the arguments from the command line using process.argv, see the documentation.
• Tasks take a random time to execute. When done, they must call callback which is the argument passed by async to the function, to know when the task is done. It takes 2 arguments : error, and result. If the task sends out an error, async stops and triggers the complete callback (the 3rd argument of parallelLimit). If the task was successfull, you have a send a null, then your result.
• In the parallelLimit callback, result is an array and contains the result of the tasks in their exact order (even if the first task finished after every others, result[0] is the result of the first task).

So, to scrape 10 pages in parallel, just follow the recipe :

var scrapPage = function(pageNumber) {
    return function(callback) {
        request('http://www.example.com/page/' + pageNumber, function(error, response, html) {
            var $ = cheerio.load(html);
            // when using .map() on cheerio, call .get() at the end otherwise you'll end up with a cheerio collection, not a string array
            var links = $("a").map(function() { return { link: $(this).attr('href') }; })
                              .get();
            db.insert(links, function(error) {
                console.log(error);
            });
        });
    };
};
// create 10 tasks to scrape 10 pages
var scrapTasks = _.range(10).map(function(i) { return scrapPage(i); });
async.parallelLimit(scrapTasks, 3, function(error, result) {
    console.log(result);
});

I have the scraping process, and the database. Let’s do a small webservice to expose some data to be used in my frontend.

Expose your data

So, my node program is listening to request on port 3000 thanks to expressjs : app.listen(3000);. Notice that even after executing that, NodeJS still executes the code that follows, and does not exit the program at the end, it just waits.

Let’s add a route:

app.get('/data', function(req, res) {
    // look in the database for links containing 'policies'
    // order them alphatically
    // and takes the first 10
    var data = db.find({ link: { $regex: /.*policies.*/ } })
        .sort({ link: 1 })
        .limit(10)
        .exec(function(err, result) {
            res.send(result);
        });
});

We added a route /data (http://localhost:3000/data). When it’s requested, we are looking in our database for the word ‘policies’ in the property ‘link’ we saved. We are using the $regex operator, there is no $contains operator in NeDB/MongoDB. Then we sort the result alphabetically (you just put 1 or -1 to say ascending/descending), and we take only the first 10.

We execute the query and we wait for an answer (we provided a callback). When it’s back, we send the object directly into the response (from app.get(..., function(req, res)). That will automatically convert our object to a JSON string.

After scraping a google result page for instance, I can see that in my browser :

[{"link":"/intl/fr/policies/privacy/","_id":"RW0qTZ8CoZBlW77m"},
{"link":"/intl/fr/policies/privacy/","_id":"VUUJDM78V77KoFdw"},
{"link":"/intl/fr/policies/terms/","_id":"tUWqDKuYoDBDIx2u"},
{"link":"/intl/fr/policies/terms/","_id":"wLzt5gfyDh3ydPny"}]

The Content-Type of the response is automatically set Content-Type: application/json; charset=utf-8 when you send out an object.

So, the server side is ready. I can get some data, save them, and expose them through webservice calls. Now, I need the UI that display them. But, small aside about Node debugging first.

How to debug Node JS

In browser, it’s fairly easy to debug. When an exception occurs, the browser stops you directly where that happens with the current state. And you can use debugger to stop where you need (or alert(), let’s face it).

Multiple solutions exist to debug with NodeJS.

You can debug in the console using node debug program.js where you can go through each line, watch variables, put breakpoints etc. but that’s a bit hardcore, you only have a console UI. More info here.

> node debug test.js
< debugger listening on port 5858
connecting... ok
break in C:\Temp\test\test.js:1
  1 var express = require('express'); // routing
  2 var request = require('request');
  3 var async = require('async');
debug>

The best way is to use node-inspector. You can have the full Webkit debugger UI which is very good.

You can use debugger to stop wherever you want, exactly as in a browser.

To do that, you need 2 things :
– start node-inspector which basically handle the debugging UI :

> node-inspector --web-port=8200  test.js
Node Inspector v0.5.0
   info  - socket.io started
Visit http://127.0.0.1:8200/debug?port=5858 to start debugging.

– then open a browser (a webkit one, Chrome or Opera) and go to http://localhost:8200/debug

You will have an error : Error: connect ECONNREFUSED. Is node running with --debug port 5858 ? Because it has nothing to debug for now.

Start your node program in debug mode :

> node --debug test.js
debugger listening on port 5858

Then refresh your browser. Done.

Time to React

At the beginning, I did a non-React version of my app. A jQuery one, just to focus on the server-side first. It was fairly straightforward. Using routie to handle the routing, some $.ajax for the webservices, and some a simple strategy to render each time : .empty() / .append(). The data and design were set. Let’s move all that view to React. \o/ I learned a lot doing that the right way, even with everything I read/watched. Let’s share some tips !

First, let’s handle the Hello World example. Create a file HelloWorld.jsx (I put .jsx to be handle properly in Atom, but you could use .js too), add a reference to it in a index.html file, and, because the browser has no clue what to do with your jsx syntax, add a reference to an online transformer script (later, we will transform them offline using Gulp), and of course, to React.

var HelloWorld = React.createClass({
    render: function() {
        return 
Hello Ctheu
;
    }
});
React.render(, document.getElementById('container'));

When running, be sure to render your html page using a webserver, such as node. Your url must be something using the http:// protocol such as http://localhost:8090/test.html and not file:///C:/Temp/public/test.html, that won’t work. Something like that is enough :

var app = express();
app.use(express.static('public'));
app.listen(8090);

Otherwise you’ll get :

XMLHttpRequest cannot load file:///C:/Temp/public/HelloWorld.jsx. Cross origin requests are only supported for protocol schemes: http, data, chrome, chrome-extension, https, chrome-extension-resource.
Uncaught NetworkError: Failed to execute 'send' on 'XMLHttpRequest': Failed to load

From now, you can apply everything you know about React, create components, include them, render them. Let’s just focus on certain features. I won’t explain here how to React, you can read my other post.

React Tips

HTML to JSX Compiler
setState does not modify this.state
Why immutability is important ?
How to communicate between components ?
How to handle the data coming from a webservice ?
How to use a third party plugin that updates the DOM, such as Highcharts?
What about React/DOM Performance ?

HTML to JSX Compiler

If you want to try out what is the React syntax for a particular HTML or if you want to migrate your HTML to React, give a try at http://facebook.github.io/react/html-jsx.html

setState does not modify this.state

When you code a component and you need to access its state, ensure to directly override getInitialState() to return something. this.state is null by default. this.props is not (empty object).

When you update the state of a component with

setState({ done: true })

this.state.done is not modified on the next line. React throttles all the setState() calls and only apply them before rendering again. (to avoid a double render if you do setState({ done: true }); setState({ done: false }) in the same tick.

Why immutability is important ?

When you have to modify an array that belongs to your state, you can’t just push or splice values into/from it, that’s won’t trigger the re-rendering because React cannot just listen to an array changes, it’s not possible yet (it’s part of ES7 with Array.observe(arr, callback) but we are far from it, we are still mostly running with ES5 !). That’s why you have to call this.setState({...}) to tell React that something changed.

Thus, multiple ways to do that :

• Dirty way : you modify your array directly then you call setState saying a=a to trigger the re-rendering. The thing is that you changed the reference directly so, anything that is using this same reference won’t be updated and it’s view won’t be in sync with its state:

  this.state.mylist.push(5);
  this.setState({ mylist: this.state.mylist });
  

• you slice() your array to clone it, then modify it, then update the state :

  var copy = this.state.values.slice(); // this.state.values reference in NOT modified here
  copy.push(5);
  this.setState({ values: copy });
  

• the React way, using the immutability helpers from React.addons. That does the same thing for arrays but handle the updates to subobjects too. The syntax is inspired by MongoDB’s query language ({$push:..} {$set:...}).

  this.setState({ values: React.addons.update(this.state.values, { $push: [ 5 ] }) });
  

How to communicate between components ?

For a parent to talk to its child is fairly straightforward, you can simply use the props ()

A way for a child to talk to its parent is for the parent to pass a callback to its child. You can pass any type of object in the props of a component.

var Who = React.createClass({
    render: function() {
        return { this.props.name };
    }
});
var HelloYou = React.createClass({
    getInitialState: function() {
        return { clicks: 0 };
    },
    somethingHappened: function() {
        this.setState({ clicks: this.state.clicks + 1});
    },
    render: function() {
        return  
Hello .
                     You clicked on me : { this.state.clicks }
                
;
    }
});
React.render(, document.getElementById('container'));

Hello Ctheu. You clicked on me : 0
Hello Ctheu. You clicked on me : 1
Hello Ctheu. You clicked on me : 2

Here, renders a passing a callback. triggers this callback when the user clicks on the generated . “talks” to its parent.

If you need to communicate between 2 components that don’t have a parent-child relationship, React team suggests to implement your own global event system.

How to handle the data coming from a webservice ?

Then, come the asyncness when using your webservices. How to ?
Here is a boilerplate to use for components that needs to call a webservice when they are mount into the DOM.

var HelloWho = React.createClass({
    getInitialState: function() {
        return { loading: true, error: false, name: null };
    },
    componentDidMount: function() {
        this.callWS();
    },
    callWS: function() {
        $.ajax({
            url: 'http://api.randomuser.me/',
            dataType: 'json',
            context: this,
            success: function(data) {
                if (this.isMounted()) {
                    this.setState({ loading: false, name: data.results[0].user.username });
                }
            },
            error: function() {
                if (this.isMounted()) {
                    this.setState({ loading: false, error: true });
                }
            }
        });
    },
    render: function() {
        if (this.state.loading) {
            return 
Loading...
;
        }
        if (this.state.error) {
            return 
Error retrieving the data
;
        }
        return  
Hello {this.state.name}
;
    }
});

React.render(, document.getElementById('container'));

Because the component calls a webservice, we add 2 properties to its state loading and error, to render something different according to 3 states : at first, the component calls the webservice, we need to wait ; then it comes back either successfully, or in error. Using the state allows you to render something appropriate.

The method componentDidMount is called when the component is mounted into the DOM, perfect time to call the webservice. Because loading is set, render() displays the loading message. Then, the service answers and we update its state accordingly, for render() to occur again.
In the callbacks of $.ajax, don’t forget to check if your component is still mounted otherwise you will get this error :

Uncaught Error: Invariant Violation: replaceState(...): Can only update a mounted or mounting component

How to use a third party plugin that updates the DOM, such as Highcharts?

I wanted to do a chart remember ? To create it, Highcharts needs a reference to a container in the DOM. It turns out that you have access to the DOM reference of what you are rendering in React. To do that, you use this.refs. It contains every ref you have set in the DOM you generated.

But you can’t add your Highcharts code in render() because the referenced items are even not rendered yet.
What you return in render() is not a DOM element. It’s just a JS object React is using. return