Chris Essig

Walkthroughs, tips and tricks from a data journalist in eastern Iowa

Exploring D3.js

leave a comment »

network_graph_mary_ssI’ve been wanting to use the D3.js library for some time now but I haven’t up to this point mostly because the code that runs the charts scares the crap out of me (that’s only a slight exaggeration).

Well this weekend I finally took on the scary and published my first D3 chart. The final product is a network graph that shows how one woman is helping connect people within the community. It’s not crazy awesome like most of the other D3 graphics out there but it’s a start.

The code started out as this chart from Mike Bostock and was modified to meet my needs. The final code is on GitHub.

While it wasn’t quite as intimidating as I thought it would be, a few things tripped me up. Chaining methods in D3 was something that threw me off at first. I was also unfamiliar with SVGs. This is a good primer from Scott Murray. Finally, many of the D3 methods are still mysterious me, like diagonal and the layouts. But one step at a time, right?

Some of it, however, isn’t crazy complicated. I’m going to review some of what I learned about D3 by going through some of the code that powered this project.

The code snippet below is line 47 in the chart’s script.js file. It starts the graph. It selects the empty SVG that I put in index.html, sets it’s width and height and appends a G element to it:

// Create an svg
  svg = d3.select("svg")
  	.attr("width", width + margin.right + margin.left)
  	.attr("height", height + margin.top + margin.bottom)
  	.append("g")
  	.attr("transform", "translate(" + (margin.left - line_width) + "," + margin.top + ")");

A few lines down, we create a variable called node, select all the G elements and bind data to them.

// Update the nodes
  var node = svg.selectAll("g.node")
  	.data(nodes, function(d) {
      return d.id || (d.id = ++i);

  	})

This G element contains everything in the chart and is placed just under the empty svg element in index.html

We then create a variable called nodeEnter, which appends another G element to the G element we just created. (confusing, I know).

// Enter any new nodes at the parent's previous position.
  var nodeEnter = node.enter().append("g")
    	// Set location of nodes
    	.attr("transform", function(d) {
    		if ( d['name'] === 'Mary Palmberg' && d['depth'] === 2) {
    			return "translate(" + circle_mary_location + "," + circle_mary_translate + ")";
        } else {

          
          if ( d['name'] === 'Thomas Jensen' ) {
            d.x -= 20;
          } else if ( d['name'] === 'Clarence Borck' ) {
            d.x += 20;
          } else if ( d['name'] === 'Jane Skinner' && d['connection'] === 5 ) {
            d.x -= 20;
          }
    			
          return "translate(" + d.y + "," + d.x + ")";
    		}
    	})
    	.attr("class", function(d) {
    		if (d['name'] === 'Mary Palmberg') {
    			return 'node-mary';
    		} else if (d['name'] === 'Top level') {
    			return 'node-top';
    		} else {
    			return 'node'
    		}
    	});

In our chart, these G elements are the nodes that contain our circles. Later in the code, I made lines that connect the circles. But we’re going to focus on just these G elements.

It may make more sense to inspect the graph to see what I’m talking about:

d3_mary_blog

After we append the G elements to their parent G elements, we chain a series methods to it, which set attributes for the G elements. These methods are called D3 selectors. With D3 selectors, you can select any shape in the SVG and a do a bunch of things to it. We can append new nodes to it, set CSS properties, give it a certain class name, etc. And since we bound data to the shape, we can decide what those properties will be based on the data we gave it.

This is seriously awesome, by the way.

In the code above, we used a D3 selector to set the location of the shapes using the CSS attribute transform. You’ll notice how setting that attribute is a function with a “d” parameter. “D” is equal to the data behind the shape we are drawing (remember, we appended data to it earlier). It returns an object with different properties. In this case, the property “name” is equal to the name of the person we are drawing in the circle.

For this chart, I had 16 G elements on the page. I also have objects with properties about each G element in a separate JSON file. As we loop through each G element and create it on the page, the data parameter becomes whichever G element we are on. So when we are creating Mary Palmberg’s circle, the “D” parameter will equal data about Mary Palmberg.

“D” returns an object that looks like so:

d3_mary_blog02

In the code, I’m checking to see if the G element we are on is Mary Palmberg, because her circle is placed in the middle and all other circles point to her. If we are on Mary, we set the CSS property transform to the variables called “circle_mary_location” and “circle_mary_translate”, which I set earlier in the script.js file. These basically put her circle in the middle of the page.

If it’s not Mary, we run the else part of the if-else statement. We do a couple of checks on a few people and change their x values. This is done because some of the circles were a little bit too close to together. After that, I put the data’s x and y coordinates in CSS translate property and return it. This basically puts it in the correct position on the page.

The last part should be easy to understand. All it does is set a class for the G Element depending on the person’s name. We give Mary a different class name so we can apply different styles to her circle:

d3_mary_blog03

It’s important to note that we haven’t actually created any shapes yet. Instead, G elements are just groups of shapes. The next thing we’re going to do is actually append circles to the empty G element.

nodeEnter.append("circle")
    	.attr("id", "circle-outer")
    	// Make radius smaller than inner circle
    	.attr("r", function(d) {
    		if (d['name'] === 'Mary Palmberg') {
    			if ( $(window).width() > 700) {
                    return circle_radius + 7;
                } else {
                    return circle_radius + 17;
                }
    		} else {
    			return circle_radius + 2;
    		}
    	})
    	// Color code the border around the circle
    	.style('stroke', function(d) {
    		// Color Mary differently
    		if (d['name'] === 'Mary Palmberg') {
    			// Red
    			return '#e41a1c'
    		} else if (d['connection'] === 1) {
    			// Purple
    			return "#984ea3";
    		} else if (d['connection'] === 2) {
    			// Green
    			return "#4daf4a";
    		} else if (d['connection'] === 3) {
    			// Blue
    			return "#377eb8";
    		} else if (d['connection'] === 4) {
    			// Orange
    			return "#ff7f00";
    		} else if (d['connection'] === 5) {
    			// Brown
    			return "#a65628";
    		} else {
    			// Blue
    			return "#377eb8";
    		}
    	});

  // Create the inner circle
  nodeEnter.append("circle")
    	.attr("id", "circle-inner")
    	// Radius
    	.attr("r", function(d) {
    		if (d['name'] === 'Mary Palmberg') {
    			if ( $(window).width() > 700) {
                    return circle_radius + 5;
                } else {
                    return circle_radius + 15;
                }
    		} else {
    			return circle_radius;
    		}
    	})
    	.style("fill", function(d) {
    		if ( $(window).width() > 700) {
                return "url(#" + d['name'].replace(' ', '') + ")"
            } else if ( d['name'] === 'Mary Palmberg') {
                return "url(#" + d['name'].replace(' ', '') + ")"
            }
    	});

I appended two circles to the G elements. The first is the circle around a person. I gave it an id of “circle-outer” and set it’s radius (attribute “r”) to a variable called “circle_radius”, which is set at the top of the script.js file. Once again I did a check for Mary and gave her a bigger circle since she’s in the middle. I then set the color of the circle’s stroke based on data property “connection”.

Our network graph has five total connections, so there’s a possibility of five different colors. I set the connection number in the JSON file. This is done so every connection through Mary is color-coded the same and thus the reader can see they are part of the same story. Same color = same story. And Mary stays in the middle of it all.

The second circle is the image itself. It has an id of “circle-id”. The attribute fill is set to the image of the person. The urls for each image are set in index.html under the defs tag. Each image is a pattern with a “xlink:href” property set to the url of the image. More information on how this works can be found here.

After that, I append text to have their name appear the circle. And I used this library to create the tooltips. I won’t bore you with that code as well.

There’s a few more things going on in this chart but those are the basics. Or at least as I understand them.

As with learning any new technology, my advice is to pick a project that you want to be powered by D3 and go at it. I learned a lot more about D3 making my first chart than I have with walkthroughs, video tutorials, etc. You’ll never learn it until you get your hands dirty.

Written by csessig

April 1, 2015 at 4:42 pm

Posted in d3, Javascript

Quick tip: My approach to media queries

leave a comment »

Anybody who has designed responsive websites knows that styling them is a delicate balance. You want to get the most of out of both your desktop and mobile experiences, as well as everything in between. On top of that, you’re trying to hit a ton of moving targets: Mobile and tablet sizes are always changing in size so designing for EVERY platform out there is very difficult to do.

With CSS, you target particular screen sizes — and therefore particular devices — with media queries.  I’ve seen a lot of developers write media queries to target particular devices based on their screen size. For instance, you might write a media query of 768 pixels wide to target an iPad. Or write a query of 667 pixels wide for the new iPhone 6.

The problem is: What happens when a hot, new device comes along with new dimensions? And what you wanted to design for all mobile platforms is suddenly obsolete because it’s wider or taller than the last mobile phone you were designing for?

This is why I approach media queries like so: I write a media query for every 50 pixels and simply make sure EVERYTHING works on every screen size of any pixel width. I will then add styling to only those 50-pixel benchmarks.

For instance, if the font in the header of my webpage gets too large at 630 pixels wide, I will modify to make the font smaller at 650 pixels wide. If an image is too big at 425 pixels wide, I will make it smaller at 450 pixels wide.

I also include classes that hide and show elements at each 50-pixel width to make it easier to hide and show things on the page at given widths. So, for instance “hide-400” will be a “display: none” property at 400 pixels wide. The inverse is “show-400”.

So my stylesheet ends of looking something like:

/* Styles for mobile devices */
@media (max-width: 450px) {
	.hide-450 {
		display: none;
	}

	.show-450 {
		display: block;
	}

	.show-450-inline {
		display: inline;
	}

	.header-sub-header p  {
        font-size: 13px;
        line-height: 30px;
    }
}

/* Styles from mobile devices */
@media (max-width: 400px) {
	.header h3 {
		line-height: 10px;
		font-size: 14px;
	}

	/* Universal classes */
	.hide-400 {
		display: none;
	}

	.show-400 {
		display: block;
	}

	.show-400-inline {
		display: inline;
	}
}

/* Styles for mobile devices */
@media (max-width: 350px) {
	.hide-350 {
		display: none;
	}

	.show-350 {
		display: block;
	}

	.show-350-inline {
		display: inline;
	}
}

That way if I need to hide and show something, all I need to do is add the appropriate class to it instead of going into the stylesheet every time. Bootstrap does something similar with their responsive utility classes. My approach is just more granular.

Now there is still a larger question: At what pixel width do you turn on your “mobile view” of your website (if you have one)? I use 600 pixels wide as a benchmark because that’ll target most mobile devices and give you extra room to cover the next, new device that is sure to be wider than the last. Also, articles on The Gazette are 620px wide, so I figure anything less than that I can consider a “mobile device”.

But, of course, this is all up for friendly debate.

Written by csessig

November 20, 2014 at 10:03 am

Leaflet formula: Counting markers within a radius

with 5 comments

ssThe last project I worked on at The Courier was a comprehensive guide to mental health and disability services in NE Iowa. For the project, we first wanted to explain the MHDS system to our readers and how it works. It’s a relatively new system with a lot of moving parts and no good way to search for services. We really wanted to help residents and families in need of these services, so we designed everything — from the wording we used to the way items were laid out — to be focused on our readers.

After laying out the details of the new system, we wanted to allow our readers to search for services near them. A map seemed like an obvious choice; but we wanted to do more than just have them search for their address and plot a dot on the map.

(As a side note, we used Ben Welsh‘s jquery-geocodify plugin to geocode the addresses because it’s awesome.)

Instead, we wanted to show readers how many providers were close to them, as well as how to get ahold of the service provider and directions to the facility. For the former task, we used the following process:

First, the providers are plotted on the map. Then, we ask the reader to type in their address, which is plotted on the map using jquery-geocodify. They also have a second option for radius, which will be used to determine how many providers are within whatever radius they have given. They have four options: 5, 10, 25 and 50 miles.

Then, as the markers are getting plotted on the map, we run a few Javascript functions to calculate how many markers on the map are within the radius.

I’ve pulled out just this part of the mental health project and open-sourced it. You can view the full code over at my Github page.

The demo uses data on mental health providers, which is contained in this JSON fileand plots them on the map based on each provider’s latitude and longitude coordinates.

Here’s a quick run through the Javascript that makes all this run. It’s contained within the project’s script.js file. We’ll start from the bottom of the file and move our way up.

Note: This walkthrough assumes a decent understanding of Leaflet. I’m also using Leaflet-Awesome Markers and Font Awesome in this project.

First, we’ll add the base map tiles to our map. We’ll also  add the json_group to the map, which is a FeatureGroup we will create later in the file:

// Base map
var layer = new L.StamenTileLayer('toner-background');
var map = new L.Map('map', {
    center: new L.LatLng(42,-93.3),
    minZoom: 4,
    maxZoom: 10,
    zoom: 6,
    keyboard: false,
    boxZoom: false,
    doubleClickZoom: false,
    scrollWheelZoom: false,
    maxBounds: [[33.154799,-116.586914],[50.190089,-77.563477]]
});
// Add base layer to group
map.addLayer(layer);
// Add our markers in our JSON file on the map
map.addLayer(json_group);

Next, we’ll add the code for jquery-geocodify. For the onSelect function, we will grab the location of reader’s address and then call a geocodePlaceMarkersOnMap function, which we will create later:

// jQuery Geocodify
var maxY = 43.749935;
var minY = 40.217754;
var minX = -96.459961;
var maxX = -90.175781;

var search_marker;
var search_icon = L.AwesomeMarkers.icon({
    icon: 'icon-circle',
    color: 'green'
});

$('#geocoder').geocodify({
    onSelect: function (result) {
        // Extract the location from the geocoder result
        var location = result.geometry.location;

        // Call function and place markers, circle on map
        geocodePlaceMarkersOnMap(location);
    },
    initialText: 'Zip code, city, etc...',
    regionBias: 'US',
    // Lat, long information for Cedar Valley enter here
    viewportBias: new google.maps.LatLngBounds(
        new google.maps.LatLng(40.217754, -96.459961),
        new google.maps.LatLng(43.749935, -90.175781)
    ),
    width: 300,
    height: 26,
    fontSize: '14px',
    filterResults: function (results) {
        var filteredResults = [];
        $.each(results, function (i, val) {
            var location = val.geometry.location;
            if (location.lat() > minY && location.lat() < maxY) {
                if (location.lng() > minX && location.lng() < maxX) {
                    filteredResults.push(val);
                }
            }
        });
        return filteredResults;
    }
});

Next, we’ll loop through our providers.json file, which has a variable of json_data, and create a marker for each object in our array. We’ll then add it to our json_group FeatureGroup, which is defined at the top of our script file:

// This loops through the data in our JSON file
// And puts it on the map
_.each(json_data, function(num) {
    var dataLat = num['latitude'];
    var dataLong = num['longitude'];

    // Add to our marker
    var marker_location = new L.LatLng(dataLat, dataLong);

    // Options for our circle marker
    var layer_marker = L.circleMarker(marker_location, {
        radius: 7,
        fillColor: "#984ea3",
        color: "#FFFFFF",
        weight: 1,
        opacity: 1,
        fillOpacity: 0.8
    });

    // Add events to marker
    layer_marker.on({
        // What happens when mouse hovers markers
        mouseover: function(e) {
            var layer_marker = e.target;
            layer_marker.setStyle({
                radius: 8,
                fillColor: "#FFFFFF",
                color: "#000000",
                weight: 1,
                opacity: 1,
                fillOpacity: 1
            });
        },
        // What happens when mouse leaves the marker
        mouseout: function(e) {
            var layer_marker = e.target;
            layer_marker.setStyle({
                radius: 7,
                fillColor: "#984ea3",
                color: "#FFFFFF",
                weight: 1,
                opacity: 1,
                fillOpacity: 0.8
            });
        }
    // Close event add for markers
    });

    json_group.addLayer(layer_marker);
// Close for loop
}, this);

The changeCircleRadius function is called anytime the radius is changed on the page. It first calls the pointsInCircle function, which we will define later, and then sets the radius of the circle using Leaflet’s setRadius function for circles.

Leaflet uses meters, so we will need to convert miles to meters using the milesToMeters function, which will be defined later. The mile’s value is contained within the DIV with the id of radius-selected. We’ll grab the value, which is the radius in miles, convert it to meters and then set the radius of the circle using Leaflet.

// Change circle radius when changed on page
function changeCircleRadius(e) {
    // Determine which geocode box is filled
    // And fire click event

    // This will determine how many markers are within the circle
    pointsInCircle(circle, milesToMeters( $('#radius-selected').val() ) )

    // Set radius of circle only if we already have one on the map
    if (circle) {
        circle.setRadius( milesToMeters( $('#radius-selected').val() ) );
    }
}

$('select').change(function() {
    changeCircleRadius();
});

The geocodePlaceMarkersOnMap function is called every time the reader enters an address into our geocoder and presses enter. It passes with it the location of their address, which we get from jquery-geocodify. With this function, we will create both a circle and a marker on top of the given address.

First, we set the view of the map based on this location and remove the circle on the map if there already is one. Then we actually create a circle and set how large it is based on the radius set by the user.

We then remove the marker if it’s already on the map and create a new one. When we create a new marker, we’ll set it to draggable and set its dragend method to reset the view of the map every time the reader drags the marker somewhere on the. It then calls the pointsInCircle function, which we will define next. Finally, we’ll call the same pointsInCircle function and add the marker to the map.

// This places marker, circle on map
function geocodePlaceMarkersOnMap(location) {
    // Center the map on the result
    map.setView(new L.LatLng(location.lat(), location.lng()), 10);

    // Remove circle if one is already on map
    if(circle) {
        map.removeLayer(circle);
    }
    
    // Create circle around marker with our selected radius
    circle = L.circle([location.lat(), location.lng()], milesToMeters( $('#radius-selected').val() ), {
        color: 'red',
        fillColor: '#f03',
        fillOpacity: 0.1,
        clickable: false
    }).addTo(map);
    
    // Remove marker if one is already on map
    if (search_marker) {
        map.removeLayer(search_marker);
    }
        
    // Create marker
    search_marker = L.marker([location.lat(), location.lng()], {
        // Allow user to drag marker
        draggable: true,
        icon: search_icon
    });

    // Reset map view on marker drag
    search_marker.on('dragend', function(event) {
        map.setView( event.target.getLatLng() ); 
        circle.setLatLng( event.target.getLatLng() );

        // This will determine how many markers are within the circle
        pointsInCircle( circle, milesToMeters( $('#radius-selected').val() ) );

        // Redraw: Leaflet function
        circle.redraw();

        // Clear out address in geocoder
        $('#geocoder-input').val('');
    });

    // This will determine how many markers are within the circle
    // Called when points are initially loaded
    pointsInCircle( circle, milesToMeters( $('#radius-selected').val() ) );

    // Add marker to the map
    search_marker.addTo(map);
        
// Close geocodePlaceMarkersOnMap
}

The pointsInCircle function is the last function called when one of two things happen: the user enters an address or the user drags the marker on the map. Its purpose is to find out how many markers on the map are within the circle we created and then display that number in the legend.

Fortunately, Leaflet has a really handy distanceTo method, which determines how many meters are within two points. It uses the two points’ latitude and longitude coordinates to find this out.

With the pointsInCircle function, we first capture the lat, long of the circle we’re putting on the map. We then loop through our providers.json file, find the latitude and longitude coordinates of each object and determine how far it is from our circle using the distanceTo method.

We then decide if that amount is higher or lower than the radius set by the user. If it is lower, we add it to a counter. When we’re done, that counter will equal the number of markers within the selected radius. It will be put on the map under the “Results” header.

Lastly, we will determine how many results we have and use the correct wording on the page. If we have one result, we’ll use the word “provider” (singular) in the map’s legend. Otherwise, we’ll use “providers.”

// This figures out how many points are within out circle
function pointsInCircle(circle, meters_user_set ) {
    if (circle !== undefined) {
        // Only run if we have an address entered
        // Lat, long of circle
        circle_lat_long = circle.getLatLng();

        // Singular, plural information about our JSON file
        // Which is getting put on the map
        var title_singular = 'provider';
        var title_plural = 'providers';

        var selected_provider = $('#dropdown_select').val();
        var counter_points_in_circle = 0;

        // Loop through each point in JSON file
        json_group.eachLayer(function (layer) {

            // Lat, long of current point
            layer_lat_long = layer.getLatLng();

            // Distance from our circle marker
            // To current point in meters
            distance_from_layer_circle = layer_lat_long.distanceTo(circle_lat_long);

            // See if meters is within raduis
            // The user has selected
            if (distance_from_layer_circle <= meters_user_set) {
                counter_points_in_circle += 1;
            }
        });

        // If we have just one result, we'll change the wording
        // So it reflects the category's singular form
        // I.E. facility not facilities
        if (counter_points_in_circle === 1) {
            $('#json_one_title').html( title_singular );
        // If not one, set to plural form of word
        } else {
            $('#json_one_title').html( title_plural );
        }
        
        // Set number of results on main page
        $('#json_one_results').html( counter_points_in_circle );
    }
// Close pointsInCircle
};

The final piece is to add the variables we’re using to the top of the script file. We’ll also add the milesToMeters function, which converts miles to meters.

// We'll append our markers to this global variable
var json_group = new L.FeatureGroup();
// This is the circle on the map that will be determine how many markers are around
var circle;
// Marker in the middle of the circle
var search_marker;
// Marker icon
var search_icon = L.AwesomeMarkers.icon({
    icon: 'icon-circle',
    color: 'red'
});


// Convert miles to meters to set radius of circle
function milesToMeters(miles) {
    return miles * 1069.344;
};

I hope this walkthrough helps fellow Leaflet users! If you have any questions, leave them in the comments.

Happy coding.

Written by csessig

June 22, 2014 at 1:32 pm

New adventures

leave a comment »

Some exciting personal news: I’ll be joining the Cedar Rapids Gazette and KCRG as their Interactive News Developer at the end of the month. My last day at the Waterloo-Cedar Falls Courier is Friday.

The new job will let me develop full time, meaning I will be coding more than I have. And I will continue to work in a newsroom, which is awesome. Needless to say, I’m very excited about the move.

I spent the last three years The Courier, and as this blog shows, I learned a ton in that time span. I will greatly miss working there. And more importantly, I’ve very, very proud of all the awesome work we did while I was there.

Here’s to new adventures!

Written by csessig

May 15, 2014 at 2:37 pm

How laziness led me to produce news apps quicker

with one comment

Note: This is cross-posted from Lee’s data journalism blog, which you can read over there by clicking here.

ssAn important trait for programmers to adopt, especially for those who are writing code on deadline, is laziness. While that sounds counter-productive, it can be incredibly beneficial.

When writing code, you  do a lot of repetitive tasks. If you’re using a Javascript library for mapping, for instance, you’ll need to copy and paste the Javascript and CSS files into a new directory every time you start a new project. If you want to use plugins with that map, you will need to include their files in the directory as well.

The more complicated your project, the more libraries you will likely use, which means a lot of tedious work putting all those files in the new directory.

And that’s before you even start to code. If you want your projects to have a similar look to them (which we do at the Courier), you will need to create code that you can to replicate in the future. And then every time you want to use it, you will need to rewrite the same code or, at the very least, copy code from your old projects and paste it into your new project.

I did this for a long time. Whenever I was creating  a new app, I’d open a bunch of projects I’ve already completed, go into their CSS and JS files, copy some of it and paste it into the new files. It was maddening because I knew with every new project, I would waste hours basically replicating work I had already done.

This is where laziness pays off. If you are too lazy to do the same things over and over again, you’ll start coming up with solutions that automatically does these repetitive tasks.

That’s why many news apps teams have created templates for their applications. When they create new projects, they start with this template. This allows their apps to have a similar look. It also means they don’t have to do the same things over and over again every time they are working with a project.

App templating is something I’ve dabbled with before. But those attempts were minor. For the most part, I did a lot of the same mind-numbing tasks every time I wanted to create a map, for instance.

Fortunately I found a good solution a few months ago: Yeoman, which is dubbed “the web’s scaffolding tool for modern web apps,” has helped me eliminate much of that boring, repetitive work.

Yeoman is built in Node. Before I stumbled upon it a few months ago, I knew very little about Node. I still don’t know a ton but this video helped me grasp the basics. One advantage of Node is it allows you to run Javascript on the command line. Pretty cool.

Yeoman is built on three parts. The first is yo, which helps create new applications and all the files that go with it. It also uses Grunt, which I have come to love. Grunt is used for running tasks, like minifying code, starting a local server so you test your code, copy files into new directories and a whole bunch of other things that makes coding quicker, easier and less repetitive.

Finally, Yeoman uses Bower, which is a directory of all the useful packages you use with applications, from CSS frameworks to Javascript mapping libraries. With a combination of Bower and Grunt, you can download the files you need for your projects and put them in the appropriate directories using a Grunt task. Tasks are written in a Gruntfile and run on the command line. The packages you want to download from Bower and use in your project are stored in bower.json file.

In Yeoman, generators are created. A generator makes it easy to create new apps using a template that’s already been created. You can either create your own generator or use one of the 400+ community generators. The generators themselves are directories of files, like  images, JS files, CSS stylesheets, etc. The advantage of using generators is you can create a stylesheet, for instance, save it in the generator directory and use that same stylesheet with every new project you create. If you decide to edit it, those changes will be reflected in every new project.

When you use a generator, those files are copied into a new directory, along with Grunt tasks, which are great for when you are testing and deploying your apps. Yeoman also has a built-in prompt system that asks you a series of questions whenever you create an app. This helps copy over the appropriate files and code, depending on what you are trying to make.

I decided I would take a stab at creating a generator. The code is here, along with a lot more explanation on how to use it. Basically, I created it so it would be easier to produce simple apps. Whenever I create a new Leaflet map, for instance, I can run this and a sample map with sample data will be outputted into a new directory. I also have a few options depending on how I want to style the map. I also have options for different data types: GeoJSON, JSON with latitude and longitude coordinates and Tabletop. What files are scaffolded out depends on how you responded to the prompts that are displayed when you first fire up the generator.

There is also an option for non-maps. Finally, you can use Handlebars for templating your data, if that’s your thing.

My hope is to expand upon this generator and create more options in the future. But for now, this is easy way to avoid a bunch of repetitive tasks. And thus, laziness pays off.

If you are interested in creating your own generator, Yeoman has a great tutorial that you should check out.

Written by csessig

February 19, 2014 at 12:08 pm

Using hexagon maps to show areas with high levels of reported crime

leave a comment »

Screenshot: Violent crime in WaterlooCrime maps are a big hit at news organizations these days. And with good reason: Good crime maps can tell your readers a whole lot about crime not only in their city but their backyard.

For the last year and a half, we’ve been doing just that. By collecting data every week from our local police departments, we’ve been able to provide our community a great, constantly updated resource on crime in Waterloo and Cedar Falls. And we’ve even dispelled some myths on where crime happens along the way.

The process, which I’ve blogged about in the past, has evolved since the project was first introduced. But basically it goes like this: I download PDFs of crime reports from the Waterloo and Cedar Falls police departments every week. Those PDFs are scraped using a Python script and exported into an CSV. The script pulls out reports of the most serious crimes and disregards minor reports like assistance calls.

The CSV is then uploaded to Google spreadsheets as a back up. We also use this website to get latitude, longitude coordinates from the addresses in the Google spreadsheet. Finally, the spreadsheet is converted into JSON using Mr. Data Converter and loaded onto our FTP server. We use Leaflet to map the JSON files.

We’ve broken the map into months and cities, meaning we have a map and a corresponding JSON file for each month for both Waterloo and Cedar Falls. We also have JSON files with all of the year’s crime reports for both Waterloo and Cedar Falls.

The resulting maps are great for readers looking for detailed information on crime in their neighborhood. It also provides plenty of filters so readers can see what areas have more shootings, assaults, burglaries, etc.

But it largely fails to give a good overview of what areas of the cities had the most reported crimes in 2013 for two reasons: 1) Because of the sheer number of crime reports (3,684 reports in 2013 for Waterloo alone) and the presence of multiple reports coming from the same address, it’s nearly impossible to simply look at the 2013 maps and see what areas of the cities had the most reports, and; 2) The JSON files containing all of the year’s reports runs well on modern browsers but runs more slowly outdated browsers (I’m looking at you, Internet Explorer), which aren’t equipped to iterate over JSON files with thousands of items.

As a result, I decided to create separate choropleth maps that serve one purpose: To show what areas of the cities had the most reported crimes. The cities would be divided into sections and areas with more crimes would be darker on the map, while the areas with fewer crimes would be lighter. Dividing the city into sections would also mean the resulting JSON file would have dozens of items instead of thousands. The map would serve a simple purpose and load quickly.

What would the areas of the cities actually look like? I’ve been fascinated with hex maps since I saw this gorgeous map from the L.A. Times on 9/11 response times. The map not only looked amazing but did an incredibly effective job displaying the data and telling the story. While we want our map to look nice, telling  its story and serving our journalistic purpose will always be our No. 1 goal. Fortunately, hex maps do both.

And even more fortunately, the process of creating hex maps got a whole lot easier when data journalist Kevin Schaul introduced a command line tool called binify that turns dot density shapefiles into shapefiles with hexagon shapes. Basically the map is cut into hexagon shapes, and all the points inside each hexagon are counted. Then each hexagon is given a corresponding count value for the number points that fall in between its borders.

I used binify to convert our 2013 crime maps into hex maps. As I mentioned earlier, we started out with CSVs with all the crimes. My workflow for creating the hex maps and then converting them into JSON files for Leaflet is in a gist below. It utilizes the fantastic command line tool og2ogr, which is used to convert map files, and binify.

# We start with a CSV called "crime_master_wloo_violent.csv" that has violent crime reports in Waterloo.
# Included in csv are columns for latitude ("lat") and longitude ("long"):
https://github.com/csessig86/crime_map2013/blob/master/csv/crime_master_wloo_violent.csv

# Command to turn CSV into DBF with the same name
ogr2ogr -f "ESRI Shapefile" crime_master_wloo_violent.dbf crime_master_wloo_violent.csv

# Create a file called "crime_master_wloo_violent.vrt"
# Change "SrcLayer" to the name of the source
# Change OGRVRTLayer name to "wloo_violent"
# Add name of lat, long columns to "GeometryField"
<OGRVRTDataSource>
  <OGRVRTLayer name="wloo_violent">
    <SrcDataSource relativeToVRT="1">./</SrcDataSource>
    <SrcLayer>crime_master_wloo_violent</SrcLayer>
    <GeometryType>wkbPoint</GeometryType>
    <LayerSRS>WGS84</LayerSRS>
    <GeometryField encoding="PointFromColumns" x="long" y="lat"/>
  </OGRVRTLayer>
</OGRVRTDataSource>

# Create "violent" folder to stash shp file
mkdir violent

# Convert "vrt" file into shp and put it in our "violent" folder
ogr2ogr -f "ESRI Shapefile" violent crime_master_wloo_property.vrt

# Binify the "wloo_violent" shp into a shp called "wloo_violent_binify" with three options
# The first sets the number of hexagons across to 45
# The second sets the custom extent to a box around Waterloo
# Custom extent is basically the extent of the area that the overlay of hexagons will cover
# The third tells binify to ignore hexagons that have zero points
binify wloo_violent.shp wloo_violent_binify.shp -n 45 -E -92.431873 -92.260867 42.4097259 42.558403 --exclude-empty

# Convert binified shp into JSON file called "crime_data_review_violent"
ogr2ogr -f "GeoJSON" crime_data_review_violent.json wloo_violent_binify.shp

# Add this variable to JSON file
var crime_data_review_violent

# Here's the final files:
https://github.com/csessig86/crime_map2013/tree/master/shps/wloo_shps

# And our JSON file:
https://github.com/csessig86/crime_map2013/blob/master/JSON/crime_data_review_violent.json

# And the Javascript file that is used with Leaflet to run the map.
# Lines 113 through 291 are the ones related to pulling the data from the JSON files,
# coloring them, setting the legend and setting a mouseover for each hexagon
https://github.com/csessig86/crime_map2013/blob/master/js/script_map_review.js

# And the map online:
http://wcfcourier.com/app/crime_map2013/index_wloo.php

The above shows the process I took for creating our hex map of violent crimes in Waterloo. I duplicated this process three more times to get four maps in all. One showed property crimes in Waterloo, while the other two show violent and property crime in Cedar Falls.

* Note: For more information on converting shapefiles with ogr2ogr, check out this blog post from Ben Balter. Also, this cheat sheet of ogr2ogr command line tools was invaluable as well.

Another great resource is this blog post on minifying GeoJSON files from Bjørn Sandvik. While the post deals with GeoJSON files specifically, the process works with JSON files as well. Minifying my JSON files made them about 50-150 KB smaller. While it’s not a drastic difference, every little bit counts.

To get choropleth maps up and running in Leaflet, check out this tutorial. You can also check out my Javascript code for our crime maps.

One last note: The map is responsive. Check it out.

Written by csessig

January 11, 2014 at 4:42 pm

How we used a Google spreadsheet to power our election app

with one comment

Election 2013 app screen shot

Last Tuesday was election night in the Cedar Valley. While we didn’t have a huge number of contested races on the ballot, we did have a mayor’s race in Waterloo and Waverly, Iowa, as well as city council races in several area towns.

As NPR’s Jeremy Bowers proudly proclaimed, election nights are exciting times for news nerds. This election, we decided to do a little bit of experimenting.

Before the election, we posted biographical information for all the candidates running in a contested race in Waterloo, Cedar Falls and Waverly. We also promoted other races in smaller towns. This gave our readers a good overview of the races on the ballot and information on the candidates. And best of all, it was all in one place.

Three reporters were responsible for getting the biographical information for each candidate and entering it online. We used a Google spreadsheet to store the information. This allowed the reporters to enter the information online themselves. I could then go into the spreadsheet, edit the text and make sure it was formatted correctly. We then used Tabletop.js to import the data into our app and Handlebars.js to template it.

The basic setup for the app is available on my Github page. This is a very similar setup mentioned in my last blog post.

Before the app went live, I exported all the data in the Google spreadsheet into JSON format using a method mentioned here. I did this for two reasons: 1) It spend up the load time of the app because browsers didn’t have to connect to the spreadsheet, download the data, format the data into JSON (which is what Tabletop.js does with the data in this app) and then display it online using templates rendered in Handlebars.js. Instead, it’s already downloaded, formatted into JSON and ready to be templated. And: 2) Tabletop.js has a bug that may cause some readers not to see any of the data at all. I wanted to avoid this problem.

The night of the election, we wanted to update the election results live. And we wanted to use the same app to display the results. Fortunately, the process was easy to do: I added new columns in our spreadsheet for vote totals and precincts reported. The new data was then pulled into our app and displayed with simple bar charts.

To display the results live, we had to ditch the exported JSON data and use the Google spreadsheet to power the app. This allowed us to update the spreadsheet and have the results display live on our website. We had one reporter at the county courthouse who punched in numbers for our Waterloo and Cedar Falls races. We had another reporter in Waverly who punched in results for races in that town. And we had another reporter in the newsroom who was monitoring the races in rural towns and giving me updated election results to enter into the spreadsheet.

The workflow worked great. While most news outlets were waiting for the county websites to update with election results, we were able to display the results right away. Unfortunately, the Black Hawk County website never wound up working on election, making our app the only place readers could go to get election results.

Our effort paid off: The app received about 11,500 pageviews, with about 10,000 of those pageviews coming the night of election (about 7,000 pageviews) and the following day. The app was more popular than any single story on our website for the month of October and November.

Reporters at the party headquarters said many of the candidates first tuned into the local television station to get election results but quickly went to our webpage when they realized we were the only ones with updated results. In fact, Waterloo’s mayor found out he won his race by looking at our website. Here’s a photo of him checking out our website on election night.

Which leads me to my last point: You’ll notice how our mayor is checking out the results on a smartphone. Our app (like all of our apps) was responsively designed, which means it looks great on mobile phones. It’s critical that news producers make sure their apps work on mobile. It’s pretty much mandatory. Because as our mayor shows, people love checking the news on their phones.

Written by csessig

November 10, 2013 at 10:19 pm