Chris Essig

Walkthroughs, tips and tricks from a data journalist in eastern Iowa

Archive for the ‘Javascript’ Category

Building your first Leaflet.js map

leave a comment »

Earlier this year, I had the privilege of teaching a class on building your first Leaflet.js map at the NICAR conference. I just realized I forgot to post the code in this blog so I figured I’d post it now. Better later than never.

If you’re interested in building your first map, check out my Github repo for the presentation.

Written by csessig

August 17, 2016 at 7:58 am

Introduction to Javascript

with 2 comments

A few weeks ago, I presented on the basics of Javascript to my fellow developers at FusionFarm.

If you want to check out my slides for the presentation, they are available here. I went over variables, objects, for loops, functions, and a whole host of other fun stuff.

If I missed anything, let me know in the comments.

Written by csessig

July 12, 2016 at 9:13 am

Posted in Javascript, Uncategorized

Tagged with

Exploring D3.js

leave a comment »

network_graph_mary_ssI’ve been wanting to use the D3.js library for some time now but I haven’t up to this point mostly because the code that runs the charts scares the crap out of me (that’s only a slight exaggeration).

Well this weekend I finally took on the scary and published my first D3 chart. The final product is a network graph that shows how one woman is helping connect people within the community. It’s not crazy awesome like most of the other D3 graphics out there but it’s a start.

The code started out as this chart from Mike Bostock and was modified to meet my needs. The final code is on GitHub.

While it wasn’t quite as intimidating as I thought it would be, a few things tripped me up. Chaining methods in D3 was something that threw me off at first. I was also unfamiliar with SVGs. This is a good primer from Scott Murray. Finally, many of the D3 methods are still mysterious me, like diagonal and the layouts. But one step at a time, right?

Some of it, however, isn’t crazy complicated. I’m going to review some of what I learned about D3 by going through some of the code that powered this project.

The code snippet below is line 47 in the chart’s script.js file. It starts the graph. It selects the empty SVG that I put in index.html, sets it’s width and height and appends a G element to it:

// Create an svg
  svg = d3.select("svg")
  	.attr("width", width + margin.right + margin.left)
  	.attr("height", height + margin.top + margin.bottom)
  	.append("g")
  	.attr("transform", "translate(" + (margin.left - line_width) + "," + margin.top + ")");

A few lines down, we create a variable called node, select all the G elements and bind data to them.

// Update the nodes
  var node = svg.selectAll("g.node")
  	.data(nodes, function(d) {
      return d.id || (d.id = ++i);

  	})

This G element contains everything in the chart and is placed just under the empty svg element in index.html

We then create a variable called nodeEnter, which appends another G element to the G element we just created. (confusing, I know).

// Enter any new nodes at the parent's previous position.
  var nodeEnter = node.enter().append("g")
    	// Set location of nodes
    	.attr("transform", function(d) {
    		if ( d['name'] === 'Mary Palmberg' && d['depth'] === 2) {
    			return "translate(" + circle_mary_location + "," + circle_mary_translate + ")";
        } else {

          
          if ( d['name'] === 'Thomas Jensen' ) {
            d.x -= 20;
          } else if ( d['name'] === 'Clarence Borck' ) {
            d.x += 20;
          } else if ( d['name'] === 'Jane Skinner' && d['connection'] === 5 ) {
            d.x -= 20;
          }
    			
          return "translate(" + d.y + "," + d.x + ")";
    		}
    	})
    	.attr("class", function(d) {
    		if (d['name'] === 'Mary Palmberg') {
    			return 'node-mary';
    		} else if (d['name'] === 'Top level') {
    			return 'node-top';
    		} else {
    			return 'node'
    		}
    	});

In our chart, these G elements are the nodes that contain our circles. Later in the code, I made lines that connect the circles. But we’re going to focus on just these G elements.

It may make more sense to inspect the graph to see what I’m talking about:

d3_mary_blog

After we append the G elements to their parent G elements, we chain a series methods to it, which set attributes for the G elements. These methods are called D3 selectors. With D3 selectors, you can select any shape in the SVG and a do a bunch of things to it. We can append new nodes to it, set CSS properties, give it a certain class name, etc. And since we bound data to the shape, we can decide what those properties will be based on the data we gave it.

This is seriously awesome, by the way.

In the code above, we used a D3 selector to set the location of the shapes using the CSS attribute transform. You’ll notice how setting that attribute is a function with a “d” parameter. “D” is equal to the data behind the shape we are drawing (remember, we appended data to it earlier). It returns an object with different properties. In this case, the property “name” is equal to the name of the person we are drawing in the circle.

For this chart, I had 16 G elements on the page. I also have objects with properties about each G element in a separate JSON file. As we loop through each G element and create it on the page, the data parameter becomes whichever G element we are on. So when we are creating Mary Palmberg’s circle, the “D” parameter will equal data about Mary Palmberg.

“D” returns an object that looks like so:

d3_mary_blog02

In the code, I’m checking to see if the G element we are on is Mary Palmberg, because her circle is placed in the middle and all other circles point to her. If we are on Mary, we set the CSS property transform to the variables called “circle_mary_location” and “circle_mary_translate”, which I set earlier in the script.js file. These basically put her circle in the middle of the page.

If it’s not Mary, we run the else part of the if-else statement. We do a couple of checks on a few people and change their x values. This is done because some of the circles were a little bit too close to together. After that, I put the data’s x and y coordinates in CSS translate property and return it. This basically puts it in the correct position on the page.

The last part should be easy to understand. All it does is set a class for the G Element depending on the person’s name. We give Mary a different class name so we can apply different styles to her circle:

d3_mary_blog03

It’s important to note that we haven’t actually created any shapes yet. Instead, G elements are just groups of shapes. The next thing we’re going to do is actually append circles to the empty G element.

nodeEnter.append("circle")
    	.attr("id", "circle-outer")
    	// Make radius smaller than inner circle
    	.attr("r", function(d) {
    		if (d['name'] === 'Mary Palmberg') {
    			if ( $(window).width() > 700) {
                    return circle_radius + 7;
                } else {
                    return circle_radius + 17;
                }
    		} else {
    			return circle_radius + 2;
    		}
    	})
    	// Color code the border around the circle
    	.style('stroke', function(d) {
    		// Color Mary differently
    		if (d['name'] === 'Mary Palmberg') {
    			// Red
    			return '#e41a1c'
    		} else if (d['connection'] === 1) {
    			// Purple
    			return "#984ea3";
    		} else if (d['connection'] === 2) {
    			// Green
    			return "#4daf4a";
    		} else if (d['connection'] === 3) {
    			// Blue
    			return "#377eb8";
    		} else if (d['connection'] === 4) {
    			// Orange
    			return "#ff7f00";
    		} else if (d['connection'] === 5) {
    			// Brown
    			return "#a65628";
    		} else {
    			// Blue
    			return "#377eb8";
    		}
    	});

  // Create the inner circle
  nodeEnter.append("circle")
    	.attr("id", "circle-inner")
    	// Radius
    	.attr("r", function(d) {
    		if (d['name'] === 'Mary Palmberg') {
    			if ( $(window).width() > 700) {
                    return circle_radius + 5;
                } else {
                    return circle_radius + 15;
                }
    		} else {
    			return circle_radius;
    		}
    	})
    	.style("fill", function(d) {
    		if ( $(window).width() > 700) {
                return "url(#" + d['name'].replace(' ', '') + ")"
            } else if ( d['name'] === 'Mary Palmberg') {
                return "url(#" + d['name'].replace(' ', '') + ")"
            }
    	});

I appended two circles to the G elements. The first is the circle around a person. I gave it an id of “circle-outer” and set it’s radius (attribute “r”) to a variable called “circle_radius”, which is set at the top of the script.js file. Once again I did a check for Mary and gave her a bigger circle since she’s in the middle. I then set the color of the circle’s stroke based on data property “connection”.

Our network graph has five total connections, so there’s a possibility of five different colors. I set the connection number in the JSON file. This is done so every connection through Mary is color-coded the same and thus the reader can see they are part of the same story. Same color = same story. And Mary stays in the middle of it all.

The second circle is the image itself. It has an id of “circle-id”. The attribute fill is set to the image of the person. The urls for each image are set in index.html under the defs tag. Each image is a pattern with a “xlink:href” property set to the url of the image. More information on how this works can be found here.

After that, I append text to have their name appear the circle. And I used this library to create the tooltips. I won’t bore you with that code as well.

There’s a few more things going on in this chart but those are the basics. Or at least as I understand them.

As with learning any new technology, my advice is to pick a project that you want to be powered by D3 and go at it. I learned a lot more about D3 making my first chart than I have with walkthroughs, video tutorials, etc. You’ll never learn it until you get your hands dirty.

Written by csessig

April 1, 2015 at 4:42 pm

Posted in d3, Javascript

Leaflet formula: Counting markers within a radius

with 5 comments

ssThe last project I worked on at The Courier was a comprehensive guide to mental health and disability services in NE Iowa. For the project, we first wanted to explain the MHDS system to our readers and how it works. It’s a relatively new system with a lot of moving parts and no good way to search for services. We really wanted to help residents and families in need of these services, so we designed everything — from the wording we used to the way items were laid out — to be focused on our readers.

After laying out the details of the new system, we wanted to allow our readers to search for services near them. A map seemed like an obvious choice; but we wanted to do more than just have them search for their address and plot a dot on the map.

(As a side note, we used Ben Welsh‘s jquery-geocodify plugin to geocode the addresses because it’s awesome.)

Instead, we wanted to show readers how many providers were close to them, as well as how to get ahold of the service provider and directions to the facility. For the former task, we used the following process:

First, the providers are plotted on the map. Then, we ask the reader to type in their address, which is plotted on the map using jquery-geocodify. They also have a second option for radius, which will be used to determine how many providers are within whatever radius they have given. They have four options: 5, 10, 25 and 50 miles.

Then, as the markers are getting plotted on the map, we run a few Javascript functions to calculate how many markers on the map are within the radius.

I’ve pulled out just this part of the mental health project and open-sourced it. You can view the full code over at my Github page.

The demo uses data on mental health providers, which is contained in this JSON fileand plots them on the map based on each provider’s latitude and longitude coordinates.

Here’s a quick run through the Javascript that makes all this run. It’s contained within the project’s script.js file. We’ll start from the bottom of the file and move our way up.

Note: This walkthrough assumes a decent understanding of Leaflet. I’m also using Leaflet-Awesome Markers and Font Awesome in this project.

First, we’ll add the base map tiles to our map. We’ll also  add the json_group to the map, which is a FeatureGroup we will create later in the file:

// Base map
var layer = new L.StamenTileLayer('toner-background');
var map = new L.Map('map', {
    center: new L.LatLng(42,-93.3),
    minZoom: 4,
    maxZoom: 10,
    zoom: 6,
    keyboard: false,
    boxZoom: false,
    doubleClickZoom: false,
    scrollWheelZoom: false,
    maxBounds: [[33.154799,-116.586914],[50.190089,-77.563477]]
});
// Add base layer to group
map.addLayer(layer);
// Add our markers in our JSON file on the map
map.addLayer(json_group);

Next, we’ll add the code for jquery-geocodify. For the onSelect function, we will grab the location of reader’s address and then call a geocodePlaceMarkersOnMap function, which we will create later:

// jQuery Geocodify
var maxY = 43.749935;
var minY = 40.217754;
var minX = -96.459961;
var maxX = -90.175781;

var search_marker;
var search_icon = L.AwesomeMarkers.icon({
    icon: 'icon-circle',
    color: 'green'
});

$('#geocoder').geocodify({
    onSelect: function (result) {
        // Extract the location from the geocoder result
        var location = result.geometry.location;

        // Call function and place markers, circle on map
        geocodePlaceMarkersOnMap(location);
    },
    initialText: 'Zip code, city, etc...',
    regionBias: 'US',
    // Lat, long information for Cedar Valley enter here
    viewportBias: new google.maps.LatLngBounds(
        new google.maps.LatLng(40.217754, -96.459961),
        new google.maps.LatLng(43.749935, -90.175781)
    ),
    width: 300,
    height: 26,
    fontSize: '14px',
    filterResults: function (results) {
        var filteredResults = [];
        $.each(results, function (i, val) {
            var location = val.geometry.location;
            if (location.lat() > minY && location.lat() < maxY) {
                if (location.lng() > minX && location.lng() < maxX) {
                    filteredResults.push(val);
                }
            }
        });
        return filteredResults;
    }
});

Next, we’ll loop through our providers.json file, which has a variable of json_data, and create a marker for each object in our array. We’ll then add it to our json_group FeatureGroup, which is defined at the top of our script file:

// This loops through the data in our JSON file
// And puts it on the map
_.each(json_data, function(num) {
    var dataLat = num['latitude'];
    var dataLong = num['longitude'];

    // Add to our marker
    var marker_location = new L.LatLng(dataLat, dataLong);

    // Options for our circle marker
    var layer_marker = L.circleMarker(marker_location, {
        radius: 7,
        fillColor: "#984ea3",
        color: "#FFFFFF",
        weight: 1,
        opacity: 1,
        fillOpacity: 0.8
    });

    // Add events to marker
    layer_marker.on({
        // What happens when mouse hovers markers
        mouseover: function(e) {
            var layer_marker = e.target;
            layer_marker.setStyle({
                radius: 8,
                fillColor: "#FFFFFF",
                color: "#000000",
                weight: 1,
                opacity: 1,
                fillOpacity: 1
            });
        },
        // What happens when mouse leaves the marker
        mouseout: function(e) {
            var layer_marker = e.target;
            layer_marker.setStyle({
                radius: 7,
                fillColor: "#984ea3",
                color: "#FFFFFF",
                weight: 1,
                opacity: 1,
                fillOpacity: 0.8
            });
        }
    // Close event add for markers
    });

    json_group.addLayer(layer_marker);
// Close for loop
}, this);

The changeCircleRadius function is called anytime the radius is changed on the page. It first calls the pointsInCircle function, which we will define later, and then sets the radius of the circle using Leaflet’s setRadius function for circles.

Leaflet uses meters, so we will need to convert miles to meters using the milesToMeters function, which will be defined later. The mile’s value is contained within the DIV with the id of radius-selected. We’ll grab the value, which is the radius in miles, convert it to meters and then set the radius of the circle using Leaflet.

// Change circle radius when changed on page
function changeCircleRadius(e) {
    // Determine which geocode box is filled
    // And fire click event

    // This will determine how many markers are within the circle
    pointsInCircle(circle, milesToMeters( $('#radius-selected').val() ) )

    // Set radius of circle only if we already have one on the map
    if (circle) {
        circle.setRadius( milesToMeters( $('#radius-selected').val() ) );
    }
}

$('select').change(function() {
    changeCircleRadius();
});

The geocodePlaceMarkersOnMap function is called every time the reader enters an address into our geocoder and presses enter. It passes with it the location of their address, which we get from jquery-geocodify. With this function, we will create both a circle and a marker on top of the given address.

First, we set the view of the map based on this location and remove the circle on the map if there already is one. Then we actually create a circle and set how large it is based on the radius set by the user.

We then remove the marker if it’s already on the map and create a new one. When we create a new marker, we’ll set it to draggable and set its dragend method to reset the view of the map every time the reader drags the marker somewhere on the. It then calls the pointsInCircle function, which we will define next. Finally, we’ll call the same pointsInCircle function and add the marker to the map.

// This places marker, circle on map
function geocodePlaceMarkersOnMap(location) {
    // Center the map on the result
    map.setView(new L.LatLng(location.lat(), location.lng()), 10);

    // Remove circle if one is already on map
    if(circle) {
        map.removeLayer(circle);
    }
    
    // Create circle around marker with our selected radius
    circle = L.circle([location.lat(), location.lng()], milesToMeters( $('#radius-selected').val() ), {
        color: 'red',
        fillColor: '#f03',
        fillOpacity: 0.1,
        clickable: false
    }).addTo(map);
    
    // Remove marker if one is already on map
    if (search_marker) {
        map.removeLayer(search_marker);
    }
        
    // Create marker
    search_marker = L.marker([location.lat(), location.lng()], {
        // Allow user to drag marker
        draggable: true,
        icon: search_icon
    });

    // Reset map view on marker drag
    search_marker.on('dragend', function(event) {
        map.setView( event.target.getLatLng() ); 
        circle.setLatLng( event.target.getLatLng() );

        // This will determine how many markers are within the circle
        pointsInCircle( circle, milesToMeters( $('#radius-selected').val() ) );

        // Redraw: Leaflet function
        circle.redraw();

        // Clear out address in geocoder
        $('#geocoder-input').val('');
    });

    // This will determine how many markers are within the circle
    // Called when points are initially loaded
    pointsInCircle( circle, milesToMeters( $('#radius-selected').val() ) );

    // Add marker to the map
    search_marker.addTo(map);
        
// Close geocodePlaceMarkersOnMap
}

The pointsInCircle function is the last function called when one of two things happen: the user enters an address or the user drags the marker on the map. Its purpose is to find out how many markers on the map are within the circle we created and then display that number in the legend.

Fortunately, Leaflet has a really handy distanceTo method, which determines how many meters are within two points. It uses the two points’ latitude and longitude coordinates to find this out.

With the pointsInCircle function, we first capture the lat, long of the circle we’re putting on the map. We then loop through our providers.json file, find the latitude and longitude coordinates of each object and determine how far it is from our circle using the distanceTo method.

We then decide if that amount is higher or lower than the radius set by the user. If it is lower, we add it to a counter. When we’re done, that counter will equal the number of markers within the selected radius. It will be put on the map under the “Results” header.

Lastly, we will determine how many results we have and use the correct wording on the page. If we have one result, we’ll use the word “provider” (singular) in the map’s legend. Otherwise, we’ll use “providers.”

// This figures out how many points are within out circle
function pointsInCircle(circle, meters_user_set ) {
    if (circle !== undefined) {
        // Only run if we have an address entered
        // Lat, long of circle
        circle_lat_long = circle.getLatLng();

        // Singular, plural information about our JSON file
        // Which is getting put on the map
        var title_singular = 'provider';
        var title_plural = 'providers';

        var selected_provider = $('#dropdown_select').val();
        var counter_points_in_circle = 0;

        // Loop through each point in JSON file
        json_group.eachLayer(function (layer) {

            // Lat, long of current point
            layer_lat_long = layer.getLatLng();

            // Distance from our circle marker
            // To current point in meters
            distance_from_layer_circle = layer_lat_long.distanceTo(circle_lat_long);

            // See if meters is within raduis
            // The user has selected
            if (distance_from_layer_circle <= meters_user_set) {
                counter_points_in_circle += 1;
            }
        });

        // If we have just one result, we'll change the wording
        // So it reflects the category's singular form
        // I.E. facility not facilities
        if (counter_points_in_circle === 1) {
            $('#json_one_title').html( title_singular );
        // If not one, set to plural form of word
        } else {
            $('#json_one_title').html( title_plural );
        }
        
        // Set number of results on main page
        $('#json_one_results').html( counter_points_in_circle );
    }
// Close pointsInCircle
};

The final piece is to add the variables we’re using to the top of the script file. We’ll also add the milesToMeters function, which converts miles to meters.

// We'll append our markers to this global variable
var json_group = new L.FeatureGroup();
// This is the circle on the map that will be determine how many markers are around
var circle;
// Marker in the middle of the circle
var search_marker;
// Marker icon
var search_icon = L.AwesomeMarkers.icon({
    icon: 'icon-circle',
    color: 'red'
});


// Convert miles to meters to set radius of circle
function milesToMeters(miles) {
    return miles * 1069.344;
};

I hope this walkthrough helps fellow Leaflet users! If you have any questions, leave them in the comments.

Happy coding.

Written by csessig

June 22, 2014 at 1:32 pm

How laziness led me to produce news apps quicker

with one comment

Note: This is cross-posted from Lee’s data journalism blog, which you can read over there by clicking here.

ssAn important trait for programmers to adopt, especially for those who are writing code on deadline, is laziness. While that sounds counter-productive, it can be incredibly beneficial.

When writing code, you  do a lot of repetitive tasks. If you’re using a Javascript library for mapping, for instance, you’ll need to copy and paste the Javascript and CSS files into a new directory every time you start a new project. If you want to use plugins with that map, you will need to include their files in the directory as well.

The more complicated your project, the more libraries you will likely use, which means a lot of tedious work putting all those files in the new directory.

And that’s before you even start to code. If you want your projects to have a similar look to them (which we do at the Courier), you will need to create code that you can to replicate in the future. And then every time you want to use it, you will need to rewrite the same code or, at the very least, copy code from your old projects and paste it into your new project.

I did this for a long time. Whenever I was creating  a new app, I’d open a bunch of projects I’ve already completed, go into their CSS and JS files, copy some of it and paste it into the new files. It was maddening because I knew with every new project, I would waste hours basically replicating work I had already done.

This is where laziness pays off. If you are too lazy to do the same things over and over again, you’ll start coming up with solutions that automatically does these repetitive tasks.

That’s why many news apps teams have created templates for their applications. When they create new projects, they start with this template. This allows their apps to have a similar look. It also means they don’t have to do the same things over and over again every time they are working with a project.

App templating is something I’ve dabbled with before. But those attempts were minor. For the most part, I did a lot of the same mind-numbing tasks every time I wanted to create a map, for instance.

Fortunately I found a good solution a few months ago: Yeoman, which is dubbed “the web’s scaffolding tool for modern web apps,” has helped me eliminate much of that boring, repetitive work.

Yeoman is built in Node. Before I stumbled upon it a few months ago, I knew very little about Node. I still don’t know a ton but this video helped me grasp the basics. One advantage of Node is it allows you to run Javascript on the command line. Pretty cool.

Yeoman is built on three parts. The first is yo, which helps create new applications and all the files that go with it. It also uses Grunt, which I have come to love. Grunt is used for running tasks, like minifying code, starting a local server so you test your code, copy files into new directories and a whole bunch of other things that makes coding quicker, easier and less repetitive.

Finally, Yeoman uses Bower, which is a directory of all the useful packages you use with applications, from CSS frameworks to Javascript mapping libraries. With a combination of Bower and Grunt, you can download the files you need for your projects and put them in the appropriate directories using a Grunt task. Tasks are written in a Gruntfile and run on the command line. The packages you want to download from Bower and use in your project are stored in bower.json file.

In Yeoman, generators are created. A generator makes it easy to create new apps using a template that’s already been created. You can either create your own generator or use one of the 400+ community generators. The generators themselves are directories of files, like  images, JS files, CSS stylesheets, etc. The advantage of using generators is you can create a stylesheet, for instance, save it in the generator directory and use that same stylesheet with every new project you create. If you decide to edit it, those changes will be reflected in every new project.

When you use a generator, those files are copied into a new directory, along with Grunt tasks, which are great for when you are testing and deploying your apps. Yeoman also has a built-in prompt system that asks you a series of questions whenever you create an app. This helps copy over the appropriate files and code, depending on what you are trying to make.

I decided I would take a stab at creating a generator. The code is here, along with a lot more explanation on how to use it. Basically, I created it so it would be easier to produce simple apps. Whenever I create a new Leaflet map, for instance, I can run this and a sample map with sample data will be outputted into a new directory. I also have a few options depending on how I want to style the map. I also have options for different data types: GeoJSON, JSON with latitude and longitude coordinates and Tabletop. What files are scaffolded out depends on how you responded to the prompts that are displayed when you first fire up the generator.

There is also an option for non-maps. Finally, you can use Handlebars for templating your data, if that’s your thing.

My hope is to expand upon this generator and create more options in the future. But for now, this is easy way to avoid a bunch of repetitive tasks. And thus, laziness pays off.

If you are interested in creating your own generator, Yeoman has a great tutorial that you should check out.

Written by csessig

February 19, 2014 at 12:08 pm

Using hexagon maps to show areas with high levels of reported crime

leave a comment »

Screenshot: Violent crime in WaterlooCrime maps are a big hit at news organizations these days. And with good reason: Good crime maps can tell your readers a whole lot about crime not only in their city but their backyard.

For the last year and a half, we’ve been doing just that. By collecting data every week from our local police departments, we’ve been able to provide our community a great, constantly updated resource on crime in Waterloo and Cedar Falls. And we’ve even dispelled some myths on where crime happens along the way.

The process, which I’ve blogged about in the past, has evolved since the project was first introduced. But basically it goes like this: I download PDFs of crime reports from the Waterloo and Cedar Falls police departments every week. Those PDFs are scraped using a Python script and exported into an CSV. The script pulls out reports of the most serious crimes and disregards minor reports like assistance calls.

The CSV is then uploaded to Google spreadsheets as a back up. We also use this website to get latitude, longitude coordinates from the addresses in the Google spreadsheet. Finally, the spreadsheet is converted into JSON using Mr. Data Converter and loaded onto our FTP server. We use Leaflet to map the JSON files.

We’ve broken the map into months and cities, meaning we have a map and a corresponding JSON file for each month for both Waterloo and Cedar Falls. We also have JSON files with all of the year’s crime reports for both Waterloo and Cedar Falls.

The resulting maps are great for readers looking for detailed information on crime in their neighborhood. It also provides plenty of filters so readers can see what areas have more shootings, assaults, burglaries, etc.

But it largely fails to give a good overview of what areas of the cities had the most reported crimes in 2013 for two reasons: 1) Because of the sheer number of crime reports (3,684 reports in 2013 for Waterloo alone) and the presence of multiple reports coming from the same address, it’s nearly impossible to simply look at the 2013 maps and see what areas of the cities had the most reports, and; 2) The JSON files containing all of the year’s reports runs well on modern browsers but runs more slowly outdated browsers (I’m looking at you, Internet Explorer), which aren’t equipped to iterate over JSON files with thousands of items.

As a result, I decided to create separate choropleth maps that serve one purpose: To show what areas of the cities had the most reported crimes. The cities would be divided into sections and areas with more crimes would be darker on the map, while the areas with fewer crimes would be lighter. Dividing the city into sections would also mean the resulting JSON file would have dozens of items instead of thousands. The map would serve a simple purpose and load quickly.

What would the areas of the cities actually look like? I’ve been fascinated with hex maps since I saw this gorgeous map from the L.A. Times on 9/11 response times. The map not only looked amazing but did an incredibly effective job displaying the data and telling the story. While we want our map to look nice, telling  its story and serving our journalistic purpose will always be our No. 1 goal. Fortunately, hex maps do both.

And even more fortunately, the process of creating hex maps got a whole lot easier when data journalist Kevin Schaul introduced a command line tool called binify that turns dot density shapefiles into shapefiles with hexagon shapes. Basically the map is cut into hexagon shapes, and all the points inside each hexagon are counted. Then each hexagon is given a corresponding count value for the number points that fall in between its borders.

I used binify to convert our 2013 crime maps into hex maps. As I mentioned earlier, we started out with CSVs with all the crimes. My workflow for creating the hex maps and then converting them into JSON files for Leaflet is in a gist below. It utilizes the fantastic command line tool og2ogr, which is used to convert map files, and binify.

# We start with a CSV called "crime_master_wloo_violent.csv" that has violent crime reports in Waterloo.
# Included in csv are columns for latitude ("lat") and longitude ("long"):
https://github.com/csessig86/crime_map2013/blob/master/csv/crime_master_wloo_violent.csv

# Command to turn CSV into DBF with the same name
ogr2ogr -f "ESRI Shapefile" crime_master_wloo_violent.dbf crime_master_wloo_violent.csv

# Create a file called "crime_master_wloo_violent.vrt"
# Change "SrcLayer" to the name of the source
# Change OGRVRTLayer name to "wloo_violent"
# Add name of lat, long columns to "GeometryField"
<OGRVRTDataSource>
  <OGRVRTLayer name="wloo_violent">
    <SrcDataSource relativeToVRT="1">./</SrcDataSource>
    <SrcLayer>crime_master_wloo_violent</SrcLayer>
    <GeometryType>wkbPoint</GeometryType>
    <LayerSRS>WGS84</LayerSRS>
    <GeometryField encoding="PointFromColumns" x="long" y="lat"/>
  </OGRVRTLayer>
</OGRVRTDataSource>

# Create "violent" folder to stash shp file
mkdir violent

# Convert "vrt" file into shp and put it in our "violent" folder
ogr2ogr -f "ESRI Shapefile" violent crime_master_wloo_property.vrt

# Binify the "wloo_violent" shp into a shp called "wloo_violent_binify" with three options
# The first sets the number of hexagons across to 45
# The second sets the custom extent to a box around Waterloo
# Custom extent is basically the extent of the area that the overlay of hexagons will cover
# The third tells binify to ignore hexagons that have zero points
binify wloo_violent.shp wloo_violent_binify.shp -n 45 -E -92.431873 -92.260867 42.4097259 42.558403 --exclude-empty

# Convert binified shp into JSON file called "crime_data_review_violent"
ogr2ogr -f "GeoJSON" crime_data_review_violent.json wloo_violent_binify.shp

# Add this variable to JSON file
var crime_data_review_violent

# Here's the final files:
https://github.com/csessig86/crime_map2013/tree/master/shps/wloo_shps

# And our JSON file:
https://github.com/csessig86/crime_map2013/blob/master/JSON/crime_data_review_violent.json

# And the Javascript file that is used with Leaflet to run the map.
# Lines 113 through 291 are the ones related to pulling the data from the JSON files,
# coloring them, setting the legend and setting a mouseover for each hexagon
https://github.com/csessig86/crime_map2013/blob/master/js/script_map_review.js

# And the map online:
http://wcfcourier.com/app/crime_map2013/index_wloo.php

The above shows the process I took for creating our hex map of violent crimes in Waterloo. I duplicated this process three more times to get four maps in all. One showed property crimes in Waterloo, while the other two show violent and property crime in Cedar Falls.

* Note: For more information on converting shapefiles with ogr2ogr, check out this blog post from Ben Balter. Also, this cheat sheet of ogr2ogr command line tools was invaluable as well.

Another great resource is this blog post on minifying GeoJSON files from Bjørn Sandvik. While the post deals with GeoJSON files specifically, the process works with JSON files as well. Minifying my JSON files made them about 50-150 KB smaller. While it’s not a drastic difference, every little bit counts.

To get choropleth maps up and running in Leaflet, check out this tutorial. You can also check out my Javascript code for our crime maps.

One last note: The map is responsive. Check it out.

Written by csessig

January 11, 2014 at 4:42 pm

New code: Using a Google spreadsheet to power a mobile site

with 3 comments

I’ve posted a new template on Github that I hope others will find useful. It involves using Tabletop.js to transfer data from a Google spreadsheet onto a mobile website. I’ve tried to create it so you have to do little tweaking of the code that grabs and formats the data so you can focus on collecting and cleaning the data. The template will render the same regardless of what data is in the Google spreadsheet.

It’s initial purpose was to power our guides to weekend festivals, since we have quite a few in the summer. We used it, for instance, for our guide to the National Cattle Congress fair, which was last weekend. And it was designed for mobile users, in particular. You will notice how the website kind of looks like an app you would download on your smartphone with navigation buttons at the button. But it was responsively designed so it will work on all computer sizes.

You can use it for whatever you like. You can also change the code to have the data displayed as you see fit. The whole project is open-sourced.

The code can be found here. If you use it at all, please e-mail a link to you work so I can share it with the world! My e-mail is chris.essig@wcfcourier.com.

Written by csessig

September 18, 2013 at 10:47 am

How-to: Creating a public salary database

leave a comment »

before_after_teaserNote: This is cross-posted from Lee’s data journalism blog, which you can read over there by clicking here.

People love knowing what public employees earn in a year. They always make for interesting conversation and plenty of controversy.

In honor of Sunshine Week, I’m going to go through the steps of creating a quick and dirty salary database using DataTables.

Here’s the process:

1. Find a data set and clean, clean, clean

For this example, I’m going to use the 2012 Salary Book for public employees in Iowa. The spreadsheet has records for more than 59,000 public employees. And there are plenty of things in the spreadsheet you’ll probably want to clean up.

I’m not going to dive too deeply into cleaning data here except to point out that it is VERY IMPORTANT. I like to first bring my spreadsheets into Excel and simply sort each row from A to Z to find empty cells, cells with funky text or characters, etc. Here’s a handy guide for cleaning data in Excel.

I then use a great program called Open Refine (formerly Google Refine) to help clean the data. Their website includes some video tutorials on how to use the program. Be sure to also read this post on how investigative journalists should use Refine. Finally, this walkthrough also teaches journalists to use Refine.

The process of cleaning data is probably the most important step in creating a database. It make take several hours. But just as you wouldn’t print a story without first proofreading it, you shouldn’t make data available on your site that you haven’t cleaned to the best of your abilities.

2. Hosting your data

When you’re done cleaning the data, you’ll export the spreadsheet in Google Refine. You’ll then you’ll need to upload the data and host it somewhere. You have a few options:

a. Don’t host it at all. Instead follow this method I mentioned in an earlier blog post that converts the spreadsheet into an HTML table, which is then converted into a DataTable. The disadvantage of this method is if you want to update the spreadsheet, you’ll have to do the conversion over and paste the new HTML table into your document. For spreadsheets that are changed repeatedly, this could be a pain.

b. Upload the spreadsheet into Google Drive and use an awesome service called Tabletop.JS to feed the spreadsheet to DataTables. This is quickly becoming one of my preferred ways to host data simply because it allows several journalists and editors to edit the same Google spreadsheet. All their changes will be reflected on your page, which will be updated automatically as the spreadsheet is updated.

And in fact, my hope was to upload our spreadsheet of Iowa public employee salaries to Google and use Tabletop.JS for this project. But I couldn’t because Google only allows spreadsheets with less than 400,000 fields. This spreadsheet, which was 59,000 rows and seven columns, just eclipses Google’s limit.

For an example of Tabletop.JS in action, check out this repo on my Github page. If you have any questions on how it works, shoot me an e-mail or leave a comment on this article.

c. You can also use an open source tool called DataSet from the Miso Project. Instead of hosting it with Google, you just load the CSV file on your own server and pull the data directly from the file. You can convert the CSV information into Javascript arrays, objects, etc., which allows the data to work with DataTables and tons of other awesome Javascript libraries.

The advantage to DataSet is you can work with huge spreadsheets like this one on Iowa public employee salaries and you don’t have to worry about Google’s size limits. The downfall is only one person can edit the spreadsheet at a time. And they’ll have to download the CSV, re-save it, then re-upload it to the server every time they make a change.

3. The code

The first thing you’ll need to do is go to my Github page and download the directory for this project.

I’m going to break down the Javascript file into chunks to give you an idea of what is happening:

// DataTables currency
// Use to sort the table via currency
jQuery.extend( jQuery.fn.dataTableExt.oSort, {
	"currency-pre": function ( a ) {
		a = (a==="-") ? 0 : a.replace( /[^\d\-\.]/g, "" );
		return parseFloat( a );
	},
	
	"currency-asc": function ( a, b ) {
		return a - b;
	},
	
	"currency-desc": function ( a, b ) {
		return b - a;
	}
});

This function allows us to sort the table based on currency. Basically, when the reader firsts loads the database, the employee with the highest salary will be shown. For more information on the different sorting techniques offered by DataTables, click here.

// Where we'll put the data
var newDataSet = [];
 
// Our column headers
// Change these to fit your table
var tableColumnSet =   [
  { "sTitle": "Employee" },
	{ "sTitle": "Department" },
	{ "sTitle": "Total FY12 Salary", "sType": "currency" },
	{ "sTitle": "Gender", "sClass": "center" },
	{ "sTitle": "County", "sClass": "center" },
	{ "sTitle": "Position", "sClass": "center" }
];

This creates a blank array called “newDataSet.” Later, we will push all of the content in our CSV into this array, which will be used by DataTables and displayed on the page. The “tableColumnSet” variable is where we set our column header names. For future projects, you’ll want to change those to match your own CSV file.

Notice also how we set the “Total FY12 Salary” column to currency, which allows us to sort the table by currency.

// Our DataTable information
// Don't need to change anything here
// Unless you want to customize the table format
function showInfo() {
  $('#table_div').html( '<table cellpadding="0" cellspacing="0" border="0" class="table table-striped table-bordered" id="table"></table>' );
 
	// Push the data to the table
	$('#table').dataTable( {
		"bProcessing":true,
		"sPaginationType": "bootstrap",
		"iDisplayLength": 100,
		"aaData": newDataSet,
		"aoColumns": tableColumnSet,
		// Pick which column we will sort by default
		// For this table, we'll sort ascending by Total FY12 Salary
		"aaSorting": [[ 2, "desc" ]],
		"oLanguage": {
			"sLengthMenu": "_MENU_ records per page"
		}
	});
}

DataTables provides us with several different options for displaying the data, including how to sort the data, where to the pull the data from, how many rows to show at a time, etc. We’re putting all this in a function called “showInfo”, which we will call below.

For more on the options available to us with DataTables, click here.

You may also notice that this table is styled using Twitter’s Bootstrap framework. Basically that gives the table the zebra look to it, as well as style the buttons and search field. For more information on working with Bootstrap, click here.

// Load up the CSV using Miso.Dataset
$(document).ready( function() {
  // Change URL to the right path for your CSV
	var ds = new Miso.Dataset({
  		url : 'csvs/salary_book_clean.csv',
  		delimiter : ','
  	});
 
	// Run this after we load our CSV
	ds.fetch({ success : function() {
    	this.each(function(row, rowIndex) {
    		// Change these variables to match your column names
    		var employeeData = row.Employee;
    		var departmentData = row.Department;
    		var salaryData = row.TotalFY12Salary;
    		var genderData = row.Gender;
    		var countyData = row.County;
    		var positionData = row.Position;
    		// Put information in an array and push it to our table
    		// Change these variables to match variables above
    		var myArray = [employeeData, departmentData, salaryData, genderData, countyData, positionData];
			newDataSet.push(myArray);
  		});
    	// Call DataTable function showInfo
		showInfo();
	}});
});

The last part of this file will load when our document is ready. It uses Miso’s DataSet library to pull in the information from our CSV file. Once the information is loading, a loop for each row in our data. A Javascript array is created for each row containing that row’s data and then that is pushed to our newDataSet variable, which we created earlier.

For future projects, make sure you change the variable names in each loop to fit your CSV file. So change the words “Employee”, “Department”, etc. to make the column headers in your CSV file.

Finally, once that is done, we call our “showInfo” function, which puts the content in our CSV on the page using DataTables.

If you have any questions, feel free to leave me a comment or e-mail me directly at chris.essig@wcfcourier.com.

Written by csessig

March 15, 2013 at 10:03 am

Create an easy before, after photo interactive

with one comment

before_after_teaserNote: This is cross-posted from Lee’s data journalism blog, which you can read over there by clicking here.

When we got our first big snow storm of the winter season a few months ago, we tried something a little bit different at the Courier.

Before the storm hit, we asked our Facebook fans to capture before and after photos of the outdoors taken in the same, identical spot. Then we had them submit their photos to us. With those photos, we put together this before/after interactive that was a nice little traffic boost for us and complimented our weather coverage well.

We have done something similar to this in the past: This interactive, for instance, looks at urban sprawl in the Cedar Valley. It uses a slider that the reader needs to click and drag to toggle between the photos.

The slider works very well. However, I loved what the Detroit Free Press did recently when they compared before and after photos of a car manufacturing plant in their area (which Marisa Kwiatkowski wrote about in Lee’s data journalism blog). Instead of using a slider, the Free Press had the photos change based on mouse movement. The result is incredibly smooth and loads quickly.

After digging through their code, I found out it is very easy to do. Here’s how we did it at the Courier.

1. Set up a basic HTML page

<!DOCTYPE html>
<html>
<head>
<title>Before/after demo</title>
</head>
 
<body>
</body>
 
</html>

That’s about as basic as it gets right there.

2. Add your images to the “body” of the HTML page

<div style="position:relative; width:600px; height:460px;" class="trackMe">
    <img src="http://wcfcourier.com/app/special/beforeafter/beforeafter05b.jpg" class="beforeafterphotos" />
    <img src="http://wcfcourier.com/app/special/beforeafter/beforeafter05a.jpg" class="beforeafterphotos" />
</div>

You will want to delete the URLs for our images and replace them with your own. Make sure the images are the same size. You’ll also want to make sure the before and after photos are as similar as they can be to each other. I spent quite a bit of time editing, cropping and rotating the photos users sent to us to have them both line up as best they could.

It is also important to note that the “after” image (beforeafter05b.jpg in the example above) is placed on the page before the “before” image is (beforeafter05a.jpg in the example above).

Lastly, it is very important that you wrap the images in a DIV with the class “trackMe.” And each image needs to be put in the class “beforeafterphotos.”

3. Add this Javascript code in the “head” of the HTML page

<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
<script type="text/javascript" language="JavaScript">
$(document).ready(function() {
  $('.trackMe').each(function(){
		$(this).children("img:last").mousemove(function(e) {
			var offset = $(this).offset();
			var xpos = (e.pageX - offset.left);
   			var ypos = (e.pageY - offset.top);
			//now to get the first child image width..
			var thisImage = $(this);
			var thisWidth = thisImage.width();
			var pct = Math.round((xpos/thisWidth)*100)/100;
			var ipct = Math.abs(Math.round(((xpos-thisWidth)/thisWidth)*100)/100);
			thisImage.css({ 'opacity' : ipct });
		});
	});
});
</script>

This code basically detects mouse movement over the second of the two images, which is actually the “before” image (see above). It then figures out where the mouse is in relation to the image. Then it sets the opacity of that image based on the mouse’s location.

So if the mouse is to the left of the image, the “before” image’s opacity is set to 100 percent and is fully visible. If the mouse is in the middle of the image, the image’s opacity is set to 50 percent and is half way visible. If it is to the right, the image’s opacity is set to 0 percent and is invisible.

This function is called every time the mouse is moved. The effect for the reader is as they move their mouse from the left side of the image to the right, the “before” image slowly fades out and reveals the “after” image.

4. Add this CSS code to the “head” of the HTML page

</pre>
<style>
.trackMe img.beforeafterphotos {
  top:0 !important;
	left:0 !important;
	position:absolute;
	margin:0 0 15px 0 !important;
}
</style>

This code just makes sure the images are layered on top of one another, instead of one above the other.

5. Full code is below:

<!DOCTYPE html>
<html>
<head>
<title>Before/after demo</title>
 
<style>
.trackMe img.beforeafterphotos {
  top:0 !important;
	left:0 !important;
	position:absolute;
	margin:0 0 15px 0 !important;
}
</style>
 
<script src="http://code.jquery.com/jquery-latest.min.js" type="text/javascript"></script>
<script type="text/javascript" language="JavaScript">
$(document).ready(function() {
  $('.trackMe').each(function(){
		$(this).children("img:last").mousemove(function(e) {
			var offset = $(this).offset();
			var xpos = (e.pageX - offset.left);
   			var ypos = (e.pageY - offset.top);
			//now to get the first child image width..
			var thisImage = $(this);
			var thisWidth = thisImage.width();
			var pct = Math.round((xpos/thisWidth)*100)/100;
			var ipct = Math.abs(Math.round(((xpos-thisWidth)/thisWidth)*100)/100);
			thisImage.css({ 'opacity' : ipct });
		});
	});
});
</script>
</head>
 
<body>
 
<div style="position:relative; width:600px; height:460px;" class="trackMe">
	<img src="http://wcfcourier.com/app/special/beforeafter/beforeafter05b.jpg" class="beforeafterphotos" />
	<img src="http://wcfcourier.com/app/special/beforeafter/beforeafter05a.jpg" class="beforeafterphotos" />
</div>
 
 
</body>
</html>

That’s all it takes!

6. Add more images

If you want to add more images, just repeat step 2 with the new images. For the complete code that I used on the snow before/after project, click here.

Written by csessig

February 25, 2013 at 3:09 pm

Final infographics project

leave a comment »

For the last six weeks, I’ve taken a very awesome online course on data visualization called “Introduction to Infographics and Data Visualization” It is sponsored by the Knight Center for Journalism in Americas and taught by the extremely-talented Albert Cairo. If you have a quick second, check out his website because it’s phenomenal.

Anyways, we have reached the final week of the course and Cairo had us all make a final interactive project. He gave us free reign to do whatever we want. We just had to pick a topic we’re passionate about. Given that I was a cops and courts reporter for a year in Galesburg, Illinois before moving to Waterloo, Iowa, I am passionate about crime reporting. So I decided for my final project I’d examine states with high violent crime rates and see what other characteristics they have. Do they have higher unemployment rates? Or lower education rates? What about wage rates?

Obviously, this is the type of project that could be expanded upon. I limited my final project to just four topics mostly because of time constraints. I work a full time job, you know! Anyways, here’s my final project. Let me know if you have any suggestions for improvement.

teaser

About:

Data: Information for the graphic was collected from four different sources, which are all listed when you click on the graphic. I took the spreadsheets from the listed websites and took out what I wanted, making CSVs for each of the four categories broken down in the interactive.

Map: The shapefile for the United States was taken from the U.S. Census Bureau’s website. Find it by going to this link, and selecting “States (and equivalent)” from the dropdown menu. I then simplified the shapefile by about 90 percent using this website. Simplifying basically makes the lines on the states’ polygons less precise but dramatically reduces the size of the file. This is important because people aren’t going to want to wait all day for your maps to load.

Merging data with map: I then opened that shapefile with an awesome, open-source mapping program called QGIS. I loaded up the four spreadsheets of data in QGIS as well using the green “Add vector layer” button (This is important! Don’t use the blue “Create a Layer from a Delimited Text File” button). The shapefile and all the spreadsheets will now show up on the right side of the screen in QGIS under “Layers.”

Each spreadsheet had a row for the state name, which matched the state names for each row in the shapefile.  It’s important these state names match exactly. For instance, for crime data from the FBI I had to lowercase the state names. So I turned “IOWA” into “Iowa” before loading it into QGIS because all the rows in the shapefile were lowercase (“Iowa” in the above example).

Then you can open the shapefile’s properties in QGIS and merge the data from the spreadsheets with the data in the shapefile using the “Joins” tab. Finally, right click on the shapefiles layer then “Save As” then export it as a GeoJSON file. We’ll use this with the wonderful mapping library Leaflet.

qgis_teaser

Leaflet: I used Leaflet to make the map. It’s awesome. Check it out. I won’t get into how I made the map interactive with Javascript because it’s copied very heavily off the this tutorial put out by Leaflet. Also check it out. The only thing I did differently was basically make separate functions (mentioned in the tutorial) for each of my four maps. There is probably (definitely) a better way to do this but I kind of ran out of time and went with what I had. If you’re looking for the full code, go here and click “View Page Source.”

Design: The buttons used are from Twitter’s Bootstrap. I used jQuery’s show/hide functions to show and hide all the element the page. This included DIVs for the legend, map and header.

GeoJSON: The last thing I did was modify my GeoJSON file. You’ll notice how the top 10 states for violent crime rates are highlighted in black on the maps to more easily compare their characteristics across the maps. Well, I went into the GeoJSON and put all those 10 states attributes at the bottom of the file. That way they are loaded last on the map and thus appear on top of the other states. If you don’t do this, the black outlines for the states don’t show up very well and look like crap. Here’s GeoJSON file for reference.

Hopefully that will help guide others. If you have any questions, feel free to drop me a line. Thanks!

 

Written by csessig

December 11, 2012 at 12:13 am