Chris Essig

Walkthroughs, tips and tricks from a data journalist in eastern Iowa

Archive for the ‘Multimedia clips’ Category

Creating responsive maps with Leaflet, Google Docs

with 9 comments

wloo_history_teaserNote: This is cross-posted from Lee’s data journalism blog, which you can read over there by clicking here.

Quick: Name five people you know who don’t have a smartphone.

Stumped? Yeah, me too. The fact is more and more people have smartphones and are using them to keep up with the world.

What does that mean for news app developers? We need to be especially conscience of the mobile platform and make sure everything we build for the web is compatible on the smallest of screens.

One great way to do this is creating apps through responsive design. The idea behind responsive design is creating one web page for all users, as opposed to making separate pages for desktop and mobile computers.

Then we simply add, rearrange, subtract or tweak features on a web page based on the size of the browser the user has when they are viewing the app.

Maps can be difficult to manage on mobile platforms, especially when you add in legends, info boxes, etc. But they are not impossible. Fortunately Leaflet, an alternative to Google maps, is designed to work especially well on mobile platforms.

In this example, we will be loading data into a Google spreadsheet and using Leaflet to map the data on a responsive map.

1. Learn a little bit of Leaflet

Before we start, it would probably be best if you familiarize yourself with Leaflet. Fortunately, their website has some wonderful walk-throughs. I’d recommend going through this one before going anything further.

2. Grab the code

Now, go to my Github page and download the template.

The Readme file includes instructions on how to set up a basic map using Google spreadsheets and Tabletop.js, which is a wonderful tool that allows us to do all kinds of things with data in a Google spreadsheet, including map it using Leaflet.

3. Edit your index.html page

After you have followed the instructions on my Github page, you should have a map ready to go.

All you have to do is go into the index.html page and edit the title of the map, as well as add your own information into the “sidebar_content” div. Also make sure you add your name to the credits because you deserve credit for this awesome map you are putting together.

4. How does it work?

Now open up your map in a browser. If you rearrange the browser window size, you’ll notice that the map rearranges its size. The other components on the page also automatically readjust.

Some of this is done with the Bootstrap web framework, which was designed with responsive designing in mind.

I’ve also added my own CSS. One easy thing I’ve done with elements on the page is declare their widths and heights using percentages instead of pixels. This ensures that the components will automatically be adjusted regardless of the screen size.

Take a look at our css/styles.css file to get an idea of what I’m talking about: 

/* Body */
body {
    padding-left: 0px;
    padding-right: 0px;
  margin: 0;
    height: 100%;
}

html {
	height: 100%;
}


/* Map */
#map {
	position: absolute;
	float: left;
	top: 1%;
	height: 98%;
	width: 100%;
	z-index: 1;
}

The map’s height is 98 percent and width is 100 percent, ensuring it changes its size when the browser changes its size. If we set it to 600 pixels wide, the map would stay 600 pixels wide, even when the browser was adjusted.

– You’ll notice some other changes. For instance, if you have a wide screen, the map’s sidebar will be on the right side of the screen. We did this by using absolute positioning to place the sidebar and its content on the page:

/* Sidebar */
#sidebar {
  position: absolute;
	top: 2%;
	right: 1%;
	height: 96%;
	width: 30%;
	z-index: 2;
	border: 1px solid #999;
	padding-left: 1%;
	padding-right: 1%;
	background-color: #FFFFFF;
    background-color:rgba(255,255,255,0.9);
}

#sidebar h3 {
	line-height: 30px;
}

#sidebar_content {
	float: left;
	width: 30%;
	height: 70%;
	position: fixed;
	overflow: auto;
	padding-top: 5px;
}

The sidebar’s “right” position is set to 1 percent. This ensures that the sidebar will appear only 1 percent from the right side of the page. Additionally, its “top” position is set to 2 percent. This, effectively, pushes it to the top right corner of the screen.

We also used percentages to declare widths, heights and padding lengths for the sidebar.

– You’ll also notice when your browser is reduced drastically, the content of the sidebar disappears off the page. Instead, we have just the title of the sidebar at the top of the page. This is done with CSS media queries:

/* Styles from mobile devices */
@media (max-width: 625px) {
  
  #sidebar_content {
  	display: none;
	}

  /* Sidebar */
  #sidebar {
		position: relative;
		margin-top: 0%;
		float: left;
		left: 0%;
		right: 0%;
		top: 0%;
		padding-left: 2%;
		padding-right: 2%;
		height: 35px;
		width: 96.5%;
	}

}

Basically what the above code is saying is: If the browser is 625 pixels wide or smaller, apply the following CSS styles. These styles would therefore apply to almost all mobile phones. So what you are saying to the browser is: If this is a mobile browser, apply these styles to the elements on the page.

The first thing we do is hide the “sidebar_content” div, which is within our main “sidebar” div. Besides the “sidebar_content” div, we also have a div within the “sidebar” div called “sidebar_header” for our title. The template sets the title to “Tabletop to Leaflet” initially, although you should change that to match your project.

We hide the “sidebar_content” div with the property “display: none.” Hiding it ensures that the only thing left in our “sidebar” div is the title. Then the sidebar is pushed to the top left corner of the page using absolute positioning.

So what do we do with that information we have hidden? We put it in another div using some Javascript. Then we toggle that div from hidden to visible using a button with the class “toggle_description.” This toggle feature is enabled using jQuery.

From our js/script.js file:

// Toggle for 'About this map' and X buttons
// Only visible on mobile
isVisibleDescription = false;
// Grab header, then content of sidebar
sidebarHeader = $('#sidebar_header').html();
sidebarContent = $('#sidebar_content').html();
// Then grab credit information
creditsContent = $('#credits_content').html();
$('.toggle_description').click(function() {
  if (isVisibleDescription === false) {
		$('#description_box_cover').show();
		// Add Sidebar header into our description box
		// And 'Scroll to read more...' text on wide mobile screen
		$('#description_box_header').html(sidebarHeader + '<div id="scroll_more"><strong>Scroll to read more...</strong></div>');
		// Add the rest of our sidebar content, credit information
		$('#description_box_content').html(sidebarContent + '<br />' + 'Credits:' + creditsContent);
		$('#description_box').show();
		isVisibleDescription = true;
	} else {
		$('#description_box').hide();
		$('#description_box_cover').hide();
		isVisibleDescription = false;
	}
});

The above code first grabs the information from the “sidebar_content” div, then places it in the “description_box” div. It also sets our toggle function, which is activated when the user clicks on the button with the class “toggle_description.”

– The “description_box” div is also styled similarly to the “sidebar” div. The big difference is the “description_box” div is hidden by default because we only want it shown if we are on a mobile phone. The button with the class “toggle_description” is also hidden by default.

From our css/styles.css file:

/* 'About this map' button, description box */
/* Mobile only */
.toggle_description {
  display: none;
	z-index: 8;
	position: relative;
	float: right;
    right: 0%;
    top: 0%;
}

#description_box_cover {
	display: none;
	z-index: 10;
	position: absolute;
	top: 0%;
	width: 100%;
	height: 100%;
    background-color: #444444;
    background-color:rgba(44,44,44,0.9);
}

#description_box {
	position: absolute;
	display: none;
	z-index: 11;
	width: 92%;
	height: 93%;
	padding-top: 1%;
	padding-left: 1%;
    padding-right: 1%;
    left: 2.5%;
    top: 2.5%;
    border: 1px solid #999;
    background-color: #FFFFFF;
    background-color:rgba(255,255,255,0.9);
}

#description_box h3 {
	padding-bottom: 0px;
	line-height: 15px;
}

Now we do the opposite with the “toggle_description” when compared to what we did with the “sidebar_content” div.

With the “sidebar_content” div, we had it shown by default then hidden on mobile phones using CSS media queries. With the button, we hide it by default and then show it on mobile phones using a CSS property of “display: inline.”

From our css/styles.css file:

/* Styles from mobile devices */
@media (max-width: 625px) {

	.toggle_description {
		display: inline;
	}

}

As a noted above, when someone clicks on that button, jQuery toggles between hidden and shown on the button with the class “toggle_description”. It is hidden by default, so it is shown when the user first clicks  it. Then when the user clicks the blue X button (which also has the class of “toggle description”), the box disappears.

A similar philosophy is in place to hide and show the credits box.

That should give you a good idea of what is happening with this map. Feel free to fork the repo and create your own awesome maps.

Have any questions? Don’t hesitate to leave a comment.

Final infographics project

leave a comment »

For the last six weeks, I’ve taken a very awesome online course on data visualization called “Introduction to Infographics and Data Visualization” It is sponsored by the Knight Center for Journalism in Americas and taught by the extremely-talented Albert Cairo. If you have a quick second, check out his website because it’s phenomenal.

Anyways, we have reached the final week of the course and Cairo had us all make a final interactive project. He gave us free reign to do whatever we want. We just had to pick a topic we’re passionate about. Given that I was a cops and courts reporter for a year in Galesburg, Illinois before moving to Waterloo, Iowa, I am passionate about crime reporting. So I decided for my final project I’d examine states with high violent crime rates and see what other characteristics they have. Do they have higher unemployment rates? Or lower education rates? What about wage rates?

Obviously, this is the type of project that could be expanded upon. I limited my final project to just four topics mostly because of time constraints. I work a full time job, you know! Anyways, here’s my final project. Let me know if you have any suggestions for improvement.

teaser

About:

Data: Information for the graphic was collected from four different sources, which are all listed when you click on the graphic. I took the spreadsheets from the listed websites and took out what I wanted, making CSVs for each of the four categories broken down in the interactive.

Map: The shapefile for the United States was taken from the U.S. Census Bureau’s website. Find it by going to this link, and selecting “States (and equivalent)” from the dropdown menu. I then simplified the shapefile by about 90 percent using this website. Simplifying basically makes the lines on the states’ polygons less precise but dramatically reduces the size of the file. This is important because people aren’t going to want to wait all day for your maps to load.

Merging data with map: I then opened that shapefile with an awesome, open-source mapping program called QGIS. I loaded up the four spreadsheets of data in QGIS as well using the green “Add vector layer” button (This is important! Don’t use the blue “Create a Layer from a Delimited Text File” button). The shapefile and all the spreadsheets will now show up on the right side of the screen in QGIS under “Layers.”

Each spreadsheet had a row for the state name, which matched the state names for each row in the shapefile.  It’s important these state names match exactly. For instance, for crime data from the FBI I had to lowercase the state names. So I turned “IOWA” into “Iowa” before loading it into QGIS because all the rows in the shapefile were lowercase (“Iowa” in the above example).

Then you can open the shapefile’s properties in QGIS and merge the data from the spreadsheets with the data in the shapefile using the “Joins” tab. Finally, right click on the shapefiles layer then “Save As” then export it as a GeoJSON file. We’ll use this with the wonderful mapping library Leaflet.

qgis_teaser

Leaflet: I used Leaflet to make the map. It’s awesome. Check it out. I won’t get into how I made the map interactive with Javascript because it’s copied very heavily off the this tutorial put out by Leaflet. Also check it out. The only thing I did differently was basically make separate functions (mentioned in the tutorial) for each of my four maps. There is probably (definitely) a better way to do this but I kind of ran out of time and went with what I had. If you’re looking for the full code, go here and click “View Page Source.”

Design: The buttons used are from Twitter’s Bootstrap. I used jQuery’s show/hide functions to show and hide all the element the page. This included DIVs for the legend, map and header.

GeoJSON: The last thing I did was modify my GeoJSON file. You’ll notice how the top 10 states for violent crime rates are highlighted in black on the maps to more easily compare their characteristics across the maps. Well, I went into the GeoJSON and put all those 10 states attributes at the bottom of the file. That way they are loaded last on the map and thus appear on top of the other states. If you don’t do this, the black outlines for the states don’t show up very well and look like crap. Here’s GeoJSON file for reference.

Hopefully that will help guide others. If you have any questions, feel free to drop me a line. Thanks!

 

Written by csessig

December 11, 2012 at 12:13 am

How We Did It: Waterloo crime map

with 3 comments

Note: This is cross-posted from Lee’s data journalism blog. Reporters at Lee newspapers can read my blog over there by clicking here.

Last week we launched a new feature on the Courier’s website: A crime map for the city of Waterloo that will be updated daily Monday through Friday.

The map uses data provided by the Waterloo police department. It’s presented in a way to allow readers to make their own stories out of the data.

(Note: The full code for this project is available here.)

Here’s a quick run-through of what we did to get the map up and running:

1. Turning a PDF into manageable data

The hardest part of this project was the first step: Turning a PDF into something usable. Every morning, the Waterloo police department updates their calls for service PDF with the latest service calls. It’s a rolling PDF that keeps track of about a week of calls.

The first step I took was turning the PDF into a HTML document using the command line tool PDFtoHTMLFor Mac users, you can download it by going to the command line and typing in “brew install pdftohtml.” Then run “pdftohtml -c (ENTER NAME OF PDF HERE)” to turn the PDF into an HTML document.

The PDF we are converting is basically a spreadsheet. Each cell of the spreadsheet is turned into a DIV with PDFtoHTML. Each page of the PDF is turned into its own HTML document. We will then scrape these HTML documents using the programming language Python, which I have blogged about before. The Python library that will allow us to scrape the information is Beautiful Soup.

The “-c” command adds a bunch of inline CSS properties to these DIVs based on where they are on the page. These inline properties are important because they help us get the information off the spreadsheet we want.

All dates and times, for instance, are located in the second column. As a result, all the dates and times have the exact same inline left CSS property of “107” because they are all the same distance from the left side of the page.

The same goes for the dispositions. They are in the fifth column and are farther from the left side of the page so they have an inline left CSS property of “677.”

We use these properties to find the columns of information we want. The first thing we want is the dates. With our Python scraper, we’ll grab all the data in the second column, which is all the DIVs that have an inline left CSS property of “107.”

We then have a second argument that uses regular expressions to make sure the data is in the correct format i.e. numbers and not letters. We do this to make sure we are pulling dates and not text accidently.

The second argument is basically an insurance policy. Everything we pull with the CSS property of “107” should be a date. But we want to be 100% so we’ll make sure it’s integers and not a string with regular expressions.

The third column is the reported crimes. But in our converted HTML document, crimes are actually located in the DIV previous to the date + time DIV. So once we have grabbed a date + time DIV with our Python scraper, we will check the previous DIV to see if it matches one of the seven crimes we are going to map. For this project, we decided not to map minor reports like business checks and traffic stops. Instead we are mapping the seven most serious reports.

If it is one of our seven crimes, we will run one final check to make sure it’s not a cancelled call, an unfounded call, etc. We do this by checking the disposition DIVs (column five in the spreadsheet), which are located before the crime DIVs. Also remember that all these have an inline left CSS property of “677”.

So we check these DIVs with our dispositions to make sure they don’t contain words like “NOT NEEDED” or “NO REPORT” or “CALL CANCELLED.”

Once we know it’s a crime that fits into one of our seven categories and it wasn’t a cancelled call, we add the crime, the date, the time, the disposition and the location to a CSV spreadsheet.

The full Python scraper is available here.

2. Using Google to get latitude, longitude and JSON

The mapping service I used was Leaflet, as opposed to Google Maps. But we will need to geocode our addresses to get latitude and longitude information for each point to use with Leaflet. We also need to convert our spreadsheet into a Javascript object file, also known as a JSON file.

Fortunately that is an easy and quick process thanks to two gadgets available to us using Google Docs.

The first thing we need to do is upload our CSV to Google Docs. Then we can use this gadget to get latitude and longitude points for each address. Then we can use this gadget to get the JSON file we will use with the map.

3. Powering the map with Leaflet, jQRangeSlider, DataTables and Bootstrap

As I mentioned, Leaflet powers the map. It uses the latitude and longitude points from the JSON file to map our crimes.

For this map, I created my own icons. I used a free image editor known as Seashore, which is a fantastic program for those who are too cheap to shell out the dough for Adobe’s Photoshop.

The date range slider below the map is a very awesome tool called jQRangeSlider. Basically every time the date range is moved, a Javascript function is called that will go through the JSON file and see if the crimes are between those two dates.

This Javascript function also checks to see if the crime has been selected by the user. Notice on the map the check boxes next to each crime logo under “Types of Crimes.”

If the crime is both between the dates on the slider and checked by the users, it is mapped.

While this is going on, an HTML table of this information is being created below the map. We use another awesome tool called DataTables to make that table of crimes interactive. With it, readers can display up to a 100 records on the page or search through the records.

Finally, we create a pretty basic bar chart using the Progress Bars made available by Bootstrap, an awesome interface released by the people who brought us Twitter.

Creating these bars are easy: We just need to create DIVs and give them a certain class so Bootstrap knows how to style them. We create a bar for each crime that is automatically updated when we tweak the map

For more information on progress bars, check out the documentation from Bootstrap. I also want to thank the app team at the Chicago Tribune for providing the inspiration behind the bar chart with their 2012 primary election app.

The full Javascript file is available here.

4. Daily upkeep

This map is not updated automatically so every day, Monday through Friday, I will be adding new crimes to our map.

Fortunately, this only takes about 5-10 minutes of work. Basically I scrape the last few pages of the police’s crime log PDF, pull out the crimes that are new, pull them into Google Docs, get the latitude and longitude information, output the JSON file and put that new file into our FTP server.

Trust me, it doesn’t take nearly as long as it sounds to do.

5. What’s next?

Besides minor tweaks and possible design improvements, I have two main goals for this project in the future:

A. Create a crime map for Cedar Falls – Cedar Falls is Waterloo’s sister city and like the Waterloo police department, the Cedar Falls police department keeps a daily log of calls for service. They also post PDFs, so I’m hoping the process of pulling out the data won’t be drastically different that what I did for the Waterloo map.

B. Create a mobile version for both crime maps – Maps don’t work tremendously well on the mobile phone. So I’d like to develop some sort of alternative for mobile users. Fortunately, we have all the data. We just need to figure out how to display it best for smartphones.

Have any questions? Feel free to e-mail me at chris.essig@wcfcourier.com.

Tip: Embedding Vmix videos into a Google Fusion table

leave a comment »

Note: This is cross-posted from Lee’s data journalism blog. Reporters at Lee newspapers can read my blog over there by clicking here.

For any map makers out there, here’s a walk-through on how to take a Vmix video and post it into a Google Fusion table. It’s a perfect follow-up to the tutorials Chris Keller and I held a month ago with Lee journalists on how to build maps with Google Fusion Tables.

1. First we need a Vmix video so post one onto your website like you normally would by uploading it to Vmix and pulling it into the Blox CMS. I’m going to use this video in our example.

2. View the source of the page by right clicking on the page and selecting “View page source.” Then search for a DIV with the class of “vmix-player”. You can do this by searching for “vmix-player”.

3. Under that should be a Javascript file with a source that starts with “http://media.vmixcore.com/&#8221;. Click on that link to open up the source in a new window. You should now see a screen with a huge “Not Found” warning. But don’t be discouraged.

4. Now view the source of that page by doing the same thing you did before (Right click > “View page source”).

5. You should now see a page with three variables: t, u and h. The variable we want is “h”, which is the object tag we will embed into the map.

The page should look something like this.

6. Clean up the variable tag by removing these tags:

var h = ”

(This marks the beginning of the variable.)

h += ”

(There should be several of these. Basically this adds whatever follows  it to the “h” variable, hints the plus sign.)

“;

(These are at the end of every line of code.)

7. Now we need to replace all references to the “t” and “u” variables with their actual value. You’ll notice that “t” and “u” appear in the code for the “h” variable and are surrounded by plus signs. Basically that is just telling Javascript to put whatever “t” equals into that spot. We’ll do that manually:

So replace:

” + t + ”

With:

location.href

And replace:

” + u + ”

With:

http://cdn-akm.vmixcore.com/player/2.0/player.swf?player_id=48df95747124fbbad1aea98cee6e46e4

(Your link will be different than mine)

– It’s important to note that you need to delete the equal signs and the plus signs before and after “t” and “u”. Your final code should not have a double quote next to a single quote. We should have just single quotes around “location.href” and our “http://cdn-adk.vmixcore.com&#8221; link.

– It’s also important to note that when we grab the “t” and “u” variables, we don’t grab the semi-colon at the end of the variable or the quotes around the “u” variable.

For instance, let’s say we have this for our “u” variable:

var u = "http://cdn-akm.vmixcore.com/player/2.0/player.swf?player_id=48df95747124fbbad1aea98cee6e46e4";

So on our movie parameter, we’re turned this line of code:

<param name='movie' value='" + u + "'/>

Into this line of code:

<param name='movie' value='http://cdn-akm.vmixcore.com/player/2.0/player.swf?player_id=48df95747124fbbad1aea98cee6e46e4'/>

– Repeat this for every reference of “t” and “u” in the code.

Our final piece of code should look like this garbled mess.

8. The final step is to post that object tag above into a Google Fusion Table. The easiest way to do this is create a new column called “video” and simply put the above code into that column’s row.

9. Then configure the info window (Visualize > Map > Configure info window) and make sure the “video” column name appears in the HTML of the info window.

If you want to pull in just the “video” column and nothing else, you’re HTML would look like this:

<div class="googft-info-window">{video}</div>

The result looks something like this map. Click on the blue marker labeled “3” to see the video.

I am using the Fusion Table API to make my map instead of using the embed code provided by Google. It seems to work better with videos. If you are interested in see my full code for this map, click here.

That’s it. If you have any questions or something doesn’t make sense, please leave a comment or e-mail at chris.essig@wcfcourier.com.

Written by csessig

July 28, 2012 at 5:49 pm

Data journalism resources

leave a comment »

Note: This is cross-posted from Lee’s data journalism blog. Reporters at Lee newspapers can read my blog over there by clicking here.

Okay this is going to be a quick blog post since we and every other Lee newspaper are switching over our sites to Templates 2.0. Here’s a couple of neat websites I’ve ran across in the past couple of weeks that may help with your next data story and/or visualization:

1. Data Journalism Handbook

Want to know about the importance of data journalism today and what is possible for you to do at your own newspaper? Then this handbook is a must read. One of the great aspects of the book is it shows how data can be used not only to make impressive graphics but also great stories. And best of all it is free and growing.

2. DataVisualization: Selected tools

Here’s a great resource for many of the tools used by data journalists across the world. For instance, Google Fusion Tables is the go-to service for map-makers, while Google Refine is a great program for cleaning data to use with maps, visualizations or just stories. Also check out ColorBrewer to help you find great color patterns.

One Javascript library worth checking out is D3.js, which has been used by the New York Times to create some jaw-dropping visualizations. Also worth noting are two blog posts from software developer Jim Vallandingham. One deals with making bubble charts with D3.js. The second shows how to use the library to work with past versions of Internet Explorer. This is a must read for sites like ourselves that have a high percentage of readers who use Internet Explorer.

3. Data Stories

If you are like me, you love listening to podcasts, especially after work when your eyes can no longer stare at your computer screen. This site features a series of podcasts on data visualization. Topics include how to learn data visualization, when to use animated graphics and other advice from people smarter than I will ever be. So check it out if you are looking for something new to listen to while you hit the gym, bike trail or couch.

Written by csessig

June 5, 2012 at 2:00 pm

Turning Blox assets into timelines: Part 1

with 3 comments

Note: This is cross-posted from Lee’s data journalism blog. Reporters at Lee newspapers can read my blog over there by clicking here.

Also note: You will need to run on the Blox CMS for this to work. For part two of this tutorial, click here. For part three, click here.

 

A couple weeks ago I blogged about the command line and a few reasons journalists should learn it. Among the reasons was a timeline tool made available by ProPublica called TimelineSetter, which is shown to the left. Here are two live examples to give you an idea of what the tool looks like.

To create the timeline, you will first need to make a specially-structured CSV file (CSV is a type of spreadsheet file. Excel can export CSV files). Rows in the CSV file will represent particular events on the timeline. Columns will include information on those events, like date, description, photo, etc.

ProPublica has a complete list of available columns here. To give you an idea of what the final product will look like BEFORE you make the timeline, you can download one CSV file we used to make a timeline by clicking here.

After you have your CSV file, you run a simple command and walla! A beautiful timeline will be created. For more information on the command you have to run, check out the TimelineSetter page. (Hint: The command you run is: timeline-setter -c timeline.csv)

By far the most tedious part of all this is tracking down events and articles you want to include in the timeline and making your CSV file. That is why I wrote a simple Python script that will help turn Blox assets into a CSV file you can use with TimelineSetter.

Here’s a walkthrough of how to use it:

1. The first thing you need to do is go to this GitHub page and follow the instructions in the ReadMe file. After that you will have a page set up will all of the events you want to include in the timeline. Here’s an example of what that page should look like.

2. Download the Python script (Timeline.py). What this script will actually be doing is scraping the web page we just created. And by that I mean it will be going to that page and pulling out all the information we need to create our timeline. So it will be grabbing photos, headlines, dates, etc. It will then create a CSV file with all that information. We can then use that CSV file with TimelineSetter.

3. The script uses a Python library called Beautiful Soup. If you don’t have that downloaded already, click here. It takes only a few seconds to install.

4. In the Timeline.py file on line 16 is a spot for the URL of the page we want to scrape. Make sure you change that to whatever URL you created.

5. Run the command python timeline.py from your command line in the same directory as the Python script you downloaded. This will output a CSV file.

6. You will need to download TimelineSetter, which is also really easy to do. Just run this command: gem install timeline_setter. For more information, click here.

7. Now navigate to the folder with the CSV file and run this command: timeline-setter -c timeline.csv. (Or whatever your CSV file is called).

8. You should end up with a directory of javascript files, a CSS file and a Timeline.html file. This is your timeline. Now put in on your server and embed it using an HTML asset in Blox (or whatever you want to do with it).

9. Do the happy dance (Mandatory step)

This will get you pushing out timelines in no time. On my next blog, I will be going through that Python script (Timeline.py) and what it actually does to create the CSV file.

Written by csessig

March 2, 2012 at 9:59 am

Better map rollover option

with 2 comments

A month ago, I blogged about an attempt I made to use a new feature from Google Fusion Tables that allowed map makers to customize their maps based on mouse overs. The idea for users was you could rollover a point/state/census tract/whatever on a map and some some sort of data would pop up on that map. You also customize it so the polygon changes colors, polygon borders grow in size or whatever option you decide to use to let the reader know they have rolled over a particular object. I used it on a recent map of poverty rates in Iowa. The result worked but mouseover events seem delayed and clunky. Not so user friendly, especially if you have a slow Internet connection. So I looked for a new option.

What I found was this great library from NY Time’s Albert Sun for polygons and rollover effects. Granted, this takes much longer to put together. But the result is much smoother and user friendly IMO. I used it on a recent map of heroin rates in the U.S.

Here’s brief synopsis on how I put it together:

1. First I grabbed data from the Substance Abuse and Mental Health
Data Archive
 on reported heroin cases at substance abuse centers broken down by state. I then grabbed a shapefile from the U.S. Census bureau of each state in the U.S. This is the file that maps each state based on its boundaries.

2. The data related to drug cases from the SAMHDA was sorted by state and year. You can download individual spreadsheets of data for years 2009 and before. I downloaded spreadsheets for 2005 through 2009. I then pulled out the information I needed from each one and merged them, ending up with a final spreadsheet that had the following data for each state dating back to 2005: number people in who were admitted to a substance abuse treatment center for heroin, number of people who were admitted to a substance abuse treatment center for any drug and the percentage of people who were heroin users.

3. I first uploaded that final spreadsheet to Google Fusion Tables. I then uploaded the shapefile I downloaded into Google Fusion Tables using the AWESOME Shpescape tool. Important: Make sure you select “Create a Simplified Geometry column” under advanced options before you import the shapefile, otherwise it will be too big. Finally I merged the shapefile with the spreadsheet and ended up with this final table.

4. I exported that table into a KML (If you click Vizualize > Map, you’ll see the option to download a KML file) and converted that to a GeoJSON file using this Ogre web client. I only did this because the KML wasn’t working on Internet Explorer. Anyways, converting it into a GeoJSON file was a TOTAL PAIN IN THE ASS that required me to mess with the GeoJSON file much more than I wanted. Finally, I ended up with this GeoJSON File.

5. Now for the code. My final script is here. I’m not going to run through it all but will point out a few lines of note:

  • Line 179 brings in the JSON file. Underneath it are polygon highlighting options based on Albert Sun’s code. I have it set up so the opacity on the state polygons is light enough to see the state names underneath it when you first pull up the map. When you rollover a state, the polygons gets shaded in all the way and you can’t read the text underneath it (on the Google Map). But if you click a polygon, you can once again see the text. I also have a red border pop up when you rollover a state (strokeColor and strokeWeight).
  • Line 207 is the highlightCallback function. This creates a variable for each of line of data: number of heroin users, number of drug users, percentage, etc. for 2005 to 2009. It’s what you see under “Figures” in the DataTable on the map when you rollover a state. I first made each line a string by adding quotes to each variable.
  • Each variable is called into function selected_district (line 229). This function creates the Google DataTable via “new google.visualization.DataTable().” I’ve used this table in the past on a map for prep high school football teams. Check this past blog post for more information.
  • Line 255 is a function that puts in commas for numbers in the thousands…I didn’t make it. It’s freely available online. Please take it and use it as you see fit.
  • Line 107 to 154 is the legend.

Per usual, I used Colorbrewer to come up with the colors…

I’m happy with the resulting map and hope to use this polygon feature in the future. If you have any questions, feel free to leave a comment. I’d be more than happy to pass my limited amount of knowledge along to others.

Written by csessig

January 21, 2012 at 5:07 pm

The caucus night that almost didn’t end

leave a comment »

All eyes were on Iowa last night as the Iowa caucuses took place. It was pretty much the longest work day I’ve ever had…By alot. Anyways, we did a ton of updating on WCFCourier.com all day and night…I wish I had a screen shot of all the photos/stories we put on the front of our website.  I did take one at about 2:30 a.m., which is shown above. It’s the site after Romney was (finally) declared the winner. The template is now being used on our Iowa caucus website.

Here’s a quick summary of our online coverage: It started with a general Iowa caucus coverage, switched over to our live coverage from the UNI-Dome (which hosted the largest caucus in the state Tuesday night) and then to the statewide race between Santorum-Romney-Paul and, finally, the grudge match between Santorum and Romney. At 1:30 a.m., the Iowa GOP announced Romney had won by eight flippin’ votes. At one point, Santorum was leading by ONE vote with ONE precinct left. What are the odds?

My schedule on caucus night went something like this:

– 9 a.m. – 2:30 p.m. –  Preparing for the day / arranging plans with reporters /posting stories, photos and other content. We had caucus stories going up all day, obviously. I also posted and helped monitor a live chat, which was shared with other Iowa newspapers and was active all day, as well as posted live video from KCRG, which played from about 7 p.m. to 12 a.m. I opted to put both the live video and the live chat on the same page, making it easier for readers to follow action at home.

– 2:30 p.m. – 2:31 p.m. – Lunch

– 2:31 p.m. – 4:30 p.m. – More preparing and posting. We also posted two maps with our coverage: one of live caucus results (which started coming in after the 7 p.m. caucus start). This map was provided by the Iowa Republican Party and is pictured to the left. You may have seen it on several news sites… Many had it or a variation of it.

The second map I made myself and featured caucus locations for all (I believe, although I haven’t counted) 1,700 caucus locations in the state of Iowa. The addresses were pulled from the Iowa GOP website, which listed every site. Basically, I wrote a Python program that scraped the data from their site and put it into a spreadsheet, pulled it into Google Fusion tables and mapped the locations based on their addresses. The Python scraper is based on this FANTASTIC walk-through by BuzzData on how to scrape data from websites. Check it out!

(NOTE: Here’s the code for the Python scraper. Here’s my Google Fusion Table.)

At about 3 p.m., we rolled over the site to feature one huge photo and story (see the screenshot at the top of this post). It was caucus night, after all,  so we had to go big.

– 4:30 p.m. – 5 p.m. – Mad dash to the UNI-Dome, where Black Hawk County was caucusing. The doors opened at 5:30 p.m. and I wanted to get there and set up before either Bachmann or Gingrich stopped by to speak.

– 5 p.m. – 10 p.m. – Posted up at the UNI-Dome. At about 5:30 p.m., we switched our main story to our Dome coverage…This was basically when our first photo and update came in. Throughout the evening, we posted small updates from the Dome and new photos. We also had three videos from the Dome.

At about 6:30 p.m., Bachmann and Gingrich spoke. I took a few photos for our live chat (which, BTW, had more than 5,000 viewers at one point!) and posted a fresh candidate Dome story when it came in.

At about 8:30 p.m., the Dome action was winding down and our attention turned to the statewide race between Santorum-Romney-Paul and then Santorum-Romney. We relied on the AP our Lee Des Moines bureau for our main story on the site, adding photos from the Dome and the wire with it.

– 10 p.m. – 10:30 p.m. – Mad dash back to the newsroom. I was actually afraid they might announce the winner while I was on the road back to the newsroom but I was off by about three hours.

– 10:30 p.m. – 12:30 a.m. – We waited. And waited. And made some jokes. And waited some more. The precinct results continued to flood in and amazingly the number of votes between Romney and Santorum dwindled. Santorum was actually in the lead for much of the night. By 16 votes. Then one vote. Then four votes. Just ridiculous.

– 12:30 a.m. – 1:30 a.m. – At about this time, they announced they were down to three precincts then one precinct…At that point, I knew I would be in the newsroom until the final precinct was counted. The lone holdout was in Clinton County (eastern Iowa along the river) and apparently there was some confusion about whether or not they had submitted their results to the state yet.

– 1:30 a.m. – 2 a.m. – The Iowa GOP finally announces Romney won by eight (!!!) votes. Hurray! I slapped a quick update on top of story we had online and added a new photo. At this point, I just wanted to make sure those who got up in the morning would see the final results.

– 2 a.m. – 3 a.m. – The longest day ever came to a close. I took down the big photo, big story template we had used all night (see screenshot at the top) and returned the site back to our standard carousel template with five rotating stories on the front (see WCFCourier.com). I also added a teaser to our Iowa caucus site on the top so people see all of our caucus coverage from the night/morning. Because there was a ton of it.

– 3 a.m. – Sleep

Here’s what the Courier’s front page looked like on Wednesday. We’re a morning paper so we were able to get the final results in:

Written by csessig

January 4, 2012 at 1:42 pm

Map mouseover test

with one comment

A couple of weeks ago I saw some tweets on a new feature that would allow Google Fusion Table map makers to customize their maps based on mouse overs (Here’s some background and the  ‘workable’ solution that was released earlier this month). This was exciting news. In the past, people who were looking at a map would typically have to click on a point or a polygon or whatever to open up new data about that point, polygon, etc.  Now, the mouseover effect would allow new data to pop up if a reader hovers over a point, polygon, etc. with their mouse.

It sounds great so I tried it out over the weekend with this map on poverty rates in Iowa. The map is broken into Census tracts and when a reader hovers over a polygon, poverty data about that particular tract pops up. I also set it so the polygon changes colors on a mouse rollover.

It didn’t turn out too bad. But I feel the polygon color changes take a while to load and frankly doesn’t feel that slick. There are plenty of other options for messing with polygons (I’ve heard the Raphael Javascript library works great and Albert Sun with the NY Times has a great library for polygon effects) but nothing I’ve seen is as simple and as quick to turn around as this workable solution… Overall, there seems to be a ton of promise here, especially for us in the news business who are trying to turn around maps on deadline.

Anyways, anybody who wants to build off this map should check out the code. And I’d love to hear any suggestions on how to improve on it.

Written by csessig

December 14, 2011 at 4:33 pm

Map Mania

with one comment


Here are a few maps I’ve worked on in the last couple of months:

1. City breakdown of property taxes in Iowa

– This map (shown above) includes property tax rates for every city in Iowa (900+) dating back to 2010 and color-codes them based on their rate. I won’t go into too much detail on how I went about doing it; instead, I’ll refer you to this great Poynter tutorial on how to make a heat map using Google Fusion Tables (which is what the above map is based off of). When you’re done with that, check out their more recent post on mapping data by county, cities, etc. using what is known as shapefiles (Don’t worry. I didn’t know what they were either until I read the post).

For this map, I first found tax rates for every city in Iowa on the Iowa Department of Management’s website. These rates were contained in several spreadsheets (City Tax Rates FY12, City Consolidated Tax Rates FY12, FY 11, etc.).For each spreadsheet I used (six in all: three for city tax rates dating back to 2010 and three for consolidated tax rates dating back to 2010), I basically went in and chopped out all the information in the spreadsheets I didn’t need (Example spreadsheet), which would up being most of it. I then merged the six spreadsheets by city name using Google Fusion Tables.

I then found the shapefile that maps every city in Iowa on the Census site, which contained the geographical data that Google uses when it maps out the cities. I then merged that spreadsheet with the one containing all the tax rates and pulled it into Google Fusion Tables.

Google did the heavy lifting so to speak and mapped out all the cities on the map. I then told FT to map the cities based on their tax rate. The final product is cities with higher tax rates are red; cities with lower tax rates are yellow. The legend is HTML/CSS, which I copied heavily off of the Chicago Tribune News App and their wonderful map walkthrough. When you get the hang of Fusion Tables, check it out. Their maps are much prettier than mine.

The property tax map is on my (barren) Github account. Please copy anything you find helpful.

2. 2008 Flood Buyouts

Compared to the above map, the flood buyout map was a piece of cake to make. We got a spreadsheet from the city of Cedar Falls that listed every home that was offered a buyout from the federal government and their address. Besides mapping shapefiles (like above), Google can also map simple addresses (obviously). So Google can map a spreadsheet of addresses as well.

3. Road to the Dome: A look at the prep teams in the semifinals

I think high school football is popular everywhere. The Cedar Valley is certainly no exception. In Iowa, we have six football classes based on size of the schools. Each year, the semifinals and final rounds of each class are played at the UNI Dome in Cedar Falls. This package breakdowns each of the 24 teams in the semifinals and maps them based on their school address. Readers can then click on each pointer on the map and read more about the team.

From a technical standpoint, this is the first map I’ve put together where a info box DOESN”T open up when a reader clicks on a point on the map. Instead, the information on the teams opens in a table format to the right of the map. This uses Google’s API; I borrowed heavily off of this map + chart example provided by Google. The main difference being is I called a table visualization instead of a chart visualization in the example.

The table below the map (the one where you can select a class and see the team’s playoff schedule) was borrowed heavily off this Google table example.

Here’s the table I ended up with on Google Fusion Tables. If you’re looking for the map source, go to this webpage and click view source. This is where the map is housed.

My Google-powered Javascript is pasted below for anyone who is curious:

google.load('visualization', '1', {'packages':['table']});

function initialize() { 
 var map = new google.maps.Map(document.getElementById('map_canvas'), {
 center: new google.maps.LatLng(42.3, -453.4),
 zoom: 7,
 minZoom: 5,
 maxZoom: 11,
 mapTypeControl: false,
 streetViewControl: false,
 mapTypeId: google.maps.MapTypeId.TERRAIN
 });

 var layer = new google.maps.FusionTablesLayer({
 query: {
 select: 'mappable_address',
 from: 2102557
 },
 suppressInfoWindows: true
 });
 layer.setMap(map);

 // Add a listener to the layer that constructs a chart from
 // the data returned on click
 google.maps.event.addListener(layer, 'click', function(e) {

 var data = new google.visualization.DataTable();
 data.addColumn('string', 'About:');
 data.addColumn('string', e.row['School'].value);
 data.addRows([
 ['Class', e.row['Class'].value],
 ['Mascot', e.row['Mascot'].value],
 ['Address', e.row['Address'].value],
 ['League', e.row['League'].value],
 ['League Record', e.row['League Record'].value],
 ['Regular Season Record', e.row['Season Record'].value],
 ['Playoffs - Round 1', e.row['Round 1'].value],
 ['Playoffs - Round 2', e.row['Round 2'].value],
 ['Quarterfinals', e.row['Q-Finals'].value],
 ['Semifinals', e.row['Semifinals'].value]
 ]);

 var chart = new google.visualization.Table(document.getElementById('chart'));
 var options = {
 'title': e.row['School'].value + ' ',
 };
 chart.draw(data, options);
 });
}

function changeData(team) {
 var whereClause = "";
 if(team) {
 whereClause = " WHERE 'Class' = '" + team + "'";
 }
 var queryText = encodeURIComponent("SELECT 'School', 'Round 1', 'Round 2', 'Q-Finals', 'Semifinals' FROM 2102557" + whereClause);
 var query = new google.visualization.Query('http://www.google.com/fusiontables/gvizdata?tq=' + queryText);

 query.send(getData);
}

function getData(response) {
 var table = new google.visualization.Table(document.getElementById('visualization'));
 table.draw(response.getDataTable());
}

Written by csessig

November 10, 2011 at 10:20 pm