Megan Taylor

front-end dev, volunteacher, news & data junkie, bibliophile, Flyers fan, sci-fi geek and kitteh servant

The Miso Project :: Dataset

The Miso Project :: Dataset

“Dataset is a JavaScript client-side data transformation and management library. Dataset makes managing client-side data easy by handling loading, parsing, sorting, querying & manipulating data from all sorts of sources.”

Miso project: how it will help you make your own Guardian-style infographics and data visualisations | guardian.co.uk.

April 20, 2012 | Comments Off on The Miso Project :: Dataset | Categories: Links | Tags: | Permalink

Introducing the data journalism handbook

Introducing the data journalism handbook | guardian.co.uk

“The launch of the Data Journalism Handbook next week is the result of a unique journalistic collaboration…The book’s contributors are a who’s who of data journalism. There are pieces by data journalists from the BBC, the Chicago Tribune, the Financial Times, Propublica and the New York Times. And that’s besides contributions from three of us at the Guardian.”

post

How I Learned to Stop Worrying and Love GIS

This week I started volunteering my time and skills at the Norwood News, a bi-weekly community newspaper serving the northwest Bronx communities of Norwood, Bedford Park, North Fordham and University Heights. The Norwood News is also part of the Bronx News Network, a series of community papers serving various neighborhoods in the West Bronx.

One of the projects I’m working on is a series of maps of various districts in the Bronx, starting with city council districts.

The first map was to be a simple graphic with no animation or interactivity. The second would be a map of all the city council districts in the Bronx, with clickable regions and information boxes for each district.

I started out with Photoshop and some images from the New York City Campaign Finance Board (for future reference, images are located here).

After about 45 mins of tinkering, I realized that method wouldn’t work. At the size that was required for the paper, I couldn’t get enough detail for people to easily figure out where the district lines were.

My next attempt was to start drawing the districts in Google Maps. By hand. (Using the My Maps Shape function)

I got frustrated enough with that to do what I should have done at the beginning: send out the call on Twitter asking for maps.

Derek Willis was kind enough to respond with the shapefiles from the New York City Department of City Planning. (A later search gave me this response to a question on Yahoo! Answers, with a long list of maps.)

Now, one of the reasons this hadn’t occurred to me before is that I’ve never really worked with GIS data before. I don’t have any software for it, and neither does the Norwood News.

So at this point I had to get the shapefiles into a format I could actually work with, preferably KML, which works with Google Maps.

A quick Google search brought me to Conversion of Shapefile to KML : An overview of tools available. It looked like my only option would be GeoCommons, a free online tool that lets users upload data and create maps from it.

GeoCommons will also provide a downloadable version of the data is CSV or KML format.

After downloading the KML file and uploading it to my own server, I plugged the link into Google Maps to take a look at what I had: city council districts for New York City.

Now I needed to narrow it down to just Bronx districts. A simple matter of removing the districts I didn’t need from the KML file, which conveniently labeled each data set with the correlating district number.

NYC_citycouncildistricts_BronxIn the end, this project probably took 5 or 6 hours. If I hadn’t been muddling around so much, I could have done it in one or two. But now I have a file of all the Bronx districts in KML, which can be altered to remove or add as much information as I want. And to see it, I just have to enter the URL for my KML file into Google Maps.

I’ve missed doing this kind of work (lately I’ve been doing more writing and very basic computing tasks) and really look forward to doing more projects like this one at the Norwood News.

post

New York should copy DataSF.org

New York City is in the process of opening a whole lot of data to developers as part of the BigApps competition.

Contestants will be asked to develop functional digital applications that will facilitate the dissemination of and greater access to publicly available City data. NYCEDC will manage the competition (including logistics and promotion) and the Department of Information Technology and Telecommunications (DoITT) will coordinate the formatting and release of selected City data to the public. The BigApps Competition will help to make City government more transparent, accessible and accountable and stimulate innovation in information technology that could lead to new businesses and job creation.

In order to do this, developers have to send in a request for data

These descriptions should provide as much detail as possible about the type and level of data desired. In addition, respondents are requested to describe how they envision the data being used in software applications that provide a useful service to City residents, visitors and government.

The Open Government NYC Meetup group is running BigAppsDevCamp, a workshop to help developers navigate the system of requests and proposals required by the city. They are also collecting project ideas.

Today, I saw an example of where New York should be heading. Infosthetics pointed out San Fransisco’s open data initiatives, including DataSF and San Francisco CrimeSpotting.

datasf

DataSF is an online repository of datasets available from the City & County of San Francisco. Similar to the goals of the data.gov and USASpending.gov initiatives, DataSF aims to improve access to data, help the community create innovative apps, understand what datasets the public likes to see, and receive feedback on the quality of the data. Included data ranges from all the trees located in the San Francisco streets (planting date, species, and location) to all its building permits or complaints.

In my opinion, that’s how New York should be running this competition. Don’t make developers try to guess how detailed your data is, or what you are collecting. I’m hoping there is an enterprising developer out there is who requesting ALL NYC data and will then make it all available to the public.

post

New York City addressing data availability, uses

I feel almost as if Mayor Bloomberg saw my previous post about NYC data.

we.gov PDF09 by stevegarfield

we.gov PDF09 by stevegarfield

The Sixth Annual Personal Democracy Forum (PDF) began yesterday. (Recap) The forum is examining the convergence of new media and politics, and includes speakers such as Craig’s List’s Craig Newmark, Twitter’s Jack Dorsey, venture capitalist Esther Dyson, new media evangelist Jeff Jarvis, and FiveThirtyEight’s Nate Silver.

Among the topics being discussed are:

  • State-of-the-art online politics and advocacy
  • Designing .gov for participation
  • Twitter as a platform for organizing and fundraising
  • The future of political journalism, blogging and network media
  • How to use online video for political and issue based advocacy
  • The rise of mobile politicking and organizing
  • Rethinking media campaigns and organizations from the ground up

During his keynote on how technology is improving government yesterday morning, NYC Mayor Bloomberg announced the “BiggApps competition,” challenging developers in the audience to “play with city data.”

MayorBloomberg PDF09 by magnifynet

MayorBloomberg PDF09 by magnifynet

Here’s the press release for the competition:

MAYOR BLOOMBERG ANNOUNCES FIVE TECHNOLOGY INITIATIVES TO IMPROVE ACCESSIBILITY, TRANSPARENCY AND ACCOUNTABILITY ACROSS CITY GOVERNMENT

City Providing Data to the Public to Allow for the Development of Applications for Computers and Mobile Devices as Part of “NYC Big Apps” Public Contest; 311 and NYC.gov Enhanced through Skype, Twitter and Google

Obama Administration Chief Technology Officer Aneesh Chopra Applauds City Efforts

Mayor Michael R. Bloomberg today announced a series of technology initiatives designed to increase transparency and improve access to information about City services. The City will provide data to allow for the development of software applications that can be used on websites and mobile devices, and through what will become an annual competition known as NYC Big Apps, the City will encourage innovative and useful applications. The Mayor also announced the launch of 311 Online and other improvements to 311 and NYC.gov through services provided by Skype, Twitter and Google. With call volume to 311 continuing to increase, 311 Online will allow the City to maintain the current level of service with current staffing levels, potentially avoiding more than $4 million in additional costs next fiscal year. The Mayor made the announcement in remarks delivered through Skype to the Personal Democracy Forum at Lincoln Center, an annual conference that explores how technology and the Internet are changing politics, democracy and society. New York City Chief Information Officer and Commissioner of the Department of Information Technology and Telecommunications Paul J. Cosgrave also attended the conference.

“We’ve already made great strides increasing the accessibility of City data and transparency of City government, and these initiatives will use private sector technological innovation to bolster those efforts,” said Mayor Bloomberg. “Through NYC Big Apps, 311 Online and services offered by Skype, Twitter and Google, we’re working to provide public information to New Yorkers in as many ways as possible.”

“We applaud New York City’s leadership on delivering a more open and innovative government,” said Federal Chief Technology Officer Aneesh Chopra. “These five announcements align well to President Obama’s Open Government Initiative and reflect best practices worthy of replication to achieve excellence in public sector performance.”

“Today’s package of initiatives represents an historic stride in transparency – even for systems that have made accessibility commonplace,” said Department of Information Technology and Telecommunications Commissioner Cosgrave. “As successful as we have been in opening up City government to those it serves, the key to technology deployment for any organization is to continue innovating. As 311 and NYC.gov grow, the City needs to adapt and engage New Yorkers in utilizing the data it collects to keep fresh these enduring avenues of access.”

NYC Big Apps

Through the NYC Big Apps annual competition, the City will provide an array of data sets to encourage the public to develop applications that could benefit New Yorkers. Approximately 80 data sets from across 32 City agencies and commissions may be made available on NYC.gov, including such categories as citywide events, property records and sales information, recreational facility directories and restaurant inspection information. The City will invite the public to create innovative applications, and winners will be awarded a cash prize and marketing opportunities. Mayor Bloomberg plans to congratulate the winners in person at a dinner. The contest will begin this fall. The program will be administered by New York City Economic Development Corporation, which today issued a Request for Expressions of Interest to solicit information from software developers and professionals in related fields to identify additional data sets to be aggregated.

“Finding opportunities to engage our innovative high-tech workforce is integral to the continued growth of the media sector in New York City,” said New York City Economic Development Corporation President Seth W. Pinsky. “By making City data available to a broader audience and encouraging our entrepreneurs to create new applications using that information, we leverage existing resources to stimulate investment and create jobs.”

311 Online

Mayor Bloomberg launched 311 Online, a one-stop, searchable web portal on NYC.gov for thousands of New York City services. Through the site, New Yorkers can obtain information, report problems, lodge complaints, check the status of previously-filed complaints and request City services – just as they can by calling 311. Users can browse through a directory of City services, search for available services by specific demographic or service type, and access quick links to featured services and top services. Keyword searches and advanced search options allow customers to navigate directly to the information. Users will be able to attach pictures, videos and audio files to their complaints and service requests.

311 Skype and Twitter Accounts

The Mayor announced 311 Skype and Twitter accounts. Through Skype – a software application that enables calls to be made over the Internet – people from around the world will be able to call 311 for free. The City will use Twitter – the free, social messaging service – to ‘tweet’ information regularly about such things as alternate side of the street parking status, schools closures and information about citywide events. 311NYC tweets will be 140 or fewer characters in length and can be sent to any mobile device via texting, instant messaging or the web. Information about emergency events and services will continue to be accessible via Notify NYC.

Google Collaboration to Improve Site Content on NYC.gov

The Mayor also announced that the City is working with Google to use Google search patterns to better understand the usage of NYC.gov and ultimately improve site content. By analyzing trends for New York City-related searches made by Google users, the City will tailor content to user preferences and improve costumer service.

—END PRESS RELEASE—

The competition will make about 80 data sets from 32 city agencies and commissions available to developers to create “applications to help Internet users navigate vast stores of data in areas like citywide events, property sales, recreational facilities and restaurant inspections.”

It will be run by the NYC Economic Development Corporation and the winner will get a cash prize, a dinner with Mayor Bloomberg, and marketing opportunities.

At the same time, unrelated to PDF09, a meeting on Open Data Standards in NYC was held by the New York City Council Committee on Technology in Government.

Looks like data is definitely getting some love (or at the very least, lip service) in New York. I wasn’t able to make either event, so if you did, let me know how it went in the comments!

post

Data in New York City

On top of the Empire State Building, Manhattan, New York City, by meironke.

On top of the Empire State Building, Manhattan, New York City, by meironke.

One of the things that makes doing web journalism in New York City absolutely frustrating is the lack of online data.

I’m not looking for anything strange. The first data set I wanted was crime reports that include individual crimes and locations. NYPD publishes weekly crime statistics, but not data that could easily be plotted on a map.

It absolutely stuns me that one of the biggest, most famous cities in the world is so backward. And why hasn’t the police department been slammed with FOIAs from every journalist in the city for the past 15 years?

Guardian DataStore

Guardian DataStore

About a week ago I posted to Twitter an idea for creating a data hub for NYC, in the vein of The Guardian’s Data Store. Everyblock does a good job of the collecting what data NYC does put online, but their job isn’t to track down city departments and convince them that providing clean data in multiple, usable formats would be to their benefit.

About a year ago, some journalists were complaining about the “data ghettos” that began popping up on newspaper websites. The problem was/is that newspaper organizations began publishing databases without context.

How would my NYC data hub be different?

It’s not. I don’t envision this as journalism. It is, instead a service provided TO journalists.

The idea needs some more fleshing out, some investors, and a business model. But it’s doable, and necessary. I don’t ever want to hear another editor turn down an idea because it will take 2 years and a FOIA to get the required data.

post

The Government, the Internet and Data

datagovThe launch of Data.gov yesterday was accompanied by a lot of fanfare on Twitter and blogs.

I think it’s fantastic that Obama is following through on his promise to make government more transparent, and looking forward to Data.gov being a very useful tool. Right now though, it’s a bit wimpy. While there is a lot of data available in machine-readable formats, it hasn’t been translated into visualizations that humans can easily understand.

I should mention that US Government Web Services and XML Data Sources, a non-government site, has been collecting data released by the government and releasing mobile applets for a long time now.

In reaction to the launch of Data.gov, Sunlight Labs has launched Apps for America 2:

apps-for-america-2_-the-datagov-challenge

Just as the federal government begins to provide data in Web developer-friendly formats, we’re organizing Apps for America 2: The Data.gov Challenge to demonstrate that when government makes data available, it makes itself more accountable and creates more trust and opportunity in its actions. The contest submissions will also show the creativity of developers in designing compelling applications that provide easy access and understanding for the public, while also showing how open data can save the government tens of millions of dollars by engaging the development community in application development at far cheaper rates than traditional government contractors.

A post on the White House blog, “Your Government & New Media,” encourages people to find out where agencies are getting involved online and use these venues to communicate with the government.

So, look for opportunities to jump in and connect with your government — at our websites and blogs, through videos and photos, in social networks, through widgets, podcasts, and more. Abraham Lincoln knew what he was talking about. This is government of the people, by the people, for the people.

View, comment, rate, participate, and share. The government is paying attention, even as we continue to learn ourselves. The more people engage, the more meaningful all of this becomes, and the more progress we can make.

Here is a list of examples of government being “cooler and more approachable than you think.” (Descriptions are added from each site.) I gathered these from multiple blogs and websites.

    pick-5-for-the-environment-us-epa

  • EPA’s Pick 5 to help the environment
  • Do more to protect the environment by choosing at least five actions (below) you’ll commit to. Pick 5 also helps you identify more actions you can take in the future. Then let others know what you committed to through Pick Five. Show the actions you’ve taken.

  • National Park Service Facebook App
  • You may use this application to share and view stories, experiences and galleries related to your experience our our nations wonderful National Park system.

  • Library of Congress Flickr photo stream
  • The Prints & Photographs Division takes care of 14 million of the Library’s pictures and features more than 1 million through online catalogs. Offering historical photo collections through Flickr is a welcome opportunity to share some of our most popular images more widely.

    peer-reviewed-prior-art-pilot

  • Peer-to-Patent project
  • Recently a group of academic and business professionals have proposed a collaborative, online process in which members of the public pool together their knowledge and locate potential prior art. This pilot will test whether such collaboration can effectively locate prior art that might not otherwise be located by the Office during the typical examination process.

  • Presidential Directives and Executive Orders
  • In this section you will find official actions by the President that have a significant impact on how the federal government functions but do not require legislation or Congressional approval. See listings below of the official Proclamations, Presidential Memoranda, and Executive Orders that President Obama has issued since his inauguration.

  • Freedom of Information Act
  • The Freedom of Information Act should be administered with a clear presumption: In the face of doubt, openness prevails.

  • Transparency and Open Government
  • We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration.

    recoverygov

  • Recovery.gov
  • As the centerpiece of the President’s commitment to transparency and accountability, Recovery.gov will feature information on how the Act is working, tools to help you hold the government accountable, and up-to-date data on the expenditure of funds.

  • White House Blog, Health Care Reform Forum (slideshow), Live Streaming from the White House, Open for Questions, Podcasts, President’s Weekly Address
  • Town Hall in Turkey
  • Serve.gov
  • This website is a new portal for you and all Americans to find your own ways to serve in your own communities. Just choose your keyword – “education,” “environment,” or whatever interests you – and type in your zip code to see what opportunities our partner organizations have in your area. Americans are putting their own country back on the right track, be a part of it.

  • Troop Tube
  • TroopTube is the new online video site designed to help military families connect and keep in touch while miles apart.

  • TSA blog and other federal blogs
  • U.S. Government channel on YouTube
  • The official YouTube channel of the U.S. Government, linking you to videos across government. Visit the playlists and other channels for a wide variety of interesting videos!

    dipnote

  • State Department’s DipNote blog, on Twitter, on Facebook
  • Official Blog of the U.S. Department of State – offers the public an alternative source to mainstream media for U.S. foreign policy information. This blog offers the opportunity for participants to discuss important foreign policy issues with senior Department officials.

  • USA.gov, USA.gov’s government FAQs, email and online chat, USA.gov on Twitter
  • As the U.S. government’s official web portal, USA.gov makes it easy for the public to get U.S. government information and services on the web. USA.gov also serves as the catalyst for a growing electronic government.

    federal-bureau-of-investigation-add-fbi-links-to-your-site

  • FBI widgets
  • Add links to FBI content by incorporating the widgets and modules below into your own website or blog.

  • Open Government Dialogue
  • This online brainstorming session, open from May 21st to 28th, 2009, will enable the White House to hear your most important ideas relating to open government.

  • Open Government Initiative and Innovations Gallery
  • Consistent with the President’s mandate, we want to be fully transparent in our work, participatory in soliciting your ideas and expertise, and collaborative in how we experiment together to use new tools and techniques for developing open government policy.

I hope these sites are useful to those interested in becoming involved in the direction of government for the next several years. If I missed any good ones, please leave them in the comments!

post

Visualizing State Taxes in Number of Days Worked

For the past month or so, I’ve been looking for data sets to play with. As a journalist, I really enjoy finding interesting ways to visualize data, and I needed some to play with.

I came across Visualize This: Days Spent Working to Pay Taxes, a challenge posted to Flowing Data:

About 28.2% of the average American’s income goes towards taxes, which means the first 103 days of the year is to pay for government. At the end of these 103 days – April 13 – is Tax Freedom Day. However, because of varying state-by-state tax burdens and average incomes, Tax Freedom Day varies by state. Alaska, for example, has the earliest Tax Freedom Day (March 23) because it has low state and local taxes while Connecticut is last on April 30, because of “extraordinarily high federal income taxes.” For this Visualize This we’re looking at the number of days each state spends paying taxes this year (2009).

FlowingData explores how designers, statisticians, and computer scientists are using data to understand ourselves better – mainly through data visualization. Money spent, reps at the gym, time you waste, and personal information you enter online are all forms of data. How can we understand these data flows? Data visualization lets non-experts make sense of it all.

So, I didn’t get mine done in time for the contest, and the results were posted today.

Edit: These two are my favorites:

Tax Freedom Day by State 1977-2009 from Alex Lundry on Vimeo.

Here is my belated attempt (click for larger view):days_to_pay_taxes

I wanted to do something with the state median incomes as well, but I’m having trouble getting the numbers to make sense.

post

Scaring highschoolers about the future of journalism

On Wednesday evening I spoke to a group of five students who are taking part in the Bronx Youth Journalism Initiative.

I’ve mentioned BYJI here before, mostly begging for help with my public speaking anxiety.

To my surprise, the whole thing went pretty well. The kids were Web-savvy enough to have uploaded a few videos to YouTube, and knew of Twitter, though none are using it yet.

I talked about the “newspaper crisis” caused by lack of innovation, an old business model and the problems with advertising and paywalls. (The kids’ immediate reaction to paywalls: “That won’t work.” Out of the mouths…) I went over the basics of online journalism: blogs, social networks, multimedia. I also talked about citizen journalism a little bit, in terms of how everyone can have a voice in their communities, which is a big problem in the Bronx. They really liked the concepts of “Not Just a Number,” which I showed them, along with the Las Vegas Sun Web site.

One student asked me how he could learn to code, and I directed him to the W3Schools site. Another asked about the future of news on e-readers like the Kindle. And of course the final question was “Where are we going?”

Thanks to Mindy McAdams, Craig Lee, and Tracy Boyer for their advice and inspiration. I’ve uploaded a powerpoint presentation to Slideshare which I used as a guide for my presentation, although it was really more a conversation than a speech.