The Canadian Council on Social Development (CCSD) and its Community Data Program is taking a leadership role in open data and open government and will be representing Canadian Civil Society at the Open Government Partnership meetings in Brazil.  Harvey Low will be attending, and here is the letter of submission that introduces Harvey and CCSD ideas which led to the award a seat at the table and full funding to attend.

This Open Government Partnership meeting, the current Federal Government Open Data Initiative, the latest consultations, the new Open Government Panel launched 2 days ago, will affect and may redirect some of the work you have been doing as researchers, open data advocates, scientists, geomaticians, data and map librarians and any others involved in evidence based decision making.

The open data and open government work is exciting indeed, but it may not be well grounded in well established practices in interoperability, portals, metadata standards and federated catalogs and the mechanism currently being implemented might become the norm.  Also, these are policy formulation days and your input is critical at this time, especially if  you want policy directions to reflect your endeavours.

The CCSD would like to solicit your ideas and opinions in preparation for the meetings in Brazil.  For instance if you had an opportunity to attend what would you want the Government of Canada to do on this file? What would benefit your communities of practice? What kind of funding are required to build TDRs, understand data access and open government? and What would you want international delegates to know about your work ? What kind of WGs should be created to advise the panel (data, portal, law, archiving, funding)?

Harvey will be holding an open conference call next week Wed March 7 at 11am EST, and Thur March 8 at 11am EST, you can join on one or the other calls, and we also welcome you input by email, in the comments, you can join the civicaccess.ca list and participate in the conversation there, or you may also want to write a letter or a position statement and Harvey will gladly point to it from here and bring it to Brazil.  The CCSD will also follow-up with you after the meetings to report results and discuss next steps.

Please respond to this tlauriau@gmail.com if you would like to participate in the call.

Your input is important and we hope to hear from you & please watch these initiatives.

I am happy to announce that the Canadian Council on Social Development‘s Community Data Program will be representing Canadian civil society at the Open Government Partnership meetings in Brazil.  The letter that was submitted is available here.

Harvey Low will be the Rep and he is looking for your insight!  If you have not done so already please register to the civicaccess.ca list or drop a comment here if you want to participate in a conference call.  We need your ideas.

All input will be posted here, and important and useful links will be posted here.

We look forward to hearing from you!

This is a bit more data than I would like liberated.

Also for some fun you may want to tell Vick everything cuz he really wants to know! #TellVicEverything @ToewsVic #Toews and I thought this one was pretty good:
Dear Mr. @ToewsVic Because we love you, I am asking every Canadian to fax you their grocery list at 613-995-1049 #TellVicEverything #Toews

This card was created for last year!  But I am proud to share it again, because, well, ffunction‘s work is timeless.

Canadians working with statistical and research data include government documents librarians whom we find in most university research libraries.  Many government document librarians and their colleagues, the data librarians, participate in the Data Liberation Initiative (DLI) which I introduced in a post honouring one of its founders.  They are also often members of the Canadian Association of Public Data Users (CAPDU) among many other important data related organizations.  The DLI also does much capacity building for research, data and map librarians in yearly face to face meetings and online discussion, developing expertise which is then shared among colleagues in their home institutions.

Aspi Balsara is one of the government documents librarians at the Queen Elizabeth II Library, Memorial University of Newfoundland.  He is a CAPDU member and has been kind enough to share his latest FAQ about various initiatives to disseminate Statistics Canada data.  The post is technical and specific in nature, but demonstrates quite nicely the kind of expertise we have across Canada in this area, a knowledge base that is often overlooked.  The FAQ introduces many databases and formats while also answering new dissemination policy questions.

Finally, this post also introduces a data community of practice with experts who collaborate nationally to benefit their local users using LISTSERVs technology, which ain’t fancy, but sure is effective in a place like Canada with its smart people scattered all over a big geographical expanse.  Twitter does some things well, but these lists and their archives are invaluable in fostering near real time deep collaboration.  People get to meet face to face once a year thanks to the DLI, so the relationships are quite strong.

***********************

FAQ on various dissemination initiatives from Statistics Canada

1.     Are all Public Use Microdata Files (PUMFs) available to the public, or only some of them?  All PUMFs are available, free of charge.  This has been the case for the past year and a  half.

2.     How does the public access and order a PUMF?   The public may order it directly from the Statistics Canada homepage, using the Search the site feature.  After filling out the order form, the customer will then be contacted by Statistics Canada to sign a licence agreement.  Upon receipt, the data is put on CD-ROM and shipped.

3.     Do these freely available PUMFs include SPSS and SAS command files (as they do for DLI subscribers)?  The codes are generally available in SAS which is what Statistics Canada (SC) uses.  SPSS is used mainly by the academic sector. SPSS may be available sometimes, but when derived from SAS, its quality is questionable as it does not include “missing values”.  Eventually, through the Common Tool for Social Surveys, SC will have more standardized output, including good SPSS codes.

4.     Since PUMFs are now publicly accessible, what value do DLI subscribers get for their subscription?  Revenue from DLI subscriptions pays for the infrastructure, regional and national training for the DLI contacts, prompt support through the listserv, and other initiatives.   No money goes toward paying for the data. This has been emphasized at the DLI training “bootcamps” where DLI contacts are asked to convey to their library administration the value of the training and support available from the DLI.  This is also pointed out in the DLI annual reports.

DLI subscribing institutions can share a PUMF in a classroom or lab environment.  Otherwise, a professor would have to obtain a licence from Statistics Canada that each student would be required to sign before using the PUMF.

Through the DLI, member institutions also have access to the Discharge Abstract Database (DAD) Research Analytic Files from the Canadian Institute for Health Information (CIHI).  The Discharge Abstract Database is only available to DLI members (see no. 11 below).

5.     In November 2010, Statistics Canada announced its intention to launch a subscription service to all its PUMFs.   This service was targeted to non-Canadian subscribers for an annual fee of $5000.00.   Is there any information about it?  This service aims at national and international organizations outside the DLI who wish to access SC’s complete PUMF collection, be informed of new releases, and avail of a service that answers their queries.  This service is called the “Public Use Microdata File (PUMF) Collection”.

See: http://www.statcan.gc.ca/bsolc/olc-cel/olc-cel?catno=11-625-XWE&1ang=eng

6.     With free CANSIM access beginning February 1, 2012, will the CANSIM component in E-Stat continue to be provided?   If so, will it be updated more than just once a year? In April 2012, Statistics Canada announced that E-Stat would be archived on June 30,    2012. It was last updated July 2011 and will remain so until removed permanently on June 30, 2013. (In the meantime, E-Stat can be accessed by clicking Students and teachers in the left menu bar of Statistics Canada’s homepage.)  Hence, there is no point using the E-Stat version of CANSIM anymore.

Other resources on E-Stat, such as the 1996, 2001 and 2006 censuses can be accessed from: http://www12.statcan.gc.ca/census-recensement/index-eng.cfm as well as from the library webpages at:  http://guides.library.mun.ca/canadianstatistics  and http://guides.library.mun.ca/content.php?pid=207197&sid=1734802

A new web location has yet to be determined for Census years 1665-1871, 1986 and 1991, as well as environment and elections data (currently accessible via E-Stat).

7.     Is it just the CANSIM data that will be freely available as of February 1, 2012, or all of Statistics Canada’s data? In addition to CANSIM, select census data products for 2011 will be freely available.  Statistics Canada will maintain current pricing practices for print publications, maps, CD-ROMs and custom products and services.

See: http://www42.statcan.gc.ca/smr09/smr09_035-eng.htm

8.     Are the geography products also freely available? As of November 29, 2011, geography data from the 2006 and 2011 censuses are available free of charge except for postal code products since they are provided by Canada Post. As it stands, DLI member institutions have access to postal code information products that can only be used for research and teaching purposes and cannot be shared with non-DLI institutions.  While these products are freely available to DLI subscribers, it should be noted that Statistics Canada is presently negotiating with Canada Post for continued access to postal code products.  If and when an agreement is concluded, it will be added as an appendix to the DLI licence agreement.

 9.     Does the public have to pay for DA (Dissemination Area), Block level data (basic population and dwelling counts) and FSA (Forward Sortation Area) data?   Data for DAs – for 2011 and previous census years – are now available for free upon request.  This is why you will see a “contact us” link for census tables at the DA level (whereas previously there was a $ sign since these tables were not freely available).  Block level data is available at the population and dwelling count level from GeoSearch or GeoSuite, and there is no charge.  FSAs come under postal code data covered above in no. 8.

10.  The new DLI licence (sent to subscribing institutions in September 2012) no longer states explicitly that data are restricted to research and educational purposes only.  Does this mean that commercial use of the data is now permitted?   Firstly, the majority of Statistics Canada’s standard and custom products will be disseminated under the terms and conditions of the Statistics Canada Open Licence Agreement.   See:  http://www.statcan.gc.ca/reference/licence-eng.html   It permits a worldwide, royalty-free, non-exclusive licence to use, reproduce, publish, freely distribute, or sell the information. This means that standard data products once distributed by the DLI, such as Intercorporate Ownership, SABAL – Small Area Business and Labour Database can now be made accessible to the general public and not just the       university community. However, lifting the restriction on such data is left to the discretion of the DLI member institution since it is then obliged to shoulder responsibility for providing support to outside clients. Organizations that prefer to maintain the restriction may refer non-university users to Statistics Canada for assistance.

Postal products are still restricted to DLI members (as explained in no. 8 above).

Public Use Microdata Files (PUMFs) are covered in an appendix to the new DLI Licence Agreement. Basically, bona fide members of a DLI member institution may use a PUMF for commercial purposes but cannot provide the file to outside clients.  For instance, a professor may publish the findings from a PUMF in a text book, but may not reproduce the data or share it.  Similarly, she may submit research for a client that draws upon a PUMF but cannot include the data. Should the client wish to consult the PUMF, she would follow the procedure described in no. 2 above.

11.  When will CIHI (Canadian Institute for Health Information) add its files to the DLI?

Plans are under way to make the Discharge Abstract Database (DAD) available as a DLI file (see no. 4).  The DAD focuses on inpatient acute care discharge in Canada (excluding Quebec).   The files will be available to the DLI community through the DLI FTP site once all members have signed and returned the licence agreement distributed last September.

Aspi Balsara

Feb 14, 2012

Revised:  April 17, 2012;  May 4, 2012, October 15, 2012

Today, Statistics Canada released the head count and the dwelling count of the 2011 census, the 2011 Census, the shortest decennial census in the history of Canada, the 1st official census since confederation was taken in 1871. More data on age, relationships and language to follow, and uh that is it!
The Census is the only legislated instrument that counts everyone every 5 years. Surveys come and go, are not legislated and do not have designated budgets.

Also, Statistics Canada announced a short while ago that its data were going to be disseminated for free for the first time and under a new more open and less restrictive licence (G&M article, Embassy Magazine Article). This is really good news as cost recovery was a horrid policy instrument barring access to data that we by law had to give away. Restricted access only allowed for a small subset of the population to study, discuss and know about who Canadians are. It also meant that we were not getting collectively smarter.

I and many others were and remain concerned that we do not know what data exactly will be made available, at what level of geography, will cross tabulations and special orders such as by neighbourhood or ward be more expensive than before, will that licence be as open, and as Woolley observed, how the data are disseminated is of concern, since well, right now it is clunky at best.   We all do applaud the effort.

Upon playing with the data dissemination interface today, my concerns were re-affirmed.  The data are free but not necessarily accessible, in the sense that the methods used to disseminate and discover these is complicated, unclear and there are some favourite geographies missing – most notably Dissemination Areas (DA) while others are hidden – Census Tracts (CTs).

For example, if you go to the Census Profile and you want to look up 5 cities at once you cannot! You can only look up one city at a time, which also means you can only download one geography at a time.  There are over 2000 cities in Canada and if you want to know who the top 30 are in terms of population, then its “Houston we have a problem!” sorta.

Furthermore, once you look at your city, you are provided with Census Metropolitan Areas (CMAs), Census divisions (CDs) and Census Subdivisions (CSDs), economic region (ERs), electoral districts (FEDs) and population centres (POPCTR).  CTs are hard to find and DA data more so.  CTs and DAs are smaller geographies very helpful for sub city analysis.  Now, when you do get lets say FED data for your city, you only get provided with one district at a time and not the cities FEDs at once.   So,  have to go back and download them one at a time and then assemble the file.  CT and DA geographies are also not in this list.  You have to dig for those!

To get to CTs  (no DAs to be found yet)  my friend Sara a GIS expert at the Social Planning and Research Council of Hamilton made this discovery:

1. Go here
2. Click on Thematic Maps (scroll down),
3. Go to CMA maps & choose your location.
4. Then on the following page there will be a link to the map and a table with all the pop change values for each CT.

Alternatively, and again thanks to Sara you can do the following:

1. Go here
2. Then type in a random CT (you can use the example given at the bottom of the list).
3. On the next page, click the CT number
4. On the next page, click the download tab.
5. Then scroll to Option 2, and select Census tracts and your data format,
6. and “Continue” – Voila, it will download a file for population counts for all CTs in Canada!

Which is ah, absurd. First cuz, well that is a lot of clicking to get to what should be on the first page.  Second, what CTs are in my city?  This file organizes CTs into CMAs which are not CDs or CSDs.  CDs or CSDs correspond to the legal administrative boundaries of cities and municipalities.  CMAs are much larger geographies, they are a StatCan construct and are not an official administrative city or municipality.  You have to be an analyst or a good dictionary reader to know this.  Most people report CMA results, but those miss many cities and some cities are split.

Also, what if you want 5 cities at a time and not just one at a time?

Ted, the GIS expert at Community Development Halton, who was trying to join the CT data with his geomatics files discovered the following:

Unfortunately, the CT table is a mess for GIS purposes.  For each CT, there are 7 entries (rows) for each discrete piece of information (Population in 2006, 2006 to 2011 population change (%), Total private dwellings, Private dwellings occupied by usual residents, Population density per square kilometre, Land area (square km)). When trying to perform a join, ArcGIS doesn’t know which of the rows to join on to map it.

You can however, download complete files in not well coded spreadsheets at a variety of geographies for all of Canada here  by selecting Option 2 – Comprehensive download file for a selected geographic level.  This is great, but be sure you know what you are doing with these data as there is a lot going on! For example, if you download the CT file they are organized by CMA, you do not have a way to know which are in your CD or CSD and that would be a nice addition. It would be even better if a table provided CTs, and city, or electoral districts and the CTs they contain and CSD with their postal codes, CTs or CSDs and their DAs and so on.

But where are those pesky DAs?,

Analyst will do fine with this release, after incessant digging, the GIS folks will have to play around with things and they will grumble at the waste of time incurred with coding and joining.  Journalists and the public will however find it hard to compare cities.  People default mistakenly to CMAs, but CMAs a city they are not.

Sara also pointed me to these gems

• Population and Dwelling Count Highlight Tables, 2011 Census, which was in all fairness not easy to find, but it has tables comparing CSDs.  These will help most people.

These reference maps are also excellent as these help unravel georeferences  and you can download geographic files here.   The search by postal code is a nice feature, as finally you can enter your postal code and find out which census geographies you fall into.  DAs are not there either!  People however really want that postal code file for free! It is the file that can be used to look up your elected officials and many democratic engagement tools have been developed, and they are sorta illegally page scraping that data all to foster democratic engagement, that file should be shared as broadly as possible.  If the government is going to open data then one would presume Crown Corporations and Agencies are also part of that deal!

But what if you want all the postal codes for your city, or all the CTs and DAs for you city and what if you want that for more than one city at a time, then you are out of luck as the tool does not allow for that type of access.

Anyway, there will no doubt be more discoveries and grumblings and I hope StatCan will work with users to make these things more useable.

Finally, a community of practice is really important, the Social Planning Network of Ontario (SPNO) data list folks were busy this morning communicating among analysts as they were looking for and finding things. These folks know their stuff well and have their members in their communities to answer to, who will no doubt be looking for NEW DATA arranged in a way that is meaningful.  Social Planning and Community Development councils have been working with these data for a very long time and have much of expertise. Demographic and geographic data are complicated and you need to know how to work with them, you need to be sensitive to underlying issues when communicating these and these folks do so with care.

Perhaps, as David E. pointed out, StatCan will begin training people more broadly on how to use these data! Alternatively, people may find a way to resource planning councils to enable them to train journalists and others on how to work with these data on StatCan’s behalf.

oh yeah!  DAs!  After emailing StatCan, I was directed to Geosuite for the 2011 Census.  But I could not find them in there either!  It is a nice tool that has to be downloaded, and as one Research Librarian Veteran commented, it will be nice when StatCan data products are software agnostic and operation system neutral, GeoSuite does not work on a MAC!

DA DATA FOUND – in GeoSuite you have to choose the Chart Search from the main Menu.  The data in there are not for the faint of heart though! (Thanks Amber from DLI List).

Below is the letter that was submitted today requesting that the Community Data Program be a civil society representative for Canada at the Open Government Partnership meetings in Brazil 2012.

The date of submission is Monday the 6th of February. If you have comments or would like to endorse this letters please email me at tlauriau@gmail.com, Thanks!

The CDP just received a new endorsement from Open North Inc.