Category Archives: API

Application Programming Interface

Creating Custom Demographic Datasets with API Tools

.. develop national scale spreadsheet files with virtually no learning time .. easy-to-use API operations to create national scope demographic-economic datasets based on American Community Survey 2016 1-year estimates .. custom subject matter selections. See more detail in related web sections ACS2016 and ACS2016_API.

Benefits and utility … how to acquire a spreadsheet showing the population of all cities with population estimates based on the ACS 2016 1-year data? … or, housing units, median household income, median housing value, etc.? Variations of this need frequently arise — what is the list of largest California counties sorted on total population: What are the 25 metros having the highest median household income? Which 10 congressional districts have the highest poverty incidence? Which urban areas have the highest educational attainment?

Use simple API calls described below to get answers to these types of questions — and more.  Create files that can be used for recurring applications. An example …

Urban Areas with 2016 Population 65,000+ Population
… results from using the API downloaded data … the following graphic shows urban areas with 65,000 or more 2016 population; zoom-in to Texas. The full national scope GIS project is available as described below; examine U.S. or any region. The file used to develop this view was created using the results of the API call reviewed below (requires integration of those data into the urban areas shapefile). Click graphic for larger view; expand browser window. Larger view shows urban areas labeled with name and mini profile for Dallas UA showing all subject matter items downloaded (via API) as described below.

… View developed using CV XE GIS.
… See more about Urban Population & Urban Areas.

Access ACS 2016 1-Year Data Using API Tools
Here are the API links … use these API calls to access/download selected items for selected geographies. See more about using API tools. Click a link and receive a return page with CSV-like structured data. See usage notes below. As these are ACS 2016 1 year estimates; geographies are only available for areas 65,000+ population.
Click a link:
• All U.S. cities/places
• All U.S. counties
• All U.S. CBSAs
• All U.S. Urban Areas
• All 115th Congressional Districts
• All U.S. states
• U.S. only

The following data retrieval operations are by state. These are examples using Arizona (FIPS state code 04).
• All [within state] Elementary School Districts
• All [within state] Secondary School Districts
• All [within state] Unified School Districts

API Call Returned Data Usage Notes
Clicking the All U.S. cities/places link above generates a new page with content very much like a CSV file. Try it .. click an above link.

See the related ACS2016_API web section for more details.

Items Retrieved in the API Calls
The sample header record above shows the subject matter item listed at the left in the following set of items. Modify API call and use other subject matter items. See full array of subject matter – xlsx file.
.. B01003_001E – Total population
Age
.. B01001_011E — Male: 25 to 29 years (illustrating age cohort access)
.. B01001_035E — Female: 25 to 29 years (illustrating age cohort access)
Race/Origin
.. B02001_002E – White alone
.. B02001_003E – Black or African American alone
.. B02001_004E – American Indian and Alaska Native alone
.. B02001_005E – Asian alone
.. B02001_006E – Native Hawaiian and Other Pacific Islander alone
.. B02001_007E – Some other race alone
.. B02001_008E – Two or more races
.. B03001_003E – Hispanic (of any race)
Income
.. B19013_001E – Median household income ($)
.. B19113_001E – Median family income ($)
Housing & Households
.. B25001_001E – Total housing units
.. B25002_002E – Occupied housing units (households)
.. B19001_017E — Households with household income $200,000 or more
.. B25003_002E — Owner Occupied housing units
.. B25075_025E — Housing units value $1,000,000 to $1,499,999
.. B25075_026E — Housing units with value $1,500,000 to $1,999,999
.. B25075_027E — Housing units with value $2,000,000 or more
.. B25002_003E – Vacant housing units
.. B25077_001E – Median housing value ($) – owner occupied units
.. B25064_001E – Median gross rent ($) – renter occupied units

The rightmost fields/columns in the rows/records contain the area name and geographic codes.

Using API Tools for Data Analytics
Join me in a Data Analytics Lab session to discuss more details about accessing and using wide-ranging demographic-economic data and data analytics. Learn more about using these data for areas and applications of interest.

About the Author
— Warren Glimpse is former senior Census Bureau statistician responsible for innovative data access and use operations. He is also the former associate director of the U.S. Office of Federal Statistical Policy and Standards for data access and use. He has more than 20 years of experience in the private sector developing data resources and tools for integration and analysis of geographic, demographic, economic and business data. Contact Warren. Join Warren on L

Creating & Using Location Shapefiles

.. GIS tools and methods to develop and update location shapefiles .. location shapefiles are essential to most GIS applications. Location shapefiles, or point shapefiles, enable viewing/analyzing locations on a map and attributes of these locations such store or customer ID, street address, city, date updated, value, ZIP code and wide-ranging attributes about the location. This section reviews tools and methods to develop and use location shapefiles. See more detail about topics covered in this section in the related Web page.

Viewing/Analyzing Store Locations in the Dallas, TX Area
The following graphic illustrates how store locations can be shown in context of other geography and associated demographic-economic attributes. This view shows store locations (red markers) in context of Dallas city (blue cross-hatch pattern) and broader metro area. Markers shown in this view are based on a location shapefile created using steps described below. The identify tool is used to click on a location and show attributes in a mini-profile.

.. view developed with ProximityOne CV XE GIS and related GIS project.

View the locations contextually with thematic patterns by tract or other geography. Combine views of store, customer, agent, competitor and other location shapefiles.
The following view shows patterns of median household income by census tract.

.. view developed with ProximityOne CV XE GIS and related GIS project.

Development of location shapefiles often starts with a list of addresses. Locations are not always address-oriented; they might be geographically dispersed measurement or transaction locations — having no address assigned. In applications reviewed here, locations are organized as rows in a CSV file. Each CSV file contains like-structured attributes for each location. The example used in this section uses store locations located in the Dallas, TX area.

There are two basic methods used to create location shapefiles: 1) geocoding address-data contained in the source data file or 2) using the latitude-longitude of the location included in the source data file record. The focus here is on option 2 — using the latitude-longitude of the location already present in the source data file.

Creating a Location Shapefile
The process of creating a location shapefile uses the CV XE GIS Manage Location Shapefile feature. With CV running, the process is started with File>Tools>ManageLocationShapefile. The following form appears.

.. ManageLocationShapefile feature/operation in ProximityOne CV XE GIS.

CV XE GIS provides other ways to create location shapefiles:
• Tools>AddShapes>Points — click points on the map window canvas.
• Tools>FindAddress — creates a single point shapefile based on specified address.
• Tools>FindAddress (Batch) — creates a point shapefile based on specified file of address records.
See details in User Guide.

Steps to Create a Location Shapefile
The process of creating the shapefile “C:\cvxe\1\locations1pts.shp” can be viewed by clicking the Run button on the form (with CV running). Two input CSV structured files are required:
• data definition file
• source data file

There are two sets of illustration location input files included with the CV installer:
• locations1_dd.csv and locations1.csv (7 locations in Johnson County, KS)
• locations2_dd.csv and locations2.csv (252 locations in Dallas and Houston)
These files are located in the \1 (typically c:\cvxe\1) folder. The marker/location shapefile used in the map shown above was created using the lcoations2 input files.

Data Definition File
The Data Definition (DD) file is an ASCII/text file structured as a CSV file. It may created with any text editor. The DD file is specific to the source data file. But in the case of recurring source data files for different periods the same DD file might apply to many source data files. There are several rules and guidelines for development of the DD file:
• there is one line/record for each field in the source data file.
• each line/record must be structured in an exact form:
.. each line/record is comprised of exactly 4 elements separated by a comma:
.. 1 field name for subject matter item
– must consist of 1 to 10 characters and include no blanks or special characters
.. 2 field type: C for character, N for numeric
.. 3 field length: an integer specifying the maximum with of the field
.. 4 maximum number of decimals for field (value is 0 for character fields)
The DD File must include three final fields:
LATITUDE,n,12,6
LONGITUDE,n,12,6
GEOID,c,15,0
The structure of these three DD file records must be as shown above. The source data file, described below, must have the LATITUDE and LONGITUDE fields populated with accurate values. The GEOID field may populated with either an accurate value of placeholder value like 0.

Example. Data for each store for the default DD file name “C:\cvxe\1\locations1_dd.csv” include the following fields/attributes:
  NAME,C,45,0
STORE,c,15,0
ADDRESS,c,60,0
CITY,c,40,0
LATITUDE,n,12,6
LONGITUDE,n,12,6
GEOID,c,15,0

Optionally create a DD File using the Create DD File button on the form. Clicking this button will create a DD File containing attributes of the dBase file specified in the associated edit box. The DD File name is created from the dBase file name. If the dBase file name is “c:\cvxe\1\locations1pts.dbf”, the DD File will be named “c:\cvxe\1\locations1pts_dd.csv”.

About the GEOID
The GEOID is a 15 character code which defines the Census 2010 census block containing each location. The GEOID is generally assigned by the ManageLocationShapefile operation and is one of the important and distinctive features of this tool. The GEOID is used to uniquely determine, with the GIS application, any of the following: state, county, census tract, block group, or census block.

The GEOID, as used in this section, is the 15 character Census 2010 geocode for the census block. The GEOID value 481130002011012 (see in location profile in map at top of section) is structured as:
state FIPS code: 48 (2 chars)
county FIPS code: 113 (3 chars)
census tract code 000201 (6 chars)
census block code: 1012 (4 chars) (block group code: 1 — first of 4 characters)

About the Source Data File
The Source Data File is an ASCII/text file structured as a CSV file. It is typically developed by exporting/saving an Excel or dBase file in CSV structure. There are several rules and guidelines for development of the source data file:
• fields must be structured and arranged as defined in the DD File.
• character fields must not contain embedded commas.
• final items in record sequence must be:
.. LATITUDE – must have accurate decimal degree value; 6 digit precision suggested.
.. LONGITUDE- must have accurate decimal degree value; 6 digit precision suggested.
.. GEOID – this may be 0, not assigned or the accurately assigned GEOID value.
– optionally create/rewrite the GEOID used in the new shapefile.

Updates; Combining Vintages of Location Attributes
Location based data might update frequently, even daily. The recommended method to add, update and extend the scope of location-based data is to create new address shapefiles corresponding to different vintages or dates covered. The structure of the files must be the same so that they files can be used together or separately. Suppose there is one set of data covering year to date and a second set of data covering the following month. The ManagePointShapefile operation would be run once for each time period. Two shapefiles would be created. These shapefiles may be added to a GIS project and used separately or in combination to view/analyze patterns.

Join me in a Data Analytics Lab session to discuss more details about accessing and using wide-ranging demographic-economic data and data analytics. Learn more about using these data for areas and applications of interest.

About the Author
— Warren Glimpse is former senior Census Bureau statistician responsible for innovative data access and use operations. He is also the former associate director of the U.S. Office of Federal Statistical Policy and Standards for data access and use. He has more than 20 years of experience in the private sector developing data resources and tools for integration and analysis of geographic, demographic, economic and business data. Contact Warren. Join Warren on LinkedIn.

Accessing & Using ZIP Code Demographics

.. tip of the day .. a continuing weekly or more frequent tip on developing, integrating, accessing and using geographic, demographic, economic and statistical data. Join in .. tip of the day posts are added to the Data Analytics Blog on an irregular basis, normally weekly. Follow the blog to receive updates as they occur.

… February 2017 updates .. 5 ways to access/analyze the most recent estimates of median housing value and other subject matter by ZIP Code area .. updates on accessing/downloading/using American Community Survey (ACS1115) 5-year estimates. See more detail in related Web section.

Site analysis (Option 5 below)
– create site analysis profiles from a location/ZIP code.

Contents
Five data access and use options, listed in the links below, are reviewed. Each method illustrates how ZIP code demographic-economic data can be accessed/ analyzed/used in different contexts. The most basic data access/data download is illustrated in Option 3. The following links open new windows that take you to the related section with more detail.
Option 1 – View the data as a thematic pattern map.
Option 2 – View, compare, rank query data in interactive tables.
Option 3 – Access data using API Tools; create datasets.
.. Option 3a – Extended ZIP Code subject matter access.
.. Option 3b – ZIP code urban/rural data access.
.. Option 3c – Additional API ACS data access resources.
Option 4 – View $MHI in structured profile in context of related data.
Option 5 – Site analysis – view circular area profile from a location.

Related ZIP Code Data Access & Use sections
Interactive access to demographics based on an address
Summary of ZIP Code Data Resources & Tools
10 Reasons to use Census Tracts Versus ZIP Codes
Analyzing Census Tract Demographics by ZIP Code Area
ZIP Code to Census Tract Equivalence Table
ZIP Code Urban/Rural Geography & Demographics
Mapping ZIP Code Demograhics
Housing Price Index by 5-digit ZIP Code – time series; annual updates
Housing Price Index by 3-digit ZIP Code – time series; quarterly updates
ZIP Code Business Establishment, Employment & Earnings by industry
ZIP Code Retail Trade Establishment & Sales by industry
ZIP Code Equivalence Tables — ZIP Code to School District

Join me in a Data Analytics Lab session to discuss more details about accessing and using wide-ranging demographic-economic data and data analytics. Learn more about using these data for areas and applications of interest.

About the Author
— Warren Glimpse is former senior Census Bureau statistician responsible for innovative data access and use operations. He is also the former associate director of the U.S. Office of Federal Statistical Policy and Standards for data access and use. He has more than 20 years of experience in the private sector developing data resources and tools for integration and analysis of geographic, demographic, economic and business data. Contact Warren. Join Warren on LinkedIn.

State and Regional Decision-Making Information

Organized on a state-by-state basis, use tools and geographic, demographic and economic data resources in these sections to facilitate planning and analysis. Updated frequently, these sections provide a unique means to access to multi-sourced data to develop insights into patterns, characteristics and trends on wide-ranging issues. Bookmark the related main Web page; keep up-to-date.

Using these Resources
Knowing “where we are” and “how things have changed” are key factors in knowing about the where, when and how of future change — and how that change might impact you. There are many sources of this knowledge. Often the required data do not knit together in an ideal manner. Key data are available for different types of geography, become available at different points in time and are often not the perfect subject matter. These sections provide access to relevant data and a means to consume the data more effectively than might otherwise be possible. Use these data, tools and resources in combination with other data to perform wide-ranging data analytics. See examples.

Select a State/Area

Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
D.C.
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin
Wyoming

Topics for each State — with drill-down to census block
Visual pattern analysis tools … using GIS resources
Digital Map Database
Situation & Outlook
Metropolitan Areas
Congressional Districts
Counties
Cities/Places
Census Tracts
ZIP Code Areas
K-12 Education, Schools & School Districts
Block Groups
Census Blocks

Join me in a Data Analytics Lab session to discuss more details about accessing and using wide-ranging demographic-economic data and data analytics. Learn more about using these data for areas and applications of interest.

About the Author
— Warren Glimpse is former senior Census Bureau statistician responsible for innovative data access and use operations. He is also the former associate director of the U.S. Office of Federal Statistical Policy and Standards for data access and use. He has more than 20 years of experience in the private sector developing data resources and tools for integration and analysis of geographic, demographic, economic and business data. Contact Warren. Join Warren on LinkedIn.

Tip of the Day — Census Tract Data Analytics

.. tip of the day .. a continuing weekly or more frequent tip on developing, integrating, accessing and using geographic, demographic, economic and statistical data. Join in .. tip of the day posts are added to the Data Analytics Blog on an irregular basis, normally weekly. Follow the blog to receive updates as they occur.

This section is focused on tools and methods to access and use census tract demographic-economic measures. Median household income ($MHI), median housing value ($MHV) and other selected items are used to illustrate operations and options.

This section illustrates use of census tract data from the 2014 American Community Survey (ACS1014) 5-year estimates. These are the most comprehensive demographic-economic data from the Census Bureau at the census tract level. These “5-year estimates” are centric to mid-2012. See more about 2010-2021 annual estimates and projections.

Methods described here apply to many other geographies; see related tip sections. See related section on ZIP code applications.

Five data access and use options are reviewed. Each method illustrates how $$MHI, $MHV and other data can be analyzed/used in different contexts.

Option 1 – View the data as a thematic pattern map.
Option 2 – View, compare, rank query data in interactive tables.
Option 3 – Access data using API Tools; create datasets.
Option 4 – View $MHI in structured profile in context of related data.
Option 5 – Site analysis – view circular area profile from a location.

Related sections:
Census tracts main section
Evolution of Census Tracts: 1970-2010
Demographic-Economic Estimates & Projections
Census tract and ZIP code equivalencing
Using census tracts versus ZIP code areas
Single year of age demographics

Option 1. View the data as a thematic pattern map; use the GIS tools:
Patterns of Economic Prosperity ($MHI) by Census Tract … the following graphic shows $MHI for a portion of the Los Angeles metro. Accommodating different demographic-economic thresholds/patterns, different legend color/data intervals are used. The pattern layer is set to 80% transparency enabling a view of earth features. Click graphic for larger view, more detail and legend color/data intervals; expand browser window for best quality view.

– View developed using CV XE GIS and related GIS project.

See details about each option in the related Web page.

Join me in a Data Analytics Lab session to discuss more details about accessing and using wide-ranging demographic-economic data and data analytics. Learn more about using these data for areas and applications of interest.

About the Author
— Warren Glimpse is former senior Census Bureau statistician responsible for innovative data access and use operations. He is also the former associate director of the U.S. Office of Federal Statistical Policy and Standards for data access and use. He has more than 20 years of experience in the private sector developing data resources and tools for integration and analysis of geographic, demographic, economic and business data. Contact Warren. Join Warren on LinkedIn.

Tip of the Day: Median Rent by Metro

.. tip of the day .. a continuing weekly or more frequent tip on developing, integrating, accessing and using geographic, demographic, economic and statistical data. Join in .. click Follow button at upper right.

.. the most recent estimate of median gross rent [for occupied housing units paying rent] for metropolitan areas (core-based statistical areas) is as of 2014. These data are based on the American Community Survey (ACS2014) 1-year estimates.

Option 1. Use the interactive table:
– go to http://proximityone.com/usstcbsa14dp4.htm
– median gross rent is item H132.
– scroll left on the table until H132 appears in the header column
– this column shows the 2014 ACS 1-year estimate for H132 for all CBSAs
    (and states)
– see usage notes below table.

Option 2. Use the API operation:
click this link to get H132 (median rent) using the API tool.
– a new page displays showing a line/row for each metro.
– median rent appears on the left, then metro name and code.
– optionally save this file and import the data into a preferred program.
– more about API tools.

Option 3. View the median rent in context of other attributes for a metro.
click a link in this table to select a metro of interest.
– when the metro profile displays, see section 5.2.

See this related section on metro rental markets.

This section is focused on median rent. Many other subject matter items will be apparent when these methods are used. Optionally adjust above details to view different subject matter for metros.

Join me in a Data Analytics Lab session to discuss more details about accessing demographic-economic data. Learn more about using these data and tools to meet specific interests.

About the Author
— Warren Glimpse is former senior Census Bureau statistician responsible for innovative data access and use operations. He is also the former associate director of the U.S. Office of Federal Statistical Policy and Standards for data access and use. He has more than 20 years of experience in the private sector developing data resources and tools for integration and analysis of geographic, demographic, economic and business data. Contact Warren. Join Warren on LinkedIn.

Relating Addresses to Digital Road Segments

.. enter an address on a Google Maps page, or many other similar Web-based “find address” tools, and view that address on a map. View street detail easily. But a view of the digital road segment is not available. Answers to these questions are not available:
• what are the demographic-economic attributes for that location?
• what are the left and right side address ranges for that road segment?
• what are the geocodes on the left and right sides of that road segment?
• what are the end-point coordinates for the road segment?
• how does that road segment (the route) extend through the city or county?

Use methods described in this section to answer these types of questions and examine “Address to Digital Road Segment” relationships. An example is used that features an address shown as a red marker in the banner graphic at the top of this page (Kansas City, MO area). That address is also used in the interactive data access form below. Use these resources and methods for most addresses in the U.S. Tools reviewed here are available at no fee. Use related Web section for full functionality.

Viewing Address/Location in Context of Digital Roads
In the following graphic, a geocoded address (point shapefile) is shown as a red marker (see pointer) in context of roads (streets/lines shapefile). The Identify tool is used to show the profile of associated road segment (TLID=91447676) — an intersection to intersection segment of Oak St. Performing a query on the roads shapefile/layer to locate ID 91447676, the segment is displayed as yellow-highlighted. This application/view is reviewed in this section.

Finding Address Attributes
Use the form below to find attributes for an address. To get started, click the Find button with default settings. Results are returned/displayed on this same page. More in general, key in an address of interest, select the type of geography (more about this below) and click Find. Try your own addresses of interest; use this tool to meet recurring address lookups. Not clear on all the steps? Join us in a Data Analytics Lab session, get answers to questions.

Click following graphic for full functionality:

About the Data Content and Structure
When the Find button (above) is clicked, this page refreshes with returned data based on your query — the values entered/selected in the section above the Find button. The first portion of the data displayed provide a structured display of selected subject matter items (ACS 5-year estimates) for the type of geography selected (e.g. tracts) for that area in which the address is located. The scope of those items could be substantially expanded.

Following that portion of the display are the geographic attributes displayed as JSON output resulting from the geocoder processing of the address. This is the display content below the text “Summary of address sent and matched results:”.

Road Segment Attributes
See road segment attributes for this address under “Summary of address sent and matched results:” and “Address: {“. See that the road segment ID is shown by “tigerLineId”: “91447676”. This ID uniquely identifies this road segment among all of the more than 45 million road segments in the U.S. (see more detail in related section).

Viewing the Address with GIS Tools
1. Install CV XE GIS software (if already installed, skip step 1).
.. Install package — Windows 32/64
.. Start-up Readme
2. After installation, with CV XE GIS running, open a GIS project:
.. Use File>Open>Dialog and open the GIS project named c:\cvxe\1\cvxe_us2.gis. Perform these steps:
2.1. In the Legend Panel, uncheck layers $MHI x BG and Locations.
2.2. Use the Add Layer button to add the layer c:\cvxe\1\$$address1.shp.
.. address used in this application now appears as red marker
.. this single point shapefile was created using the CV XE Tools>FindAddress
2.3. In Legend Panel, check layer Jackson Cty MO roads.
.. now, all roads in the county can be viewed with marker.
.. optionally zoom-in to develop view like shown at top of this page.

.. view developed with ProximityOne CV XE GIS and related GIS project.

2.4. Use GIS Tools>FindShape tool to show all occurrences of “Oak St”
.. on the FindShape form the FULLNAME field is selected and set to “like” Oak St% .. all road segments having name like “Oak St%” are highlighted in yellow.

.. view developed with ProximityOne CV XE GIS and related GIS project.

2.5. Use GIS data table feature to show all occurrences of “Oak St” as table.
.. there are 155 road segment, shown in the next graphic with left and right-side address ranges .. optionally export for use in other applications.

Not clear on all the steps? Join us in a Data Analytics Lab session, get answers to questions.

Census Block Code
See section starting with “2010 Census Blocks”. In the default address run, the census block shows as “GEOID”: “290950066002016”: … state: 29 … county: 095 … tract: 006600 … block: 2016. Other items, such as the TIGER/Line segment ID and segment side, can also be important for some applications.

Any given address or location is contained with several types of statistical areas (e.g. census tract or block group) and political areas (e.g. city or county). We may want to know the demographic-economic characteristics of a location for any one or several of these geographies. Use the interactive tool on this page to access those data. For example, access/view the median household income of the location/address block group or the median household income the location/address city.

Join me in a Data Analytics Lab session to discuss more details about analyzing citizen voting age population and use of data analytics to develop further detail related to your interests.

About the Author
— Warren Glimpse is former senior Census Bureau statistician responsible for innovative data access and use operations. He is also the former associate director of the U.S. Office of Federal Statistical Policy and Standards for data access and use. He has more than 20 years of experience in the private sector developing data resources and tools for integration and analysis of geographic, demographic, economic and business data. Contact Warren. Join Warren on LinkedIn.