Difference between revisions of "CEN Data Migration"
| (One intermediate revision by the same user not shown) | |||
| Line 1: | Line 1: | ||
| − | {{Template:Important_Warning|content=AFF Obsolete}} | + | {{Template:Important_Warning|content=AFF Obsolete as of March 31, 2020}} |
{{Template:Data_Source_Maintenance_Has_Keytip}} | {{Template:Data_Source_Maintenance_Has_Keytip}} | ||
| Line 9: | Line 9: | ||
<li>For dataZoa, this means the location and names of data selected with the dZ-Dot simply no longer exist, and therefore cannot be automatically updated.</li> | <li>For dataZoa, this means the location and names of data selected with the dZ-Dot simply no longer exist, and therefore cannot be automatically updated.</li> | ||
<li>The good news is that the underlying data still exists and is accessible via a new Census.gov Application Programming Interface (Census.gov API).</li> | <li>The good news is that the underlying data still exists and is accessible via a new Census.gov Application Programming Interface (Census.gov API).</li> | ||
| − | <li>Further, dataZoa has an application page to help you update your existing links for dataZoa displays and calculations. Use [https://www.datazoa.com/utils/ | + | <li>Further, dataZoa has an application page to help you update your existing links for dataZoa displays and calculations. Use [https://www.datazoa.com/utils/cenapi.asp this tool] to convert old Census data locations to new ones.</li> |
<li>This process does involve manual review and confirmation of how to translate legacy links to the new style, but this is a one-time process for the series. The utility is by and large very good at mapping the old to the new, with a bare minimum of user intervention.</li> | <li>This process does involve manual review and confirmation of how to translate legacy links to the new style, but this is a one-time process for the series. The utility is by and large very good at mapping the old to the new, with a bare minimum of user intervention.</li> | ||
</ul> | </ul> | ||
</div> | </div> | ||
| + | <br><br><br> | ||
{{Template:Data_Source_Maintenance_General_Discussion}} | {{Template:Data_Source_Maintenance_General_Discussion}} | ||
Latest revision as of 11:57, 2 March 2020
dataZoa cooperates with many different sources of data, and generally speaking, keeps up well as the sources evolve.
In some cases, however, the sources may change so radically that user guidance and approval are needed to deal with some changes.
Key Tips for Census.gov/American Fact Finder (AFF) Data Migration:
- In March of 2020, the US Census Bureau de-commissioned its legacy American Fact Finder data catalog.
- For dataZoa, this means the location and names of data selected with the dZ-Dot simply no longer exist, and therefore cannot be automatically updated.
- The good news is that the underlying data still exists and is accessible via a new Census.gov Application Programming Interface (Census.gov API).
- Further, dataZoa has an application page to help you update your existing links for dataZoa displays and calculations. Use this tool to convert old Census data locations to new ones.
- This process does involve manual review and confirmation of how to translate legacy links to the new style, but this is a one-time process for the series. The utility is by and large very good at mapping the old to the new, with a bare minimum of user intervention.
Contents
Discussion
When you use the dZ-Dot to put data into your dataZoa account, you are actually capturing a pathway to the data, so we can go back and look for changes. Capturing straight from the website is convenient, but websites can change and the pathways can disappear. When the pathway changes, you can use a dataZoa data migration tool to update the path to that data.
Importantly, you will be migrating from a path on a website to a (hopefully) more permanent path in a database behind the website, using the site's data access API.
Select Series to Migrate section
The first section of the migration tool helps you identify series that need updating in your dataZoa account. You can set various search criteria and get a list of possible candidates, or put in a specific series key, (like MyAccount/0003417).
Whenever you use the ListSeries button, dataZoa will generate rows of Series that match your current criteria. You can then use the PickSeries button on any of these rows to begin the actual conversion.
Set Parameters section
Once you have picked a series, you will set parameters that re-identify the data series at the source database. The details of this section vary widely based on the particulars of the data Series being converted.
dataZoa will examine the stored legacy path to suggest new path specifications. Often this will happen automatically, but there in some cases there will be a button to Auto-Suggest the parameters based on clues in the dataZoa series.
Data Comparison Columns section
When all of parameters are selected, you do a trial data fetch and examine the results. You can tweak parameters if needed and re-fetch. When you are satisfied that the trial fetch is right for the data series, you can Accept the Parameters and Commit to dataZoa.
The three data comparison columns show:
- The legacy Current data for this Series as stored at dataZoa.
-
The proposed Trial data for this Series as it will be fetched with the current parameters.
- Hover over the Current URL row to see the raw legacy data path.
- Hover over the Current Title row to see the full description of the legacy data.
- An optional Raw Fetch direct from the data source, with minimal processing, to help resolve any subtleties about the data as stored at the source.
