Upload Dataset is a self-service Slate tool for importing historical and live data.

It may sometimes be hard to find an explanation for discrepancies you find between incoming data and the data that you ultimately see in Slate.

The following tips will help you identify the reason for these discrepancies and guide you toward a fix.

Before you begin:

• Verify that your source file includes all data points you wish to import.

• Any tests should be completed in your up-to-date test environment.

Verify source settings

• Confirm that the Source Format is mapped to the dataset you want to update, such as Persons / Applications or Organizations.

• Check the Safe / Unsafe settings to determine if the dataset is updating active applicants when specified.

• Reconcile your file format and specified File Format in Upload Dataset.

• Select Update Only if the file should not create new Slate records.

Check dataset field and value mappings

• Confirm that the source fields in your import have been mapped to the correct destination in Slate.

• Check your value mappings to confirm that all expected values are mapped, and map or append values as appropriate.

• Make sure that your dataset includes the necessary data points to create or match on records, such as basic biographic data (First, Last, Birthdate, Email) or relevant unique identifiers (Slate ID, App ID, custom field set as “Unique for Merging”).

• When attempting to create an application, an Application Round must be mapped.

Check settings on Slate destination field

• Safe / Unsafe

• Prompt vs. Store Value

• Single Value / Multiple Values

• Append / Replace

Check the snapshot history​

The snapshot history of a source format is a great way to see what the source format appeared like in the past.

If, starting on a specific date, files started returning errors upon import, checking the snapshots immediately preceding and following that date can uncover changes to the source format that may require attention.

Similarly, if the issue has not been identified, viewing the snapshot history for recent changes made to the source format can help uncover the source of an issue.

File size

There is no size limit to importing records/data/materials into Slate. However, there is a time limit of 15 minutes. After 15 minutes, an import will fail.

We recommend breaking large files into 50,000 record segments. This will significantly improve performance.

Feel free to try the import of these smaller files in your test environment first, to ensure that the results are as you expect. We recommend importing larger files early in the morning or late in the evening as this is the window where activity is likely at its lowest point in your Slate instance.

File formatting

From the source page in Upload Dataset, you can download files that were imported recently.

Once downloaded, opening the file in a text or code editor will allow you to confirm if the file specifications align to what the source format expects.

For example, if you are expecting a fixed-width file, downloading the file and opening it on your local machine is a great way to confirm the fixed-width perimeters and visualize whether adjustments to the raw file are necessary.

While we don't recommend any specific text or code editors, common text editors include Notepad, TextEdit, Notepad++, and common code editors include Visual Studio Code and Sublime Text.

Items to check include:

• Is the file formatted correctly? (.csv, .txt, .xlsx)​

• Is the file delimited as expected? (comma, tab)​

• Does the value exist in the file?​

• Are date values in the right format?​

• Are leading zeros present?

Use test records

Create test person and application records and import values to those records.

To differentiate test records from actual records, add a test tag and append the first name, last name, and email address values with TEST to make the record stand out.

Example: Hamilton TEST, Alexander TEST

Use a Time Warp Environment

If records are merging or changing, provision a Time Warp Environment to before the records were merged/changed.

This will help your team visualize how the records appeared like prior to the event that modified them.

📖 Further reading

Provision Clean Slate, Test, and Time Warp Environments

Unicode & UTF-8 support

• UTF-8 (Unicode Transformation Format – 8-bit) consists of more than 128,000 characters, accounting for Greek, Chinese, Cyrillic, Japanese, and many other non-Latin alphabets.

• Slate supports Unicode; however, Excel does not support saving .csv files in UTF-8.

• As a result, non-English characters need to be exported by third-party vendors as text files.

• Slate currently fully supports the western European character set (Latin).

Leading zeros are truncated

If the leading zeros in your data are truncated, there is likely one of two reasons:

1. The file is opened in excel prior to uploading to Slate.

   • General is the default format in excel and excel will drop the leading zeros.

   • Do not open the file prior to uploading in Slate.

2. You have the XML in the format definition set to convert and during the conversion process the leading zeros are dropped.

   • Convert should only be used for an excel file.

   • Set the proper XML code in the format definition for the source format so that the file can be processed correctly.

Error messages

Below is a list of error messages that pertain to sources and sources formats within Upload Dataset.

📖 Further reading

Format Definition XML